Hi Natasha - I believe that Knox drops that on the floor because we rewrite the response and therefore invalidate the content length that is received from WebHDFS. Sending an incorrect Content-Length causes other problems.
Recalculating the size from scratch would be a performance issue perhaps we can calculate it based on the difference in the replaced content and rewritten content but we would need to assess that possibility. Would love to work on getting support for Knox in the hadoop CLIs! If you would like to file a JIRA to assess that work and/or contribute a fix for it that would be fantastic. thanks, --larry On Fri, Nov 27, 2015 at 8:32 PM, Natasha d'silva <[email protected]> wrote: > > Using Knox 0.6.0 with hadoop 2.7.1. > I am using the Hadoop FileSystem api but I have made some very minor > modifications to the WebHDFSFileSystem class: > 1) I hardcoded the username/password credentials for the server such that > the connection to the server succeeds. > 2) I also changed the default path for the connections such that the URL > path is > https://<HOST>:<PORT>/*gateway/default/*webhdfs/v1/<PATH> > instead of > https://<HOST>:<PORT>/webhdfs/v1/<PATH> > With these changes, I am able to successfully write to the server. > However, I cannot read. I get the following exception: > > java.io.IOException: Content-Length is missing: {null=[HTTP/1.1 200 OK], > Server=[Jetty(8.1.14.v20131031)], Access-Control-Allow-Origin=[*], > Access-Control-Allow-Methods=[GET], Connection=[close], > Set-Cookie=[JSESSIONID=eknosvics9ou1s6ttv2kbfscl;Path=/gateway/default;Secure;HttpOnly], > Expires=[Thu, 01 Jan 1970 00:00:00 GMT], > Content-Type=[application/octet-stream]} > at > org.apache.hadoop.hdfs.web.ByteRangeInputStream.getStreamLength(ByteRangeInputStream.java:153) > at > org.apache.hadoop.hdfs.web.ByteRangeInputStream.openInputStream(ByteRangeInputStream.java:131) > at > org.apache.hadoop.hdfs.web.ByteRangeInputStream.getInputStream(ByteRangeInputStream.java:105) > at > org.apache.hadoop.hdfs.web.ByteRangeInputStream.<init>(ByteRangeInputStream.java:90) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem$OffsetUrlInputStream.<init>(WebHdfsFileSystem.java:1275) > at > org.apache.hadoop.hdfs.web.WebHdfsFileSystem.open(WebHdfsFileSystem.java:1189) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) > > > > Comparing the headers between Knox 0.6 and Knox 0.5 shows that in v0.5 the > Content header *is* sent normally, but missing in the response to the > client from Knox 0.6. > I have turned on debugging in the Knox server and what is interesting is > that the trace shows that the file is read successfully and the right > headers seem to be generated, as is shown in the following excerpt from > gateway.log: > > > 2015-11-27 23:22:48,757 DEBUG conn.HttpClientConnectionOperator > (HttpClientConnectionOperator.java:connect(129)) - Connection established > 10.143.177.241:57593<->10.143.177.15:50075 > 2015-11-27 23:22:48,757 DEBUG execchain.MainClientExec > (MainClientExec.java:execute(238)) - Executing request GET > /webhdfs/v1/user/admin/hdfs_test_2015_20151127_182243.txt?user.name=admin&namenoderpcaddress=ehaascluster&buffersize=4096&offset=0&op=OPEN > HTTP/1.1 > 2015-11-27 23:22:48,758 DEBUG execchain.MainClientExec > (MainClientExec.java:execute(243)) - Target auth state: UNCHALLENGED > 2015-11-27 23:22:48,758 DEBUG execchain.MainClientExec > (MainClientExec.java:execute(249)) - Proxy auth state: UNCHALLENGED > 2015-11-27 23:22:48,758 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onRequestSubmitted(124)) - > http-outgoing-16 >> GET > /webhdfs/v1/user/admin/hdfs_test_2015_20151127_182243.txt?user.name=admin&namenoderpcaddress=ehaascluster&buffersize=4096&offset=0&op=OPEN > HTTP/1.1 > 2015-11-27 23:22:48,758 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) - > http-outgoing-16 >> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; > q=.2 > 2015-11-27 23:22:48,758 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) - > http-outgoing-16 >> Connection: keep-alive > 2015-11-27 23:22:48,759 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) - > http-outgoing-16 >> User-Agent: Java/1.8.0 > 2015-11-27 23:22:48,759 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) - > http-outgoing-16 >> Host: sampleserver.com:50075 > 2015-11-27 23:22:48,759 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onRequestSubmitted(127)) - > http-outgoing-16 >> Accept-Encoding: gzip,deflate > 2015-11-27 23:22:48,759 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 >> "GET > /webhdfs/v1/user/admin/hdfs_test_2015_20151127_182243.txt?user.name=admin&namenoderpcaddress=ehaascluster&buffersize=4096&offset=0&op=OPEN > HTTP/1.1[\r][\n]" > 2015-11-27 23:22:48,759 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 >> "Accept: text/html, image/gif, image/jpeg, *; q=.2, > */*; q=.2[\r][\n]" > 2015-11-27 23:22:48,759 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 >> "Connection: keep-alive[\r][\n]" > 2015-11-27 23:22:48,760 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 >> "User-Agent: Java/1.8.0[\r][\n]" > 2015-11-27 23:22:48,760 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 >> "Host: sampleserver.com:50075[\r][\n]" > 2015-11-27 23:22:48,760 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 >> "Accept-Encoding: gzip,deflate[\r][\n]" > 2015-11-27 23:22:48,760 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 >> "[\r][\n]" > 2015-11-27 23:22:48,765 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HTTP/1.1 200 OK[\r][\n]" > 2015-11-27 23:22:48,766 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "Access-Control-Allow-Methods: GET[\r][\n]" > 2015-11-27 23:22:48,766 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "Access-Control-Allow-Origin: *[\r][\n]" > 2015-11-27 23:22:48,766 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "Content-Type: application/octet-stream[\r][\n]" > 2015-11-27 23:22:48,766 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "Connection: close[\r][\n]" > 2015-11-27 23:22:48,767 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "Content-Length: 450[\r][\n]" > 2015-11-27 23:22:48,767 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "[\r][\n]" > 2015-11-27 23:22:48,767 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onResponseReceived(113)) - > http-outgoing-16 << HTTP/1.1 200 OK > 2015-11-27 23:22:48,767 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onResponseReceived(116)) - > http-outgoing-16 << Access-Control-Allow-Methods: GET > 2015-11-27 23:22:48,767 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onResponseReceived(116)) - > http-outgoing-16 << Access-Control-Allow-Origin: * > 2015-11-27 23:22:48,768 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onResponseReceived(116)) - > http-outgoing-16 << Content-Type: application/octet-stream > 2015-11-27 23:22:48,768 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onResponseReceived(116)) - > http-outgoing-16 << Connection: close > 2015-11-27 23:22:48,768 DEBUG http.headers > (LoggingManagedHttpClientConnection.java:onResponseReceived(116)) - > http-outgoing-16 << Content-Length: 450 > 2015-11-27 23:22:48,769 DEBUG hadoop.gateway > (DefaultDispatch.java:executeOutboundRequest(136)) - Dispatch response > status: 200 > 2015-11-27 23:22:48,769 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 0[\n]" > 2015-11-27 23:22:48,769 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 1[\n]" > 2015-11-27 23:22:48,769 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 2[\n]" > 2015-11-27 23:22:48,769 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 3[\n]" > 2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 4[\n]" > 2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 5[\n]" > 2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 6[\n]" > 2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 7[\n]" > 2015-11-27 23:22:48,770 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 8[\n]" > 2015-11-27 23:22:48,771 DEBUG http.wire (Wire.java:wire(72)) - > http-outgoing-16 << "HDFS and knox read and write test New LINE 9[\n]" > 2015-11-27 23:22:48,771 DEBUG conn.DefaultManagedHttpClientConnection > (LoggingManagedHttpClientConnection.java:shutdown(87)) - http-outgoing-16: > Shutdown connection > 2015-11-27 23:22:48,771 DEBUG execchain.MainClientExec > (ConnectionHolder.java:abortConnection(126)) - Connection discarded > 2015-11-27 23:22:48,771 DEBUG conn.DefaultManagedHttpClientConnection > (LoggingManagedHttpClientConnection.java:close(79)) - http-outgoing-16: > Close connection > 2015-11-27 23:22:48,771 DEBUG conn.PoolingHttpClientConnectionManager > (PoolingHttpClientConnectionManager.java:releaseConnection(287)) - > Connection released: [id: 16][route: {}->http://sampleserver.com:50075][total > kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 20] > 2015-11-27 23:22:48,772 DEBUG nio.ssl (SslConnection.java:process(347)) - > [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL > NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false > {AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=307,b=450,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1} > NOT_HANDSHAKING filled=0/0 flushed=0/0 > 2015-11-27 23:22:48,772 DEBUG nio.ssl (SslConnection.java:wrap(462)) - > [Session-1, SSL_NULL_WITH_NULL_NULL] wrap OK NOT_HANDSHAKING consumed=307 > produced=341 > 2015-11-27 23:22:48,772 DEBUG nio.ssl (SslConnection.java:process(347)) - > [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL > NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false > {AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=450,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1} > NOT_HANDSHAKING filled=0/0 flushed=341/0 > 2015-11-27 23:22:48,773 DEBUG nio.ssl (SslConnection.java:process(347)) - > [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL > NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false > {AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=450,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1} > NOT_HANDSHAKING filled=0/0 flushed=0/0 > 2015-11-27 23:22:48,773 DEBUG nio.ssl (SslConnection.java:process(347)) - > [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL > NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false > {AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=450,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1} > NOT_HANDSHAKING filled=0/0 flushed=0/0 > 2015-11-27 23:22:48,773 DEBUG nio.ssl (SslConnection.java:wrap(462)) - > [Session-1, SSL_NULL_WITH_NULL_NULL] wrap OK NOT_HANDSHAKING consumed=450 > produced=522 > 2015-11-27 23:22:48,773 DEBUG nio.ssl (SslConnection.java:process(347)) - > [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL > NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false > {AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=0,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1} > NOT_HANDSHAKING filled=0/0 flushed=522/0 > 2015-11-27 23:22:48,774 DEBUG nio.ssl (SslConnection.java:process(347)) - > [Session-1, SSL_NULL_WITH_NULL_NULL] SslConnection@172f1603 SSL > NOT_HANDSHAKING i/o/u=0/0/0 ishut=false oshut=false > {AsyncHttpConnection@c8a8734,g=HttpGenerator{s=2,h=0,b=0,c=-1},p=HttpParser{s=-5,l=10,c=0},r=1} > NOT_HANDSHAKING filled=0/0 flushed=0/0 > 2015-11-27 23:22:48,774 DEBUG server.Server (Server.java:handle(367)) - > RESPONSE > /gateway/default/webhdfs/data/v1/webhdfs/v1/user/admin/hdfs_test_2015_20151127_182243.txt > 200 handled=true > > > > Is there something I'm missing? How come the final response from Knox does > not contain the content-length header when it seems the response from > Hadoop does? > > > > -- > -Natasha D'Silva >
