[
https://issues.apache.org/jira/browse/HDFS-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13415403#comment-13415403
]
Alejandro Abdelnur commented on HDFS-3577:
------------------------------------------
@Eli, chunked encoding is enabled by default in modern servlet containers, it
normally kicks in after a threshold (I believe in Tomcat this threshold is 8K),
don't know what is the threshold of jetty.
Another thing to keep in mind, is that the HttpURLConnection fully caches in
memory the outputstream (uploading data) and this seems hardcoded in the JDK.
This would pose a limitation in writing files through webhdfs/httpfs. A
solution may be to refactor WebHdfsFileSystem to use HttpClient (arghh, we may
need to tweak hadoop-auth to work with HttpClient).
> WebHdfsFileSystem can not read files larger than 24KB
> -----------------------------------------------------
>
> Key: HDFS-3577
> URL: https://issues.apache.org/jira/browse/HDFS-3577
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 0.23.3, 2.0.0-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Tsz Wo (Nicholas), SZE
> Priority: Blocker
> Attachments: h3577_20120705.patch, h3577_20120708.patch,
> h3577_20120714.patch
>
>
> If reading a file large enough for which the httpserver running
> webhdfs/httpfs uses chunked transfer encoding (more than 24K in the case of
> webhdfs), then the WebHdfsFileSystem client fails with an IOException with
> message *Content-Length header is missing*.
> It looks like WebHdfsFileSystem is delegating opening of the inputstream to
> *ByteRangeInputStream.URLOpener* class, which checks for the *Content-Length*
> header, but when using chunked transfer encoding the *Content-Length* header
> is not present and the *URLOpener.openInputStream()* method thrown an
> exception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
