[
https://issues.apache.org/jira/browse/HDFS-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15718649#comment-15718649
]
Oscar Guadilla commented on HDFS-11189:
---------------------------------------
Hi Weiwei Yang,
I will try to explain it better. Let's focus on a GET operation:
- step 1 --> http request to the namenode from the client to get a
concrete file
- step 2 --> the namenode looks up where is the file and sends to the
client a 307 (forward) response with the datanode uri in the location
- step 3 --> the client reads the response (307) and makes a second http
client to the suggested datanode
- step 4 --> the datanode reads the requested file and gives it back the
content to the client
It works fine if the httpconnection has the default value (true) in the
getFollowRedirects atributte; otherwise if fails because it stops in the step3.
I know that this atributte followRedirects should be by default true but we
detected that certain libraries modify that behaviour and webhdfs worked
poorly.
To solve this problem what we did was change the FileSystem class to
instead of supposing that FollowRedirects is true always set the value to true.
Of course we used the instance scope using setInstanceFollowRedirects instead
of using the global parameter.
PS: It was a nighmare of two weeks for us and we wanted to contribute with
the solution we found. We are just users of hadoop and we wanted to give back
something from us to the project.
Regards,
Oscar
> WebHDFS and FollowRedirects
> ---------------------------
>
> Key: HDFS-11189
> URL: https://issues.apache.org/jira/browse/HDFS-11189
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
> Affects Versions: 2.7.3
> Environment: We are using webhdfs from a j2ee environment with java
> 6. In the web application we have a lot of libraries: spring, jersey,
> jackson...
> Reporter: Oscar Guadilla
>
> In some cases with simple operations (get file - non two step operation) we
> detect that FollowRedirects flag of http comes as "false" and it makes
> webhdfs crash.
> We don't really know which library change this behavior and it is really
> strange because the change is done directly in the jvm because in other
> wars/ears of the same j2ee server it fails also. If we restart the j2ee
> server it starts working again.
> To fix the problem we changed the WebHdfsFileSystem class adding
> "setInstanceFollowRedirects(true)" in the connection management instead of
> supposing that it should be true and it works fine.
> The same problem arises both in 1.x and 2.x webhdfs. We didn't test in 3.x
> Could you fix it? We did it in our environment but it would fine in it could
> be included in the next releases.
> Thanks in advance,
> Oscar
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
