[
https://issues.apache.org/jira/browse/HDFS-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701179#comment-16701179
]
Siyao Meng edited comment on HDFS-14092 at 11/28/18 12:50 AM:
--------------------------------------------------------------
I've taken a look at the client.
{code:java}
/** Expects HTTP response 307 "Temporary Redirect". */
public static class TemporaryRedirectOp implements Op {
static final TemporaryRedirectOp CREATE = new TemporaryRedirectOp(
PutOpParam.Op.CREATE);
static final TemporaryRedirectOp APPEND = new TemporaryRedirectOp(
PostOpParam.Op.APPEND);
static final TemporaryRedirectOp OPEN = new TemporaryRedirectOp(
GetOpParam.Op.OPEN);
static final TemporaryRedirectOp GETFILECHECKSUM = new TemporaryRedirectOp(
GetOpParam.Op.GETFILECHECKSUM);
...
{code}
Only for those 4 operations the WebHDFS client would: 1. Send a request to
server and expect 307 Temporary Redirect; 2. Grab the new URL from HTTP header
Location key and send a second request to it (which points to a DN).
WebHdfsFileSystem#connect(URL) performs the logic.
-So my understanding for the problem is that, the Java 6 library was faulty so
it couldn't redirect the request automatically (like user browsers normally do
when encountered HTTP 3xx). Therefore, we had to do it manually. CMIIW-
I just read HDFS-2540 which added this two-step code. And I am a bit confused.
What is the expected logic if we want to remove two-step?
Any comments?
CC [~szetszwo]
was (Author: smeng):
I've taken a look at the client.
{code:java}
/** Expects HTTP response 307 "Temporary Redirect". */
public static class TemporaryRedirectOp implements Op {
static final TemporaryRedirectOp CREATE = new TemporaryRedirectOp(
PutOpParam.Op.CREATE);
static final TemporaryRedirectOp APPEND = new TemporaryRedirectOp(
PostOpParam.Op.APPEND);
static final TemporaryRedirectOp OPEN = new TemporaryRedirectOp(
GetOpParam.Op.OPEN);
static final TemporaryRedirectOp GETFILECHECKSUM = new TemporaryRedirectOp(
GetOpParam.Op.GETFILECHECKSUM);
...
{code}
Only for those 4 operations the WebHDFS client would: 1. Send a request to
server and expect 307 Temporary Redirect; 2. Grab the new URL from HTTP header
Location key and send a second request to it (which points to a DN).
WebHdfsFileSystem#connect(URL) performs the logic.
So my understanding for the problem is that, the Java 6 library was faulty so
it couldn't redirect the request automatically (like user browsers normally do
when encountered HTTP 3xx). Therefore, we had to do it manually. CMIIW
> Remove two-step create/append in WebHdfsFileSystem
> --------------------------------------------------
>
> Key: HDFS-14092
> URL: https://issues.apache.org/jira/browse/HDFS-14092
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
> Affects Versions: 3.2.0
> Reporter: Daniel Templeton
> Assignee: Siyao Meng
> Priority: Major
>
> Per the javadoc on the {{WebHdfsFileSystem.connect()}} method:
> {code} /**
> * Two-step requests redirected to a DN
> *
> * Create/Append:
> * Step 1) Submit a Http request with neither auto-redirect nor data.
> * Step 2) Submit another Http request with the URL from the Location
> header
> * with data.
> *
> * The reason of having two-step create/append is for preventing clients
> to
> * send out the data before the redirect. This issue is addressed by the
> * "Expect: 100-continue" header in HTTP/1.1; see RFC 2616, Section 8.2.3.
> * Unfortunately, there are software library bugs (e.g. Jetty 6 http
> server
> * and Java 6 http client), which do not correctly implement "Expect:
> * 100-continue". The two-step create/append is a temporary workaround for
> * the software library bugs.
> *
> * Open/Checksum
> * Also implements two-step connects for other operations redirected to
> * a DN such as open and checksum
> */{code}
> We should validate that it's safe to remove the two-step process and do so.
> FYI, [~smeng].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
