[
https://issues.apache.org/jira/browse/HDFS-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-3405:
--------------------------------
Attachment: HDFS-3405.patch
Attaching the updated patch as per Andrew's comments.
1. {{GetImageServlet}} renamed to {{ImageServlet}}, *but patch shows complete
deletion and addition of files.*
2. url "/getimage" changed to "/imagetransfer"
All other comments also addressed except below.
bq. We also should update the hdfs-default.xml description of
dfs.image.transfer.timeout, since it is in fact a socket timeout now that the
GETs aren't nested. I think we should also lower it back down to 60s, a more
normal value for a socket timeout.
Javadoc updated. But changing default value back may need some more test and
conclusion for the optimal value.
bq. In writeFileToPutRequest, we can use IOUtils.copy instead of doing our own
buffering. The existing for loop syntax is also messy, it'd be better as a
while loop.
Instead of this, re-used existing code to use throttler.
bq. Have you tried this with a large fsimage, e.g. 2GB+? Last time I looked
into this, we could sometimes run into issues with HttpURLConnection.
This is not done yet. I will do it and post results.
bq. as well as unit tests for secure HA/non-HA environments if we don't already
have that, and the SPNEGO stuff.
I am not sure these tests are already there is secure mode or not.
But I have tested manually. Works fine. I will try to findout how to add secure
tests.
> Checkpointing should use HTTP POST or PUT instead of GET-GET to send merged
> fsimages
> ------------------------------------------------------------------------------------
>
> Key: HDFS-3405
> URL: https://issues.apache.org/jira/browse/HDFS-3405
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 3.0.0, 2.0.5-alpha
> Reporter: Aaron T. Myers
> Assignee: Vinayakumar B
> Attachments: HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch,
> HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch,
> HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch,
> HDFS-3405.patch, HDFS-3405.patch, HDFS-3405.patch
>
>
> As Todd points out in [this
> comment|https://issues.apache.org/jira/browse/HDFS-3404?focusedCommentId=13272986&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13272986],
> the current scheme for a checkpointing daemon to upload a merged fsimage
> file to an NN is to issue an HTTP get request to tell the target NN to issue
> another GET request back to the checkpointing daemon to retrieve the merged
> fsimage file. There's no fundamental reason the checkpointing daemon can't
> just use an HTTP POST or PUT to send back the merged fsimage file, rather
> than the double-GET scheme.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
