[
https://issues.apache.org/jira/browse/MAPREDUCE-7032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324710#comment-16324710
]
Robert Kanter commented on MAPREDUCE-7032:
------------------------------------------
The approach looks good overall. Here's some comments:
- {{fileSystem.getFileBlockLocations(targetPath, 0, 1024).length <
acceptableReplication}} only checks for the first block (well, that sort of
depends on the block size, but it's likely the first block). We should
probably check all of the blocks to be safe in case the first block is
replicated well, but some other block is not.
-- Unfortunately, HDFS doesn't have a simple API for that. You're probably
going to have to get all the block locations and then reorganize them to figure
out if any blocks are lower than {{acceptableReplication}}
- Instead of "Operation timed out in %d seconds", we should use a more helpful
message like "Timed out after %d seconds while waiting for acceptable
replication of %d (current replication is %d)" or something like that.
- The description in the arguments for the "timeout" should say what the
timeout is for. i.e. "Desired timeout for the acceptable replication in
seconds"
- The description for the desired replications and timeout should say what
their default values are.
- It would be good to add or update the unit tests to test the replication
changes
This isn't part of this patch, but I was just thinking about the failure
scenario when the uploader fails halfway (e.g. network issue). The tool will
print an error, but the partially uploaded file is left on the system and the
tool still has a clean exit code. Obviously if the network is out the tool
can't delete the file, but we should at least try to and also return a bad exit
code so automation tools know that it failed.
> Add the ability to specify a delayed replication count
> ------------------------------------------------------
>
> Key: MAPREDUCE-7032
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7032
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Miklos Szegedi
> Assignee: Miklos Szegedi
> Attachments: MAPREDUCE-7032.000.patch
>
>
> Setting the delayed replication count is more robust
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
