[
https://issues.apache.org/jira/browse/HADOOP-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17320445#comment-17320445
]
Ayush Saxena commented on HADOOP-17611:
---------------------------------------
Seems there are two PRs, More or less doing the same thing I guess, I just had
a glance on the second one,
So, who ever plans to chase this, Couple of points to keep in mind:
* We need a test in {{AbstractContractDistCpTest}} which all the
{{FileSystems}} can also use
* Should cover *two scenarios.* First When preserve Time is specified it
should preserve time and when not it shouldn't in case of parallel copy. The
latter case is working ok as of now, To make sure we don't change the behaviour
* The parent modification time is to be preserved when the parent is in the
scope of copy, not always. say your are copying /dir/fil1 to /dir1/file2 using
parallel copy, then we don't touch /dir1 AFAIK
The above are the basic requirements, Now the below stuff, If possible we
should do:
* For parent directories preserve only once, say if you have 10K files under
that parent, then do that setTimes 10K times.
* And if the parallel copy is enabled, there is no point of preserving before
concat operation, we can save that call.
This isn't a one liner, and throw some challenges, So, please decide who wants
to chase this and together work on one PR only.
> Distcp parallel file copy breaks the modification time
> ------------------------------------------------------
>
> Key: HADOOP-17611
> URL: https://issues.apache.org/jira/browse/HADOOP-17611
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Adam Maroti
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> The commit HADOOP-11794. Enable distcp to copy blocks in parallel.
> (bf3fb585aaf2b179836e139c041fc87920a3c886) broke the modification time of
> large files.
>
> In CopyCommitter.java inside concatFileChunks Filesystem.concat is called
> which changes the modification time therefore the modification times of files
> copeid by distcp will not match the source files. However this only occurs
> for large enough files, which are copied by splitting them up by distcp.
> In concatFileChunks before calling concat extract the modification time and
> apply that to the concatenated result-file after the concat. (probably best
> -after- before the rename()).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
