[
https://issues.apache.org/jira/browse/MAPREDUCE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174316#comment-13174316
]
Konstantin Shvachko commented on MAPREDUCE-3592:
------------------------------------------------
This happened in -update mode, when the source and destination turned out to be
in sync.
I think DistCp does not need to update anything at all. But it does and gets
this exception. The exception is confusing, but it means setTimes() is not
supported for directories.
{code}
11/12/20 15:55:49 INFO tools.DistCp: sourcePathsCount(files+directories)=11602
11/12/20 15:55:49 INFO tools.DistCp: filesToCopyCount=0
11/12/20 15:55:49 INFO tools.DistCp: bytesToCopyCount=0
11/12/20 15:55:49 INFO tools.DistCp: filesToSkipCopyCount=5676
11/12/20 15:55:49 INFO tools.DistCp: bytesToSkipCopyCount=5339.3g
11/12/20 15:56:01 INFO tools.DistCp:
deletedPathsFromDestCount(files+directories)=0
11/12/20 15:56:01 INFO tools.DistCp: sourcePathsCount=11602
11/12/20 15:56:01 INFO tools.DistCp: filesToCopyCount=0
11/12/20 15:56:01 INFO tools.DistCp: bytesToCopyCount=0
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.FileNotFoundException: File /my-backup/my-data-input does
not exist.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimes(FSNamesystem.java:1096)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.setTimes(NameNode.java:1269)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:349)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1482)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1476)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at org.apache.hadoop.hdfs.DFSClient.setTimes(DFSClient.java:1347)
at
org.apache.hadoop.hdfs.DistributedFileSystem.setTimes(DistributedFileSystem.java:777)
at org.apache.hadoop.tools.DistCp.updateDestStatus(DistCp.java:827)
at org.apache.hadoop.tools.DistCp.finalize(DistCp.java:853)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:794)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:1028)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:1055)
{code}
> DistCp should not setTimes on directories
> -----------------------------------------
>
> Key: MAPREDUCE-3592
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3592
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
> Affects Versions: 0.22.0
> Reporter: Konstantin Shvachko
>
> On finalize() stage DistCp attempts to updateDestStatus(), which includes
> calling setTimes(). In HDFS setTimes() is not supported for directories,
> therefore DistCp fails if there are directories in the tree.
> It looks the failure occurs only when there are no files to copy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
