[
https://issues.apache.org/jira/browse/HADOOP-17763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364247#comment-17364247
]
Ayush Saxena commented on HADOOP-17763:
---------------------------------------
Thanx [~BilwaST] for the report and fix.
{code:java}
- metaFolder = createMetaFolderPath();
{code}
I think this removal from {{createAndSubmitJob}} is gonna fetch you a bunch of
test failures due to NPE. Moreover that is a public method, we can let it stay
there as is and we can only tackle the path stuff here.
The new path:
{code:java}
+ Path metaFolderPath = new Path(DistCp.class.getSimpleName() + "_"
+ + System.currentTimeMillis() + "_" + rand.nextInt(Integer.MAX_VALUE));
{code}
I am not sure if changing the path directly from the staging directory to
something in user home directory will be a safe move from compatibility point
of view. If the user has some quotas set or has some certain permissions, or in
case of federation, which prevents creation of a file there, we would land up
in a mess.
Thoughts on having a custom option for {{MetaFolderPath}}, if this is specified
it can be used else the default behaviour? Will that solve your purpose?
> DistCp job fails when AM is killed
> ----------------------------------
>
> Key: HADOOP-17763
> URL: https://issues.apache.org/jira/browse/HADOOP-17763
> Project: Hadoop Common
> Issue Type: Bug
> Reporter: Bilwa S T
> Assignee: Bilwa S T
> Priority: Major
> Attachments: HADOOP-17763.001.patch
>
>
> Job fails as tasks fail with below exception
> {code:java}
> 2021-06-11 18:48:47,047 | ERROR | IPC Server handler 0 on 27101 | Task:
> attempt_1623387358383_0006_m_000000_1000 - exited :
> java.io.FileNotFoundException: File does not exist:
> hdfs://hacluster/staging-dir/dsperf/.staging/_distcp-646531269/fileList.seq
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1637)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1630)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1645)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1863)
> at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1886)
> at
> org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$1.run(YarnChild.java:183)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1761)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:177)
> | TaskAttemptListenerImpl.java:304{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
