Xiaomin Zhang created SQOOP-3171:
------------------------------------
Summary: Import as parquet jobs failed randomly while multiple
jobs concurrently importing into targets with same parent
Key: SQOOP-3171
URL: https://issues.apache.org/jira/browse/SQOOP-3171
Project: Sqoop
Issue Type: Bug
Reporter: Xiaomin Zhang
Running multiple parquet import jobs concurrently into below target directories:
hdfs://ns/path/dataset1
hdfs://ns/path/dataset2
In some cases, one of the sqoop job will be failed with below error:
17/03/19 08:21:21 INFO mapreduce.Job: Job job_1488289274600_188649 failed with
state FAILED due to: Job commit failed: org.kitesdk.data.DatasetIOException:
Could not cleanly delete path:hdfs://ns/path/.temp/job_1488289274600_188649
at
org.kitesdk.data.spi.filesystem.FileSystemUtil.cleanlyDelete(FileSystemUtil.java:239)
at
org.kitesdk.data.spi.filesystem.TemporaryFileSystemDatasetRepository.delete(TemporaryFileSystemDatasetRepository.java:61)
at
org.kitesdk.data.mapreduce.DatasetKeyOutputFormat$MergeOutputCommitter.commitJob(DatasetKeyOutputFormat.java:395)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274)
at
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.FileNotFoundException: File hdfs://ns/path/.temp does not
exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:705)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:106)
at
org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:763)
at
org.apache.hadoop.hdfs.DistributedFileSystem$15.doCall(DistributedFileSystem.java:759)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:759)
at
org.kitesdk.data.spi.filesystem.FileSystemUtil.cleanlyDelete(FileSystemUtil.java:226)
This is due to:
https://issues.cloudera.org/browse/KITE-1155
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
