[
https://issues.apache.org/jira/browse/MAPREDUCE-6766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark S updated MAPREDUCE-6766:
------------------------------
Summary: Concurrent local job failures due to uniqueNumberGenerator = new
AtomicLong(System.currentTimeMillis()) (was: Concurrent Local Job Failures due
to uniqueNumberGenerator = new AtomicLong(System.currentTimeMillis()))
> Concurrent local job failures due to uniqueNumberGenerator = new
> AtomicLong(System.currentTimeMillis())
> -------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6766
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6766
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.3.0, 2.7.1
> Environment: Druid 0.8.3
> Reporter: Mark S
>
> I am seeing the following exception when attempting to execute multiple
> Hadoop Local Jobs with Druid.
> {code}
> java.io.IOException: Rename cannot overwrite non empty destination directory
> /tmp/hadoop-username/mapred/local/1472019105135
> {code}
> From a quick look at the Hadoop code base, it seems that the
> uniqueNumberGenerator for the LocalDistributedCacheManager is based on the
> System time, and this appears to cause problems for concurrent jobs.
> {code}
> // Generating unique numbers for FSDownload.
> AtomicLong uniqueNumberGenerator =
> new AtomicLong(System.currentTimeMillis());
> {code}
> I am pretty sure the following line of code is responsible, and this seems to
> exist in latter versions of such as 2.7.1:
> * [Hadoop 2.3.0 -
> LocalDistributedCacheManager.java#L96|https://github.com/apache/hadoop/blob/release-2.3.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L96]
> * [Hadoop 2.7.1 -
> LocalDistributedCacheManager.java#L96|https://github.com/apache/hadoop/blob/release-2.7.1/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalDistributedCacheManager.java#L96]
> h3. Full Stack Trace
> {code}
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> ~[guava-16.0.1.jar:?]
> at
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:138)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> at
> io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:206)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> at
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:285)
> [druid-indexing-service-0.8.3.jar:0.8.3]
> at
> io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:265)
> [druid-indexing-service-0.8.3.jar:0.8.3]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_71]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_71]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_71]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_71]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_71]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
> at
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> ... 7 more
> Caused by: java.lang.RuntimeException: java.io.IOException:
> java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot
> overwrite non empty destination directory
> /tmp/hadoop-username/mapred/local/1472019105135
> at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:211)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at
> io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at
> io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_71]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_71]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_71]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
> at
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> ... 7 more
> Caused by: java.io.IOException: java.util.concurrent.ExecutionException:
> java.io.IOException: Rename cannot overwrite non empty destination directory
> /tmp/hadoop-username/mapred/local/1472019105135
> at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:143)
> ~[?:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
> ~[?:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> ~[?:?]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
> ~[?:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_71]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_71]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> ~[?:?]
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]
> at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:199)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at
> io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at
> io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_71]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_71]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_71]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
> at
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> ... 7 more
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
> Rename cannot overwrite non empty destination directory
> /tmp/hadoop-username/mapred/local/1472019105135
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> ~[?:1.8.0_71]
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> ~[?:1.8.0_71]
> at
> org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:139)
> ~[?:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
> ~[?:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
> ~[?:?]
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
> ~[?:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_71]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_71]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> ~[?:?]
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]
> at io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:199)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at io.druid.indexer.JobHelper.runJobs(JobHelper.java:321)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at
> io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96)
> ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]
> at
> io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_71]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_71]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_71]
> at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]
> at
> io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135)
> ~[druid-indexing-service-0.8.3.jar:0.8.3]
> ... 7 more
> Caused by: java.io.IOException: Rename cannot overwrite non empty destination
> directory /tmp/hadoop-username/mapred/local/1472019105135
> at
> org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735)
> ~[?:?]
> at org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236)
> ~[?:?]
> at
> org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678)
> ~[?:?]
> at org.apache.hadoop.fs.FileContext.rename(FileContext.java:958) ~[?:?]
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366)
> ~[?:?]
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
> ~[?:?]
> ... 4 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
