[
https://issues.apache.org/jira/browse/IGNITE-418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Veselovsky updated IGNITE-418:
-----------------------------------
Component/s: hadoop
> "terasort" example does not work on secondary HDFS with DUAL_SYNC, DUAL_ASYNC
> mode
> ----------------------------------------------------------------------------------
>
> Key: IGNITE-418
> URL: https://issues.apache.org/jira/browse/IGNITE-418
> Project: Ignite
> Issue Type: Bug
> Components: hadoop
> Affects Versions: sprint-1
> Reporter: Ivan Veselovsky
> Assignee: Ivan Veselovsky
> Priority: Critical
> Fix For: sprint-2
>
>
> On attempt to run "terasort" Hadoop example on top of IGFS backed by HDFS in
> DUAL_SYNC, DUAL_ASYNC mode we have failure: hadoop job finished Ok, but its
> fesult files are not found:
> + ./hadoop jar ../share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar
> teragen 1000000 /tmp/teragen
> 15/03/05 19:07:26 INFO terasort.TeraSort: Generating 1000000 using 2
> 15/03/05 19:07:26 INFO mapreduce.JobSubmitter: number of splits:2
> 15/03/05 19:07:26 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
> 15/03/05 19:07:27 INFO mapreduce.Job: The url to track the job: N/A
> 15/03/05 19:07:27 INFO mapreduce.Job: Running job:
> job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001
> 15/03/05 19:07:29 INFO mapreduce.Job: Job
> job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001 running in uber mode : false
> 15/03/05 19:07:29 INFO mapreduce.Job: map 0% reduce 0%
> 15/03/05 19:07:31 INFO mapreduce.Job: map 50% reduce 0%
> 15/03/05 19:07:31 INFO mapreduce.Job: map 100% reduce 100%
> 15/03/05 19:07:31 INFO mapreduce.Job: Job
> job_774fb810-21a7-48d9-81d1-ce1ac727841f_0001 completed successfully
> 15/03/05 19:07:31 INFO mapreduce.Job: Counters: 1
> org.apache.hadoop.examples.terasort.TeraGen$Counters
> CHECKSUM=2148987642402270
> + ./hadoop fs -ls /tmp/teragen
> ls: `/tmp/teragen': No such file or directory
> At the same time the following exception is on the Ignite node:
> [19:07:16] Topology snapshot [ver=1, nodes=1, CPUs=8, heap=1.0GB]
> [19:07:31,288][ERROR][Hadoop-task-774fb810-21a7-48d9-81d1-ce1ac727841f_1-COMMIT-0-0-#110%null%][IgfsMetaManager]
> Path rename in DUAL mode failed
> [source=/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000,
> destination=/tmp/teragen]
> class org.apache.ignite.IgniteCheckedException: Failed to rename (source path
> not found):
> /tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.fsException(IgfsMetaManager.java:2668)
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.access$1100(IgfsMetaManager.java:53)
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager$7.onSuccess(IgfsMetaManager.java:2052)
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager$7.onSuccess(IgfsMetaManager.java:2043)
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2527)
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.synchronizeAndExecute(IgfsMetaManager.java:2337)
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.renameDual(IgfsMetaManager.java:2103)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.rename(IgfsImpl.java:673)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.rename(HadoopIgfsInProc.java:128)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$5.apply(HadoopIgfsWrapper.java:133)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$5.apply(HadoopIgfsWrapper.java:130)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:289)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.rename(HadoopIgfsWrapper.java:130)
> at
> org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.rename(IgniteHadoopFileSystem.java:698)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:345)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
> at
> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
> at
> org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:56)
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.igfs.IgfsFileNotFoundException: Failed to
> rename (source path not found):
> /tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000
> ... 23 more
> [19:07:31,294][ERROR][Hadoop-task-774fb810-21a7-48d9-81d1-ce1ac727841f_1-COMMIT-0-0-#110%null%][HadoopRunnableTask]
> Task execution failed.
> class org.apache.ignite.IgniteCheckedException: class
> org.apache.ignite.IgniteCheckedException: Failed to rename FileStatus
> [path=igfs://igfs@localhost/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000,
> isDir=false, len=0, mtime=1425571651255, atime=1425571651255] to
> igfs://igfs@localhost/tmp/teragen
> at
> org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:59)
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:193)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed to rename FileStatus
> [path=igfs://igfs@localhost/tmp/teragen/_temporary/0/task_774fb810-21a7-48d9-81d1-ce1ac727841f_0001_m_000000,
> isDir=false, len=0, mtime=1425571651255, atime=1425571651255] to
> igfs://igfs@localhost/tmp/teragen
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:346)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310)
> at
> org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136)
> at
> org.apache.ignite.internal.processors.hadoop.v1.HadoopV1CleanupTask.run(HadoopV1CleanupTask.java:56)
> ... 7 more
> at
> org.apache.ignite.internal.processors.hadoop.HadoopUtils.transformException(HadoopUtils.java:273)
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:196)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:176)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:120)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:36)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:183)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:145)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
