[
https://issues.apache.org/jira/browse/IGNITE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Ozerov reassigned IGNITE-3312:
---------------------------------------
Assignee: Vladimir Ozerov (was: Ivan Veselovsky)
> [Test]
> HadoopMapReduceEmbeddedSelfTest.testMultiReducerWholeMapReduceExecution
> flakily fails.
> ---------------------------------------------------------------------------------------------
>
> Key: IGNITE-3312
> URL: https://issues.apache.org/jira/browse/IGNITE-3312
> Project: Ignite
> Issue Type: Bug
> Components: IGFS
> Reporter: Ivan Veselovsky
> Assignee: Vladimir Ozerov
> Labels: test
>
> HadoopMapReduceEmbeddedSelfTest.testMultiReducerWholeMapReduceExecution fails
> with ~20% probability .
> Failure cause is either the 1st or 2nd marked line in the following code
> (org.apache.ignite.internal.processors.igfs.IgfsMetaManager#create) :
> {code}
> // Check: can we overwrite it?
> if (!overwrite)
> throw new
> IgfsPathAlreadyExistsException("Failed to create a file: " + path); // *****
> #1
> // Check if file already opened for write.
> if (oldInfo.lockId() != null)
> throw new IgfsException("File is already
> opened for write: " + path); // ***** #2
> {code}
> Diagnostic shows that the same file really attempted to be created several
> times on one thread.
> Situation #2 reproducible with the following stack when IGFS is used as a
> secondary fs:
> {code}
> Hadoop-task-efea2ae1-09cb-4c49-9465-dcbbedee1835_1-REDUCE-2-0-#679%hadoop.HadoopMapReduceEmbeddedSelfTest2%@20675,
> prio=5, in group 'ignite', status: 'RUNNING'
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.create(IgfsMetaManager.java:2857)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1051)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1019)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1823)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:1019)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:990)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:359)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:357)
> at
> org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:49)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.create(HadoopIgfsInProc.java:357)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:258)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:255)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:255)
> at
> org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:632)
> at
> org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.create(IgniteHadoopIgfsSecondaryFileSystem.java:406)
> at
> org.apache.ignite.internal.processors.igfs.IgfsSecondaryFileSystemCreateContext.create(IgfsSecondaryFileSystemCreateContext.java:87)
> at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.create(IgfsMetaManager.java:2922)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1051)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1019)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1823)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:1019)
> at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:990)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:359)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:357)
> at
> org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:54)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.create(HadoopIgfsInProc.java:357)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:258)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:255)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)
> at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:255)
> at
> org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:632)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:799)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at
> org.apache.ignite.internal.processors.hadoop.v1.HadoopV1OutputCollector.<init>(HadoopV1OutputCollector.java:68)
> at
> org.apache.ignite.internal.processors.hadoop.v1.HadoopV1Task$1.<init>(HadoopV1Task.java:73)
> at
> org.apache.ignite.internal.processors.hadoop.v1.HadoopV1Task.collector(HadoopV1Task.java:72)
> at
> org.apache.ignite.internal.processors.hadoop.v1.HadoopV1ReduceTask.run(HadoopV1ReduceTask.java:65)
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:249)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:201)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0(HadoopRunnableTask.java:144)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:116)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:114)
> at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.runAsJobOwner(HadoopV2TaskContext.java:544)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:114)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:46)
> at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:186)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> situation #1 also happens when IGFS plays role of the 2ndary file system with
> the following stack :
> {code}
> 72
> Hadoop-task-aebecef6-5cf2-4b66-9674-9fd8644e1183_1-REDUCE-0-0-#368%hadoop.HadoopMapReduceEmbeddedSelfTest0%@14531,
> prio=5, in group 'ignite', status: 'RUNNING'
> 73 at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.create(IgfsMetaManager.java:2853)
> 74 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1051)
> 75 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1019)
> 76 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1823)
> 77 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:1019)
> 78 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:990)
> 79 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:359)
> 80 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:357)
> 81 at
> org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:49)
> 82 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.create(HadoopIgfsInProc.java:357)
> 83 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:258)
> 84 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:255)
> 85 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)
> 86 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:255)
> 87 at
> org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:632)
> 88 at
> org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem.create(IgniteHadoopIgfsSecondaryFileSystem.java:406)
> 89 at
> org.apache.ignite.internal.processors.igfs.IgfsSecondaryFileSystemCreateContext.create(IgfsSecondaryFileSystemCreateContext.java:87)
> 90 at
> org.apache.ignite.internal.processors.igfs.IgfsMetaManager.create(IgfsMetaManager.java:2922)
> 91 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1051)
> 92 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl$15.call(IgfsImpl.java:1019)
> 93 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.safeOp(IgfsImpl.java:1823)
> 94 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.create0(IgfsImpl.java:1019)
> 95 at
> org.apache.ignite.internal.processors.igfs.IgfsImpl.create(IgfsImpl.java:990)
> 96 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:359)
> 97 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc$15.apply(HadoopIgfsInProc.java:357)
> 98 at
> org.apache.ignite.igfs.IgfsUserContext.doAs(IgfsUserContext.java:54)
> 99 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.create(HadoopIgfsInProc.java:357)
> 100 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:258)
> 101 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper$15.apply(HadoopIgfsWrapper.java:255)
> 102 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.withReconnectHandling(HadoopIgfsWrapper.java:310)
> 103 at
> org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsWrapper.create(HadoopIgfsWrapper.java:255)
> 104 at
> org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem.create(IgniteHadoopFileSystem.java:632)
> 105 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:906)
> 106 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:887)
> 107 at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:784)
> 108 at
> org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:132)
> 109 at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.prepareWriter(HadoopV2Task.java:103)
> 110 at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2ReduceTask.run0(HadoopV2ReduceTask.java:57)
> 111 at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.run(HadoopV2Task.java:54)
> 112 at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:249)
> 113 at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:201)
> 114 at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0(HadoopRunnableTask.java:144)
> 115 at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:116)
> 116 at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:114)
> 117 at
> org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.runAsJobOwner(HadoopV2TaskContext.java:544)
> 118 at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:114)
> 119 at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:46)
> 120 at
> org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:186)
> 121 at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> 122 at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
