[
https://issues.apache.org/jira/browse/TEZ-1421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278597#comment-14278597
]
Tsuyoshi OZAWA commented on TEZ-1421:
-------------------------------------
Attaching a stack trace I faced recently:
{quote}
15/01/09 08:10:41 INFO mapreduce.Job: map 98% reduce 0%
15/01/09 08:10:42 INFO mapreduce.Job: map 99% reduce 0%
15/01/09 08:10:44 INFO mapreduce.Job: map 100% reduce 0%
15/01/09 08:10:47 INFO mapreduce.Job: Job job_1420765352344_0022 failed with
state FAILED due to: Vertex failed, vertexName=finalreduce,
vertexId=vertex_1420765352344_0022_1_01, diagnostics=[Task failed, taskId=task_
1420765352344_0022_1_01_000000, diagnostics=[TaskAttempt 0 failed, info=[Error:
exceptionThrown=org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
error in shuffle in MemtoDiskMerger [
initialmap]
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:347)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:327)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:127)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:117)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.runCombineProcessor(MergeManager.java:480)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$InMemoryMerger.merge(MergeManager.java:615)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:123)
... 5 more
, errorMessage=Shuffle Runner
Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
error in shuffle in MemtoDiskMerger [initialmap]
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:347)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:327)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:127)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:117)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.runCombineProcessor(MergeManager.java:480)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$InMemoryMerger.merge(MergeManager.java:615)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:123)
... 5 more
], TaskAttempt 1 failed, info=[Error:
exceptionThrown=org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
error in shuffle in MemtoDiskMerger [initialmap]
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:347)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:327)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:127)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:117)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.runCombineProcessor(MergeManager.java:480)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$InMemoryMerger.merge(MergeManager.java:615)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:123)
... 5 more
, errorMessage=Shuffle Runner
Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
error in shuffle in MemtoDiskMerger [initialmap]
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:347)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:327)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:127)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:117)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.runCombineProcessor(MergeManager.java:480)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$InMemoryMerger.merge(MergeManager.java:615)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:123)
... 5 more
], TaskAttempt 2 failed, info=[Error:
exceptionThrown=org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
error in shuffle in MemtoDiskMerger [initialmap]
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:347)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:327)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:127)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:117)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.runCombineProcessor(MergeManager.java:480)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$InMemoryMerger.merge(MergeManager.java:615)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:123)
... 5 more
, errorMessage=Shuffle Runner
Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
error in shuffle in MemtoDiskMerger [initialmap]
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:347)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:327)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:127)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:117)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.runCombineProcessor(MergeManager.java:480)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$InMemoryMerger.merge(MergeManager.java:615)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:123)
... 5 more
], TaskAttempt 3 failed, info=[Error:
exceptionThrown=org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
error in shuffle in MemtoDiskMerger [initialmap]
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:347)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:327)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:127)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:117)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.runCombineProcessor(MergeManager.java:480)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$InMemoryMerger.merge(MergeManager.java:615)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:123)
... 5 more
, errorMessage=Shuffle Runner
Failed:org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$ShuffleError:
error in shuffle in MemtoDiskMerger [initialmap]
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:347)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.Shuffle$RunShuffleCallable.call(Shuffle.java:327)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
at
org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:127)
at
org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:117)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.runCombineProcessor(MergeManager.java:480)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager$InMemoryMerger.merge(MergeManager.java:615)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeThread.run(MergeThread.java:89)
Caused by: java.lang.NullPointerException
at
java.util.concurrent.ConcurrentHashMap.hash(ConcurrentHashMap.java:333)
at
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:988)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:123)
... 5 more
]], Vertex failed as one or more tasks failed. failedTasks:1, Vertex
vertex_1420765352344_0022_1_01 [finalreduce] killed/failed due to:null]. DAG
failed due to vertex failure. failedVertices:1 killedVertices:0
{code}
> MRCombiner throws NPE in MapredWordCount on master branch
> ---------------------------------------------------------
>
> Key: TEZ-1421
> URL: https://issues.apache.org/jira/browse/TEZ-1421
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Tsuyoshi OZAWA
> Priority: Blocker
>
> I tested MapredWordCount against 70GB generated by RandowTextWriter. When a
> Combiner runs, it throws NPE. It looks setCombinerClass doesn't work
> correctly.
> {quote}
> Caused by: java.lang.RuntimeException: java.lang.NullPointerException
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131)
> at
> org.apache.tez.mapreduce.combine.MRCombiner.runOldCombiner(MRCombiner.java:122)
> at org.apache.tez.mapreduce.combine.MRCombiner.combine(MRCombiner.java:112)
> at
> org.apache.tez.runtime.library.common.shuffle.impl.MergeManager.runCombineProcessor(MergeManager.java:472)
> at
> org.apache.tez.runtime.library.common.shuffle.impl.MergeManager$InMemoryMerger.merge(MergeManager.java:605)
> at
> org.apache.tez.runtime.library.common.shuffle.impl.MergeThread.run(MergeThread.java:89)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
