Hi all, My Hive over Tez job fails with the following error while the same Hive over MR job succeeds, given the same input data and configurations.
*2014-01-26 13:46:19,823 ERROR [main] org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hive (auth:SIMPLE) cause:org.apache.tez.runtime.library.common.shuffle.impl.Shuffle$ShuffleError: error in shuffle in fetcher#115* * 2014-01-26 13:46:19,824 FATAL [main] org.apache.hadoop.mapred.YarnTezDagChild: Error running child : org.apache.tez.runtime.library.common.shuffle.impl.Shuffle$ShuffleError: error in shuffle in fetcher#115* * at org.apache.tez.runtime.library.common.shuffle.impl.Shuffle$RunShuffleCallable.call(Shuffle.java:224)* * at org.apache.tez.runtime.library.common.shuffle.impl.Shuffle$RunShuffleCallable.call(Shuffle.java:204)* * at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)* * at java.util.concurrent.FutureTask.run(FutureTask.java:138)* * at java.lang.Thread.run(Thread.java:662)* * Caused by: java.lang.OutOfMemoryError: Java heap space* * at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:56)* * at org.apache.hadoop.io.BoundedByteArrayOutputStream.<init>(BoundedByteArrayOutputStream.java:46)* * at org.apache.tez.runtime.library.common.shuffle.impl.MapOutput.<init>(MapOutput.java:98)* * at org.apache.tez.runtime.library.common.shuffle.impl.MergeManager.unconditionalReserve(MergeManager.java:295)* * at org.apache.tez.runtime.library.common.shuffle.impl.MergeManager.reserve(MergeManager.java:285)* * at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyMapOutput(Fetcher.java:395)* * at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.copyFromHost(Fetcher.java:325)* * at org.apache.tez.runtime.library.common.shuffle.impl.Fetcher.run(Fetcher.java:173)* Is it merely a jvm configuration issue (like a larger heap memory) ? Thanks, Manu Zhang
