Hey guys, in an customer environment certain Tez jobs fail to start
On the client side it looks like: —————————————————————————— INFO [2015-04-08 15:19:30.213] [MrPlanRunnerV2] (YarnClientImpl.java:204) - Submitted application application_1428177121154_0065 INFO [2015-04-08 15:19:30.214] [MrPlanRunnerV2] (TezClient.java:357) - The url to track the Tez Session: http://master:8088/proxy/application_1428177121154_0065/ INFO [2015-04-08 15:19:33.219] [MrPlanRunnerV2] (TezClient.java:556) - App did not succeed. Diagnostics: Application application_1428177121154_0065 failed 2 times due to AM Container for appattempt_1428177121154_0065_000002 exited with exitCode: 134 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 1: 10818 Aborted (core dumped) /opt/teradata/jvm64/jdk7/bin/java -Xmx819m -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dapple.awt.UIElement=true -Djava.awt.headless=true -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/data3/hadoop/yarn/log/application_1428177121154_0065/container_1428177121154_0065_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session > /data3/hadoop/yarn/log/application_1428177121154_0065/container_1428177121154_0065_02_000001/stdout 2> /data3/hadoop/yarn/log/application_1428177121154_0065/container_1428177121154_0065_02_000001/stderr org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 1: 10818 Aborted (core dumped) /opt/teradata/jvm64/jdk7/bin/java -Xmx819m -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps -XX:+UseNUMA -XX:+UseParallelGC -Dapple.awt.UIElement=true -Djava.awt.headless=true -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/data3/hadoop/yarn/log/application_1428177121154_0065/container_1428177121154_0065_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session > /data3/hadoop/yarn/log/application_1428177121154_0065/container_1428177121154_0065_02_000001/stdout 2> /data3/hadoop/yarn/log/application_1428177121154_0065/container_1428177121154_0065_02_000001/stderr at org.apache.hadoop.util.Shell.runCommand(Shell.java:505) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 134 .Failing this attempt.. Failing the application. —————————————————————————— Then you have that for the task: —————————————————————————— Log Type: stderr Log Length: 429 java: malloc.c:3090: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. Log Type: stdout Log Length: 0 —————————————————————————— Any ideas ? Johannes
