Re: Failing attemption at org.apache.tez.client.TezClient.waitTillReady

[no jihun]

Thanks Hitesh Shah.

It claims

2016-02-12 14:59:07,388 [ERROR] [main] |app.DAGAppMaster|: Error starting
DAGAppMaster

java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
at
org.apache.hadoop.yarn.conf.YarnConfiguration.addDeprecatedKeys(YarnConfiguration.java:
79)
at
org.apache.hadoop.yarn.conf.YarnConfiguration.(YarnConfiguration.java:
73)
at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2271)

I am not sure but according to this thread(
http://grokbase.com/t/cloudera/cdh-user/12765svj61/libjars-and-hadoop-jar-command
)
this perhaps caused by $HADOOP_CLASSPATH problem.

But I wander should I copy "tez-dist/target/tez-0.8.2" to all cluster then
export below?
export TEZ_JARS=/home1/apps/tez-0.8.2
export TEZ_CONF_DIR=$TEZ_JARS/conf
export
HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS/*:$TEZ_JARS/lib/*:$HADOOP_CLASSPATH

I did this only on name nodes.


2016-02-12 16:47 GMT+09:00 Hitesh Shah :

> Run the following command: “bin/yarn logs -applicationId
> application_1452243782005_0292” . This should give you the logs for
> container_1452243782005_0292_02_01 which may shed more light on why the
> Tez ApplicationMaster is failing to launch when triggered via Hive.
>
> thanks
> — Hitesh
>
>
>
> On Feb 11, 2016, at 10:48 PM, no jihun  wrote:
>
> > Hi all.
> >
> > When I execute a query on hive I got an error below.(so do in hive cli)
> > no more detailed log found.
> >
> > what should I check?
> > any advice will be appreciated.
> >
> > versions
> > - tez-0.8.2
> > - hadoop 2.6.0
> >
> > ---
> >
> > hive > set hive.execution.engine=tez;
> > hive > select count(*) from contents;
> >
> > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated.
> Please use org.apache.hadoop.log.metrics.EventCounter in all the
> log4j.properties files.
> >
> > Logging initialized using configuration in
> file:/home1/eco/hive/conf/hive-log4j.properties
> > hive> set hive.execution.engine=tez;
> > hive> select count(*) from agg_band_contents;
> > Query ID = irteam_20160212145903_9300f3b2-3942-4423-8586-73d2eaff9e58
> > Total jobs = 1
> > Launching Job 1 out of 1
> > Exception in thread "Thread-10" java.lang.RuntimeException:
> org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown.
> Application application_1452243782005_0292 failed 2 times due to AM
> Container for appattempt_1452243782005_0292_02 exited with  exitCode: 1
> > For more detailed output, check application tracking page:
> http://xstathn003:8088/proxy/application_1452243782005_0292/Then, click
> on links to logs of each attempt.
> > Diagnostics: Exception from container-launch.
> > Container id: container_1452243782005_0292_02_01
> > Exit code: 1
> > Stack trace: ExitCodeException exitCode=1:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> > at org.apache.hadoop.util.Shell.run(Shell.java:455)
> > at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> > at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> > at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> > at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > Container exited with a non-zero exit code 1
> > Failing this attempt. Failing the application.
> > at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
> > at
> org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:74)
> > Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has
> already shutdown. Application application_1452243782005_0292 failed 2 times
> due to AM Container for appattempt_1452243782005_0292_02 exited with
> exitCode: 1
> > For more detailed output, check application tracking page:
> http://xstathn003:8088/proxy/application_1452243782005_0292/Then, click
> on links to logs of each attempt.
> > Diagnostics: Exception from container-launch.
> > Container id: container_1452243782005_0292_02_01
> > Exit code: 1
> > Stack trace: ExitCodeException exitCode=1:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> > at org.apache.hadoop.util.Shell.run(Shell.java:455)
> > at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> > at
> 

Re: Failing attemption at org.apache.tez.client.TezClient.waitTillReady

[Hitesh Shah]

It seems to me that the Tez AM classpath somehow has a hadoop-common jar that 
does not have the Configuration.addDeprecations() api that YARN needs.

For the Tez AM, the classpath is fully constructed based on the tez tarball ( 
from HDFS using distributed cache ) and additional jars that Hive adds ( 
hive-exec.jar, etc ). It does not use HADOOP_CLASSPATH or anything else from 
the cluster nodes. HADOOP_CLASSPATH is only used on the client node where the 
hive shell runs. Can you confirm that hive was also compiled against 
hadoop-2.6.0 as that might be pulling in a different version of hadoop-common? 

thanks
— Hitesh

On Feb 12, 2016, at 12:16 AM, no jihun  wrote:

> Thanks Hitesh Shah.
> 
> It claims 
> 
> 2016-02-12 14:59:07,388 [ERROR] [main] |app.DAGAppMaster|: Error starting 
> DAGAppMaster
> 
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDelta;)V
> at 
> org.apache.hadoop.yarn.conf.YarnConfiguration.addDeprecatedKeys(YarnConfiguration.java:79)
> at 
> org.apache.hadoop.yarn.conf.YarnConfiguration.(YarnConfiguration.java:73)
> at org.apache.tez.dag.app.DAGAppMaster.main(DAGAppMaster.java:2271)
> 
> I am not sure but according to this 
> thread(http://grokbase.com/t/cloudera/cdh-user/12765svj61/libjars-and-hadoop-jar-command)
> this perhaps caused by $HADOOP_CLASSPATH problem.
> 
> But I wander should I copy "tez-dist/target/tez-0.8.2" to all cluster then 
> export below?
> export TEZ_JARS=/home1/apps/tez-0.8.2
> export TEZ_CONF_DIR=$TEZ_JARS/conf
> export 
> HADOOP_CLASSPATH=$TEZ_CONF_DIR:$TEZ_JARS/*:$TEZ_JARS/lib/*:$HADOOP_CLASSPATH
> 
> I did this only on name nodes.
> 
> 
> 2016-02-12 16:47 GMT+09:00 Hitesh Shah :
> Run the following command: “bin/yarn logs -applicationId 
> application_1452243782005_0292” . This should give you the logs for 
> container_1452243782005_0292_02_01 which may shed more light on why the 
> Tez ApplicationMaster is failing to launch when triggered via Hive.
> 
> thanks
> — Hitesh
> 
> 
> 
> On Feb 11, 2016, at 10:48 PM, no jihun  wrote:
> 
> > Hi all.
> >
> > When I execute a query on hive I got an error below.(so do in hive cli)
> > no more detailed log found.
> >
> > what should I check?
> > any advice will be appreciated.
> >
> > versions
> > - tez-0.8.2
> > - hadoop 2.6.0
> >
> > ---
> >
> > hive > set hive.execution.engine=tez;
> > hive > select count(*) from contents;
> >
> > WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please 
> > use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties 
> > files.
> >
> > Logging initialized using configuration in 
> > file:/home1/eco/hive/conf/hive-log4j.properties
> > hive> set hive.execution.engine=tez;
> > hive> select count(*) from agg_band_contents;
> > Query ID = irteam_20160212145903_9300f3b2-3942-4423-8586-73d2eaff9e58
> > Total jobs = 1
> > Launching Job 1 out of 1
> > Exception in thread "Thread-10" java.lang.RuntimeException: 
> > org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. 
> > Application application_1452243782005_0292 failed 2 times due to AM 
> > Container for appattempt_1452243782005_0292_02 exited with  exitCode: 1
> > For more detailed output, check application tracking 
> > page:http://xstathn003:8088/proxy/application_1452243782005_0292/Then, 
> > click on links to logs of each attempt.
> > Diagnostics: Exception from container-launch.
> > Container id: container_1452243782005_0292_02_01
> > Exit code: 1
> > Stack trace: ExitCodeException exitCode=1:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> > at org.apache.hadoop.util.Shell.run(Shell.java:455)
> > at 
> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> > at 
> > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> > at 
> > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> > at 
> > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > at 
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at 
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> > Container exited with a non-zero exit code 1
> > Failing this attempt. Failing the application.
> > at 
> > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
> > at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:74)
> > Caused by: 

Re: Failing attemption at org.apache.tez.client.TezClient.waitTillReady

[no jihun]

* I did nothing with hive-exec.jar, hive.jar.directory..

* hive-default.xml

$ cat eco/hive/conf/hive-default.xml

  
hive.jar.directory


  This is the location hive in tez mode will look for to find a site
wide
  installed hive instance.

  


* tez/lib have hadoop-common-2.6.0.jar
* tez tarball on hdfs also exactly same jar list
$  pwd
/home1/apps/tez-0.8.2/lib
$ ll
-rw-rw-r-- 1124846 2016-02-11 13:37 RoaringBitmap-0.4.9.jar
-rw-rw-r-- 1 62983 2016-02-11 13:37 activation-1.1.jar
-rw-rw-r-- 1  4467 2016-02-11 13:37 aopalliance-1.0.jar
-rw-rw-r-- 1 44925 2016-02-11 13:37 apacheds-i18n-2.0.0-M15.jar
-rw-rw-r-- 1691479 2016-02-11 13:37
apacheds-kerberos-codec-2.0.0-M15.jar
-rw-rw-r-- 1 16560 2016-02-11 13:37 api-asn1-api-1.0.0-M20.jar
-rw-rw-r-- 1 79912 2016-02-11 13:37 api-util-1.0.0-M20.jar
-rw-rw-r-- 1535731 2016-02-11 13:37 async-http-client-1.8.16.jar
-rw-rw-r-- 1303139 2016-02-11 13:37 avro-1.7.4.jar
-rw-rw-r-- 1188671 2016-02-11 13:37 commons-beanutils-1.7.0.jar
-rw-rw-r-- 1206035 2016-02-11 13:37 commons-beanutils-core-1.8.0.jar
-rw-rw-r-- 1 41123 2016-02-11 13:37 commons-cli-1.2.jar
-rw-rw-r-- 1 58160 2016-02-11 13:37 commons-codec-1.4.jar
-rw-rw-r-- 1588337 2016-02-11 13:37 commons-collections-3.2.2.jar
-rw-rw-r-- 1751238 2016-02-11 13:37 commons-collections4-4.1.jar
-rw-rw-r-- 1241367 2016-02-11 13:37 commons-compress-1.4.1.jar
-rw-rw-r-- 1298829 2016-02-11 13:37 commons-configuration-1.6.jar
-rw-rw-r-- 1143602 2016-02-11 13:37 commons-digester-1.8.jar
-rw-rw-r-- 1305001 2016-02-11 13:37 commons-httpclient-3.1.jar
-rw-rw-r-- 1185140 2016-02-11 13:37 commons-io-2.4.jar
-rw-rw-r-- 1284220 2016-02-11 13:37 commons-lang-2.6.jar
-rw-rw-r-- 1 62050 2016-02-11 13:37 commons-logging-1.1.3.jar
-rw-rw-r-- 1   1599627 2016-02-11 13:37 commons-math3-3.1.1.jar
-rw-rw-r-- 1273370 2016-02-11 13:37 commons-net-3.1.jar
-rw-rw-r-- 1 68866 2016-02-11 13:37 curator-client-2.6.0.jar
-rw-rw-r-- 1185245 2016-02-11 13:37 curator-framework-2.6.0.jar
-rw-rw-r-- 1248171 2016-02-11 13:37 curator-recipes-2.6.0.jar
-rw-rw-r-- 1190432 2016-02-11 13:37 gson-2.2.4.jar
-rw-rw-r-- 1   1648200 2016-02-11 13:37 guava-11.0.2.jar
-rw-rw-r-- 1710492 2016-02-11 13:37 guice-3.0.jar
-rw-rw-r-- 1 65012 2016-02-11 13:37 guice-servlet-3.0.jar
-rw-rw-r-- 1 17035 2016-02-11 13:37 hadoop-annotations-2.6.0.jar
-rw-rw-r-- 1 67167 2016-02-11 13:37 hadoop-auth-2.6.0.jar
-rw-rw-r-- 1   3360985 2016-02-11 13:37 hadoop-common-2.6.0.jar
-rw-rw-r-- 1   7822670 2016-02-11 13:37 hadoop-hdfs-2.6.0.jar
-rw-rw-r-- 1664918 2016-02-11 13:37
hadoop-mapreduce-client-common-2.6.0.jar
-rw-rw-r-- 1   1509399 2016-02-11 13:37
hadoop-mapreduce-client-core-2.6.0.jar
-rw-rw-r-- 1   1870176 2016-02-11 13:37 hadoop-yarn-api-2.6.0.jar
-rw-rw-r-- 1127986 2016-02-11 13:37 hadoop-yarn-client-2.6.0.jar
-rw-rw-r-- 1   1602059 2016-02-11 13:37 hadoop-yarn-common-2.6.0.jar
-rw-rw-r-- 1289619 2016-02-11 13:37 hadoop-yarn-server-common-2.6.0.jar
-rw-rw-r-- 1 28501 2016-02-11 13:37
hadoop-yarn-server-web-proxy-2.6.0.jar
-rw-rw-r-- 1 31212 2016-02-11 13:37 htrace-core-3.0.4.jar
-rw-rw-r-- 1433368 2016-02-11 13:37 httpclient-4.2.5.jar
-rw-rw-r-- 1227275 2016-02-11 13:37 httpcore-4.2.4.jar
-rw-rw-r-- 1232248 2016-02-11 13:37 jackson-core-asl-1.9.13.jar
-rw-rw-r-- 1 18336 2016-02-11 13:37 jackson-jaxrs-1.9.13.jar
-rw-rw-r-- 1780664 2016-02-11 13:37 jackson-mapper-asl-1.9.13.jar
-rw-rw-r-- 1 27084 2016-02-11 13:37 jackson-xc-1.9.13.jar
-rw-rw-r-- 1  2497 2016-02-11 13:37 javax.inject-1.jar
-rw-rw-r-- 1105134 2016-02-11 13:37 jaxb-api-2.2.2.jar
-rw-rw-r-- 1890168 2016-02-11 13:37 jaxb-impl-2.2.3-1.jar
-rw-rw-r-- 1130458 2016-02-11 13:37 jersey-client-1.9.jar
-rw-rw-r-- 1458739 2016-02-11 13:37 jersey-core-1.9.jar
-rw-rw-r-- 1 14786 2016-02-11 13:37 jersey-guice-1.9.jar
-rw-rw-r-- 1147952 2016-02-11 13:37 jersey-json-1.9.jar
-rw-rw-r-- 1 81743 2016-02-11 13:37 jettison-1.3.4.jar
-rw-rw-r-- 1539912 2016-02-11 13:37 jetty-6.1.26.jar
-rw-rw-r-- 1177131 2016-02-11 13:37 jetty-util-6.1.26.jar
-rw-rw-r-- 1 33031 2016-02-11 13:37 jsr305-3.0.0.jar
-rw-rw-r-- 1   1045744 2016-02-11 13:37 leveldbjni-all-1.8.jar
-rw-rw-r-- 1489884 2016-02-11 13:37 log4j-1.2.17.jar
-rw-rw-r-- 1111908 2016-02-11 13:37 metrics-core-3.1.0.jar
-rw-rw-r-- 1   1303237 2016-02-11 13:37 netty-3.9.2.Final.jar
-rw-rw-r-- 1 29555 2016-02-11 13:37 paranamer-2.3.jar
-rw-rw-r-- 1533455 2016-02-11 13:37 protobuf-java-2.5.0.jar
-rw-rw-r-- 1134133 2016-02-11 13:37 servlet-api-2.5-20081211.jar
-rw-rw-r-- 1105112 2016-02-11 13:37 servlet-api-2.5.jar
-rw-rw-r-- 1 32119 2016-02-11 13:37 slf4j-api-1.7.10.jar
-rw-rw-r-- 1  8866 2016-02-11 13:37 slf4j-log4j12-1.7.10.jar
-rw-rw-r-- 1995968 2016-02-11 13:37 snappy-java-1.0.4.1.jar
-rw-rw-r-- 1 23346 2016-02-11 13:37 

Re: Failing attemption at org.apache.tez.client.TezClient.waitTillReady

[Hitesh Shah]

I think there is a mix of classes of hadoop-common getting on to the classpath. 
This might possibly be due to hive-exec.jar.

One option for you to try is put the tez-minimal tarball on HDFS instead of the 
full tarball and set “tez.use.cluster.hadoop-libs” to true in your 
tez-site.xml. This will use hadoop jars from the cluster and hopefully solve 
the mixed version problem. 

thanks
— Hitesh 


On Feb 12, 2016, at 1:01 AM, no jihun  wrote:

> * I did nothing with hive-exec.jar, hive.jar.directory..
> 
> * hive-default.xml
> 
> $ cat eco/hive/conf/hive-default.xml
> 
>   
> hive.jar.directory
> 
> 
>   This is the location hive in tez mode will look for to find a site wide
>   installed hive instance.
> 
>   
> 
> 
> * tez/lib have hadoop-common-2.6.0.jar
> * tez tarball on hdfs also exactly same jar list
> $  pwd
> /home1/apps/tez-0.8.2/lib
> $ ll
> -rw-rw-r-- 1124846 2016-02-11 13:37 RoaringBitmap-0.4.9.jar
> -rw-rw-r-- 1 62983 2016-02-11 13:37 activation-1.1.jar
> -rw-rw-r-- 1  4467 2016-02-11 13:37 aopalliance-1.0.jar
> -rw-rw-r-- 1 44925 2016-02-11 13:37 apacheds-i18n-2.0.0-M15.jar
> -rw-rw-r-- 1691479 2016-02-11 13:37 apacheds-kerberos-codec-2.0.0-M15.jar
> -rw-rw-r-- 1 16560 2016-02-11 13:37 api-asn1-api-1.0.0-M20.jar
> -rw-rw-r-- 1 79912 2016-02-11 13:37 api-util-1.0.0-M20.jar
> -rw-rw-r-- 1535731 2016-02-11 13:37 async-http-client-1.8.16.jar
> -rw-rw-r-- 1303139 2016-02-11 13:37 avro-1.7.4.jar
> -rw-rw-r-- 1188671 2016-02-11 13:37 commons-beanutils-1.7.0.jar
> -rw-rw-r-- 1206035 2016-02-11 13:37 commons-beanutils-core-1.8.0.jar
> -rw-rw-r-- 1 41123 2016-02-11 13:37 commons-cli-1.2.jar
> -rw-rw-r-- 1 58160 2016-02-11 13:37 commons-codec-1.4.jar
> -rw-rw-r-- 1588337 2016-02-11 13:37 commons-collections-3.2.2.jar
> -rw-rw-r-- 1751238 2016-02-11 13:37 commons-collections4-4.1.jar
> -rw-rw-r-- 1241367 2016-02-11 13:37 commons-compress-1.4.1.jar
> -rw-rw-r-- 1298829 2016-02-11 13:37 commons-configuration-1.6.jar
> -rw-rw-r-- 1143602 2016-02-11 13:37 commons-digester-1.8.jar
> -rw-rw-r-- 1305001 2016-02-11 13:37 commons-httpclient-3.1.jar
> -rw-rw-r-- 1185140 2016-02-11 13:37 commons-io-2.4.jar
> -rw-rw-r-- 1284220 2016-02-11 13:37 commons-lang-2.6.jar
> -rw-rw-r-- 1 62050 2016-02-11 13:37 commons-logging-1.1.3.jar
> -rw-rw-r-- 1   1599627 2016-02-11 13:37 commons-math3-3.1.1.jar
> -rw-rw-r-- 1273370 2016-02-11 13:37 commons-net-3.1.jar
> -rw-rw-r-- 1 68866 2016-02-11 13:37 curator-client-2.6.0.jar
> -rw-rw-r-- 1185245 2016-02-11 13:37 curator-framework-2.6.0.jar
> -rw-rw-r-- 1248171 2016-02-11 13:37 curator-recipes-2.6.0.jar
> -rw-rw-r-- 1190432 2016-02-11 13:37 gson-2.2.4.jar
> -rw-rw-r-- 1   1648200 2016-02-11 13:37 guava-11.0.2.jar
> -rw-rw-r-- 1710492 2016-02-11 13:37 guice-3.0.jar
> -rw-rw-r-- 1 65012 2016-02-11 13:37 guice-servlet-3.0.jar
> -rw-rw-r-- 1 17035 2016-02-11 13:37 hadoop-annotations-2.6.0.jar
> -rw-rw-r-- 1 67167 2016-02-11 13:37 hadoop-auth-2.6.0.jar
> -rw-rw-r-- 1   3360985 2016-02-11 13:37 hadoop-common-2.6.0.jar
> -rw-rw-r-- 1   7822670 2016-02-11 13:37 hadoop-hdfs-2.6.0.jar
> -rw-rw-r-- 1664918 2016-02-11 13:37 
> hadoop-mapreduce-client-common-2.6.0.jar
> -rw-rw-r-- 1   1509399 2016-02-11 13:37 hadoop-mapreduce-client-core-2.6.0.jar
> -rw-rw-r-- 1   1870176 2016-02-11 13:37 hadoop-yarn-api-2.6.0.jar
> -rw-rw-r-- 1127986 2016-02-11 13:37 hadoop-yarn-client-2.6.0.jar
> -rw-rw-r-- 1   1602059 2016-02-11 13:37 hadoop-yarn-common-2.6.0.jar
> -rw-rw-r-- 1289619 2016-02-11 13:37 hadoop-yarn-server-common-2.6.0.jar
> -rw-rw-r-- 1 28501 2016-02-11 13:37 hadoop-yarn-server-web-proxy-2.6.0.jar
> -rw-rw-r-- 1 31212 2016-02-11 13:37 htrace-core-3.0.4.jar
> -rw-rw-r-- 1433368 2016-02-11 13:37 httpclient-4.2.5.jar
> -rw-rw-r-- 1227275 2016-02-11 13:37 httpcore-4.2.4.jar
> -rw-rw-r-- 1232248 2016-02-11 13:37 jackson-core-asl-1.9.13.jar
> -rw-rw-r-- 1 18336 2016-02-11 13:37 jackson-jaxrs-1.9.13.jar
> -rw-rw-r-- 1780664 2016-02-11 13:37 jackson-mapper-asl-1.9.13.jar
> -rw-rw-r-- 1 27084 2016-02-11 13:37 jackson-xc-1.9.13.jar
> -rw-rw-r-- 1  2497 2016-02-11 13:37 javax.inject-1.jar
> -rw-rw-r-- 1105134 2016-02-11 13:37 jaxb-api-2.2.2.jar
> -rw-rw-r-- 1890168 2016-02-11 13:37 jaxb-impl-2.2.3-1.jar
> -rw-rw-r-- 1130458 2016-02-11 13:37 jersey-client-1.9.jar
> -rw-rw-r-- 1458739 2016-02-11 13:37 jersey-core-1.9.jar
> -rw-rw-r-- 1 14786 2016-02-11 13:37 jersey-guice-1.9.jar
> -rw-rw-r-- 1147952 2016-02-11 13:37 jersey-json-1.9.jar
> -rw-rw-r-- 1 81743 2016-02-11 13:37 jettison-1.3.4.jar
> -rw-rw-r-- 1539912 2016-02-11 13:37 jetty-6.1.26.jar
> -rw-rw-r-- 1177131 2016-02-11 13:37 jetty-util-6.1.26.jar
> -rw-rw-r-- 1 33031 2016-02-11 13:37 jsr305-3.0.0.jar
> -rw-rw-r-- 1   1045744 2016-02-11 13:37 leveldbjni-all-1.8.jar
> -rw-rw-r-- 1489884 

Re: Failing attemption at org.apache.tez.client.TezClient.waitTillReady

[Hitesh Shah]

Run the following command: “bin/yarn logs -applicationId 
application_1452243782005_0292” . This should give you the logs for 
container_1452243782005_0292_02_01 which may shed more light on why the Tez 
ApplicationMaster is failing to launch when triggered via Hive. 

thanks
— Hitesh



On Feb 11, 2016, at 10:48 PM, no jihun  wrote:

> Hi all.
> 
> When I execute a query on hive I got an error below.(so do in hive cli)
> no more detailed log found.
> 
> what should I check?
> any advice will be appreciated.
> 
> versions
> - tez-0.8.2
> - hadoop 2.6.0
> 
> ---
> 
> hive > set hive.execution.engine=tez;
> hive > select count(*) from contents;
> 
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use 
> org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> 
> Logging initialized using configuration in 
> file:/home1/eco/hive/conf/hive-log4j.properties
> hive> set hive.execution.engine=tez;
> hive> select count(*) from agg_band_contents;
> Query ID = irteam_20160212145903_9300f3b2-3942-4423-8586-73d2eaff9e58
> Total jobs = 1
> Launching Job 1 out of 1
> Exception in thread "Thread-10" java.lang.RuntimeException: 
> org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. 
> Application application_1452243782005_0292 failed 2 times due to AM Container 
> for appattempt_1452243782005_0292_02 exited with  exitCode: 1
> For more detailed output, check application tracking 
> page:http://xstathn003:8088/proxy/application_1452243782005_0292/Then, click 
> on links to logs of each attempt.
> Diagnostics: Exception from container-launch.
> Container id: container_1452243782005_0292_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1: 
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 
> 
> Container exited with a non-zero exit code 1
> Failing this attempt. Failing the application.
> at 
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:74)
> Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already 
> shutdown. Application application_1452243782005_0292 failed 2 times due to AM 
> Container for appattempt_1452243782005_0292_02 exited with  exitCode: 1
> For more detailed output, check application tracking 
> page:http://xstathn003:8088/proxy/application_1452243782005_0292/Then, click 
> on links to logs of each attempt.
> Diagnostics: Exception from container-launch.
> Container id: container_1452243782005_0292_02_01
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1: 
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at 
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at 
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 
> 
> Container exited with a non-zero exit code 1
> Failing this attempt. Failing the application.
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:784)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:205)
> at 
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
> at 
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532)
> ... 1 more
>