[
https://issues.apache.org/jira/browse/TEZ-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved TEZ-3421.
-------------------------------
Resolution: Invalid
Closing as Invalid, it is related to Cloudera not related to Apache Tez, & cdh
5.5.2 is like pre medieval era...
> cdh 5.5.2 cann't use tez 0.8.4
> ------------------------------
>
> Key: TEZ-3421
> URL: https://issues.apache.org/jira/browse/TEZ-3421
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.1, 0.8.4
> Environment: cdh 5.5.2, hive 1.1.0, tez 0.8.4
> Reporter: Jake.zhang
> Priority: Major
>
> following the install guide, and it don't work.
> {code}
> 2016-08-31 15:31:20,614 [INFO] [IPC Server handler 0 on 43797]
> |app.DAGAppMaster|: Running DAG:
> root_20160831153131_0c02cc1a-1447-4ce9-a4b1-f66c2b8e1354:2
> 2016-08-31 15:31:20,849 [INFO] [IPC Server handler 0 on 43797]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_SUBMITTED]:
> dagID=dag_1470895429219_0037_1, submitTime=1472628680567
> 2016-08-31 15:31:20,886 [INFO] [IPC Server handler 0 on 43797]
> |impl.VertexImpl|: Default container context for
> vertex_1470895429219_0037_1_00 [Map 1]=LocalResources: [[ name=tezlib,
> value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020
> file: "/user/tez/tez-0.8.4.tar.gz" } size: 48523006 timestamp: 1472625394859
> type: ARCHIVE visibility: PUBLIC],[ name=hbase-client.jar, value=resource {
> scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-client.jar"
> } size: 1294765 timestamp: 1472625755599 type: FILE visibility: PRIVATE],[
> name=hbase-server.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-server.jar"
> } size: 4299872 timestamp: 1472625755903 type: FILE visibility: PRIVATE],[
> name=hive-hbase-handler-1.1.0-cdh5.8.0.jar, value=resource { scheme: "hdfs"
> host: "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hive-hbase-handler-1.1.0-cdh5.8.0.jar"
> } size: 118398 timestamp: 1472625755443 type: FILE visibility: PRIVATE],[
> name=postgresql-9.0-801.jdbc4.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/postgresql-9.0-801.jdbc4.jar"
> } size: 539705 timestamp: 1472625756348 type: FILE visibility: PRIVATE],[
> name=hbase-common.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-common.jar"
> } size: 581623 timestamp: 1472625755634 type: FILE visibility: PRIVATE],[
> name=htrace-core-3.2.0-incubating.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core-3.2.0-incubating.jar"
> } size: 1483913 timestamp: 1472625756002 type: FILE visibility: PRIVATE],[
> name=hbase-hadoop-compat.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop-compat.jar"
> } size: 39226 timestamp: 1472625755671 type: FILE visibility: PRIVATE],[
> name=htrace-core4-4.0.1-incubating.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core4-4.0.1-incubating.jar"
> } size: 1485102 timestamp: 1472625756117 type: FILE visibility: PRIVATE],[
> name=hbase-protocol.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-protocol.jar"
> } size: 4378452 timestamp: 1472625755802 type: FILE visibility: PRIVATE],[
> name=htrace-core.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core.jar"
> } size: 1483913 timestamp: 1472625756052 type: FILE visibility: PRIVATE],[
> name=hbase-hadoop2-compat.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop2-compat.jar"
> } size: 97555 timestamp: 1472625755713 type: FILE visibility: PRIVATE],[
> name=tez-conf.pb, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/.tez/application_1470895429219_0037/tez-conf.pb"
> } size: 66799 timestamp: 1472628677372 type: FILE visibility: APPLICATION],[
> name=hive-exec-1.1.0-cdh5.8.0-a7a75fba1322d9f2c8d2af1c5461acb91e9006758e4a4b1a3f83f2a0bd646623.jar,
> value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020
> file:
> "/user/root/.hiveJars/hive-exec-1.1.0-cdh5.8.0-a7a75fba1322d9f2c8d2af1c5461acb91e9006758e4a4b1a3f83f2a0bd646623.jar"
> } size: 19306196 timestamp: 1472610430690 type: FILE visibility: PRIVATE],],
> environment: [[ SHELL=/bin/bash ],[
> CLASSPATH=$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR: ],[
> HADOOP_ROOT_LOGGER=INFO,CLA ],[
> LD_LIBRARY_PATH=$PWD:$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:$PWD:$HADOOP_COMMON_HOME/lib/native/
> ],], credentials(token kinds): [tez.job,], javaOpts: -server
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN
> -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator
> -Dlog4j.configuration=tez-container-log4j.properties
> -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN
> -Djava.net.preferIPv4Stack=true
> -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator
> -Dlog4j.configuration=tez-container-log4j.properties
> -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA , vertex:
> vertex_1470895429219_0037_1_00 [Map 1], Default Resources=<memory:0, vCores:1>
> 2016-08-31 15:31:20,886 [INFO] [IPC Server handler 0 on 43797]
> |impl.VertexImpl|: Setting 1 additional inputs for
> vertexvertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,887 [INFO] [IPC Server handler 0 on 43797]
> |impl.VertexImpl|: Running vertex: vertex_1470895429219_0037_1_00 [Map 1] :
> TaskScheduler=0:TezYarn, ContainerLauncher=0:TezYarn,
> TaskCommunicator=0:TezYarn
> 2016-08-31 15:31:20,894 [INFO] [IPC Server handler 0 on 43797]
> |impl.VertexImpl|: Default container context for
> vertex_1470895429219_0037_1_01 [Reducer 2]=LocalResources: [[ name=tezlib,
> value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020
> file: "/user/tez/tez-0.8.4.tar.gz" } size: 48523006 timestamp: 1472625394859
> type: ARCHIVE visibility: PUBLIC],[ name=hbase-client.jar, value=resource {
> scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-client.jar"
> } size: 1294765 timestamp: 1472625755599 type: FILE visibility: PRIVATE],[
> name=hbase-server.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-server.jar"
> } size: 4299872 timestamp: 1472625755903 type: FILE visibility: PRIVATE],[
> name=hive-hbase-handler-1.1.0-cdh5.8.0.jar, value=resource { scheme: "hdfs"
> host: "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hive-hbase-handler-1.1.0-cdh5.8.0.jar"
> } size: 118398 timestamp: 1472625755443 type: FILE visibility: PRIVATE],[
> name=postgresql-9.0-801.jdbc4.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/postgresql-9.0-801.jdbc4.jar"
> } size: 539705 timestamp: 1472625756348 type: FILE visibility: PRIVATE],[
> name=hbase-common.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-common.jar"
> } size: 581623 timestamp: 1472625755634 type: FILE visibility: PRIVATE],[
> name=htrace-core-3.2.0-incubating.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core-3.2.0-incubating.jar"
> } size: 1483913 timestamp: 1472625756002 type: FILE visibility: PRIVATE],[
> name=hbase-hadoop-compat.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop-compat.jar"
> } size: 39226 timestamp: 1472625755671 type: FILE visibility: PRIVATE],[
> name=htrace-core4-4.0.1-incubating.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core4-4.0.1-incubating.jar"
> } size: 1485102 timestamp: 1472625756117 type: FILE visibility: PRIVATE],[
> name=hbase-protocol.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-protocol.jar"
> } size: 4378452 timestamp: 1472625755802 type: FILE visibility: PRIVATE],[
> name=htrace-core.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/htrace-core.jar"
> } size: 1483913 timestamp: 1472625756052 type: FILE visibility: PRIVATE],[
> name=hbase-hadoop2-compat.jar, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/hbase-hadoop2-compat.jar"
> } size: 97555 timestamp: 1472625755713 type: FILE visibility: PRIVATE],[
> name=tez-conf.pb, value=resource { scheme: "hdfs" host:
> "name2.test.kingnetdc.com" port: 8020 file:
> "/tmp/hive/root/_tez_session_dir/298b98e0-8658-4045-a0db-d8ca0b20f932/.tez/application_1470895429219_0037/tez-conf.pb"
> } size: 66799 timestamp: 1472628677372 type: FILE visibility: APPLICATION],[
> name=hive-exec-1.1.0-cdh5.8.0-a7a75fba1322d9f2c8d2af1c5461acb91e9006758e4a4b1a3f83f2a0bd646623.jar,
> value=resource { scheme: "hdfs" host: "name2.test.kingnetdc.com" port: 8020
> file:
> "/user/root/.hiveJars/hive-exec-1.1.0-cdh5.8.0-a7a75fba1322d9f2c8d2af1c5461acb91e9006758e4a4b1a3f83f2a0bd646623.jar"
> } size: 19306196 timestamp: 1472610430690 type: FILE visibility: PRIVATE],],
> environment: [[ SHELL=/bin/bash ],[
> CLASSPATH=$PWD:$PWD/*:$PWD/tezlib/*:$PWD/tezlib/lib/*:$HADOOP_CONF_DIR: ],[
> HADOOP_ROOT_LOGGER=INFO,CLA ],[
> LD_LIBRARY_PATH=$PWD:$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:$PWD:$PWD:$HADOOP_COMMON_HOME/lib/native:$JAVA_LIBRARY_PATH:$PWD:$HADOOP_COMMON_HOME/lib/native/
> ],], credentials(token kinds): [tez.job,], javaOpts: -server
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN
> -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator
> -Dlog4j.configuration=tez-container-log4j.properties
> -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA
> -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN
> -Djava.net.preferIPv4Stack=true
> -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator
> -Dlog4j.configuration=tez-container-log4j.properties
> -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA , vertex:
> vertex_1470895429219_0037_1_01 [Reducer 2], Default Resources=<memory:0,
> vCores:1>
> 2016-08-31 15:31:20,894 [INFO] [IPC Server handler 0 on 43797]
> |impl.VertexImpl|: Setting 1 additional outputs for vertex
> vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:20,895 [INFO] [IPC Server handler 0 on 43797]
> |impl.VertexImpl|: Running vertex: vertex_1470895429219_0037_1_01 [Reducer 2]
> : TaskScheduler=0:TezYarn, ContainerLauncher=0:TezYarn,
> TaskCommunicator=0:TezYarn
> 2016-08-31 15:31:20,900 [INFO] [IPC Server handler 0 on 43797]
> |impl.DAGImpl|: Using DAG Scheduler:
> org.apache.tez.dag.app.dag.impl.DAGSchedulerNaturalOrder
> 2016-08-31 15:31:20,901 [INFO] [IPC Server handler 0 on 43797]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_INITIALIZED]:
> dagID=dag_1470895429219_0037_1, initTime=1472628680901
> 2016-08-31 15:31:20,901 [INFO] [IPC Server handler 0 on 43797]
> |impl.DAGImpl|: dag_1470895429219_0037_1 transitioned from NEW to INITED due
> to event DAG_INIT
> 2016-08-31 15:31:20,901 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> Added additional resources : [[]] to classpath
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_STARTED]:
> dagID=dag_1470895429219_0037_1, startTime=1472628680901
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> dag_1470895429219_0037_1 transitioned from INITED to RUNNING due to event
> DAG_START
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Root Inputs exist for Vertex: Map 1 :
> {tab={InputName=tab},
> {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInputLegacy,
> hasPayload=true},
> {ControllerDescriptor=ClassName=org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator,
> hasPayload=false}}
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Starting root input initializer for input: tab, with
> class: [org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator]
> 2016-08-31 15:31:20,903 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Setting vertexManager to RootInputVertexManager for
> vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,907 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Num tasks is -1. Expecting
> VertexManager/InputInitializers/1-1 split to set #tasks for the vertex
> vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,907 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Vertex will initialize from input initializer.
> vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,908 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Starting 1 inputInitializers for vertex
> vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,916 [INFO] [InputInitializer {Map 1} #0]
> |dag.RootInputInitializerManager|: Starting InputInitializer for Input: tab
> on vertex vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:20,924 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: vertex_1470895429219_0037_1_00 [Map 1] transitioned from
> NEW to INITIALIZING due to event V_INIT
> 2016-08-31 15:31:20,924 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Setting vertexManager to ShuffleVertexManager for
> vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:20,924 [INFO] [InputInitializer {Map 1} #0]
> |Configuration.deprecation|: mapred.committer.job.setup.cleanup.needed is
> deprecated. Instead, use mapreduce.job.committer.setup.cleanup.needed
> 2016-08-31 15:31:20,950 [INFO] [Dispatcher thread {Central}]
> |vertexmanager.ShuffleVertexManager|: Shuffle Vertex Manager: settings
> minFrac:0.25 maxFrac:0.75 auto:false desiredTaskIput:104857600 minTasks:1
> 2016-08-31 15:31:20,950 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Creating 1 tasks for vertex:
> vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:20,958 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Directly initializing vertex:
> vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:20,959 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:VERTEX_CONFIGURE_DONE]:
> vertexId=vertex_1470895429219_0037_1_01, reconfigureDoneTime=0, numTasks=1,
> vertexLocationHint=null, edgeManagersCount=1, rootInputSpecUpdateCount=0,
> setParallelismCalledFlag=false
> 2016-08-31 15:31:20,959 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Setting up committers for vertex
> vertex_1470895429219_0037_1_01 [Reducer 2], numAdditionalOutputs=1
> 2016-08-31 15:31:20,959 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:VERTEX_INITIALIZED]:
> vertexName=Reducer 2, vertexId=vertex_1470895429219_0037_1_01,
> initRequestedTime=1472628680924, initedTime=1472628680959, numTasks=1,
> processorName=org.apache.hadoop.hive.ql.exec.tez.ReduceTezProcessor,
> additionalInputsCount=0, initGeneratedEventsCount=0,
> servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn,
> taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn,
> containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl,
> taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService,
> taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl }
> 2016-08-31 15:31:20,959 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: vertex_1470895429219_0037_1_01 [Reducer 2] transitioned
> from NEW to INITED due to event V_INIT
> 2016-08-31 15:31:20,976 [INFO] [InputInitializer {Map 1} #0]
> |exec.Utilities|: PLAN PATH =
> hdfs://name2.test.kingnetdc.com:8020/tmp/hive/root/c933a804-f7ea-4a31-ae8e-ba47378e84b2/hive_2016-08-31_15-31-16_521_8741335194912077181-1/root/_tez_scratch_dir/0822a3c4-f015-4485-91d7-168ec4f3eacc/map.xml
> 2016-08-31 15:31:20,976 [INFO] [InputInitializer {Map 1} #0]
> |exec.Utilities|: ***************non-local mode***************
> 2016-08-31 15:31:20,976 [INFO] [InputInitializer {Map 1} #0]
> |exec.Utilities|: local path =
> hdfs://name2.test.kingnetdc.com:8020/tmp/hive/root/c933a804-f7ea-4a31-ae8e-ba47378e84b2/hive_2016-08-31_15-31-16_521_8741335194912077181-1/root/_tez_scratch_dir/0822a3c4-f015-4485-91d7-168ec4f3eacc/map.xml
> 2016-08-31 15:31:20,981 [INFO] [InputInitializer {Map 1} #0]
> |log.PerfLogger|: <PERFLOG method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.Utilities>
> 2016-08-31 15:31:20,982 [INFO] [InputInitializer {Map 1} #0]
> |exec.Utilities|: Deserializing MapWork via kryo
> 2016-08-31 15:31:21,082 [INFO] [InputInitializer {Map 1} #0]
> |log.PerfLogger|: </PERFLOG method=deserializePlan start=1472628680981
> end=1472628681082 duration=101 from=org.apache.hadoop.hive.ql.exec.Utilities>
> 2016-08-31 15:31:21,085 [INFO] [InputInitializer {Map 1} #0]
> |dag.RootInputInitializerManager|: Failed InputInitializer for Input: tab on
> vertex vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:21,086 [ERROR] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Vertex Input: tab initializer failed,
> vertex=vertex_1470895429219_0037_1_00 [Map 1]
> org.apache.tez.dag.app.dag.impl.AMUserCodeException:
> java.lang.ArithmeticException: / by zero
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallback.onFailure(RootInputInitializerManager.java:327)
> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1140)
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
> at
> com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:150)
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:135)
> at
> com.google.common.util.concurrent.ListenableFutureTask.done(ListenableFutureTask.java:91)
> at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:380)
> at java.util.concurrent.FutureTask.setException(FutureTask.java:247)
> at java.util.concurrent.FutureTask.run(FutureTask.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:123)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ... 3 more
> 2016-08-31 15:31:21,094 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:VERTEX_FINISHED]:
> vertexName=Map 1, vertexId=vertex_1470895429219_0037_1_00,
> initRequestedTime=1472628680903, initedTime=0,
> startRequestedTime=1472628680924, startedTime=0, finishTime=1472628681089,
> timeTaken=1472628681089, status=FAILED, diagnostics=Vertex
> vertex_1470895429219_0037_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE
> Vertex Input: tab initializer failed, vertex=vertex_1470895429219_0037_1_00
> [Map 1], java.lang.ArithmeticException: / by zero
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:123)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> , counters=Counters: 0, vertexStats=firstTaskStartTime=-1,
> firstTasksToStart=[ ], lastTaskFinishTime=-1, lastTasksToFinish=[ ],
> minTaskDuration=-1, maxTaskDuration=-1, avgTaskDuration=-1.0,
> numSuccessfulTasks=0, shortestDurationTasks=[ ], longestDurationTasks=[ ],
> vertexTaskStats={numFailedTasks=0, numSucceededTasks=0,
> numKilledTaskAttempts=0, numKilledTasks=0, numFailedTaskAttempts=0,
> numCompletedTasks=0}, servicePluginInfo=ServicePluginInfo
> {containerLauncherName=TezYarn, taskSchedulerName=TezYarn,
> taskCommunicatorName=TezYarn,
> containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl,
> taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService,
> taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl }
> 2016-08-31 15:31:21,094 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: vertex_1470895429219_0037_1_00 [Map 1] transitioned from
> INITIALIZING to FAILED due to event V_ROOT_INPUT_FAILED
> 2016-08-31 15:31:21,095 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> Vertex vertex_1470895429219_0037_1_00 [Map 1] completed.,
> numCompletedVertices=1, numSuccessfulVertices=0, numFailedVertices=1,
> numKilledVertices=0, numVertices=2
> 2016-08-31 15:31:21,095 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> Checking vertices for DAG completion, numCompletedVertices=1,
> numSuccessfulVertices=0, numFailedVertices=1, numKilledVertices=0,
> numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
> 2016-08-31 15:31:21,095 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> dag_1470895429219_0037_1 transitioned from RUNNING to TERMINATING due to
> event DAG_VERTEX_COMPLETED
> 2016-08-31 15:31:21,095 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Invoking committer abort for vertex,
> vertexId=vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:21,097 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:VERTEX_FINISHED]:
> vertexName=Reducer 2, vertexId=vertex_1470895429219_0037_1_01,
> initRequestedTime=1472628680924, initedTime=1472628680959,
> startRequestedTime=0, startedTime=0, finishTime=1472628681095,
> timeTaken=1472628681095, status=KILLED, diagnostics=Vertex received Kill in
> INITED state.
> Vertex vertex_1470895429219_0037_1_01 [Reducer 2] killed/failed due
> to:OTHER_VERTEX_FAILURE, counters=Counters: 0,
> vertexStats=firstTaskStartTime=0, firstTasksToStart=[
> task_1470895429219_0037_1_01_000000 ], lastTaskFinishTime=-1,
> lastTasksToFinish=[ ], minTaskDuration=-1, maxTaskDuration=-1,
> avgTaskDuration=-1.0, numSuccessfulTasks=0, shortestDurationTasks=[ ],
> longestDurationTasks=[ ], vertexTaskStats={numFailedTasks=0,
> numSucceededTasks=0, numKilledTaskAttempts=0, numKilledTasks=0,
> numFailedTaskAttempts=0, numCompletedTasks=0},
> servicePluginInfo=ServicePluginInfo {containerLauncherName=TezYarn,
> taskSchedulerName=TezYarn, taskCommunicatorName=TezYarn,
> containerLauncherClassName=org.apache.tez.dag.app.launcher.TezContainerLauncherImpl,
> taskSchedulerClassName=org.apache.tez.dag.app.rm.YarnTaskSchedulerService,
> taskCommunicatorClassName=org.apache.tez.dag.app.TezTaskCommunicatorImpl }
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: vertex_1470895429219_0037_1_01 [Reducer 2] transitioned
> from INITED to KILLED due to event V_TERMINATE
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> Vertex vertex_1470895429219_0037_1_01 [Reducer 2] completed.,
> numCompletedVertices=2, numSuccessfulVertices=0, numFailedVertices=1,
> numKilledVertices=1, numVertices=2
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> Checking vertices for DAG completion, numCompletedVertices=2,
> numSuccessfulVertices=0, numFailedVertices=1, numKilledVertices=1,
> numVertices=2, commitInProgress=0, terminationCause=VERTEX_FAILURE
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1
> 2016-08-31 15:31:21,098 [INFO] [Dispatcher thread {Central}]
> |counters.Limits|: Counter limits initialized with parameters:
> GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
> 2016-08-31 15:31:21,111 [INFO] [Dispatcher thread {Central}]
> |recovery.RecoveryService|: DAG completed, dagId=dag_1470895429219_0037_1,
> queueSize=0
> {code}
> {code}
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}]
> |history.HistoryEventHandler|:
> [HISTORY][DAG:dag_1470895429219_0037_1][Event:DAG_FINISHED]:
> dagId=dag_1470895429219_0037_1, startTime=1472628680901,
> finishTime=1472628681098, timeTaken=197, status=FAILED, diagnostics=Vertex
> failed, vertexName=Map 1, vertexId=vertex_1470895429219_0037_1_00,
> diagnostics=[Vertex vertex_1470895429219_0037_1_00 [Map 1] killed/failed due
> to:ROOT_INPUT_INIT_FAILURE, Vertex Input: tab initializer failed,
> vertex=vertex_1470895429219_0037_1_00 [Map 1], java.lang.ArithmeticException:
> / by zero
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:123)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> ]
> Vertex killed, vertexName=Reducer 2, vertexId=vertex_1470895429219_0037_1_01,
> diagnostics=[Vertex received Kill in INITED state., Vertex
> vertex_1470895429219_0037_1_01 [Reducer 2] killed/failed due
> to:OTHER_VERTEX_FAILURE]
> DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1,
> counters=Counters: 2, org.apache.tez.common.counters.DAGCounter,
> AM_CPU_MILLISECONDS=450, AM_GC_TIME_MILLIS=0
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Ignoring multiple aborts for vertex:
> vertex_1470895429219_0037_1_01 [Reducer 2]
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}]
> |impl.VertexImpl|: Ignoring multiple aborts for vertex:
> vertex_1470895429219_0037_1_00 [Map 1]
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> DAG: dag_1470895429219_0037_1 finished with state: FAILED
> 2016-08-31 15:31:21,125 [INFO] [Dispatcher thread {Central}] |impl.DAGImpl|:
> dag_1470895429219_0037_1 transitioned from TERMINATING to FAILED due to event
> DAG_VERTEX_COMPLETED
> 2016-08-31 15:31:21,126 [INFO] [Dispatcher thread {Central}]
> |app.DAGAppMaster|: DAG completed, dagId=dag_1470895429219_0037_1,
> dagState=FAILED
> 2016-08-31 15:31:21,126 [INFO] [Dispatcher thread {Central}]
> |common.TezUtilsInternal|: Redirecting log file based on addend:
> dag_1470895429219_0037_1_post
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
