Pradarttana created AMBARI-21568:
------------------------------------
Summary: Pig-Service check failing in post EU validation (IOP-HDP)
Key: AMBARI-21568
URL: https://issues.apache.org/jira/browse/AMBARI-21568
Project: Ambari
Issue Type: Bug
Components: ambari-upgrade
Affects Versions: 2.5.2
Environment: OS:- RHEL 7
Ambari Upgraded 2.2.0 to 2.5.2.0-174
Express Upgrade:- BigInsights-4.2.0.0 to HDP-2.6.2.0-107
Reporter: Pradarttana
Priority: Blocker
Fix For: 2.5.2
Steps to reproduce:-
1. Installed a IOP cluster ambari-version:-
2.2.0/20160616_1658,BigInsights-4.2.0.0
2. Upgrade the ambari from 2.2.0 to 2.5.2.0-174(IOP Clusters)
3. Remove IOP Select.
4. Register HDP Stack to HDP-2.6.2.0-107.
5. EU
6. Post EU
Pig service check is failing
{code}
java.lang.RuntimeException: org.apache.tez.dag.api.TezUncheckedException:
Invalid configuration of tez jars, tez.lib.uris is not defined in the
configuration
{code}
stderr:-
{code}
Traceback (most recent call last):
File
"/var/lib/ambari-agent/cache/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py",
line 141, in <module>
PigServiceCheck().execute()
File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 329, in execute
method(env)
File
"/var/lib/ambari-agent/cache/common-services/PIG/0.12.0.2.0/package/scripts/service_check.py",
line 122, in service_check
logoutput = True
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
line 166, in __init__
self.env.run()
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 160, in run
self.run_action(resource, action)
File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
line 124, in run_action
provider_action()
File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 102, in checked_call
tries=tries, try_sleep=try_sleep,
timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'pig -x tez
/var/lib/ambari-agent/tmp/pigSmoke.sh' returned 2. ######## Hortonworks
#############
This is MOTD message, added for testing in qe infra
17/07/23 17:17:38 INFO pig.ExecTypeProvider: Trying ExecType : LOCAL
17/07/23 17:17:38 INFO pig.ExecTypeProvider: Trying ExecType : MAPREDUCE
17/07/23 17:17:38 INFO pig.ExecTypeProvider: Trying ExecType : TEZ_LOCAL
17/07/23 17:17:38 INFO pig.ExecTypeProvider: Trying ExecType : TEZ
17/07/23 17:17:38 INFO pig.ExecTypeProvider: Picked TEZ as the ExecType
2017-07-23 17:17:39,099 [main] INFO org.apache.pig.Main - Apache Pig version
0.16.0.2.6.2.0-107 (rUnversioned directory) compiled Jul 21 2017, 21:55:11
2017-07-23 17:17:39,099 [main] INFO org.apache.pig.Main - Logging error
messages to: /home/ambari-qa/pig_1500830259097.log
2017-07-23 17:17:40,471 [main] INFO org.apache.pig.impl.util.Utils - Default
bootup file /home/ambari-qa/.pigbootup not found
2017-07-23 17:17:40,677 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: hdfs://vs-iop420tofnsec-re-6.openstacklocal:8020
2017-07-23 17:17:41,450 [main] INFO org.apache.pig.PigServer - Pig Script ID
for the session: PIG-pigSmoke.sh-7d069f75-d04a-43ca-93b0-01ea92655fcc
2017-07-23 17:17:41,945 [main] INFO
org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service
address: http://vs-iop420tofnsec-re-4.openstacklocal:8188/ws/v1/timeline/
2017-07-23 17:17:42,218 [main] INFO org.apache.pig.backend.hadoop.PigATSClient
- Created ATS Hook
2017-07-23 17:17:43,408 [main] INFO org.apache.hadoop.hdfs.DFSClient - Created
HDFS_DELEGATION_TOKEN token 290 for ambari-qa on 172.22.115.62:8020
2017-07-23 17:17:43,447 [main] INFO
org.apache.hadoop.mapreduce.security.TokenCache - Got dt for
hdfs://vs-iop420tofnsec-re-6.openstacklocal:8020; Kind: HDFS_DELEGATION_TOKEN,
Service: 172.22.115.62:8020, Ident: (HDFS_DELEGATION_TOKEN token 290 for
ambari-qa)
2017-07-23 17:17:43,477 [main] INFO org.apache.pig.tools.pigstats.ScriptState
- Pig features used in the script: UNKNOWN
2017-07-23 17:17:43,545 [main] INFO org.apache.pig.data.SchemaTupleBackend -
Key [pig.schematuple] was not set... will not generate code.
2017-07-23 17:17:43,592 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator,
GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,
MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2017-07-23 17:17:43,670 [main] INFO
org.apache.pig.impl.util.SpillableMemoryManager - Selected heap (PS Old Gen) of
size 699400192 to monitor. collectionUsageThreshold = 489580128, usageThreshold
= 489580128
2017-07-23 17:17:43,809 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Tez staging
directory is /tmp/temp1553709011 and resources directory is /tmp/temp1553709011
2017-07-23 17:17:43,881 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler - File
concatenation threshold: 100 optimistic? false
2017-07-23 17:17:44,068 [main] INFO org.apache.pig.builtin.PigStorage - Using
PigTextInputFormat
2017-07-23 17:17:44,074 [main] INFO org.apache.hadoop.hdfs.DFSClient - Created
HDFS_DELEGATION_TOKEN token 291 for ambari-qa on 172.22.115.62:8020
2017-07-23 17:17:44,075 [main] INFO
org.apache.hadoop.mapreduce.security.TokenCache - Got dt for
hdfs://vs-iop420tofnsec-re-6.openstacklocal:8020; Kind: HDFS_DELEGATION_TOKEN,
Service: 172.22.115.62:8020, Ident: (HDFS_DELEGATION_TOKEN token 291 for
ambari-qa)
2017-07-23 17:17:44,078 [main] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2017-07-23 17:17:44,078 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2017-07-23 17:17:44,131 [main] INFO
com.hadoop.compression.lzo.GPLNativeCodeLoader - Loaded native gpl library
2017-07-23 17:17:44,134 [main] INFO com.hadoop.compression.lzo.LzoCodec -
Successfully loaded & initialized native-lzo library [hadoop-lzo rev
ed99f623f101fee9f5d4e1fbad4bcdf6e9a60277]
2017-07-23 17:17:44,142 [main] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2017-07-23 17:17:45,227 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local
resource: joda-time-2.9.4.jar
2017-07-23 17:17:45,227 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local
resource: pig-0.16.0.2.6.2.0-107-core-h2.jar
2017-07-23 17:17:45,227 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local
resource: antlr-runtime-3.4.jar
2017-07-23 17:17:45,227 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local
resource: automaton-1.11-8.jar
2017-07-23 17:17:45,498 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - For vertex -
scope-5: parallelism=1, memory=2560, java opts=-server -XX:NewRatio=8
-Djava.net.preferIPv4Stack=true -Dhdp.version=2.6.2.0-107 -Xmx2048m
-Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator
-Dlog4j.configuration=tez-container-log4j.properties
-Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA
2017-07-23 17:17:45,498 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Processing
aliases: A,B
2017-07-23 17:17:45,498 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Detailed
locations: A[16,4],B[17,4]
2017-07-23 17:17:45,498 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezDagBuilder - Pig features
in the vertex:
2017-07-23 17:17:45,600 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Total
estimated parallelism is 1
2017-07-23 17:17:45,685 [PigTezLauncher-0] INFO
org.apache.pig.tools.pigstats.tez.TezScriptState - Pig script settings are
added to the job
2017-07-23 17:17:45,704 [PigTezLauncher-0] INFO
org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api,
version=0.7.0.2.6.2.0-107, revision=bbcfb9e8d9cc93fb586b32199eb9492528449f7c,
SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git,
buildTime=2017-07-21T21:09:31Z ]
2017-07-23 17:17:45,764 [PigTezLauncher-0] INFO
org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at
vs-iop420tofnsec-re-4.openstacklocal/172.22.115.61:8050
2017-07-23 17:17:45,937 [PigTezLauncher-0] INFO
org.apache.hadoop.yarn.client.AHSProxy - Connecting to Application History
server at vs-iop420tofnsec-re-4.openstacklocal/172.22.115.61:10200
2017-07-23 17:17:45,954 [PigTezLauncher-0] INFO
org.apache.tez.client.TezClient - Session mode. Starting session.
2017-07-23 17:17:45,955 [PigTezLauncher-0] ERROR
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager - Exception
while waiting for Tez client to be ready
org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez
jars, tez.lib.uris is not defined in the configuration
at
org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:166)
at
org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:831)
at org.apache.tez.client.TezClient.start(TezClient.java:355)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:102)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:234)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:203)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:210)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2017-07-23 17:17:45,964 [PigTezLauncher-0] ERROR
org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG
java.lang.RuntimeException: org.apache.tez.dag.api.TezUncheckedException:
Invalid configuration of tez jars, tez.lib.uris is not defined in the
configuration
at
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:111)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:234)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:203)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:210)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.tez.dag.api.TezUncheckedException: Invalid configuration
of tez jars, tez.lib.uris is not defined in the configuration
at
org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:166)
at
org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:831)
at org.apache.tez.client.TezClient.start(TezClient.java:355)
at
org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:102)
... 8 more
2017-07-23 17:17:46,641 [main] WARN
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Ooops! Some job
has failed! Specify -stop_on_failure if you want Pig to stop immediately on
failure.
2017-07-23 17:17:46,649 [main] INFO
org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script Statistics:
HadoopVersion: 2.7.3.2.6.2.0-107
PigVersion: 0.16.0.2.6.2.0-107
TezVersion: 0.7.0.2.6.2.0-107
UserId: ambari-qa
FileName: /var/lib/ambari-agent/tmp/pigSmoke.sh
StartedAt: 2017-07-23 17:17:43
FinishedAt: 2017-07-23 17:17:46
Features: UNKNOWN
Failed!
DAG 0:
Name: PigLatin:pigSmoke.sh-0_scope-0
ApplicationId: null
TotalLaunchedTasks: -1
FileBytesRead: -1
FileBytesWritten: -1
HdfsBytesRead: 0
HdfsBytesWritten: 0
SpillableMemoryManager spill count: 0
Bags proactively spilled: 0
Records proactively spilled: 0
DAG Plan:
Tez vertex scope-5
Input(s):
Output(s):
2017-07-23 17:17:46,653 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /home/ambari-qa/pig_1500830259097.log
2017-07-23 17:17:46,676 [main] INFO org.apache.pig.Main - Pig script completed
in 7 seconds and 913 milliseconds (7913 ms)
2017-07-23 17:17:46,677 [main] INFO
org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down
thread pool
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
