Hi Dev List,
I'm trying to best understand how to troubleshoot this error (is it Oozie,
or Falcon induced?).
I have a Pig script that I am using as the workflow for my Falcon process.
The pig script uses HCatalogStorer to write to a HCatalog URI that is the
output feed defined in my Falcon Process Entity. The Pig action in the
resulting Ooozie Workflow generated by Falcon fails with the attached stack
trace. The root is that it is *missing a class definitions of
org/apache/hadoop/hive/shims/ShimLoader.*
Running the script manually using pig -x tex -useHCatalog <all the -params
passed by Oozie> <path to pig script> results in a successful execution.
It's only once this is called as a Pig activity in the Falcon-generated
Oozie workflow that the missing class definitions manifests.
I am running the following stack:
HDP-2.3.2.0-2950
Pig 0.15.0.2.3
Hive 1.2.1.2.3
Oozie 4.2.0.2.3
Falcon 0.6.1.2.3
--
Mark Greene
*E:* [email protected]
Log Type: stderr
Log Upload Time: Fri Jan 22 15:51:16 +0000 2016
Log Length: 4758
Showing 4096 bytes of 4758 total. Click here for the full log.
014348_0046_01_000002/pig-job_1453418014348_0046.log
Pig logfile dump:
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error. org/apache/hadoop/hive/shims/ShimLoader
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoader
at
org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
at
org.apache.hive.hcatalog.pig.PigHCatUtil.getHCatServerUri(PigHCatUtil.java:134)
at
org.apache.hive.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:217)
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:175)
at
org.apache.pig.newplan.logical.relational.LOLoad.<init>(LOLoad.java:89)
at
org.apache.pig.parser.LogicalPlanBuilder.buildLoadOp(LogicalPlanBuilder.java:901)
at
org.apache.pig.parser.LogicalPlanGenerator.load_clause(LogicalPlanGenerator.java:3568)
at
org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1625)
at
org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at
org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at
org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at
org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:191)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1735)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1443)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:387)
at org.apache.pig.PigServer.executeBatch(PigServer.java:412)
at org.apache.pig.PigServer.executeBatch(PigServer.java:398)
at
org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:171)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:502)
at org.apache.pig.PigRunner.run(PigRunner.java:49)
at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:288)
at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:231)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.shims.ShimLoader
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 40 more
================================================================================
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain],
exit code [2]
Log Type: stdout
Log Upload Time: Fri Jan 22 15:51:16 +0000 2016
Log Length: 226786
Showing 4096 bytes of 226786 total. Click here for the full log.
et:9083
-param
falcon_output_storage_type=TABLE
-param
falcon_output_partitions='last_updated=2016-01-19'
-log4jconf
/usr/hdp/hadoop/yarn/local/usercache/falcon/appcache/application_1453418014348_0046/container_e07_1453418014348_0046_01_000002/piglog4j.properties
-logfile
pig-job_1453418014348_0046.log
Fetching child yarn jobs
tag id : oozie-52d9e877011a2c7ee77b5ace12dfbda0
Child yarn jobs are found -
=================================================================
>>> Invoking Pig command line now >>>
Run pig script using PigRunner.run() for Pig version 0.8+
Apache Pig version 0.15.0.2.3.2.0-2950 (rexported)
compiled Sep 30 2015, 19:33:55
Run pig script using PigRunner.run() for Pig version 0.8+
2016-01-22 15:51:03,639 [main] INFO org.apache.pig.Main - Apache Pig version
0.15.0.2.3.2.0-2950 (rexported) compiled Sep 30 2015, 19:33:55
2016-01-22 15:51:03,642 [main] INFO org.apache.pig.Main - Logging error
messages to:
/usr/hdp/hadoop/yarn/local/usercache/falcon/appcache/application_1453418014348_0046/container_e07_1453418014348_0046_01_000002/pig-job_1453418014348_0046.log
2016-01-22 15:51:03,892 [main] INFO org.apache.pig.impl.util.Utils - Default
bootup file /home/yarn/.pigbootup not found
2016-01-22 15:51:04,163 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is
deprecated. Instead, use mapreduce.jobtracker.address
2016-01-22 15:51:04,163 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2016-01-22 15:51:04,163 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
hadoop file system at: hdfs://c1t11346.itcs.[redacted].net:8020
2016-01-22 15:51:04,193 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to
map-reduce job tracker at: c1t11347.[redacted]:8050
2016-01-22 15:51:05,791 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2016-01-22 15:51:06,228 [main] INFO
org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is
deprecated. Instead, use fs.defaultFS
2016-01-22 15:51:06,337 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
2998: Unhandled internal error. org/apache/hadoop/hive/shims/ShimLoader
2016-01-22 15:51:06,427 [main] INFO org.apache.pig.Main - Pig script
completed in 2 seconds and 854 milliseconds (2854 ms)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain],
exit code [2]
Oozie Launcher failed, finishing Hadoop job gracefully<configuration>
<property>
<name>falconInputNames</name>
<value>input</value>
</property>
<property>
<name>mapreduce.job.user.name</name>
<value>falcon</value>
</property>
<property>
<name>falconInPaths</name>
<value>hdfs://c1t11346.itcs.[redacted].net:8020/edl_common/loading/0092_SCOUT/DLVR_RCPT/2016/01/19</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://c1t11355.itcs.[redacted].net:9083</value>
</property>
<property>
<name>feedNames</name>
<value>EDL-COMMON-CERTIFIED-PRF-OF-DLVR-TEST</value>
</property>
<property>
<name>falcon.libpath</name>
<value>/apps/falcon/working/lib</value>
</property>
<property>
<name>entityType</name>
<value>PROCESS</value>
</property>
<property>
<name>hcatNode</name>
<value>hcat://c1t11355.itcs.[redacted].net:9083</value>
</property>
<property>
<name>feedInstancePaths</name>
<value>hcat://c1t11355.itcs.[redacted].net:9083/default/PRF_OF_DLVR/last_updated=2016-01-19</value>
</property>
<property>
<name>oozie.bundle.application.path</name>
<value>hdfs://c1t11346.itcs.[redacted].net:8020/apps/falcon/staging/falcon/workflows/process/EDL-COMMON-PRF-OF-DLVR-LOAD-TEST/eca16f164ea9ba040cfa7166d807ce70_1453477798765</value>
</property>
<property>
<name>logDir</name>
<value>hdfs://c1t11346.itcs.[redacted].net:8020/apps/falcon/staging/falcon/workflows/process/EDL-COMMON-PRF-OF-DLVR-LOAD-TEST/logs</value>
</property>
<property>
<name>falconInputFeedStorageTypes</name>
<value>FILESYSTEM</value>
</property>
<property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property>
<property>
<name>userJMSNotificationEnabled</name>
<value>true</value>
</property>
<property>
<name>oozie.wf.external.id</name>
<value>EDL-COMMON-PRF-OF-DLVR-LOAD-TEST/DEFAULT/2016-01-20T15:02Z</value>
</property>
<property>
<name>brokerUrl</name>
<value>tcp://c1t11349.itcs.[redacted].net:61616</value>
</property>
<property>
<name>brokerTTL</name>
<value>4320</value>
</property>
<property>
<name>userWorkflowName</name>
<value>EDL-COMMON-PRF-OF-DLVR-LOAD-TEST</value>
</property>
<property>
<name>input</name>
<value>hdfs://c1t11346.itcs.[redacted].net:8020/edl_common/loading/0092_SCOUT/DLVR_RCPT/2016/01/19</value>
</property>
<property>
<name>srcClusterName</name>
<value>NA</value>
</property>
<property>
<name>userBrokerUrl</name>
<value>tcp://c1t11349.itcs.[redacted].net:61616?daemon=true</value>
</property>
<property>
<name>user.name</name>
<value>falcon</value>
</property>
<property>
<name>oozie.libpath</name>
<value>/apps/falcon/staging/falcon/workflows/process/EDL-COMMON-PRF-OF-DLVR-LOAD-TEST/eca16f164ea9ba040cfa7166d807ce70_1453477798765/DEFAULT/lib</value>
</property>
<property>
<name>falcon_output_storage_type</name>
<value>TABLE</value>
</property>
<property>
<name>oozie.bundle.id</name>
<value>0000129-160121005003884-oozie-oozi-B</value>
</property>
<property>
<name>jobPriority</name>
<value>NORMAL</value>
</property>
<property>
<name>oozie.wf.application.path</name>
<value>hdfs://c1t11346.itcs.[redacted].net:8020/apps/falcon/staging/falcon/workflows/process/EDL-COMMON-PRF-OF-DLVR-LOAD-TEST/eca16f164ea9ba040cfa7166d807ce70_1453477798765/DEFAULT</value>
</property>
<property>
<name>falcon_output_catalog_url</name>
<value>thrift://c1t11355.itcs.[redacted].net:9083</value>
</property>
<property>
<name>oozie.coord.application.path</name>
<value>hdfs://c1t11346.itcs.[redacted].net:8020/apps/falcon/staging/falcon/workflows/process/EDL-COMMON-PRF-OF-DLVR-LOAD-TEST/eca16f164ea9ba040cfa7166d807ce70_1453477798765/DEFAULT/coordinator.xml</value>
</property>
<property>
<name>shouldRecord</name>
<value>false</value>
</property>
<property>
<name>timeStamp</name>
<value>2016-01-22-15-50</value>
</property>
<property>
<name>ENTITY_PATH</name>
<value>/apps/falcon/staging/falcon/workflows/process/EDL-COMMON-PRF-OF-DLVR-LOAD-TEST/eca16f164ea9ba040cfa7166d807ce70_1453477798765/DEFAULT/coordinator.xml</value>
</property>
<property>
<name>falcon_output_partitions_hive</name>
<value>last_updated='2016-01-19'</value>
</property>
<property>
<name>nominalTime</name>
<value>2016-01-20-15-02</value>
</property>
<property>
<name>falcon_output_table</name>
<value>PRF_OF_DLVR</value>
</property>
<property>
<name>userWorkflowEngine</name>
<value>pig</value>
</property>
<property>
<name>hcat.metastore.uri</name>
<value>thrift://c1t11355.itcs.[redacted].net:9083</value>
</property>
<property>
<name>falcon_output_dated_partition_value_last_updated</name>
<value>2016-01-19</value>
</property>
<property>
<name>queueName</name>
<value>default</value>
</property>
<property>
<name>falconDataOperation</name>
<value>GENERATE</value>
</property>
<property>
<name>workflowEngineUrl</name>
<value>http://c1t11355.itcs.[redacted].net:11000/oozie/</value>
</property>
<property>
<name>brokerImplClass</name>
<value>org.apache.activemq.ActiveMQConnectionFactory</value>
</property>
<property>
<name>userBrokerImplClass</name>
<value>org.apache.activemq.ActiveMQConnectionFactory</value>
</property>
<property>
<name>falcon_output_partitions_pig</name>
<value>'last_updated=2016-01-19'</value>
</property>
<property>
<name>ENTITY_NAME</name>
<value>FALCON_PROCESS_DEFAULT_EDL-COMMON-PRF-OF-DLVR-LOAD-TEST</value>
</property>
<property>
<name>availabilityFlag</name>
<value>NA</value>
</property>
<property>
<name>falcon_output_database</name>
<value>default</value>
</property>
<property>
<name>entityName</name>
<value>EDL-COMMON-PRF-OF-DLVR-LOAD-TEST</value>
</property>
<property>
<name>hive.metastore.execute.setugi</name>
<value>true</value>
</property>
<property>
<name>falconInputFeeds</name>
<value>EDL-COMMON-LOAD-DLVR-RCPT-TEST</value>
</property>
<property>
<name>userWorkflowVersion</name>
<value>pig-0.15.0</value>
</property>
<property>
<name>falcon_output_partitions_java</name>
<value>'last_updated=2016-01-19'</value>
</property>
<property>
<name>cluster</name>
<value>DevHDPPOC-Test</value>
</property>
<property>
<name>colo.name</name>
<value>Austin 1</value>
</property>
<property>
<name>nameNode</name>
<value>hdfs://c1t11346.itcs.[redacted].net:8020</value>
</property>
<property>
<name>jobTracker</name>
<value>c1t11347.itcs.[redacted].net:8050</value>
</property>
</configuration><pig xmlns="uri:oozie:workflow:0.3">
<job-tracker>c1t11347.itcs.[redacted].net:8050</job-tracker>
<name-node>hdfs://c1t11346.itcs.[redacted].net:8020</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
<property>
<name>oozie.launcher.mapred.job.priority</name>
<value>NORMAL</value>
</property>
<property>
<name>oozie.action.sharelib.for.pig</name>
<value>pig,hcatalog</value>
</property>
</configuration>
<script>hdfs://c1t11346.itcs.[redacted].net:8020/edl_common/executables/prf_of_dlvr_lz_to_cz.pig</script>
<param>input=hdfs://c1t11346.itcs.[redacted].net:8020/edl_common/loading/0092_SCOUT/DLVR_RCPT/2016/01/19</param>
<param>falcon_output_partitions_java='last_updated=2016-01-19'</param>
<param>falcon_output_database=default</param>
<param>falcon_output_dated_partition_value_last_updated=2016-01-19</param>
<param>falcon_output_partitions_pig='last_updated=2016-01-19'</param>
<param>falcon_output_partitions_hive=last_updated='2016-01-19'</param>
<param>falcon_output_table=PRF_OF_DLVR</param>
<param>falcon_output_catalog_url=thrift://c1t11355.itcs.[redacted].net:9083</param>
<param>falcon_output_storage_type=TABLE</param>
<param>falcon_output_partitions='last_updated=2016-01-19'</param>
<file>hdfs://c1t11346.itcs.[redacted].net:8020/apps/falcon/staging/falcon/workflows/process/EDL-COMMON-PRF-OF-DLVR-LOAD-TEST/eca16f164ea9ba040cfa7166d807ce70_1453477798765/DEFAULT/conf/hive-site.xml</file>
</pig><?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cluster name="DevHDPPOC-Test" description="Development Cluster" colo="" xmlns="uri:falcon:cluster:0.1">
<interfaces>
<interface type="readonly" endpoint="hftp://c1t11346.itcs.[redacted].net:50070"; version="2.7.1"/>
<interface type="write" endpoint="hdfs://c1t11346.itcs.[redacted].net:8020" version="2.7.1"/>
<interface type="execute" endpoint="c1t11347.itcs.[redacted].net:8050" version="2.7.1"/>
<interface type="workflow" endpoint="http://c1t11355.itcs.[redacted].net:11000/oozie/"; version="4.2.0"/>
<interface type="messaging" endpoint="tcp://c1t11349.itcs.[redacted].net:61616?daemon=true" version="5.1.6"/>
<interface type="registry" endpoint="thrift://c1t11355.itcs.[redacted].net:9083" version="1.2.1"/>
</interfaces>
<locations>
<location name="staging" path="/apps/falcon/staging"/>
<location name="temp" path="/apps/falcon/temp"/>
<location name="working" path="/apps/falcon/working"/>
</locations>
<ACL owner="falcon" group="users" permission="0x755"/>
</cluster><feed xmlns='uri:falcon:feed:0.1' name='EDL-COMMON-CERTIFIED-PRF-OF-DLVR-TEST' description='Certified zone for PRF OF DLVR'>
<frequency>days(1)</frequency>
<timezone>UTC</timezone>
<clusters>
<cluster name='DevHDPPOC-Test' type='source'>
<validity start='2015-01-20T14:21Z' end='2099-01-20T14:21Z'/>
<retention limit='months(72)' action='delete'/>
<table uri='catalog:default:PRF_OF_DLVR#last_updated=${YEAR}-${MONTH}-${DAY}'/>
</cluster>
</clusters>
<table uri='catalog:default:PRF_OF_DLVR#last_updated=${YEAR}-${MONTH}-${DAY}'/>
<ACL owner='falcon' group='users' permission='0x755'/>
<schema location='hcat' provider='hcat'/>
<properties>
<property name='jobPriority' value='NORMAL'/>
<property name='timeout' value='hours(1)'/>
</properties>
</feed><?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="EDL-COMMON-LOAD-DLVR-RCPT-TEST" description="Loading directory for DLVR RCPT" xmlns="uri:falcon:feed:0.1">
<frequency>days(1)</frequency>
<timezone>UTC</timezone>
<clusters>
<cluster name="DevHDPPOC-Test" type="source">
<validity start="2015-01-20T13:06Z" end="2099-01-20T13:06Z"/>
<retention limit="days(7)" action="delete"/>
</cluster>
</clusters>
<locations>
<location type="data" path="/edl_common/loading/0092_SCOUT/DLVR_RCPT/${YEAR}/${MONTH}/${DAY}"/>
<location type="stats" path="/edl_common/loading/0092_SCOUT/DLVR_RCPT/stats/${YEAR}/${MONTH}/${DAY}"/>
<location type="meta" path="/edl_common/loading/0092_SCOUT/DLVR_RCPT/meta/${YEAR}/${MONTH}/${DAY}"/>
</locations>
<ACL owner="falcon" group="users" permission="0x755"/>
<schema location="/NONE" provider="NONE"/>
<properties>
<property name="jobPriority" value="NORMAL"/>
<property name="timeout" value="minutes(30)"/>
</properties>
</feed>
