Based on the stack trace, the following issue seems to be the cause:
Caused by: java.lang.NoSuchMethodError:
com.google.common.base.Stopwatch.elapsedTime(Ljava/util/concurrent/TimeUnit;)J
at
org.apache.tez.runtime.library.common.shuffle.HttpConnection.validate(HttpConnection.java:221)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:328)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:245)
at
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run(FetcherOrderedGrouped.java:167)
This means that the guava library version is picking up a new version at
runtime.
A quick test is to run say orderedwordcount from tez-examples to verify
standalone Tez has no issues. Also, you may wish to check the contents of
“tez.lib.uris” to verify that it has guava-11.0.2.
If you are familiar with using/debugging YARN, set
"yarn.nodemanager.delete.debug-delay-sec” to a value such as 1200. Now, pick a
host on which the Tez container ran in which a task failed.
Using the directories specified in "yarn.nodemanager.local-dirs” in
yarn-site.xml, search for a “launch_container.sh” under the container-specific
directory mapping to the tez container above. The contents of this will tell
you which guava library is being symlinked into the container space and used in
the classpath. Having 2 guava jars is also a problem as either could be picked.
thanks
— Hitesh
On Sep 2, 2015, at 5:18 AM, Sandeep Kumar <[email protected]> wrote:
> Thanks for your responses. I was mistaken that there is any compatibility
> issue. Its the same error when i run PIG-0.14.0 over Tez-0.5.2.
>
> HadoopVersion: 2.6.0-cdh5.4.4
>
> PigVersion: 0.14.0
>
> TezVersion: 0.5.2
>
>
> PFA the exception stack trace.
>
>
> On Wed, Sep 2, 2015 at 3:11 PM, Jianfeng (Jeff) Zhang
> <[email protected]> wrote:
> >>> I could not use tez-0.5.2 because it was not compatible with
> >>> Hadoop-2.6.0.
>
> What incompatible do you see ?
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Sandeep Kumar <[email protected]>
> Reply-To: "[email protected]" <[email protected]>
> Date: Wednesday, September 2, 2015 at 5:18 PM
>
> To: "[email protected]" <[email protected]>
> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
>
> Yes i did change PIG/ivy/libraries.propeties to compile it with tez-0.7.0 and
> also changed pig to compile with Hadoop-core-2.6.0.
>
> I could not use tez-0.5.2 because it was not compatible with Hadoop-2.6.0.
>
> I'm compiling my code of PIG using same command: ant clean jar
> -Dhadoopversion=23
>
>
>
> On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang
> <[email protected]> wrote:
>
> Not sure how did you compile pig with tez 0.7.0, did you change the tez
> version in PIG/ivy/libraries.propeties ?
>
> And make sure you build pig with hadoop version, by default, pig build with
> hadoop-1.x. Use the following command to build pig with hadoop-2.x
>
> >> ant clean jar -Dhadoopversion=23
>
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Sandeep Kumar <[email protected]>
> Reply-To: "[email protected]" <[email protected]>
> Date: Wednesday, September 2, 2015 at 4:27 PM
> To: "[email protected]" <[email protected]>
> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
>
> Hi Jeff,
>
> The cloudera Hadoop is using guava-11.0.2.jar.
> I've also exported one environment variable before running pig:
>
> export HADOOP_USER_CLASSPATH_FIRST=true
>
> Should i recompile my PIG code with new guava jar?
>
> Regards,
> Sandeep
>
> On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang
> <[email protected]> wrote:
>
> It looks like classpath issue due guava version. Could you check the guava
> version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use guava
> 11.0
>
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Sandeep Kumar <[email protected]>
> Reply-To: "[email protected]" <[email protected]>
> Date: Wednesday, September 2, 2015 at 2:28 PM
> To: "[email protected]" <[email protected]>
> Subject: Pig(0.14.0) on Tez(0.7.0)
>
> Hi Guys,
>
> I've a pig script which is as follows:
>
> map = LOAD 'input/' using
> com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
> normalized_map_data = foreach r360map generate flatten(com.Map($0..));
>
> data_grouped_over_event_time = GROUP normalized_map_data by
> (((startTime/1000) / 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
> final_data = foreach data_grouped_over_event_time generate flatten($1);
>
> store final_data into 'tmp/200MB' using com.StorageModel();
>
>
> I'm trying to run my Pig job using Tez but i'm facing an exception which is
> attached. I searched over net and found some vague solutions. I've only two
> configuration entries in my tez-site.xml which are as follows:
>
> <property>
> <name>tez.lib.uris</name>
> <value>hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz</value>
> </property>
> <property>
> <description>URL for where the Tez UI is hosted</description>
> <name>tez.tez-ui.history-url.base</name>
> <value>http://node4:9999/tez-ui/</value>
> </property>
>
> Is there anything else which i'm missing and supposed to be added in Tez
> configuration file.
> I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided with
> Tez-0.5.2 but the exception is coming in both the versions of tez.
>
> Following are the version of softwares i'm using.
>
> HadoopVersion: 2.6.0-cdh5.4.4
>
> PigVersion: 0.14.0-SNAPSHOT
>
> TezVersion: 0.7.0
>
>
>
>
> Regards,
> Sandeep
>
>
>
> <Pig-Exception>
