As you correctly pointed out there was issue of guava library only. In my code there were some UDFs which were using guava-0.16.0.jar.
I've removed it and now there are no exceptions. Just for curiosity. Can i use tez-0.7.0 with latest PIG-0.14.0? Is it tested earlier? Regards, Sandeep On Wed, Sep 2, 2015 at 8:44 PM, Hitesh Shah <[email protected]> wrote: > Based on the stack trace, the following issue seems to be the cause: > > Caused by: java.lang.NoSuchMethodError: > com.google.common.base.Stopwatch.elapsedTime(Ljava/util/concurrent/TimeUnit;)J > at > org.apache.tez.runtime.library.common.shuffle.HttpConnection.validate(HttpConnection.java:221) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:328) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:245) > at > org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run(FetcherOrderedGrouped.java:167) > > This means that the guava library version is picking up a new version at > runtime. > > A quick test is to run say orderedwordcount from tez-examples to verify > standalone Tez has no issues. Also, you may wish to check the contents of > “tez.lib.uris” to verify that it has guava-11.0.2. > > If you are familiar with using/debugging YARN, set > "yarn.nodemanager.delete.debug-delay-sec” to a value such as 1200. Now, > pick a host on which the Tez container ran in which a task failed. > > Using the directories specified in "yarn.nodemanager.local-dirs” in > yarn-site.xml, search for a “launch_container.sh” under the > container-specific directory mapping to the tez container above. The > contents of this will tell you which guava library is being symlinked into > the container space and used in the classpath. Having 2 guava jars is also > a problem as either could be picked. > > thanks > — Hitesh > > > On Sep 2, 2015, at 5:18 AM, Sandeep Kumar <[email protected]> > wrote: > > > Thanks for your responses. I was mistaken that there is any > compatibility issue. Its the same error when i run PIG-0.14.0 over > Tez-0.5.2. > > > > HadoopVersion: 2.6.0-cdh5.4.4 > > PigVersion: 0.14.0 > > TezVersion: 0.5.2 > > > > PFA the exception stack trace. > > > > > > On Wed, Sep 2, 2015 at 3:11 PM, Jianfeng (Jeff) Zhang < > [email protected]> wrote: > > >>> I could not use tez-0.5.2 because it was not compatible with > Hadoop-2.6.0. > > > > What incompatible do you see ? > > > > > > Best Regard, > > Jeff Zhang > > > > > > From: Sandeep Kumar <[email protected]> > > Reply-To: "[email protected]" <[email protected]> > > Date: Wednesday, September 2, 2015 at 5:18 PM > > > > To: "[email protected]" <[email protected]> > > Subject: Re: Pig(0.14.0) on Tez(0.7.0) > > > > Yes i did change PIG/ivy/libraries.propeties to compile it with > tez-0.7.0 and also changed pig to compile with Hadoop-core-2.6.0. > > > > I could not use tez-0.5.2 because it was not compatible with > Hadoop-2.6.0. > > > > I'm compiling my code of PIG using same command: ant clean jar > -Dhadoopversion=23 > > > > > > > > On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang < > [email protected]> wrote: > > > > Not sure how did you compile pig with tez 0.7.0, did you change the tez > version in PIG/ivy/libraries.propeties ? > > > > And make sure you build pig with hadoop version, by default, pig build > with hadoop-1.x. Use the following command to build pig with hadoop-2.x > > > > >> ant clean jar -Dhadoopversion=23 > > > > > > > > Best Regard, > > Jeff Zhang > > > > > > From: Sandeep Kumar <[email protected]> > > Reply-To: "[email protected]" <[email protected]> > > Date: Wednesday, September 2, 2015 at 4:27 PM > > To: "[email protected]" <[email protected]> > > Subject: Re: Pig(0.14.0) on Tez(0.7.0) > > > > Hi Jeff, > > > > The cloudera Hadoop is using guava-11.0.2.jar. > > I've also exported one environment variable before running pig: > > > > export HADOOP_USER_CLASSPATH_FIRST=true > > > > Should i recompile my PIG code with new guava jar? > > > > Regards, > > Sandeep > > > > On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang < > [email protected]> wrote: > > > > It looks like classpath issue due guava version. Could you check the > guava version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use > guava 11.0 > > > > > > > > Best Regard, > > Jeff Zhang > > > > > > From: Sandeep Kumar <[email protected]> > > Reply-To: "[email protected]" <[email protected]> > > Date: Wednesday, September 2, 2015 at 2:28 PM > > To: "[email protected]" <[email protected]> > > Subject: Pig(0.14.0) on Tez(0.7.0) > > > > Hi Guys, > > > > I've a pig script which is as follows: > > > > map = LOAD 'input/' using > com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv'); > > normalized_map_data = foreach r360map generate flatten(com.Map($0..)); > > > > data_grouped_over_event_time = GROUP normalized_map_data by > (((startTime/1000) / 3600) * 3600) PARTITION BY com.CustomTimePartitioner; > > final_data = foreach data_grouped_over_event_time generate flatten($1); > > > > store final_data into 'tmp/200MB' using com.StorageModel(); > > > > > > I'm trying to run my Pig job using Tez but i'm facing an exception which > is attached. I searched over net and found some vague solutions. I've only > two configuration entries in my tez-site.xml which are as follows: > > > > <property> > > <name>tez.lib.uris</name> > > <value>hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz</value> > > </property> > > <property> > > <description>URL for where the Tez UI is hosted</description> > > <name>tez.tez-ui.history-url.base</name> > > <value>http://node4:9999/tez-ui/</value> > > </property> > > > > Is there anything else which i'm missing and supposed to be added in Tez > configuration file. > > I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided > with Tez-0.5.2 but the exception is coming in both the versions of tez. > > > > Following are the version of softwares i'm using. > > > > HadoopVersion: 2.6.0-cdh5.4.4 > > PigVersion: 0.14.0-SNAPSHOT > > TezVersion: 0.7.0 > > > > > > > > Regards, > > Sandeep > > > > > > > > <Pig-Exception> > >
