Re: Pig(0.14.0) on Tez(0.7.0)

[Hitesh Shah]

Pig 0.14 was released around the time when tez-0.5 was the stable release. Tez 
0.7 is compatible with tez 0.5 so pig 0.14 should work with it. This is a 
question which you should also ask on the pig mailing lists ( I don’t believe 
anyone from the Pig community has raised any bugs in this area ). 

thanks
— Hitesh

On Sep 2, 2015, at 9:48 PM, Sandeep Kumar  wrote:

> As you correctly pointed out there was issue of guava library only. In my 
> code there were some UDFs which were using guava-0.16.0.jar. 
> 
> I've removed it and now there are no exceptions. 
> 
> Just for curiosity. Can i use tez-0.7.0 with latest PIG-0.14.0? Is it tested 
> earlier?
> 
> Regards,
> Sandeep
> 
> On Wed, Sep 2, 2015 at 8:44 PM, Hitesh Shah  wrote:
> Based on the stack trace, the following issue seems to be the cause:
> 
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Stopwatch.elapsedTime(Ljava/util/concurrent/TimeUnit;)J
> at 
> org.apache.tez.runtime.library.common.shuffle.HttpConnection.validate(HttpConnection.java:221)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:328)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:245)
> at 
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run(FetcherOrderedGrouped.java:167)
> 
> This means that the guava library version is picking up a new version at 
> runtime.
> 
> A quick test is to run say orderedwordcount from tez-examples to verify 
> standalone Tez has no issues. Also, you may wish to check the contents of 
> “tez.lib.uris” to verify that it has guava-11.0.2.
> 
> If you are familiar with using/debugging YARN, set 
> "yarn.nodemanager.delete.debug-delay-sec” to a value such as 1200. Now, pick 
> a host on which the Tez container ran in which a task failed.
> 
> Using the directories specified in "yarn.nodemanager.local-dirs” in 
> yarn-site.xml, search for a “launch_container.sh” under the 
> container-specific directory mapping to the tez container above. The contents 
> of this will tell you which guava library is being symlinked into the 
> container space and used in the classpath. Having 2 guava jars is also a 
> problem as either could be picked.
> 
> thanks
> — Hitesh
> 
> 
> On Sep 2, 2015, at 5:18 AM, Sandeep Kumar  wrote:
> 
> > Thanks for your responses. I was mistaken that there is any compatibility 
> > issue. Its the same error when i run PIG-0.14.0 over Tez-0.5.2.
> >
> > HadoopVersion: 2.6.0-cdh5.4.4
> > PigVersion: 0.14.0
> > TezVersion: 0.5.2
> >
> > PFA the exception stack trace.
> >
> >
> > On Wed, Sep 2, 2015 at 3:11 PM, Jianfeng (Jeff) Zhang 
> >  wrote:
> > >>> I could not use tez-0.5.2 because it was not compatible with 
> > >>> Hadoop-2.6.0.
> >
> > What incompatible do you see ?
> >
> >
> > Best Regard,
> > Jeff Zhang
> >
> >
> > From: Sandeep Kumar 
> > Reply-To: "user@tez.apache.org" 
> > Date: Wednesday, September 2, 2015 at 5:18 PM
> >
> > To: "user@tez.apache.org" 
> > Subject: Re: Pig(0.14.0) on Tez(0.7.0)
> >
> > Yes i did change PIG/ivy/libraries.propeties to compile it with tez-0.7.0 
> > and also changed pig to compile with Hadoop-core-2.6.0.
> >
> > I could not use tez-0.5.2 because it was not compatible with Hadoop-2.6.0.
> >
> > I'm compiling my code of PIG using same command: ant clean jar 
> > -Dhadoopversion=23
> >
> >
> >
> > On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang 
> >  wrote:
> >
> > Not sure how did you compile pig with tez 0.7.0, did you change the tez 
> > version in PIG/ivy/libraries.propeties ?
> >
> > And make sure you build pig with hadoop version, by default, pig build with 
> > hadoop-1.x.  Use the following command to build pig with hadoop-2.x
> >
> > >> ant clean jar -Dhadoopversion=23
> >
> >
> >
> > Best Regard,
> > Jeff Zhang
> >
> >
> > From: Sandeep Kumar 
> > Reply-To: "user@tez.apache.org" 
> > Date: Wednesday, September 2, 2015 at 4:27 PM
> > To: "user@tez.apache.org" 
> > Subject: Re: Pig(0.14.0) on Tez(0.7.0)
> >
> > Hi Jeff,
> >
> > The cloudera Hadoop is using guava-11.0.2.jar.
> > I've also exported one environment variable before running pig:
> >
> > export HADOOP_USER_CLASSPATH_FIRST=true
> >
> > Should i recompile my PIG code with new guava jar?
> >
> > Regards,
> > Sandeep
> >
> > On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang 
> >  wrote:
> >
> > It looks like classpath issue due guava version. Could you check the guava 
> > version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use guava 
> > 11.0
> >
> >
> >
> > Best Regard,
> > Jeff Zhang
> >
> >
> > From: Sandeep Kumar 
> > Reply-To: "user@tez.apache.org" 
> > Date: Wednesday, September 2, 2015 at 2:28 PM
> > To: "user@tez.apache.org" 
> > Subject: Pig(0.14.0) on Tez(0.7.0)
> >
> > Hi Guys,
> >
> > I've a pig script which is as follows:
> >
> > map = LOAD 'input/' u

Re: Pig(0.14.0) on Tez(0.7.0)

[Sandeep Kumar]

As you correctly pointed out there was issue of guava library only. In my
code there were some UDFs which were using guava-0.16.0.jar.

I've removed it and now there are no exceptions.

Just for curiosity. Can i use tez-0.7.0 with latest PIG-0.14.0? Is it
tested earlier?

Regards,
Sandeep

On Wed, Sep 2, 2015 at 8:44 PM, Hitesh Shah  wrote:

> Based on the stack trace, the following issue seems to be the cause:
>
> Caused by: java.lang.NoSuchMethodError:
> com.google.common.base.Stopwatch.elapsedTime(Ljava/util/concurrent/TimeUnit;)J
> at
> org.apache.tez.runtime.library.common.shuffle.HttpConnection.validate(HttpConnection.java:221)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:328)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:245)
> at
> org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run(FetcherOrderedGrouped.java:167)
>
> This means that the guava library version is picking up a new version at
> runtime.
>
> A quick test is to run say orderedwordcount from tez-examples to verify
> standalone Tez has no issues. Also, you may wish to check the contents of
> “tez.lib.uris” to verify that it has guava-11.0.2.
>
> If you are familiar with using/debugging YARN, set
> "yarn.nodemanager.delete.debug-delay-sec” to a value such as 1200. Now,
> pick a host on which the Tez container ran in which a task failed.
>
> Using the directories specified in "yarn.nodemanager.local-dirs” in
> yarn-site.xml, search for a “launch_container.sh” under the
> container-specific directory mapping to the tez container above. The
> contents of this will tell you which guava library is being symlinked into
> the container space and used in the classpath. Having 2 guava jars is also
> a problem as either could be picked.
>
> thanks
> — Hitesh
>
>
> On Sep 2, 2015, at 5:18 AM, Sandeep Kumar 
> wrote:
>
> > Thanks for your responses. I was mistaken that there is any
> compatibility issue. Its the same error when i run PIG-0.14.0 over
> Tez-0.5.2.
> >
> > HadoopVersion: 2.6.0-cdh5.4.4
> > PigVersion: 0.14.0
> > TezVersion: 0.5.2
> >
> > PFA the exception stack trace.
> >
> >
> > On Wed, Sep 2, 2015 at 3:11 PM, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
> > >>> I could not use tez-0.5.2 because it was not compatible with
> Hadoop-2.6.0.
> >
> > What incompatible do you see ?
> >
> >
> > Best Regard,
> > Jeff Zhang
> >
> >
> > From: Sandeep Kumar 
> > Reply-To: "user@tez.apache.org" 
> > Date: Wednesday, September 2, 2015 at 5:18 PM
> >
> > To: "user@tez.apache.org" 
> > Subject: Re: Pig(0.14.0) on Tez(0.7.0)
> >
> > Yes i did change PIG/ivy/libraries.propeties to compile it with
> tez-0.7.0 and also changed pig to compile with Hadoop-core-2.6.0.
> >
> > I could not use tez-0.5.2 because it was not compatible with
> Hadoop-2.6.0.
> >
> > I'm compiling my code of PIG using same command: ant clean jar
> -Dhadoopversion=23
> >
> >
> >
> > On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
> >
> > Not sure how did you compile pig with tez 0.7.0, did you change the tez
> version in PIG/ivy/libraries.propeties ?
> >
> > And make sure you build pig with hadoop version, by default, pig build
> with hadoop-1.x.  Use the following command to build pig with hadoop-2.x
> >
> > >> ant clean jar -Dhadoopversion=23
> >
> >
> >
> > Best Regard,
> > Jeff Zhang
> >
> >
> > From: Sandeep Kumar 
> > Reply-To: "user@tez.apache.org" 
> > Date: Wednesday, September 2, 2015 at 4:27 PM
> > To: "user@tez.apache.org" 
> > Subject: Re: Pig(0.14.0) on Tez(0.7.0)
> >
> > Hi Jeff,
> >
> > The cloudera Hadoop is using guava-11.0.2.jar.
> > I've also exported one environment variable before running pig:
> >
> > export HADOOP_USER_CLASSPATH_FIRST=true
> >
> > Should i recompile my PIG code with new guava jar?
> >
> > Regards,
> > Sandeep
> >
> > On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
> >
> > It looks like classpath issue due guava version. Could you check the
> guava version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use
> guava 11.0
> >
> >
> >
> > Best Regard,
> > Jeff Zhang
> >
> >
> > From: Sandeep Kumar 
> > Reply-To: "user@tez.apache.org" 
> > Date: Wednesday, September 2, 2015 at 2:28 PM
> > To: "user@tez.apache.org" 
> > Subject: Pig(0.14.0) on Tez(0.7.0)
> >
> > Hi Guys,
> >
> > I've a pig script which is as follows:
> >
> > map = LOAD 'input/' using
> com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
> > normalized_map_data = foreach r360map generate flatten(com.Map($0..));
> >
> > data_grouped_over_event_time = GROUP normalized_map_data by
> (((startTime/1000) / 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
> > final_data = foreach data_grouped_over_event_time generate flatten($1);
> >

RE: how to run tez-tools?

[Bikas Saha]

Thanks for trying. The only tool that can currently be run from the cmd line is 
the CriticalPath tool. Example command line is

hadoop jar 
tez/tez-dist/target/tez-0.8.1-SNAPSHOT/tez-job-analyzer-0.8.1-SNAPSHOT.jar 
CriticalPath --dagId=dag_1439860407967_0080_1 
--atsFileName=tmp/test/dag_1439860407967_0080_1.zip --outputDir=tmp/

The ATS zip directory can be obtained by running the following example
hadoop jar ./tez/dist/tez/tez-history-parser-0.8.1-SNAPSHOT.jar 
org.apache.tez.history.ATSImportTool -dagId dag_1439860407967_0080_1 
--downloadDir tmp/

As more tools get added to the command line, they would replace CriticalPath in 
the first command line with their own tool name.

These tools are still work in progress. So we appreciate your feedback as you 
try them out. Patches for fixes (in usability/bugs/features) are welcome :)

Bikas

From: Xiaoyong Zhu [mailto:xiaoy...@microsoft.com]
Sent: Wednesday, September 02, 2015 6:27 AM
To: user@tez.apache.org
Subject: how to run tez-tools?

I found the job analyzer in tez-tools is quite interesting and want to give it 
a try. What's the correct way to use them?

I tried
java -cp 
./tez-tools/analyzers/job-analyzer/target/tez-job-analyzer-0.8.1-SNAPSHOT.jar 
org.apache.tez.analyzer.plugins.AnalyzerDriver
but it throws the exception
Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/hadoop/util/ProgramDriver
at 
org.apache.tez.analyzer.plugins.AnalyzerDriver.main(AnalyzerDriver.java:27)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.util.ProgramDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 1 more


I guess I missed something but not sure where to find the correct ways Will 
appreciate if someone could offer help, thanks!

Xiaoyong

[ANNOUNCE] Apache Tez 0.8.0-alpha released

[Siddharth Seth]

The Apache Tez team is proud to announce the latest release of Apache Tez -
version 0.8.0-alpha.

Apache Tez is an application framework which allows for a complex
directed-acyclic-graph of tasks for processing data and is built atop
Apache Hadoop YARN. More details on Apache Tez can be found at
http://tez.incubator.apache.org/

More information, and release bits are available at
https://tez.apache.org/releases/apache-tez-0-8-0-alpha.html

The released maven artifacts have also been made available on
repository.apache.org.

We would like to thank all the contributors that made this release possible.

Thanks
Siddharth Seth on behalf of The Apache Tez Team

Re: Pig(0.14.0) on Tez(0.7.0)

[Rohini Palaniswamy]

Are you sure you don't have any other version of guava apart from the 11.0
from Pig and 11.0.2 from Tez? This is a very basic problem and the Pig
release will not work for anyone if that is the case. None of us have
encountered the error and no one else has reported in pig-usergroup or a
pig jira.

-Rohini

On Wed, Sep 2, 2015 at 5:18 AM, Sandeep Kumar 
wrote:

> Thanks for your responses. I was mistaken that there is any compatibility
> issue. Its the same error when i run PIG-0.14.0 over Tez-0.5.2.
>
> HadoopVersion:
> 2.6.0-cdh5.4.4
>
> PigVersion:
> 0.14.0
>
> TezVersion:
> 0.5.2
>
>
> PFA the exception stack trace.
>
>
> On Wed, Sep 2, 2015 at 3:11 PM, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
>
>> >>> I could not use tez-0.5.2 because it was not compatible with
>> Hadoop-2.6.0.
>>
>> What incompatible do you see ?
>>
>>
>> Best Regard,
>> Jeff Zhang
>>
>>
>> From: Sandeep Kumar 
>> Reply-To: "user@tez.apache.org" 
>> Date: Wednesday, September 2, 2015 at 5:18 PM
>>
>> To: "user@tez.apache.org" 
>> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
>>
>> Yes i did change PIG/ivy/libraries.propeties to compile it with tez-0.7.0
>> and also changed pig to compile with Hadoop-core-2.6.0.
>>
>> I could not use tez-0.5.2 because it was not compatible with
>> Hadoop-2.6.0.
>>
>> I'm compiling my code of PIG using same command: ant clean jar
>> -Dhadoopversion=23
>>
>>
>>
>> On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang <
>> jzh...@hortonworks.com> wrote:
>>
>>>
>>> Not sure how did you compile pig with tez 0.7.0, did you change the tez
>>> version in PIG/ivy/libraries.propeties ?
>>>
>>> And make sure you build pig with hadoop version, by default, pig build
>>> with hadoop-1.x.  Use the following command to build pig with hadoop-2.x
>>>
>>> >> ant clean jar -Dhadoopversion=23
>>>
>>>
>>>
>>> Best Regard,
>>> Jeff Zhang
>>>
>>>
>>> From: Sandeep Kumar 
>>> Reply-To: "user@tez.apache.org" 
>>> Date: Wednesday, September 2, 2015 at 4:27 PM
>>> To: "user@tez.apache.org" 
>>> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
>>>
>>> Hi Jeff,
>>>
>>> The cloudera Hadoop is using guava-11.0.2.jar.
>>> I've also exported one environment variable before running pig:
>>>
>>> export HADOOP_USER_CLASSPATH_FIRST=true
>>>
>>> Should i recompile my PIG code with new guava jar?
>>>
>>> Regards,
>>> Sandeep
>>>
>>> On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang <
>>> jzh...@hortonworks.com> wrote:
>>>

 It looks like classpath issue due guava version. Could you check the
 guava version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use
 guava 11.0



 Best Regard,
 Jeff Zhang


 From: Sandeep Kumar 
 Reply-To: "user@tez.apache.org" 
 Date: Wednesday, September 2, 2015 at 2:28 PM
 To: "user@tez.apache.org" 
 Subject: Pig(0.14.0) on Tez(0.7.0)

 Hi Guys,

 I've a pig script which is as follows:

 map = LOAD 'input/' using
 com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
 normalized_map_data = foreach r360map generate flatten(com.Map($0..));

 data_grouped_over_event_time = GROUP normalized_map_data by
 (((startTime/1000) / 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
 final_data = foreach data_grouped_over_event_time generate flatten($1);

 store final_data into 'tmp/200MB' using com.StorageModel();


 I'm trying to run my Pig job using Tez but i'm facing an exception
 which is attached. I searched over net and found some vague solutions. I've
 only two configuration entries in my tez-site.xml which are as follows:

 
   tez.lib.uris
   hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz
 
 
   URL for where the Tez UI is hosted
   tez.tez-ui.history-url.base
   http://node4:/tez-ui/
 

 Is there anything else which i'm missing and supposed to be added in
 Tez configuration file.
 I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided
 with Tez-0.5.2 but the exception is coming in both the versions of tez.

 Following are the version of softwares i'm using.

 HadoopVersion:
 2.6.0-cdh5.4.4

 PigVersion:
 0.14.0-SNAPSHOT

 TezVersion:
 0.7.0




 Regards,
 Sandeep

>>>
>>>
>>
>

Re: Pig(0.14.0) on Tez(0.7.0)

[Hitesh Shah]

Based on the stack trace, the following issue seems to be the cause:

Caused by: java.lang.NoSuchMethodError: 
com.google.common.base.Stopwatch.elapsedTime(Ljava/util/concurrent/TimeUnit;)J
at 
org.apache.tez.runtime.library.common.shuffle.HttpConnection.validate(HttpConnection.java:221)
at 
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.setupConnection(FetcherOrderedGrouped.java:328)
at 
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.copyFromHost(FetcherOrderedGrouped.java:245)
at 
org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run(FetcherOrderedGrouped.java:167)

This means that the guava library version is picking up a new version at 
runtime. 

A quick test is to run say orderedwordcount from tez-examples to verify 
standalone Tez has no issues. Also, you may wish to check the contents of 
“tez.lib.uris” to verify that it has guava-11.0.2.

If you are familiar with using/debugging YARN, set 
"yarn.nodemanager.delete.debug-delay-sec” to a value such as 1200. Now, pick a 
host on which the Tez container ran in which a task failed.

Using the directories specified in "yarn.nodemanager.local-dirs” in 
yarn-site.xml, search for a “launch_container.sh” under the container-specific 
directory mapping to the tez container above. The contents of this will tell 
you which guava library is being symlinked into the container space and used in 
the classpath. Having 2 guava jars is also a problem as either could be picked. 

thanks
— Hitesh


On Sep 2, 2015, at 5:18 AM, Sandeep Kumar  wrote:

> Thanks for your responses. I was mistaken that there is any compatibility 
> issue. Its the same error when i run PIG-0.14.0 over Tez-0.5.2.
> 
> HadoopVersion: 2.6.0-cdh5.4.4 
>  
> PigVersion: 0.14.0
>   
> TezVersion: 0.5.2 
>   
> 
> PFA the exception stack trace. 
> 
> 
> On Wed, Sep 2, 2015 at 3:11 PM, Jianfeng (Jeff) Zhang 
>  wrote:
> >>> I could not use tez-0.5.2 because it was not compatible with 
> >>> Hadoop-2.6.0. 
> 
> What incompatible do you see ?
> 
> 
> Best Regard,
> Jeff Zhang
> 
> 
> From: Sandeep Kumar 
> Reply-To: "user@tez.apache.org" 
> Date: Wednesday, September 2, 2015 at 5:18 PM
> 
> To: "user@tez.apache.org" 
> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
> 
> Yes i did change PIG/ivy/libraries.propeties to compile it with tez-0.7.0 and 
> also changed pig to compile with Hadoop-core-2.6.0. 
> 
> I could not use tez-0.5.2 because it was not compatible with Hadoop-2.6.0. 
> 
> I'm compiling my code of PIG using same command: ant clean jar 
> -Dhadoopversion=23 
> 
> 
> 
> On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang 
>  wrote:
> 
> Not sure how did you compile pig with tez 0.7.0, did you change the tez 
> version in PIG/ivy/libraries.propeties ?
> 
> And make sure you build pig with hadoop version, by default, pig build with 
> hadoop-1.x.  Use the following command to build pig with hadoop-2.x 
> 
> >> ant clean jar -Dhadoopversion=23
> 
> 
> 
> Best Regard,
> Jeff Zhang
> 
> 
> From: Sandeep Kumar 
> Reply-To: "user@tez.apache.org" 
> Date: Wednesday, September 2, 2015 at 4:27 PM
> To: "user@tez.apache.org" 
> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
> 
> Hi Jeff,
> 
> The cloudera Hadoop is using guava-11.0.2.jar. 
> I've also exported one environment variable before running pig:
> 
> export HADOOP_USER_CLASSPATH_FIRST=true
> 
> Should i recompile my PIG code with new guava jar?
> 
> Regards,
> Sandeep
> 
> On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang 
>  wrote:
> 
> It looks like classpath issue due guava version. Could you check the guava 
> version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use guava 
> 11.0 
> 
> 
> 
> Best Regard,
> Jeff Zhang
> 
> 
> From: Sandeep Kumar 
> Reply-To: "user@tez.apache.org" 
> Date: Wednesday, September 2, 2015 at 2:28 PM
> To: "user@tez.apache.org" 
> Subject: Pig(0.14.0) on Tez(0.7.0)
> 
> Hi Guys,
> 
> I've a pig script which is as follows:
> 
> map = LOAD 'input/' using 
> com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
> normalized_map_data = foreach r360map generate flatten(com.Map($0..));
> 
> data_grouped_over_event_time = GROUP normalized_map_data by 
> (((startTime/1000) / 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
> final_data = foreach data_grouped_over_event_time generate flatten($1);
> 
> store final_data into 'tmp/200MB' using com.StorageModel();
> 
> 
> I'm trying to run my Pig job using Tez but i'm facing an exception which is 
> attached. I searched over net and found some vague solutions. I've only two 
> configuration entries in my tez-site.xml which are as follows:
> 
> 
>   tez.lib

Re: building Tez master branch fail

[Hitesh Shah]

Actually protobuf version is still 2.5.0 on master. Hadoop forces everyone to 
use 2.5.0 for compatibility.

2.5.0

Please check to see whether protoc is firstly in your path and confirm that 
"protoc —version” returns 2.5.0

thanks
— Hitesh

On Sep 2, 2015, at 1:47 AM, Sandeep Kumar  wrote:

> In case you are trying to compile master branch then its pom.xml refers to 
> 2.6.0 protoc. So, you have to install protoc 2.6.0 on your host to compile 
> tez master branch. AFAIK tez 0.7.0 uses older version of protoc 2.5.0.
> 
> 
> 
> On Wed, Sep 2, 2015 at 2:08 PM, Xiaoyong Zhu  wrote:
> Hi experts
> 
>  
> 
>  
> 
> I am trying to build the tez master branch but it always gives me the error 
> message (it seems that the pom file has something wrong?)
> 
>  
> 
> I am sure I am using protobuf 2.5, and not sure why there’s such kind of 
> issue…the log is also attached.
> 
>  
> 
> Thanks!
> 
>  
> 
> [INFO] --- hadoop-maven-plugins:2.6.0:protoc (compile-protoc) @ tez-api ---
> 
> [DEBUG] org.apache.hadoop:hadoop-maven-plugins:jar:2.6.0:
> 
> [DEBUG]org.apache.maven:maven-plugin-api:jar:3.0:compile
> 
> [DEBUG]   org.apache.maven:maven-model:jar:3.0:compile
> 
> [DEBUG]   org.apache.maven:maven-artifact:jar:3.0:compile
> 
> [DEBUG]   org.sonatype.sisu:sisu-inject-plexus:jar:1.4.2:compile
> 
> [DEBUG]  org.sonatype.sisu:sisu-inject-bean:jar:1.4.2:compile
> 
> [DEBUG] org.sonatype.sisu:sisu-guice:jar:noaop:2.1.7:compile
> 
> [DEBUG]org.apache.maven:maven-core:jar:3.0:compile
> 
> [DEBUG]   org.apache.maven:maven-settings:jar:3.0:compile
> 
> [DEBUG]   org.apache.maven:maven-settings-builder:jar:3.0:compile
> 
> [DEBUG]   org.apache.maven:maven-repository-metadata:jar:3.0:compile
> 
> [DEBUG]   org.apache.maven:maven-model-builder:jar:3.0:compile
> 
> [DEBUG]   org.apache.maven:maven-aether-provider:jar:3.0:runtime
> 
> [DEBUG]   org.sonatype.aether:aether-impl:jar:1.7:compile
> 
> [DEBUG]  org.sonatype.aether:aether-spi:jar:1.7:compile
> 
> [DEBUG]   org.sonatype.aether:aether-api:jar:1.7:compile
> 
> [DEBUG]   org.sonatype.aether:aether-util:jar:1.7:compile
> 
> [DEBUG]   org.codehaus.plexus:plexus-interpolation:jar:1.14:compile
> 
> [DEBUG]   org.codehaus.plexus:plexus-utils:jar:2.0.5:compile (version 
> managed from 2.0.4)
> 
> [DEBUG]   org.codehaus.plexus:plexus-classworlds:jar:2.2.3:compile
> 
> [DEBUG]   
> org.codehaus.plexus:plexus-component-annotations:jar:1.5.5:compile
> 
> [DEBUG]   org.sonatype.plexus:plexus-sec-dispatcher:jar:1.3:compile
> 
>  
> 
>  
> 
>  
> 
> [ERROR] Failed to execute goal 
> org.apache.hadoop:hadoop-maven-plugins:2.6.0:protoc (compile-protoc) on 
> project tez-api: org.apache.maven.plugin.MojoExecutionException: protoc 
> failure -> [Help 1]
> 
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.hadoop:hadoop-maven-plugins:2.6.0:protoc (compile-protoc) on 
> project tez-api: org.apache.maven.plugin.MojoExecutionException: protoc 
> failure
> 
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
> 
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> 
> at 
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> 
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> 
> at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> 
> at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> 
> at 
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> 
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> 
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> 
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> 
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> 
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> 
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 
> at java.lang.reflect.Method.invoke(Method.java:606)
> 
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> 
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> 
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> 
> at 
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> 
> Caused by: org.apache.maven.plugin.MojoExecutionExcep

how to run tez-tools?

[Xiaoyong Zhu]

I found the job analyzer in tez-tools is quite interesting and want to give it 
a try. What's the correct way to use them?

I tried
java -cp 
./tez-tools/analyzers/job-analyzer/target/tez-job-analyzer-0.8.1-SNAPSHOT.jar 
org.apache.tez.analyzer.plugins.AnalyzerDriver
but it throws the exception
Exception in thread "main" java.lang.NoClassDefFoundError: 
org/apache/hadoop/util/ProgramDriver
at 
org.apache.tez.analyzer.plugins.AnalyzerDriver.main(AnalyzerDriver.java:27)
Caused by: java.lang.ClassNotFoundException: 
org.apache.hadoop.util.ProgramDriver
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 1 more


I guess I missed something but not sure where to find the correct ways Will 
appreciate if someone could offer help, thanks!

Xiaoyong

Re: Pig(0.14.0) on Tez(0.7.0)

[Sandeep Kumar]

Thanks for your responses. I was mistaken that there is any compatibility
issue. Its the same error when i run PIG-0.14.0 over Tez-0.5.2.

HadoopVersion:
2.6.0-cdh5.4.4

PigVersion:
0.14.0

TezVersion:
0.5.2


PFA the exception stack trace.


On Wed, Sep 2, 2015 at 3:11 PM, Jianfeng (Jeff) Zhang <
jzh...@hortonworks.com> wrote:

> >>> I could not use tez-0.5.2 because it was not compatible with
> Hadoop-2.6.0.
>
> What incompatible do you see ?
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Sandeep Kumar 
> Reply-To: "user@tez.apache.org" 
> Date: Wednesday, September 2, 2015 at 5:18 PM
>
> To: "user@tez.apache.org" 
> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
>
> Yes i did change PIG/ivy/libraries.propeties to compile it with tez-0.7.0
> and also changed pig to compile with Hadoop-core-2.6.0.
>
> I could not use tez-0.5.2 because it was not compatible with Hadoop-2.6.0.
>
> I'm compiling my code of PIG using same command: ant clean jar
> -Dhadoopversion=23
>
>
>
> On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
>
>>
>> Not sure how did you compile pig with tez 0.7.0, did you change the tez
>> version in PIG/ivy/libraries.propeties ?
>>
>> And make sure you build pig with hadoop version, by default, pig build
>> with hadoop-1.x.  Use the following command to build pig with hadoop-2.x
>>
>> >> ant clean jar -Dhadoopversion=23
>>
>>
>>
>> Best Regard,
>> Jeff Zhang
>>
>>
>> From: Sandeep Kumar 
>> Reply-To: "user@tez.apache.org" 
>> Date: Wednesday, September 2, 2015 at 4:27 PM
>> To: "user@tez.apache.org" 
>> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
>>
>> Hi Jeff,
>>
>> The cloudera Hadoop is using guava-11.0.2.jar.
>> I've also exported one environment variable before running pig:
>>
>> export HADOOP_USER_CLASSPATH_FIRST=true
>>
>> Should i recompile my PIG code with new guava jar?
>>
>> Regards,
>> Sandeep
>>
>> On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang <
>> jzh...@hortonworks.com> wrote:
>>
>>>
>>> It looks like classpath issue due guava version. Could you check the
>>> guava version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use
>>> guava 11.0
>>>
>>>
>>>
>>> Best Regard,
>>> Jeff Zhang
>>>
>>>
>>> From: Sandeep Kumar 
>>> Reply-To: "user@tez.apache.org" 
>>> Date: Wednesday, September 2, 2015 at 2:28 PM
>>> To: "user@tez.apache.org" 
>>> Subject: Pig(0.14.0) on Tez(0.7.0)
>>>
>>> Hi Guys,
>>>
>>> I've a pig script which is as follows:
>>>
>>> map = LOAD 'input/' using
>>> com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
>>> normalized_map_data = foreach r360map generate flatten(com.Map($0..));
>>>
>>> data_grouped_over_event_time = GROUP normalized_map_data by
>>> (((startTime/1000) / 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
>>> final_data = foreach data_grouped_over_event_time generate flatten($1);
>>>
>>> store final_data into 'tmp/200MB' using com.StorageModel();
>>>
>>>
>>> I'm trying to run my Pig job using Tez but i'm facing an exception which
>>> is attached. I searched over net and found some vague solutions. I've only
>>> two configuration entries in my tez-site.xml which are as follows:
>>>
>>> 
>>>   tez.lib.uris
>>>   hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz
>>> 
>>> 
>>>   URL for where the Tez UI is hosted
>>>   tez.tez-ui.history-url.base
>>>   http://node4:/tez-ui/
>>> 
>>>
>>> Is there anything else which i'm missing and supposed to be added in Tez
>>> configuration file.
>>> I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided
>>> with Tez-0.5.2 but the exception is coming in both the versions of tez.
>>>
>>> Following are the version of softwares i'm using.
>>>
>>> HadoopVersion:
>>> 2.6.0-cdh5.4.4
>>>
>>> PigVersion:
>>> 0.14.0-SNAPSHOT
>>>
>>> TezVersion:
>>> 0.7.0
>>>
>>>
>>>
>>>
>>> Regards,
>>> Sandeep
>>>
>>
>>
>


Pig-Exception
Description: Binary data

Re: Pig(0.14.0) on Tez(0.7.0)

[Jianfeng (Jeff) Zhang]

>>> I could not use tez-0.5.2 because it was not compatible with Hadoop-2.6.0.

What incompatible do you see ?


Best Regard,
Jeff Zhang


From: Sandeep Kumar mailto:sandeepdas@gmail.com>>
Reply-To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Date: Wednesday, September 2, 2015 at 5:18 PM
To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Subject: Re: Pig(0.14.0) on Tez(0.7.0)

Yes i did change PIG/ivy/libraries.propeties to compile it with tez-0.7.0 and 
also changed pig to compile with Hadoop-core-2.6.0.

I could not use tez-0.5.2 because it was not compatible with Hadoop-2.6.0.

I'm compiling my code of PIG using same command: ant clean jar 
-Dhadoopversion=23



On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang 
mailto:jzh...@hortonworks.com>> wrote:

Not sure how did you compile pig with tez 0.7.0, did you change the tez version 
in PIG/ivy/libraries.propeties ?

And make sure you build pig with hadoop version, by default, pig build with 
hadoop-1.x.  Use the following command to build pig with hadoop-2.x

>> ant clean jar -Dhadoopversion=23



Best Regard,
Jeff Zhang


From: Sandeep Kumar mailto:sandeepdas@gmail.com>>
Reply-To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Date: Wednesday, September 2, 2015 at 4:27 PM
To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Subject: Re: Pig(0.14.0) on Tez(0.7.0)

Hi Jeff,

The cloudera Hadoop is using guava-11.0.2.jar.
I've also exported one environment variable before running pig:

export HADOOP_USER_CLASSPATH_FIRST=true

Should i recompile my PIG code with new guava jar?

Regards,
Sandeep

On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang 
mailto:jzh...@hortonworks.com>> wrote:

It looks like classpath issue due guava version. Could you check the guava 
version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use guava 11.0



Best Regard,
Jeff Zhang


From: Sandeep Kumar mailto:sandeepdas@gmail.com>>
Reply-To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Date: Wednesday, September 2, 2015 at 2:28 PM
To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Subject: Pig(0.14.0) on Tez(0.7.0)

Hi Guys,

I've a pig script which is as follows:

map = LOAD 'input/' using 
com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
normalized_map_data = foreach r360map generate flatten(com.Map($0..));

data_grouped_over_event_time = GROUP normalized_map_data by (((startTime/1000) 
/ 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
final_data = foreach data_grouped_over_event_time generate flatten($1);

store final_data into 'tmp/200MB' using com.StorageModel();


I'm trying to run my Pig job using Tez but i'm facing an exception which is 
attached. I searched over net and found some vague solutions. I've only two 
configuration entries in my tez-site.xml which are as follows:


  tez.lib.uris
  hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz


  URL for where the Tez UI is hosted
  tez.tez-ui.history-url.base
  http://node4:/tez-ui/


Is there anything else which i'm missing and supposed to be added in Tez 
configuration file.
I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided with 
Tez-0.5.2 but the exception is coming in both the versions of tez.

Following are the version of softwares i'm using.

HadoopVersion: 2.6.0-cdh5.4.4
PigVersion: 0.14.0-SNAPSHOT
TezVersion: 0.7.0



Regards,
Sandeep

Re: Pig(0.14.0) on Tez(0.7.0)

[Sandeep Kumar]

Yes i did change PIG/ivy/libraries.propeties to compile it with tez-0.7.0
and also changed pig to compile with Hadoop-core-2.6.0.

I could not use tez-0.5.2 because it was not compatible with Hadoop-2.6.0.

I'm compiling my code of PIG using same command: ant clean jar
-Dhadoopversion=23



On Wed, Sep 2, 2015 at 2:36 PM, Jianfeng (Jeff) Zhang <
jzh...@hortonworks.com> wrote:

>
> Not sure how did you compile pig with tez 0.7.0, did you change the tez
> version in PIG/ivy/libraries.propeties ?
>
> And make sure you build pig with hadoop version, by default, pig build
> with hadoop-1.x.  Use the following command to build pig with hadoop-2.x
>
> >> ant clean jar -Dhadoopversion=23
>
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Sandeep Kumar 
> Reply-To: "user@tez.apache.org" 
> Date: Wednesday, September 2, 2015 at 4:27 PM
> To: "user@tez.apache.org" 
> Subject: Re: Pig(0.14.0) on Tez(0.7.0)
>
> Hi Jeff,
>
> The cloudera Hadoop is using guava-11.0.2.jar.
> I've also exported one environment variable before running pig:
>
> export HADOOP_USER_CLASSPATH_FIRST=true
>
> Should i recompile my PIG code with new guava jar?
>
> Regards,
> Sandeep
>
> On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang <
> jzh...@hortonworks.com> wrote:
>
>>
>> It looks like classpath issue due guava version. Could you check the
>> guava version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use
>> guava 11.0
>>
>>
>>
>> Best Regard,
>> Jeff Zhang
>>
>>
>> From: Sandeep Kumar 
>> Reply-To: "user@tez.apache.org" 
>> Date: Wednesday, September 2, 2015 at 2:28 PM
>> To: "user@tez.apache.org" 
>> Subject: Pig(0.14.0) on Tez(0.7.0)
>>
>> Hi Guys,
>>
>> I've a pig script which is as follows:
>>
>> map = LOAD 'input/' using
>> com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
>> normalized_map_data = foreach r360map generate flatten(com.Map($0..));
>>
>> data_grouped_over_event_time = GROUP normalized_map_data by
>> (((startTime/1000) / 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
>> final_data = foreach data_grouped_over_event_time generate flatten($1);
>>
>> store final_data into 'tmp/200MB' using com.StorageModel();
>>
>>
>> I'm trying to run my Pig job using Tez but i'm facing an exception which
>> is attached. I searched over net and found some vague solutions. I've only
>> two configuration entries in my tez-site.xml which are as follows:
>>
>> 
>>   tez.lib.uris
>>   hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz
>> 
>> 
>>   URL for where the Tez UI is hosted
>>   tez.tez-ui.history-url.base
>>   http://node4:/tez-ui/
>> 
>>
>> Is there anything else which i'm missing and supposed to be added in Tez
>> configuration file.
>> I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided with
>> Tez-0.5.2 but the exception is coming in both the versions of tez.
>>
>> Following are the version of softwares i'm using.
>>
>> HadoopVersion:
>> 2.6.0-cdh5.4.4
>>
>> PigVersion:
>> 0.14.0-SNAPSHOT
>>
>> TezVersion:
>> 0.7.0
>>
>>
>>
>>
>> Regards,
>> Sandeep
>>
>
>

Re: Pig(0.14.0) on Tez(0.7.0)

[Jianfeng (Jeff) Zhang]

Not sure how did you compile pig with tez 0.7.0, did you change the tez version 
in PIG/ivy/libraries.propeties ?

And make sure you build pig with hadoop version, by default, pig build with 
hadoop-1.x.  Use the following command to build pig with hadoop-2.x

>> ant clean jar -Dhadoopversion=23



Best Regard,
Jeff Zhang


From: Sandeep Kumar mailto:sandeepdas@gmail.com>>
Reply-To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Date: Wednesday, September 2, 2015 at 4:27 PM
To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Subject: Re: Pig(0.14.0) on Tez(0.7.0)

Hi Jeff,

The cloudera Hadoop is using guava-11.0.2.jar.
I've also exported one environment variable before running pig:

export HADOOP_USER_CLASSPATH_FIRST=true

Should i recompile my PIG code with new guava jar?

Regards,
Sandeep

On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang 
mailto:jzh...@hortonworks.com>> wrote:

It looks like classpath issue due guava version. Could you check the guava 
version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use guava 11.0



Best Regard,
Jeff Zhang


From: Sandeep Kumar mailto:sandeepdas@gmail.com>>
Reply-To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Date: Wednesday, September 2, 2015 at 2:28 PM
To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Subject: Pig(0.14.0) on Tez(0.7.0)

Hi Guys,

I've a pig script which is as follows:

map = LOAD 'input/' using 
com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
normalized_map_data = foreach r360map generate flatten(com.Map($0..));

data_grouped_over_event_time = GROUP normalized_map_data by (((startTime/1000) 
/ 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
final_data = foreach data_grouped_over_event_time generate flatten($1);

store final_data into 'tmp/200MB' using com.StorageModel();


I'm trying to run my Pig job using Tez but i'm facing an exception which is 
attached. I searched over net and found some vague solutions. I've only two 
configuration entries in my tez-site.xml which are as follows:


  tez.lib.uris
  hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz


  URL for where the Tez UI is hosted
  tez.tez-ui.history-url.base
  http://node4:/tez-ui/


Is there anything else which i'm missing and supposed to be added in Tez 
configuration file.
I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided with 
Tez-0.5.2 but the exception is coming in both the versions of tez.

Following are the version of softwares i'm using.

HadoopVersion: 2.6.0-cdh5.4.4
PigVersion: 0.14.0-SNAPSHOT
TezVersion: 0.7.0



Regards,
Sandeep

RE: building Tez master branch fail

[Xiaoyong Zhu]

Oh.. I am using protoc 2.5.0. will try the 2.6.0.

Should we make this clear in the BUILDING.txt? since in the master branch’s 
BUILDING.txt it’s still referring to protoc 2.5.0

Xiaoyong

From: Sandeep Kumar [mailto:sandeepdas@gmail.com]
Sent: Wednesday, September 2, 2015 4:48 PM
To: user@tez.apache.org
Subject: Re: building Tez master branch fail

In case you are trying to compile master branch then its pom.xml refers to 
2.6.0 protoc. So, you have to install protoc 2.6.0 on your host to compile tez 
master branch. AFAIK tez 0.7.0 uses older version of protoc 2.5.0.


On Wed, Sep 2, 2015 at 2:08 PM, Xiaoyong Zhu 
mailto:xiaoy...@microsoft.com>> wrote:
Hi experts


I am trying to build the tez master branch but it always gives me the error 
message (it seems that the pom file has something wrong?)

I am sure I am using protobuf 2.5, and not sure why there’s such kind of 
issue…the log is also attached.

Thanks!

[INFO] --- hadoop-maven-plugins:2.6.0:protoc (compile-protoc) @ tez-api ---
[DEBUG] org.apache.hadoop:hadoop-maven-plugins:jar:2.6.0:
[DEBUG]org.apache.maven:maven-plugin-api:jar:3.0:compile
[DEBUG]   org.apache.maven:maven-model:jar:3.0:compile
[DEBUG]   org.apache.maven:maven-artifact:jar:3.0:compile
[DEBUG]   org.sonatype.sisu:sisu-inject-plexus:jar:1.4.2:compile
[DEBUG]  org.sonatype.sisu:sisu-inject-bean:jar:1.4.2:compile
[DEBUG] org.sonatype.sisu:sisu-guice:jar:noaop:2.1.7:compile
[DEBUG]org.apache.maven:maven-core:jar:3.0:compile
[DEBUG]   org.apache.maven:maven-settings:jar:3.0:compile
[DEBUG]   org.apache.maven:maven-settings-builder:jar:3.0:compile
[DEBUG]   org.apache.maven:maven-repository-metadata:jar:3.0:compile
[DEBUG]   org.apache.maven:maven-model-builder:jar:3.0:compile
[DEBUG]   org.apache.maven:maven-aether-provider:jar:3.0:runtime
[DEBUG]   org.sonatype.aether:aether-impl:jar:1.7:compile
[DEBUG]  org.sonatype.aether:aether-spi:jar:1.7:compile
[DEBUG]   org.sonatype.aether:aether-api:jar:1.7:compile
[DEBUG]   org.sonatype.aether:aether-util:jar:1.7:compile
[DEBUG]   org.codehaus.plexus:plexus-interpolation:jar:1.14:compile
[DEBUG]   org.codehaus.plexus:plexus-utils:jar:2.0.5:compile (version 
managed from 2.0.4)
[DEBUG]   org.codehaus.plexus:plexus-classworlds:jar:2.2.3:compile
[DEBUG]   org.codehaus.plexus:plexus-component-annotations:jar:1.5.5:compile
[DEBUG]   org.sonatype.plexus:plexus-sec-dispatcher:jar:1.3:compile



[ERROR] Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:2.6.0:protoc (compile-protoc) on project 
tez-api: org.apache.maven.plugin.MojoExecutionException: protoc failure -> 
[Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.hadoop:hadoop-maven-plugins:2.6.0:protoc (compile-protoc) on project 
tez-api: org.apache.maven.plugin.MojoExecutionException: protoc failure
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.MojoExecutionException: 
org.apache.maven.plugin.MojoExecutionException: protoc failure
at 
org.apache.hadoop.maven.plugin.protoc.ProtocMojo.execute(ProtocMojo.java:105)
at 
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java

Re: building Tez master branch fail

[Sandeep Kumar]

In case you are trying to compile master branch then its pom.xml refers to
2.6.0 protoc. So, you have to install protoc 2.6.0 on your host to compile
tez master branch. AFAIK tez 0.7.0 uses older version of protoc 2.5.0.



On Wed, Sep 2, 2015 at 2:08 PM, Xiaoyong Zhu  wrote:

> Hi experts
>
>
>
>
>
> I am trying to build the tez master branch but it always gives me the
> error message (it seems that the pom file has something wrong?)
>
>
>
> I am sure I am using protobuf 2.5, and not sure why there’s such kind of
> issue…the log is also attached.
>
>
>
> Thanks!
>
>
>
> [INFO] --- hadoop-maven-plugins:2.6.0:protoc (compile-protoc) @ tez-api ---
>
> [DEBUG] org.apache.hadoop:hadoop-maven-plugins:jar:2.6.0:
>
> [DEBUG]org.apache.maven:maven-plugin-api:jar:3.0:compile
>
> [DEBUG]   org.apache.maven:maven-model:jar:3.0:compile
>
> [DEBUG]   org.apache.maven:maven-artifact:jar:3.0:compile
>
> [DEBUG]   org.sonatype.sisu:sisu-inject-plexus:jar:1.4.2:compile
>
> [DEBUG]  org.sonatype.sisu:sisu-inject-bean:jar:1.4.2:compile
>
> [DEBUG] org.sonatype.sisu:sisu-guice:jar:noaop:2.1.7:compile
>
> [DEBUG]org.apache.maven:maven-core:jar:3.0:compile
>
> [DEBUG]   org.apache.maven:maven-settings:jar:3.0:compile
>
> [DEBUG]   org.apache.maven:maven-settings-builder:jar:3.0:compile
>
> [DEBUG]   org.apache.maven:maven-repository-metadata:jar:3.0:compile
>
> [DEBUG]   org.apache.maven:maven-model-builder:jar:3.0:compile
>
> [DEBUG]   org.apache.maven:maven-aether-provider:jar:3.0:runtime
>
> [DEBUG]   org.sonatype.aether:aether-impl:jar:1.7:compile
>
> [DEBUG]  org.sonatype.aether:aether-spi:jar:1.7:compile
>
> [DEBUG]   org.sonatype.aether:aether-api:jar:1.7:compile
>
> [DEBUG]   org.sonatype.aether:aether-util:jar:1.7:compile
>
> [DEBUG]   org.codehaus.plexus:plexus-interpolation:jar:1.14:compile
>
> [DEBUG]   org.codehaus.plexus:plexus-utils:jar:2.0.5:compile (version
> managed from 2.0.4)
>
> [DEBUG]   org.codehaus.plexus:plexus-classworlds:jar:2.2.3:compile
>
> [DEBUG]
> org.codehaus.plexus:plexus-component-annotations:jar:1.5.5:compile
>
> [DEBUG]   org.sonatype.plexus:plexus-sec-dispatcher:jar:1.3:compile
>
>
>
>
>
>
>
> [ERROR] Failed to execute goal
> org.apache.hadoop:hadoop-maven-plugins:2.6.0:protoc (compile-protoc) on
> project tez-api: org.apache.maven.plugin.MojoExecutionException: protoc
> failure -> [Help 1]
>
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.hadoop:hadoop-maven-plugins:2.6.0:protoc (compile-protoc)
> on project tez-api: org.apache.maven.plugin.MojoExecutionException: protoc
> failure
>
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
>
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
>
> at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
>
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
>
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
>
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
>
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
>
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
>
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
>
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
>
> at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
>
> Caused by: org.apache.maven.plugin.MojoExecutionException:
> org.apache.maven.plugin.MojoExecutionException: protoc failure
>
> at
> org.apache.hadoop.maven.plugin.protoc.ProtocMojo.execute(ProtocMojo.java:105)
>
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>
> Xiaoyong
>
>
>

Re: Pig(0.14.0) on Tez(0.7.0)

[Sandeep Kumar]

Hi Jeff,

The cloudera Hadoop is using guava-11.0.2.jar.
I've also exported one environment variable before running pig:

export HADOOP_USER_CLASSPATH_FIRST=true

Should i recompile my PIG code with new guava jar?

Regards,
Sandeep

On Wed, Sep 2, 2015 at 1:20 PM, Jianfeng (Jeff) Zhang <
jzh...@hortonworks.com> wrote:

>
> It looks like classpath issue due guava version. Could you check the guava
> version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use guava
> 11.0
>
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Sandeep Kumar 
> Reply-To: "user@tez.apache.org" 
> Date: Wednesday, September 2, 2015 at 2:28 PM
> To: "user@tez.apache.org" 
> Subject: Pig(0.14.0) on Tez(0.7.0)
>
> Hi Guys,
>
> I've a pig script which is as follows:
>
> map = LOAD 'input/' using
> com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
> normalized_map_data = foreach r360map generate flatten(com.Map($0..));
>
> data_grouped_over_event_time = GROUP normalized_map_data by
> (((startTime/1000) / 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
> final_data = foreach data_grouped_over_event_time generate flatten($1);
>
> store final_data into 'tmp/200MB' using com.StorageModel();
>
>
> I'm trying to run my Pig job using Tez but i'm facing an exception which
> is attached. I searched over net and found some vague solutions. I've only
> two configuration entries in my tez-site.xml which are as follows:
>
> 
>   tez.lib.uris
>   hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz
> 
> 
>   URL for where the Tez UI is hosted
>   tez.tez-ui.history-url.base
>   http://node4:/tez-ui/
> 
>
> Is there anything else which i'm missing and supposed to be added in Tez
> configuration file.
> I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided with
> Tez-0.5.2 but the exception is coming in both the versions of tez.
>
> Following are the version of softwares i'm using.
>
> HadoopVersion:
> 2.6.0-cdh5.4.4
>
> PigVersion:
> 0.14.0-SNAPSHOT
>
> TezVersion:
> 0.7.0
>
>
>
>
> Regards,
> Sandeep
>

Re: Pig(0.14.0) on Tez(0.7.0)

[Jianfeng (Jeff) Zhang]

It looks like classpath issue due guava version. Could you check the guava 
version CDH 5.4.4 is using ? AFAIK, Tez use guava 11.02 and pig use guava 11.0



Best Regard,
Jeff Zhang


From: Sandeep Kumar mailto:sandeepdas@gmail.com>>
Reply-To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Date: Wednesday, September 2, 2015 at 2:28 PM
To: "user@tez.apache.org" 
mailto:user@tez.apache.org>>
Subject: Pig(0.14.0) on Tez(0.7.0)

Hi Guys,

I've a pig script which is as follows:

map = LOAD 'input/' using 
com.RawPigLoader('conf/Map.xml','conf/R360MapSignalling.json','csv');
normalized_map_data = foreach r360map generate flatten(com.Map($0..));

data_grouped_over_event_time = GROUP normalized_map_data by (((startTime/1000) 
/ 3600) * 3600) PARTITION BY com.CustomTimePartitioner;
final_data = foreach data_grouped_over_event_time generate flatten($1);

store final_data into 'tmp/200MB' using com.StorageModel();


I'm trying to run my Pig job using Tez but i'm facing an exception which is 
attached. I searched over net and found some vague solutions. I've only two 
configuration entries in my tez-site.xml which are as follows:


  tez.lib.uris
  hdfs://node3:8020/user/hdfs/tez-0.7.0.tar.gz


  URL for where the Tez UI is hosted
  tez.tez-ui.history-url.base
  http://node4:/tez-ui/


Is there anything else which i'm missing and supposed to be added in Tez 
configuration file.
I've compiled PIG-0.14.0 manually with Tez-0.7.0 whereas it provided with 
Tez-0.5.2 but the exception is coming in both the versions of tez.

Following are the version of softwares i'm using.

HadoopVersion: 2.6.0-cdh5.4.4
PigVersion: 0.14.0-SNAPSHOT
TezVersion: 0.7.0



Regards,
Sandeep