Re: My first TEZ job fails

[Hitesh Shah]

To clarify, 

if you do the following: 

hdfs -mkdir /tez/
hdfs -put $TEZ_HOME/share/tez-0.7.1.tar.gz /tez/

Then tez.lib.uris should be “/tez/tez-0.7.1.tar.gz” 

— HItesh 


> On May 20, 2016, at 5:11 PM, Mich Talebzadeh  
> wrote:
> 
> I think the instructions can be simplified as follows
> 
> wherever you have your tarball say $TEZ_HOME/share/tez-0.7.1.tar.gz
> 
> you create the said absolute directory in your hdfs and put you tarball there
> 
> Also your uris should refer to the said absolute path (a single directory 
> structure)
> 
>   tez.lib.uris
> /usr/lib/apache-tez-0.7.1-bin/share
>  
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
> On 21 May 2016 at 01:06, Mich Talebzadeh  wrote:
> Thanks Hitesh,
> 
> Here
> 
> tez.lib.uris should be a single value and the value should be the absolute 
> path pointing the tez tarball on HDFS
> 
> OK
> 
> are we talking about the one under $TEZ_HOME/share?
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr
> -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
> 
> 
> 
> And my uris would be the abosulte path to the above tarball under 
> /usr/lib/apache-tez-0.7.1-bin/share
> 
> cat tez-site.xml
> 
>   
> tez.version
> 0.7.1
>   
>   
> tez.lib.uris
> /usr/lib/apache-tez-0.7.1-bin/share
>   
> 
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/Hadoop>
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
> On 21 May 2016 at 00:58, Hitesh Shah  wrote:
> Hello Mich,
> 
> Yes those are the instructions.
> 
> tez.lib.uris should be a single value and the value should be the absolute 
> path pointing the tez tarball on HDFS. The tarball to be uploaded to HDFS can 
> be found at apache-tez-{x.y.z}/share/tez.tar.gz ( or 
> tez-dist/target/tez-x.y.x.tar.gz if you are using your local build ).
> 
> thanks
> — Hitesh
> 
> > On May 20, 2016, at 4:39 PM, Mich Talebzadeh  
> > wrote:
> >
> > This is the instruction?
> >
> > Created by Hitesh Shah, last modified on May 02, 2016 Go to start of 
> > metadata
> > Making use of the Tez Binary Release tarball
> >
> >   • If the binary tarball's name does not include anything referring to 
> > a hadoop version, then this implies that the tarball was compiled against 
> > the hadoop version that the Tez release compiles against by default. For 
> > example, for 0.7.0 and 0.8.0, the default hadoop version used is 2.6.0 ( 
> > this can be found by looking for the hadoop.version property in the 
> > top-level pom.xml in the source tarball for the release).
> >
> >   • The tarball structure is as follows:
> >
> >
> > ?
> > apache-tez-{x.y.z}/
> >   /tez*.jar
> >   /lib/*.jar
> >   /conf/tez*.xml.template
> >   /share/tez.tar.gz
> >   • Set up Tez by following INSTALL.txt and use 
> > apache-tez-{x.y.z}/share/tez.tar.gz as the full tarball to be uploaded to 
> > HDFS.
> >   • Use the config templates under apache-tez-{x.y.z}/conf/ to create 
> > the tez-site.xml as needed in an appropriate conf directory. If you end up 
> > using apache-tez-{x.y.z}/conf/, then do an export 
> > TEZ_CONF_DIR="apache-tez-{x.y.z}/conf/"
> >   • Add "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}" 
> > to HADOOP_CLASSPATH so as to get the tez client jars onto the classpath 
> > invoked when using the "bin/hadoop jar" command to run an example job.
> >
> >
> > Dr Mich Talebzadeh
> >
> > LinkedIn  
> > https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > On 21 May 2016 at 00:37, Mich Talebzadeh  wrote:
> > Thanks both
> >
> > so this is the file that needs to go in hdfs correct?
> >
> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr tez-0.7.1.tar.gz
> > -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
> >
> >
> > In hdfs I have now
> >
> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls 
> > /usr/lib/apache-tez-0.7.1-bin
> >
> > -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31 
> > /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
> >
> >
> > Now I only installed tez under
> >
> > /usr/lib/apache-tez-0.7.1-bin
> >
> > MY Hadoop is installed in
> >
> > echo $HADOOP_HOME
> > /home/hduser/hadoop-2.6.0
> >
> > and my xml-site in in
> >
> > $HADOOP_HOME/etc/hadoop
> >
> > OK and this is my sml-site content
> >
> > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> > 
> >   
> > tez.version
> > 0.7.1
> >   
> >   
> > tez.lib.uris
> > 
> > /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
> >   
> > 
> >
> > Is the red correct please?
> >
> >
> >
> > Dr Mich Talebzadeh
> >
> > LinkedIn  
> > htt

Re: My first TEZ job fails

[Hitesh Shah]

Sorry - I forgot to call that out in the instructions as the config templates 
are being generated only from 0.8.x onwards. Will update the wiki.

— Hitesh 

> On May 20, 2016, at 4:51 PM, Mich Talebzadeh  
> wrote:
> 
> With reference to
> 
> ... then do an export TEZ_CONF_DIR="apache-tez-{x.y.z}/conf/"
> 
> By they way there is no conf directory under apache-tez-0.7.1-bin! There are 
> only lib, and share
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> ltr|grep '^d'
> drwxr-xr-x  2 hduser hadoop4096 May 20 22:58 share
> drwxr-xr-x  2 hduser hadoop4096 May 20 22:58 lib
> drwxr-xr-x 85 root   root 40960 May 20 22:58 ..
> drwxr-xr-x  4 hduser hadoop4096 May 20 23:18 .
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin>
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
> On 21 May 2016 at 00:42, Mich Talebzadeh  wrote:
> Please bear in mind that all I want is to test Hive with TEZ engine. I 
> already have Hive working OK with Spark 1.3.1 engine and I compiled it spark 
> from source code. so hopefully I can use TEZ as Spark engine as well.
> 
> 
> thanks
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
> On 21 May 2016 at 00:39, Mich Talebzadeh  wrote:
> This is the instruction?
> 
> Created by Hitesh Shah, last modified on May 02, 2016 Go to start of metadata
> Making use of the Tez Binary Release tarball
> 
>   • If the binary tarball's name does not include anything referring to a 
> hadoop version, then this implies that the tarball was compiled against the 
> hadoop version that the Tez release compiles against by default. For example, 
> for 0.7.0 and 0.8.0, the default hadoop version used is 2.6.0 ( this can be 
> found by looking for the hadoop.version property in the top-level pom.xml in 
> the source tarball for the release).
> 
>   • The tarball structure is as follows:
> 
> 
> ?
> apache-tez-{x.y.z}/
>   /tez*.jar
>   /lib/*.jar
>   /conf/tez*.xml.template
>   /share/tez.tar.gz
>   • Set up Tez by following INSTALL.txt and use 
> apache-tez-{x.y.z}/share/tez.tar.gz as the full tarball to be uploaded to 
> HDFS.
>   • Use the config templates under apache-tez-{x.y.z}/conf/ to create the 
> tez-site.xml as needed in an appropriate conf directory. If you end up using 
> apache-tez-{x.y.z}/conf/, then do an export 
> TEZ_CONF_DIR="apache-tez-{x.y.z}/conf/"
>   • Add "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}" 
> to HADOOP_CLASSPATH so as to get the tez client jars onto the classpath 
> invoked when using the "bin/hadoop jar" command to run an example job.
>  
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
> On 21 May 2016 at 00:37, Mich Talebzadeh  wrote:
> Thanks both
> 
> so this is the file that needs to go in hdfs correct?
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr tez-0.7.1.tar.gz
> -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
> 
> 
> In hdfs I have now
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin
> 
> -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31 
> /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
> 
> 
> Now I only installed tez under
> 
> /usr/lib/apache-tez-0.7.1-bin
> 
> MY Hadoop is installed in
> 
> echo $HADOOP_HOME
> /home/hduser/hadoop-2.6.0
> 
> and my xml-site in in
> 
> $HADOOP_HOME/etc/hadoop
> 
> OK and this is my sml-site content
> 
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> 
>   
> tez.version
> 0.7.1
>   
>   
> tez.lib.uris
> 
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>   
> 
> 
> Is the red correct please?
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
> On 21 May 2016 at 00:24, Bikas Saha  wrote:
> >> tez.lib.uris assumes that paths are based on the default fs and therefore 
> >> if your setup is using HDFS as default, the paths /usr/lib would be invalid
> 
> Are you sure? The below paths looks right to me except that the contents of 
> the directories are wrong.
> 
> tez.lib.uris
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin Found 2 items
> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15 
> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/li

Re: My first TEZ job fails

[Mich Talebzadeh]

I think the instructions can be simplified as follows

wherever you have your tarball say $TEZ_HOME/share/tez-0.7.1.tar.gz

you create the said absolute directory in your hdfs and put you tarball
there

Also your uris should refer to the said absolute path (a single directory
structure)

  tez.lib.uris
/usr/lib/apache-tez-0.7.1-bin/share


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 May 2016 at 01:06, Mich Talebzadeh  wrote:

> Thanks Hitesh,
>
> Here
>
> tez.lib.uris should be a single value and the value should be the absolute
> path pointing the tez tarball on HDFS
>
> OK
>
> are we talking about the one under $TEZ_HOME/share?
>
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr
> -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
>
>
>
> And my uris would be the abosulte path to the above tarball under
> /usr/lib/apache-tez-0.7.1-bin/share
>
> cat tez-site.xml
> 
>   
> tez.version
> 0.7.1
>   
>   
> tez.lib.uris
> /usr/lib/apache-tez-0.7.1-bin/share
>   
> 
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/Hadoop>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 May 2016 at 00:58, Hitesh Shah  wrote:
>
>> Hello Mich,
>>
>> Yes those are the instructions.
>>
>> tez.lib.uris should be a single value and the value should be the
>> absolute path pointing the tez tarball on HDFS. The tarball to be uploaded
>> to HDFS can be found at apache-tez-{x.y.z}/share/tez.tar.gz ( or
>> tez-dist/target/tez-x.y.x.tar.gz if you are using your local build ).
>>
>> thanks
>> — Hitesh
>>
>> > On May 20, 2016, at 4:39 PM, Mich Talebzadeh 
>> wrote:
>> >
>> > This is the instruction?
>> >
>> > Created by Hitesh Shah, last modified on May 02, 2016 Go to start of
>> metadata
>> > Making use of the Tez Binary Release tarball
>> >
>> >   • If the binary tarball's name does not include anything
>> referring to a hadoop version, then this implies that the tarball was
>> compiled against the hadoop version that the Tez release compiles against
>> by default. For example, for 0.7.0 and 0.8.0, the default hadoop version
>> used is 2.6.0 ( this can be found by looking for the hadoop.version
>> property in the top-level pom.xml in the source tarball for the release).
>> >
>> >   • The tarball structure is as follows:
>> >
>> >
>> > ?
>> > apache-tez-{x.y.z}/
>> >   /tez*.jar
>> >   /lib/*.jar
>> >   /conf/tez*.xml.template
>> >   /share/tez.tar.gz
>> >   • Set up Tez by following INSTALL.txt and use
>> apache-tez-{x.y.z}/share/tez.tar.gz as the full tarball to be uploaded to
>> HDFS.
>> >   • Use the config templates under apache-tez-{x.y.z}/conf/ to
>> create the tez-site.xml as needed in an appropriate conf directory. If you
>> end up using apache-tez-{x.y.z}/conf/, then do an export
>> TEZ_CONF_DIR="apache-tez-{x.y.z}/conf/"
>> >   • Add
>> "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}" to
>> HADOOP_CLASSPATH so as to get the tez client jars onto the classpath
>> invoked when using the "bin/hadoop jar" command to run an example job.
>> >
>> >
>> > Dr Mich Talebzadeh
>> >
>> > LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> >
>> > http://talebzadehmich.wordpress.com
>> >
>> >
>> > On 21 May 2016 at 00:37, Mich Talebzadeh 
>> wrote:
>> > Thanks both
>> >
>> > so this is the file that needs to go in hdfs correct?
>> >
>> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr
>> tez-0.7.1.tar.gz
>> > -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
>> >
>> >
>> > In hdfs I have now
>> >
>> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls
>> /usr/lib/apache-tez-0.7.1-bin
>> >
>> > -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31
>> /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
>> >
>> >
>> > Now I only installed tez under
>> >
>> > /usr/lib/apache-tez-0.7.1-bin
>> >
>> > MY Hadoop is installed in
>> >
>> > echo $HADOOP_HOME
>> > /home/hduser/hadoop-2.6.0
>> >
>> > and my xml-site in in
>> >
>> > $HADOOP_HOME/etc/hadoop
>> >
>> > OK and this is my sml-site content
>> >
>> > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
>> > 
>> >   
>> > tez.version
>> > 0.7.1
>> >   
>> >   
>> > tez.lib.uris
>> >
>>  
>> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>> >   
>> > 
>> >
>> > Is the red correct please?
>> >
>> >
>> >
>> > Dr Mich Talebzadeh
>> >
>> > LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP

Re: My first TEZ job fails

[Mich Talebzadeh]

Thanks Hitesh,

Here

tez.lib.uris should be a single value and the value should be the absolute
path pointing the tez tarball on HDFS

OK

are we talking about the one under $TEZ_HOME/share?

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr
-rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz



And my uris would be the abosulte path to the above tarball under
/usr/lib/apache-tez-0.7.1-bin/share

cat tez-site.xml

  
tez.version
0.7.1
  
  
tez.lib.uris
/usr/lib/apache-tez-0.7.1-bin/share
  

hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/Hadoop>



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 May 2016 at 00:58, Hitesh Shah  wrote:

> Hello Mich,
>
> Yes those are the instructions.
>
> tez.lib.uris should be a single value and the value should be the absolute
> path pointing the tez tarball on HDFS. The tarball to be uploaded to HDFS
> can be found at apache-tez-{x.y.z}/share/tez.tar.gz ( or
> tez-dist/target/tez-x.y.x.tar.gz if you are using your local build ).
>
> thanks
> — Hitesh
>
> > On May 20, 2016, at 4:39 PM, Mich Talebzadeh 
> wrote:
> >
> > This is the instruction?
> >
> > Created by Hitesh Shah, last modified on May 02, 2016 Go to start of
> metadata
> > Making use of the Tez Binary Release tarball
> >
> >   • If the binary tarball's name does not include anything referring
> to a hadoop version, then this implies that the tarball was compiled
> against the hadoop version that the Tez release compiles against by
> default. For example, for 0.7.0 and 0.8.0, the default hadoop version used
> is 2.6.0 ( this can be found by looking for the hadoop.version property in
> the top-level pom.xml in the source tarball for the release).
> >
> >   • The tarball structure is as follows:
> >
> >
> > ?
> > apache-tez-{x.y.z}/
> >   /tez*.jar
> >   /lib/*.jar
> >   /conf/tez*.xml.template
> >   /share/tez.tar.gz
> >   • Set up Tez by following INSTALL.txt and use
> apache-tez-{x.y.z}/share/tez.tar.gz as the full tarball to be uploaded to
> HDFS.
> >   • Use the config templates under apache-tez-{x.y.z}/conf/ to
> create the tez-site.xml as needed in an appropriate conf directory. If you
> end up using apache-tez-{x.y.z}/conf/, then do an export
> TEZ_CONF_DIR="apache-tez-{x.y.z}/conf/"
> >   • Add
> "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}" to
> HADOOP_CLASSPATH so as to get the tez client jars onto the classpath
> invoked when using the "bin/hadoop jar" command to run an example job.
> >
> >
> > Dr Mich Talebzadeh
> >
> > LinkedIn
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > On 21 May 2016 at 00:37, Mich Talebzadeh 
> wrote:
> > Thanks both
> >
> > so this is the file that needs to go in hdfs correct?
> >
> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr
> tez-0.7.1.tar.gz
> > -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
> >
> >
> > In hdfs I have now
> >
> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls
> /usr/lib/apache-tez-0.7.1-bin
> >
> > -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31
> /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
> >
> >
> > Now I only installed tez under
> >
> > /usr/lib/apache-tez-0.7.1-bin
> >
> > MY Hadoop is installed in
> >
> > echo $HADOOP_HOME
> > /home/hduser/hadoop-2.6.0
> >
> > and my xml-site in in
> >
> > $HADOOP_HOME/etc/hadoop
> >
> > OK and this is my sml-site content
> >
> > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> > 
> >   
> > tez.version
> > 0.7.1
> >   
> >   
> > tez.lib.uris
> >
>  
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
> >   
> > 
> >
> > Is the red correct please?
> >
> >
> >
> > Dr Mich Talebzadeh
> >
> > LinkedIn
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > On 21 May 2016 at 00:24, Bikas Saha  wrote:
> > >> tez.lib.uris assumes that paths are based on the default fs and
> therefore if your setup is using HDFS as default, the paths /usr/lib would
> be invalid
> >
> > Are you sure? The below paths looks right to me except that the contents
> of the directories are wrong.
> >
> > tez.lib.uris
> >
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
> >
> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
> /usr/lib/apache-tez-0.7.1-bin Found 2 items
> > -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15
> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
> > drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib
> >
> >

Re: My first TEZ job fails

[Hitesh Shah]

Hello Mich, 

Yes those are the instructions. 

tez.lib.uris should be a single value and the value should be the absolute path 
pointing the tez tarball on HDFS. The tarball to be uploaded to HDFS can be 
found at apache-tez-{x.y.z}/share/tez.tar.gz ( or 
tez-dist/target/tez-x.y.x.tar.gz if you are using your local build ).

thanks
— Hitesh

> On May 20, 2016, at 4:39 PM, Mich Talebzadeh  
> wrote:
> 
> This is the instruction?
> 
> Created by Hitesh Shah, last modified on May 02, 2016 Go to start of metadata
> Making use of the Tez Binary Release tarball
> 
>   • If the binary tarball's name does not include anything referring to a 
> hadoop version, then this implies that the tarball was compiled against the 
> hadoop version that the Tez release compiles against by default. For example, 
> for 0.7.0 and 0.8.0, the default hadoop version used is 2.6.0 ( this can be 
> found by looking for the hadoop.version property in the top-level pom.xml in 
> the source tarball for the release).
> 
>   • The tarball structure is as follows:
> 
> 
> ?
> apache-tez-{x.y.z}/
>   /tez*.jar
>   /lib/*.jar
>   /conf/tez*.xml.template
>   /share/tez.tar.gz
>   • Set up Tez by following INSTALL.txt and use 
> apache-tez-{x.y.z}/share/tez.tar.gz as the full tarball to be uploaded to 
> HDFS.
>   • Use the config templates under apache-tez-{x.y.z}/conf/ to create the 
> tez-site.xml as needed in an appropriate conf directory. If you end up using 
> apache-tez-{x.y.z}/conf/, then do an export 
> TEZ_CONF_DIR="apache-tez-{x.y.z}/conf/"
>   • Add "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}" 
> to HADOOP_CLASSPATH so as to get the tez client jars onto the classpath 
> invoked when using the "bin/hadoop jar" command to run an example job.
>  
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
> On 21 May 2016 at 00:37, Mich Talebzadeh  wrote:
> Thanks both
> 
> so this is the file that needs to go in hdfs correct?
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr tez-0.7.1.tar.gz
> -rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz
> 
> 
> In hdfs I have now
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin
> 
> -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31 
> /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
> 
> 
> Now I only installed tez under
> 
> /usr/lib/apache-tez-0.7.1-bin
> 
> MY Hadoop is installed in
> 
> echo $HADOOP_HOME
> /home/hduser/hadoop-2.6.0
> 
> and my xml-site in in
> 
> $HADOOP_HOME/etc/hadoop
> 
> OK and this is my sml-site content
> 
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> 
>   
> tez.version
> 0.7.1
>   
>   
> tez.lib.uris
> 
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>   
> 
> 
> Is the red correct please?
> 
> 
> 
> Dr Mich Talebzadeh
>  
> LinkedIn  
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>  
> http://talebzadehmich.wordpress.com
>  
> 
> On 21 May 2016 at 00:24, Bikas Saha  wrote:
> >> tez.lib.uris assumes that paths are based on the default fs and therefore 
> >> if your setup is using HDFS as default, the paths /usr/lib would be invalid
> 
> Are you sure? The below paths looks right to me except that the contents of 
> the directories are wrong.
> 
> tez.lib.uris
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin Found 2 items
> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15 
> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib
> 
> 
> -Original Message-
> From: Hitesh Shah [mailto:hit...@apache.org]
> Sent: Friday, May 20, 2016 4:18 PM
> To: user@tez.apache.org
> Subject: Re: My first TEZ job fails
> 
> Can you try the instructions mentioned at 
> https://cwiki.apache.org/confluence/display/TEZ/Tez+Release+FAQ ?
> 
> tez.lib.uris assumes that paths are based on the default fs and therefore if 
> your setup is using HDFS as default, the paths /usr/lib would be invalid.
> 
> — HItesh
> 
> > On May 20, 2016, at 3:39 PM, Mich Talebzadeh  
> > wrote:
> >
> > Still failing with /apache-tez-0.7.1-bin I am afraid.
> >
> > OK this is my tez-site.xml
> >
> > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> > 
> >   
> > tez.version
> > 0.7.1
> >   
> >   
> > tez.lib.uris
> > 
> > /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
> >   
> > 
> >
> > This is what I have put in hdfs directory
> >
> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
> > /usr/lib/apache-tez-0.7.1-b

Re: My first TEZ job fails

[Mich Talebzadeh]

With reference to

... then do an export TEZ_CONF_DIR="apache-tez-{x.y.z}/conf/"

By they way there is no conf directory under apache-tez-0.7.1-bin! There
are only lib, and share

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> ltr|grep '^d'
drwxr-xr-x  2 hduser hadoop4096 May 20 22:58 share
drwxr-xr-x  2 hduser hadoop4096 May 20 22:58 lib
drwxr-xr-x 85 root   root 40960 May 20 22:58 ..
drwxr-xr-x  4 hduser hadoop4096 May 20 23:18 .
hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin>

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 May 2016 at 00:42, Mich Talebzadeh  wrote:

> Please bear in mind that all I want is to test Hive with TEZ engine. I
> already have Hive working OK with Spark 1.3.1 engine and I compiled it
> spark from source code. so hopefully I can use TEZ as Spark engine as well.
>
>
> thanks
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 May 2016 at 00:39, Mich Talebzadeh 
> wrote:
>
>> This is the instruction?
>>
>> Created by Hitesh Shah
>> , last modified on May
>> 02, 2016
>> 
>>  Go
>> to start of metadata
>> 
>> Making use of the Tez Binary Release tarball
>>
>>-
>>
>>If the binary tarball's name does not include anything referring to a
>>hadoop version, then this implies that the tarball was compiled against 
>> the
>>hadoop version that the Tez release compiles against by default. For
>>example, for 0.7.0 and 0.8.0, the default hadoop version used is 2.6.0 (
>>this can be found by looking for the hadoop.version property in the
>>top-level pom.xml in the source tarball for the release).
>>-
>>
>>The tarball structure is as follows:
>>
>>? 
>>apache-tez-{x.y.z}/
>>  /tez*.jar
>>  /lib/*.jar
>>  /conf/tez*.xml.template
>>  /share/tez.tar.gz
>>- Set up Tez by following INSTALL.txt and use 
>> apache-tez-{x.y.z}/share/tez.tar.gz
>>as the full tarball to be uploaded to HDFS.
>>- Use the config templates under apache-tez-{x.y.z}/conf/ to create
>>the tez-site.xml as needed in an appropriate conf directory. If you end up
>>using apache-tez-{x.y.z}/conf/, then do an export TEZ_CONF_DIR="
>>apache-tez-{x.y.z}/conf/"
>>- Add "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}"
>>to HADOOP_CLASSPATH so as to get the tez client jars onto the classpath
>>invoked when using the "bin/hadoop jar" command to run an example job.
>>
>>
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 May 2016 at 00:37, Mich Talebzadeh 
>> wrote:
>>
>>> Thanks both
>>>
>>> so this is the file that needs to go in hdfs correct?
>>>
>>> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr
>>> tez-0.7.1.tar.gz
>>> -*rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz*
>>>
>>>
>>> In hdfs I have now
>>>
>>> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls
>>> /usr/lib/apache-tez-0.7.1-bin
>>>
>>> -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31
>>> /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
>>>
>>>
>>> Now I only installed tez under
>>>
>>> /usr/lib/apache-tez-0.7.1-bin
>>>
>>> MY Hadoop is installed in
>>>
>>> echo $HADOOP_HOME
>>> /home/hduser/hadoop-2.6.0
>>>
>>> and my xml-site in in
>>>
>>> $HADOOP_HOME/etc/hadoop
>>>
>>> OK and this is my sml-site content
>>>
>>> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
>>> 
>>>   
>>> tez.version
>>> 0.7.1
>>>   
>>>   
>>> tez.lib.uris
>>>
>>> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>>>   
>>> 
>>>
>>> Is the red correct please?
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn * 
>>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> *
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>>
>>> On 21 May 2016 at 00:24, Bikas Saha  wrote:
>>>
 >> tez.lib.uris assumes that 

Re: My first TEZ job fails

[Mich Talebzadeh]

Please bear in mind that all I want is to test Hive with TEZ engine. I
already have Hive working OK with Spark 1.3.1 engine and I compiled it
spark from source code. so hopefully I can use TEZ as Spark engine as well.


thanks



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 May 2016 at 00:39, Mich Talebzadeh  wrote:

> This is the instruction?
>
> Created by Hitesh Shah
> , last modified on May
> 02, 2016
> 
>  Go
> to start of metadata
> 
> Making use of the Tez Binary Release tarball
>
>-
>
>If the binary tarball's name does not include anything referring to a
>hadoop version, then this implies that the tarball was compiled against the
>hadoop version that the Tez release compiles against by default. For
>example, for 0.7.0 and 0.8.0, the default hadoop version used is 2.6.0 (
>this can be found by looking for the hadoop.version property in the
>top-level pom.xml in the source tarball for the release).
>-
>
>The tarball structure is as follows:
>
>? 
>apache-tez-{x.y.z}/
>  /tez*.jar
>  /lib/*.jar
>  /conf/tez*.xml.template
>  /share/tez.tar.gz
>- Set up Tez by following INSTALL.txt and use 
> apache-tez-{x.y.z}/share/tez.tar.gz
>as the full tarball to be uploaded to HDFS.
>- Use the config templates under apache-tez-{x.y.z}/conf/ to create
>the tez-site.xml as needed in an appropriate conf directory. If you end up
>using apache-tez-{x.y.z}/conf/, then do an export TEZ_CONF_DIR="
>apache-tez-{x.y.z}/conf/"
>- Add "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}"
>to HADOOP_CLASSPATH so as to get the tez client jars onto the classpath
>invoked when using the "bin/hadoop jar" command to run an example job.
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 May 2016 at 00:37, Mich Talebzadeh 
> wrote:
>
>> Thanks both
>>
>> so this is the file that needs to go in hdfs correct?
>>
>> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr tez-0.7.1.tar.gz
>> -*rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz*
>>
>>
>> In hdfs I have now
>>
>> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls
>> /usr/lib/apache-tez-0.7.1-bin
>>
>> -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31
>> /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
>>
>>
>> Now I only installed tez under
>>
>> /usr/lib/apache-tez-0.7.1-bin
>>
>> MY Hadoop is installed in
>>
>> echo $HADOOP_HOME
>> /home/hduser/hadoop-2.6.0
>>
>> and my xml-site in in
>>
>> $HADOOP_HOME/etc/hadoop
>>
>> OK and this is my sml-site content
>>
>> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
>> 
>>   
>> tez.version
>> 0.7.1
>>   
>>   
>> tez.lib.uris
>>
>> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>>   
>> 
>>
>> Is the red correct please?
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * 
>> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> *
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>> On 21 May 2016 at 00:24, Bikas Saha  wrote:
>>
>>> >> tez.lib.uris assumes that paths are based on the default fs and
>>> therefore if your setup is using HDFS as default, the paths /usr/lib would
>>> be invalid
>>>
>>> Are you sure? The below paths looks right to me except that the contents
>>> of the directories are wrong.
>>>
>>> tez.lib.uris
>>>
>>> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>>>
>>> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
>>> /usr/lib/apache-tez-0.7.1-bin Found 2 items
>>> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15
>>> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
>>> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27
>>> /usr/lib/apache-tez-0.7.1-bin/lib
>>>
>>>
>>> -Original Message-
>>> From: Hitesh Shah [mailto:hit...@apache.org]
>>> Sent: Friday, May 20, 2016 4:18 PM
>>> To: user@tez.apache.org
>>> Subject: Re: My first TEZ job fails
>>>
>>> Can you try the instructions mentioned at
>>> https://cwiki.apache.org/con

Re: My first TEZ job fails

[Mich Talebzadeh]

This is the instruction?

Created by Hitesh Shah ,
last modified on May 02, 2016

Go
to start of metadata

Making use of the Tez Binary Release tarball

   -

   If the binary tarball's name does not include anything referring to a
   hadoop version, then this implies that the tarball was compiled against the
   hadoop version that the Tez release compiles against by default. For
   example, for 0.7.0 and 0.8.0, the default hadoop version used is 2.6.0 (
   this can be found by looking for the hadoop.version property in the
   top-level pom.xml in the source tarball for the release).
   -

   The tarball structure is as follows:

   ? 
   apache-tez-{x.y.z}/
 /tez*.jar
 /lib/*.jar
 /conf/tez*.xml.template
 /share/tez.tar.gz
   - Set up Tez by following INSTALL.txt and use
apache-tez-{x.y.z}/share/tez.tar.gz
   as the full tarball to be uploaded to HDFS.
   - Use the config templates under apache-tez-{x.y.z}/conf/ to create the
   tez-site.xml as needed in an appropriate conf directory. If you end up
   using apache-tez-{x.y.z}/conf/, then do an export TEZ_CONF_DIR="
   apache-tez-{x.y.z}/conf/"
   - Add "apache-tez-{x.y.z}/*:apache-tez-{x.y.z}/lib/*:${TEZ_CONF_DIR}" to
   HADOOP_CLASSPATH so as to get the tez client jars onto the classpath
   invoked when using the "bin/hadoop jar" command to run an example job.





Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 May 2016 at 00:37, Mich Talebzadeh  wrote:

> Thanks both
>
> so this is the file that needs to go in hdfs correct?
>
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr tez-0.7.1.tar.gz
> -*rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz*
>
>
> In hdfs I have now
>
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls
> /usr/lib/apache-tez-0.7.1-bin
>
> -rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31
> /usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz
>
>
> Now I only installed tez under
>
> /usr/lib/apache-tez-0.7.1-bin
>
> MY Hadoop is installed in
>
> echo $HADOOP_HOME
> /home/hduser/hadoop-2.6.0
>
> and my xml-site in in
>
> $HADOOP_HOME/etc/hadoop
>
> OK and this is my sml-site content
>
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> 
>   
> tez.version
> 0.7.1
>   
>   
> tez.lib.uris
>
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>   
> 
>
> Is the red correct please?
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 21 May 2016 at 00:24, Bikas Saha  wrote:
>
>> >> tez.lib.uris assumes that paths are based on the default fs and
>> therefore if your setup is using HDFS as default, the paths /usr/lib would
>> be invalid
>>
>> Are you sure? The below paths looks right to me except that the contents
>> of the directories are wrong.
>>
>> tez.lib.uris
>>
>> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>>
>> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
>> /usr/lib/apache-tez-0.7.1-bin Found 2 items
>> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15
>> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
>> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27
>> /usr/lib/apache-tez-0.7.1-bin/lib
>>
>>
>> -Original Message-
>> From: Hitesh Shah [mailto:hit...@apache.org]
>> Sent: Friday, May 20, 2016 4:18 PM
>> To: user@tez.apache.org
>> Subject: Re: My first TEZ job fails
>>
>> Can you try the instructions mentioned at
>> https://cwiki.apache.org/confluence/display/TEZ/Tez+Release+FAQ ?
>>
>> tez.lib.uris assumes that paths are based on the default fs and therefore
>> if your setup is using HDFS as default, the paths /usr/lib would be invalid.
>>
>> — HItesh
>>
>> > On May 20, 2016, at 3:39 PM, Mich Talebzadeh 
>> wrote:
>> >
>> > Still failing with /apache-tez-0.7.1-bin I am afraid.
>> >
>> > OK this is my tez-site.xml
>> >
>> > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
>> > 
>> >   
>> > tez.version
>> > 0.7.1
>> >   
>> >   
>> > tez.lib.uris
>> >
>>  
>> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>> >   
>> > 
>> >
>> > This is what I have put in hdfs directory
>> >
>> > hduser@rhes564: /

Re: My first TEZ job fails

[Hitesh Shah]

Sorry - missed the fact that the /usr/lib paths were also being used on HDFS.  

In any case, the contents of 
/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib will be only 
from the minimal tarball and therefore will not work out of the box as the 
relevant hadoop-yarn jars would be missing. The Release FAQ wiki has the 
correct instructions on how to use the binary release. 

— Hitesh


> On May 20, 2016, at 4:24 PM, Bikas Saha  wrote:
> 
>>> tez.lib.uris assumes that paths are based on the default fs and therefore 
>>> if your setup is using HDFS as default, the paths /usr/lib would be invalid
> 
> Are you sure? The below paths looks right to me except that the contents of 
> the directories are wrong.
> 
> tez.lib.uris
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin Found 2 items
> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15 
> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib
> 
> 
> -Original Message-
> From: Hitesh Shah [mailto:hit...@apache.org] 
> Sent: Friday, May 20, 2016 4:18 PM
> To: user@tez.apache.org
> Subject: Re: My first TEZ job fails
> 
> Can you try the instructions mentioned at 
> https://cwiki.apache.org/confluence/display/TEZ/Tez+Release+FAQ ?
> 
> tez.lib.uris assumes that paths are based on the default fs and therefore if 
> your setup is using HDFS as default, the paths /usr/lib would be invalid.
> 
> — HItesh
> 
>> On May 20, 2016, at 3:39 PM, Mich Talebzadeh  
>> wrote:
>> 
>> Still failing with /apache-tez-0.7.1-bin I am afraid.
>> 
>> OK this is my tez-site.xml
>> 
>> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml 
>> 
>>  
>>tez.version
>>0.7.1
>>  
>>  
>>tez.lib.uris
>>
>> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>>  
>> 
>> 
>> This is what I have put in hdfs directory
>> 
>> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
>> /usr/lib/apache-tez-0.7.1-bin Found 2 items
>> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15 
>> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
>> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib
>> 
>> Also I put all /usr/lib/apache-tez-0.7.1-bin/lib/*.jar in 
>> /usr/lib/apache-tez-0.7.1-bin/lib
>> 
>> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
>> /usr/lib/apache-tez-0.7.1-bin/lib Found 22 items
>> -rw-r--r--   2 hduser supergroup 124846 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/RoaringBitmap-0.4.9.jar
>> -rw-r--r--   2 hduser supergroup  41123 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-cli-1.2.jar
>> -rw-r--r--   2 hduser supergroup  58160 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-codec-1.4.jar
>> -rw-r--r--   2 hduser supergroup 588337 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections-3.2.2.jar
>> -rw-r--r--   2 hduser supergroup 751238 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections4-4.1.jar
>> -rw-r--r--   2 hduser supergroup 185140 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-io-2.4.jar
>> -rw-r--r--   2 hduser supergroup 284220 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-lang-2.6.jar
>> -rw-r--r--   2 hduser supergroup1599627 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/commons-math3-3.1.1.jar
>> -rw-r--r--   2 hduser supergroup1648200 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/guava-11.0.2.jar
>> -rw-r--r--   2 hduser supergroup 664918 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-common-2.6.0.jar
>> -rw-r--r--   2 hduser supergroup1509399 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-core-2.6.0.jar
>> -rw-r--r--   2 hduser supergroup 130458 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-client-1.9.jar
>> -rw-r--r--   2 hduser supergroup 147952 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-json-1.9.jar
>> -rw-r--r--   2 hduser supergroup  81743 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/jettison-1.3.4.jar
>> -rw-r--r--   2 hduser supergroup 539912 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/jetty-6.1.26.jar
>> -rw-r--r--   2 hduser supergroup 177131 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/jetty-util-6.1.26.jar
>> -rw-r--r--   2 hduser supergroup  33031 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/jsr305-2.0.3.jar
>> -rw-r--r--   2 hduser supergroup 111908 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/metrics-core-3.1.0.jar
>> -rw-r--r--   2 hduser supergroup 533455 2016-05-20 23:27 
>> /usr/lib/apache-tez-0.7.1-bin/lib/protobuf-

Re: My first TEZ job fails

[Mich Talebzadeh]

Thanks both

so this is the file that needs to go in hdfs correct?

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> ltr tez-0.7.1.tar.gz
-*rw-r--r-- 1 hduser hadoop 39694439 May  4 18:47 tez-0.7.1.tar.gz*


In hdfs I have now

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin/share> hdfs dfs -ls
/usr/lib/apache-tez-0.7.1-bin

-rw-r--r--   2 hduser supergroup   39694439 2016-05-21 00:31
/usr/lib/apache-tez-0.7.1-bin/tez-0.7.1.tar.gz


Now I only installed tez under

/usr/lib/apache-tez-0.7.1-bin

MY Hadoop is installed in

echo $HADOOP_HOME
/home/hduser/hadoop-2.6.0

and my xml-site in in

$HADOOP_HOME/etc/hadoop

OK and this is my sml-site content

hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml

  
tez.version
0.7.1
  
  
tez.lib.uris

/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
  


Is the red correct please?



Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 21 May 2016 at 00:24, Bikas Saha  wrote:

> >> tez.lib.uris assumes that paths are based on the default fs and
> therefore if your setup is using HDFS as default, the paths /usr/lib would
> be invalid
>
> Are you sure? The below paths looks right to me except that the contents
> of the directories are wrong.
>
> tez.lib.uris
>
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
> /usr/lib/apache-tez-0.7.1-bin Found 2 items
> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15
> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib
>
>
> -Original Message-
> From: Hitesh Shah [mailto:hit...@apache.org]
> Sent: Friday, May 20, 2016 4:18 PM
> To: user@tez.apache.org
> Subject: Re: My first TEZ job fails
>
> Can you try the instructions mentioned at
> https://cwiki.apache.org/confluence/display/TEZ/Tez+Release+FAQ ?
>
> tez.lib.uris assumes that paths are based on the default fs and therefore
> if your setup is using HDFS as default, the paths /usr/lib would be invalid.
>
> — HItesh
>
> > On May 20, 2016, at 3:39 PM, Mich Talebzadeh 
> wrote:
> >
> > Still failing with /apache-tez-0.7.1-bin I am afraid.
> >
> > OK this is my tez-site.xml
> >
> > hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> > 
> >   
> > tez.version
> > 0.7.1
> >   
> >   
> > tez.lib.uris
> >
>  
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
> >   
> > 
> >
> > This is what I have put in hdfs directory
> >
> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
> > /usr/lib/apache-tez-0.7.1-bin Found 2 items
> > -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15
> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
> > drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib
> >
> > Also I put all /usr/lib/apache-tez-0.7.1-bin/lib/*.jar in
> > /usr/lib/apache-tez-0.7.1-bin/lib
> >
> > hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
> > /usr/lib/apache-tez-0.7.1-bin/lib Found 22 items
> > -rw-r--r--   2 hduser supergroup 124846 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/RoaringBitmap-0.4.9.jar
> > -rw-r--r--   2 hduser supergroup  41123 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-cli-1.2.jar
> > -rw-r--r--   2 hduser supergroup  58160 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-codec-1.4.jar
> > -rw-r--r--   2 hduser supergroup 588337 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections-3.2.2.jar
> > -rw-r--r--   2 hduser supergroup 751238 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections4-4.1.jar
> > -rw-r--r--   2 hduser supergroup 185140 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-io-2.4.jar
> > -rw-r--r--   2 hduser supergroup 284220 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-lang-2.6.jar
> > -rw-r--r--   2 hduser supergroup1599627 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-math3-3.1.1.jar
> > -rw-r--r--   2 hduser supergroup1648200 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/guava-11.0.2.jar
> > -rw-r--r--   2 hduser supergroup 664918 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-common-2.6.0.jar
> > -rw-r--r--   2 hduser supergroup1509399 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-core-2.6.0.jar
> > -rw-r--r--   2 hduser supergroup 130458 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-client-1.9.jar
> > -rw-r--r--   2 hduser supergroup 147952 2016-05-20 23:27
> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-json-1.9.jar
> > -rw-r--r--   2 

RE: My first TEZ job fails

[Bikas Saha]

>> tez.lib.uris assumes that paths are based on the default fs and therefore if 
>> your setup is using HDFS as default, the paths /usr/lib would be invalid

Are you sure? The below paths looks right to me except that the contents of the 
directories are wrong.

tez.lib.uris
/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
/usr/lib/apache-tez-0.7.1-bin Found 2 items
-rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15 
/usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27 
/usr/lib/apache-tez-0.7.1-bin/lib


-Original Message-
From: Hitesh Shah [mailto:hit...@apache.org] 
Sent: Friday, May 20, 2016 4:18 PM
To: user@tez.apache.org
Subject: Re: My first TEZ job fails

Can you try the instructions mentioned at 
https://cwiki.apache.org/confluence/display/TEZ/Tez+Release+FAQ ?

tez.lib.uris assumes that paths are based on the default fs and therefore if 
your setup is using HDFS as default, the paths /usr/lib would be invalid.

— HItesh

> On May 20, 2016, at 3:39 PM, Mich Talebzadeh  
> wrote:
> 
> Still failing with /apache-tez-0.7.1-bin I am afraid.
> 
> OK this is my tez-site.xml
> 
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml 
> 
>   
> tez.version
> 0.7.1
>   
>   
> tez.lib.uris
> 
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>   
> 
> 
> This is what I have put in hdfs directory
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin Found 2 items
> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15 
> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib
> 
> Also I put all /usr/lib/apache-tez-0.7.1-bin/lib/*.jar in 
> /usr/lib/apache-tez-0.7.1-bin/lib
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin/lib Found 22 items
> -rw-r--r--   2 hduser supergroup 124846 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/RoaringBitmap-0.4.9.jar
> -rw-r--r--   2 hduser supergroup  41123 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-cli-1.2.jar
> -rw-r--r--   2 hduser supergroup  58160 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-codec-1.4.jar
> -rw-r--r--   2 hduser supergroup 588337 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections-3.2.2.jar
> -rw-r--r--   2 hduser supergroup 751238 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections4-4.1.jar
> -rw-r--r--   2 hduser supergroup 185140 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-io-2.4.jar
> -rw-r--r--   2 hduser supergroup 284220 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-lang-2.6.jar
> -rw-r--r--   2 hduser supergroup1599627 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-math3-3.1.1.jar
> -rw-r--r--   2 hduser supergroup1648200 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/guava-11.0.2.jar
> -rw-r--r--   2 hduser supergroup 664918 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-common-2.6.0.jar
> -rw-r--r--   2 hduser supergroup1509399 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-core-2.6.0.jar
> -rw-r--r--   2 hduser supergroup 130458 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-client-1.9.jar
> -rw-r--r--   2 hduser supergroup 147952 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-json-1.9.jar
> -rw-r--r--   2 hduser supergroup  81743 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jettison-1.3.4.jar
> -rw-r--r--   2 hduser supergroup 539912 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jetty-6.1.26.jar
> -rw-r--r--   2 hduser supergroup 177131 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jetty-util-6.1.26.jar
> -rw-r--r--   2 hduser supergroup  33031 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jsr305-2.0.3.jar
> -rw-r--r--   2 hduser supergroup 111908 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/metrics-core-3.1.0.jar
> -rw-r--r--   2 hduser supergroup 533455 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/protobuf-java-2.5.0.jar
> -rw-r--r--   2 hduser supergroup 105112 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/servlet-api-2.5.jar
> -rw-r--r--   2 hduser supergroup  26084 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/slf4j-api-1.7.5.jar
> -rw-r--r--   2 hduser supergroup   8869 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/slf4j-log4j12-1.7.5.jar
> 
> Still getting this error
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hadoop jar 
> ./tez-examples-0.7.1.jar orderedwordcount /tmp/input/test.txt /tmp/out
> 
> Application application_146377570

RE: My first TEZ job fails

[Bikas Saha]

Please follow the instructions from Step 3 onwards at 
http://tez.apache.org/install.html since you already have a build.

 

Step 3 - The tarball to be uploaded to HDFS is at 
apache-tez-0.7.1-bin/share/tez-0.7.1.tar.gz

Step 5 - The result of minimal tarball extraction is what you already have on 
apache-tez-0.7.1-bin

 

@Hitesh, is there already a jira that tracks updating the install page above 
for the new binary release?

 

Thanks for your patience Mitch while we work this out!

Bikas

 

From: Mich Talebzadeh [mailto:mich.talebza...@gmail.com] 
Sent: Friday, May 20, 2016 3:05 PM
To: user@tez.apache.org
Subject: Re: My first TEZ job fails

 

Thanks Bikas,

 

Downloaded it and installed under /usr/lib/apache-tez-0.7.1-bin

 

Are these env variables correct?

 

export TEZ_HOME=/usr/lib/apache-tez-0.7.1-bin
export TEZ_JARS=/usr/lib/apache-tez-0.7.1-bin

 




Dr Mich Talebzadeh

 

LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com  

 

 

On 20 May 2016 at 22:43, Bikas Saha mailto:bi...@apache.org> 
> wrote:

Yes. Almost all Hadoop ecosystem components are on 2.5.1 for protoc.

 

Btw, unless you need something specific from 0.8 branch and are on the right 
Hadoop version 2.6.0 or higher, you could use the recent 0.7.1 binary release 
so you wont have to build the code yourself. May save you from these mismatch 
issues for the most part.

 

Bikas

 

From: Mich Talebzadeh [mailto:mich.talebza...@gmail.com 
 ] 
Sent: Friday, May 20, 2016 2:25 PM
To: user@tez.apache.org  
Subject: Re: My first TEZ job fails

 

the issue is that I have protobuf-2.6.1 and I believe I modified the pom.xml 
under apache-tez-0.8.3-src and changed it from 2.5.0 to 2.6.1.

 

Now I have to install protobuf-2.5.0 and It is not working.




Dr Mich Talebzadeh

 

LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com  

 

 

On 20 May 2016 at 21:46, Mich Talebzadeh mailto:mich.talebza...@gmail.com> > wrote:

sounds like this may ve the issue

 

java.lang.NoSuchMethodError: 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I




Dr Mich Talebzadeh

 

LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com  

 

 

On 20 May 2016 at 19:28, Mich Talebzadeh mailto:mich.talebza...@gmail.com> > wrote:

Hi Hitesh,

 

This is the content of tez-site.xml]

 

hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml

  
tez.version
0.8.3
  

  
tez.lib.uris
/usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/
  


 

 

 

This is what is in hdfs

 

drwxr-xr-x   - hduser supergroup  0 2016-05-20 19:25 
/usr/lib/tez-0.8.3/lib
-rw-r--r--   2 hduser supergroup   42821282 2016-05-20 09:03 
/usr/lib/tez-0.8.3/tez-0.8.3.tar.gz

 

 

Tez is installed in 

 

cd $TEZ_HOME
hduser@rhes564: /usr/lib/tez-0.8.3>

 

tez-site.xml is soft linked in $TEZ_HOME as follows

 

hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml
lrwxrwxrwx 1 hduser hadoop 49 May 20 19:22 tez-site.xml -> 
/home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml

 

This is error

 


Application application_1463758195355_0004 failed 2 times due to Error 
launching appattempt_1463758195355_0004_02. Got exception: 
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError): 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I

at 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)

at 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)

at 
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)

at 
org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)

at java.util.HashMap.hash(HashMap.java:338)

at java.util.HashMap.put(HashMap.java:611)

at java.util.HashSet.add(HashSet.java:219)

at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)

at javax.security.auth.Subject$ClassSet.(Subject.java:1369)

at javax.security.auth.Subject.getPublicCredentials(Subject.java:720)

at 
org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)

at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)

at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)

at 
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)

at

Re: My first TEZ job fails

[Hitesh Shah]

Can you try the instructions mentioned at 
https://cwiki.apache.org/confluence/display/TEZ/Tez+Release+FAQ ?

tez.lib.uris assumes that paths are based on the default fs and therefore if 
your setup is using HDFS as default, the paths /usr/lib would be invalid.

— HItesh

> On May 20, 2016, at 3:39 PM, Mich Talebzadeh  
> wrote:
> 
> Still failing with /apache-tez-0.7.1-bin I am afraid.
> 
> OK this is my tez-site.xml
> 
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> 
>   
> tez.version
> 0.7.1
>   
>   
> tez.lib.uris
> 
> /usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
>   
> 
> 
> This is what I have put in hdfs directory
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin
> Found 2 items
> -rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15 
> /usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
> drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib
> 
> Also I put all /usr/lib/apache-tez-0.7.1-bin/lib/*.jar in 
> /usr/lib/apache-tez-0.7.1-bin/lib
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls 
> /usr/lib/apache-tez-0.7.1-bin/lib
> Found 22 items
> -rw-r--r--   2 hduser supergroup 124846 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/RoaringBitmap-0.4.9.jar
> -rw-r--r--   2 hduser supergroup  41123 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-cli-1.2.jar
> -rw-r--r--   2 hduser supergroup  58160 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-codec-1.4.jar
> -rw-r--r--   2 hduser supergroup 588337 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections-3.2.2.jar
> -rw-r--r--   2 hduser supergroup 751238 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-collections4-4.1.jar
> -rw-r--r--   2 hduser supergroup 185140 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-io-2.4.jar
> -rw-r--r--   2 hduser supergroup 284220 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-lang-2.6.jar
> -rw-r--r--   2 hduser supergroup1599627 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/commons-math3-3.1.1.jar
> -rw-r--r--   2 hduser supergroup1648200 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/guava-11.0.2.jar
> -rw-r--r--   2 hduser supergroup 664918 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-common-2.6.0.jar
> -rw-r--r--   2 hduser supergroup1509399 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-core-2.6.0.jar
> -rw-r--r--   2 hduser supergroup 130458 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-client-1.9.jar
> -rw-r--r--   2 hduser supergroup 147952 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jersey-json-1.9.jar
> -rw-r--r--   2 hduser supergroup  81743 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jettison-1.3.4.jar
> -rw-r--r--   2 hduser supergroup 539912 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jetty-6.1.26.jar
> -rw-r--r--   2 hduser supergroup 177131 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jetty-util-6.1.26.jar
> -rw-r--r--   2 hduser supergroup  33031 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/jsr305-2.0.3.jar
> -rw-r--r--   2 hduser supergroup 111908 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/metrics-core-3.1.0.jar
> -rw-r--r--   2 hduser supergroup 533455 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/protobuf-java-2.5.0.jar
> -rw-r--r--   2 hduser supergroup 105112 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/servlet-api-2.5.jar
> -rw-r--r--   2 hduser supergroup  26084 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/slf4j-api-1.7.5.jar
> -rw-r--r--   2 hduser supergroup   8869 2016-05-20 23:27 
> /usr/lib/apache-tez-0.7.1-bin/lib/slf4j-log4j12-1.7.5.jar
> 
> Still getting this error
> 
> hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hadoop jar 
> ./tez-examples-0.7.1.jar orderedwordcount /tmp/input/test.txt /tmp/out
> 
> Application application_1463775706014_0008 failed 2 times due to Error 
> launching appattempt_1463775706014_0008_02. Got exception: 
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError): 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
> at 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
> at 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
> at 
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
> at 
> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
> at java.util.HashMap.hash(HashMap.java:338)
> 
> My protoc is 2.6.1 but everything else Hadoop, spark, hive, etx all work fine!
> 
> hduser@rhes564: /usr

Re: My first TEZ job fails

[Mich Talebzadeh]

Still failing with /apache-tez-0.7.1-bin I am afraid.

OK this is my tez-site.xml

hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml

  
tez.version
0.7.1
  
  
tez.lib.uris

/usr/lib/apache-tez-0.7.1-bin,/usr/lib/apache-tez-0.7.1-bin/lib
  


This is what I have put in hdfs directory

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
/usr/lib/apache-tez-0.7.1-bin
Found 2 items
-rw-r--r--   2 hduser supergroup   53092828 2016-05-20 23:15
/usr/lib/apache-tez-0.7.1-bin/apache-tez-0.7.1-bin.tar.gz
drwxr-xr-x   - hduser supergroup  0 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib

Also I put all /usr/lib/apache-tez-0.7.1-bin/lib/*.jar in
/usr/lib/apache-tez-0.7.1-bin/lib

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hdfs dfs -ls
/usr/lib/apache-tez-0.7.1-bin/lib
Found 22 items
-rw-r--r--   2 hduser supergroup 124846 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/RoaringBitmap-0.4.9.jar
-rw-r--r--   2 hduser supergroup  41123 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/commons-cli-1.2.jar
-rw-r--r--   2 hduser supergroup  58160 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/commons-codec-1.4.jar
-rw-r--r--   2 hduser supergroup 588337 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/commons-collections-3.2.2.jar
-rw-r--r--   2 hduser supergroup 751238 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/commons-collections4-4.1.jar
-rw-r--r--   2 hduser supergroup 185140 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/commons-io-2.4.jar
-rw-r--r--   2 hduser supergroup 284220 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/commons-lang-2.6.jar
-rw-r--r--   2 hduser supergroup1599627 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/commons-math3-3.1.1.jar
-rw-r--r--   2 hduser supergroup1648200 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/guava-11.0.2.jar
-rw-r--r--   2 hduser supergroup 664918 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-common-2.6.0.jar
-rw-r--r--   2 hduser supergroup1509399 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/hadoop-mapreduce-client-core-2.6.0.jar
-rw-r--r--   2 hduser supergroup 130458 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/jersey-client-1.9.jar
-rw-r--r--   2 hduser supergroup 147952 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/jersey-json-1.9.jar
-rw-r--r--   2 hduser supergroup  81743 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/jettison-1.3.4.jar
-rw-r--r--   2 hduser supergroup 539912 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/jetty-6.1.26.jar
-rw-r--r--   2 hduser supergroup 177131 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/jetty-util-6.1.26.jar
-rw-r--r--   2 hduser supergroup  33031 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/jsr305-2.0.3.jar
-rw-r--r--   2 hduser supergroup 111908 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/metrics-core-3.1.0.jar
-rw-r--r--   2 hduser supergroup 533455 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/protobuf-java-2.5.0.jar
-rw-r--r--   2 hduser supergroup 105112 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/servlet-api-2.5.jar
-rw-r--r--   2 hduser supergroup  26084 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/slf4j-api-1.7.5.jar
-rw-r--r--   2 hduser supergroup   8869 2016-05-20 23:27
/usr/lib/apache-tez-0.7.1-bin/lib/slf4j-log4j12-1.7.5.jar

Still getting this error

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> hadoop jar
./tez-examples-0.7.1.jar orderedwordcount /tmp/input/test.txt /tmp/out

Application application_1463775706014_0008 failed 2 times due to Error
launching appattempt_1463775706014_0008_02. Got exception:
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
at
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
at
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
at
org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
at java.util.HashMap.hash(HashMap.java:338)

My protoc is 2.6.1 but everything else Hadoop, spark, hive, etx all
work fine!

hduser@rhes564: /usr/lib/apache-tez-0.7.1-bin> protoc --version
libprotoc 2.6.1

Thanks

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 20 May 2016 at 23:04, Mich Talebzadeh  wrote:

> Thanks Bikas,
>
> Downloaded it and installed under /usr/lib/apache-tez-0.7.1-bin
>
> Are these env variables correct?
>
> export TEZ_HOME=/usr/lib/apache-tez-0.7.1-bin
> export TEZ_JARS=/usr/lib/apache-tez-0.7.

Re: My first TEZ job fails

[Mich Talebzadeh]

Thanks Bikas,

Downloaded it and installed under /usr/lib/apache-tez-0.7.1-bin

Are these env variables correct?

export TEZ_HOME=/usr/lib/apache-tez-0.7.1-bin
export TEZ_JARS=/usr/lib/apache-tez-0.7.1-bin


Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 20 May 2016 at 22:43, Bikas Saha  wrote:

> Yes. Almost all Hadoop ecosystem components are on 2.5.1 for protoc.
>
>
>
> Btw, unless you need something specific from 0.8 branch and are on the
> right Hadoop version 2.6.0 or higher, you could use the recent 0.7.1 binary
> release so you wont have to build the code yourself. May save you from
> these mismatch issues for the most part.
>
>
>
> Bikas
>
>
>
> *From:* Mich Talebzadeh [mailto:mich.talebza...@gmail.com]
> *Sent:* Friday, May 20, 2016 2:25 PM
> *To:* user@tez.apache.org
> *Subject:* Re: My first TEZ job fails
>
>
>
> the issue is that I have protobuf-2.6.1 and I believe I modified the
> pom.xml under apache-tez-0.8.3-src and changed it from 2.5.0 to 2.6.1.
>
>
>
> Now I have to install protobuf-2.5.0 and It is not working.
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn  
> *https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
>
>
> On 20 May 2016 at 21:46, Mich Talebzadeh 
> wrote:
>
> sounds like this may ve the issue
>
>
>
> java.lang.NoSuchMethodError: org.apache.hadoop.*yarn.proto.YarnProtos$*
> ApplicationIdProto.hashLong(J)I
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn  
> *https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
>
>
> On 20 May 2016 at 19:28, Mich Talebzadeh 
> wrote:
>
> Hi Hitesh,
>
>
>
> This is the content of tez-site.xml]
>
>
>
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> 
>   
> tez.version
> 0.8.3
>   
>
>   
> tez.lib.uris
> /usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/
>   
> 
>
>
>
>
>
>
>
> This is what is in hdfs
>
>
>
> drwxr-xr-x   - hduser supergroup  0 2016-05-20 19:25
> /usr/lib/tez-0.8.3/lib
> -rw-r--r--   2 hduser supergroup   42821282 2016-05-20 09:03
> /usr/lib/tez-0.8.3/tez-0.8.3.tar.gz
>
>
>
>
>
> Tez is installed in
>
>
>
> cd $TEZ_HOME
> hduser@rhes564: /usr/lib/tez-0.8.3>
>
>
>
> tez-site.xml is soft linked in $TEZ_HOME as follows
>
>
>
> hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml
> lrwxrwxrwx 1 hduser hadoop 49 May 20 19:22 tez-site.xml ->
> /home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml
>
>
>
> This is error
>
>
>
> Application application_1463758195355_0004 failed 2 times due to Error
> launching appattempt_1463758195355_0004_02. Got exception:
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
>
> at
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
>
> at
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
>
> at
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
>
> at
> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
>
> at java.util.HashMap.hash(HashMap.java:338)
>
> at java.util.HashMap.put(HashMap.java:611)
>
> at java.util.HashSet.add(HashSet.java:219)
>
> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
>
> at javax.security.auth.Subject$ClassSet.(Subject.java:1369)
>
> at javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
>
> at
> org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
>
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
>
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
>
> at
> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>
> at java.security.AccessCon

RE: My first TEZ job fails

[Bikas Saha]

Yes. Almost all Hadoop ecosystem components are on 2.5.1 for protoc.

 

Btw, unless you need something specific from 0.8 branch and are on the right 
Hadoop version 2.6.0 or higher, you could use the recent 0.7.1 binary release 
so you wont have to build the code yourself. May save you from these mismatch 
issues for the most part.

 

Bikas

 

From: Mich Talebzadeh [mailto:mich.talebza...@gmail.com] 
Sent: Friday, May 20, 2016 2:25 PM
To: user@tez.apache.org
Subject: Re: My first TEZ job fails

 

the issue is that I have protobuf-2.6.1 and I believe I modified the pom.xml 
under apache-tez-0.8.3-src and changed it from 2.5.0 to 2.6.1.

 

Now I have to install protobuf-2.5.0 and It is not working.




Dr Mich Talebzadeh

 

LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com  

 

 

On 20 May 2016 at 21:46, Mich Talebzadeh mailto:mich.talebza...@gmail.com> > wrote:

sounds like this may ve the issue

 

java.lang.NoSuchMethodError: 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I




Dr Mich Talebzadeh

 

LinkedIn  
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

 

http://talebzadehmich.wordpress.com  

 

 

On 20 May 2016 at 19:28, Mich Talebzadeh mailto:mich.talebza...@gmail.com> > wrote:

Hi Hitesh,

 

This is the content of tez-site.xml]

 

hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml

  
tez.version
0.8.3
  

  
tez.lib.uris
/usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/
  


 

 

 

This is what is in hdfs

 

drwxr-xr-x   - hduser supergroup  0 2016-05-20 19:25 
/usr/lib/tez-0.8.3/lib
-rw-r--r--   2 hduser supergroup   42821282 2016-05-20 09:03 
/usr/lib/tez-0.8.3/tez-0.8.3.tar.gz

 

 

Tez is installed in 

 

cd $TEZ_HOME
hduser@rhes564: /usr/lib/tez-0.8.3>

 

tez-site.xml is soft linked in $TEZ_HOME as follows

 

hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml
lrwxrwxrwx 1 hduser hadoop 49 May 20 19:22 tez-site.xml -> 
/home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml

 

This is error

 


Application application_1463758195355_0004 failed 2 times due to Error 
launching appattempt_1463758195355_0004_02. Got exception: 
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError): 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I

at 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)

at 
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)

at 
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)

at 
org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)

at java.util.HashMap.hash(HashMap.java:338)

at java.util.HashMap.put(HashMap.java:611)

at java.util.HashSet.add(HashSet.java:219)

at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)

at javax.security.auth.Subject$ClassSet.(Subject.java:1369)

at javax.security.auth.Subject.getPublicCredentials(Subject.java:720)

at 
org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)

at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)

at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)

at 
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)

at 
org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)

at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)

at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)

at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

at org.apache.hadoop.ipc.Client.call(Client.java:1468)

at org.apache.hadoop.ipc.Client.call(Client.java:1399)

at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)

at com.sun.proxy.$Proxy80.startContainers(Unknown Source)

at 
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)

at 
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:11

Re: My first TEZ job fails

[Mich Talebzadeh]

the issue is that I have protobuf-2.6.1 and I believe I modified the
pom.xml under apache-tez-0.8.3-src and changed it from 2.5.0 to 2.6.1.

Now I have to install protobuf-2.5.0 and It is not working.

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 20 May 2016 at 21:46, Mich Talebzadeh  wrote:

> sounds like this may ve the issue
>
> java.lang.NoSuchMethodError: org.apache.hadoop.*yarn.proto.YarnProtos$*
> ApplicationIdProto.hashLong(J)I
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 20 May 2016 at 19:28, Mich Talebzadeh 
> wrote:
>
>> Hi Hitesh,
>>
>> This is the content of tez-site.xml]
>>
>> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
>> 
>>   
>> tez.version
>> 0.8.3
>>   
>>   
>> tez.lib.uris
>> /usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/
>>   
>> 
>>
>>
>>
>> This is what is in hdfs
>>
>> drwxr-xr-x   - hduser supergroup  0 2016-05-20 19:25
>> /usr/lib/tez-0.8.3/lib
>> -rw-r--r--   2 hduser supergroup   42821282 2016-05-20 09:03
>> /usr/lib/tez-0.8.3/tez-0.8.3.tar.gz
>>
>>
>> Tez is installed in
>>
>> cd $TEZ_HOME
>> hduser@rhes564: /usr/lib/tez-0.8.3>
>>
>> tez-site.xml is soft linked in $TEZ_HOME as follows
>>
>> hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml
>> lrwxrwxrwx 1 hduser hadoop 49 May 20 19:22 tez-site.xml ->
>> /home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml
>>
>> This is error
>>
>> Application application_1463758195355_0004 failed 2 times due to Error
>> launching appattempt_1463758195355_0004_02. Got exception:
>> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
>> at
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
>> at
>> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
>> at
>> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
>> at
>> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
>> at java.util.HashMap.hash(HashMap.java:338)
>> at java.util.HashMap.put(HashMap.java:611)
>> at java.util.HashSet.add(HashSet.java:219)
>> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
>> at javax.security.auth.Subject$ClassSet.(Subject.java:1369)
>> at javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
>> at
>> org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
>> at
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
>> at
>> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1468)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
>> at
>> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
>> at com.sun.proxy.$Proxy80.startContainers(Unknown Source)
>> at
>> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>> at
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119)
>> at
>> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:254)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.jav

Re: My first TEZ job fails

[Mich Talebzadeh]

sounds like this may ve the issue

java.lang.NoSuchMethodError: org.apache.hadoop.*yarn.proto.YarnProtos$*
ApplicationIdProto.hashLong(J)I

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 20 May 2016 at 19:28, Mich Talebzadeh  wrote:

> Hi Hitesh,
>
> This is the content of tez-site.xml]
>
> hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml
> 
>   
> tez.version
> 0.8.3
>   
>   
> tez.lib.uris
> /usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/
>   
> 
>
>
>
> This is what is in hdfs
>
> drwxr-xr-x   - hduser supergroup  0 2016-05-20 19:25
> /usr/lib/tez-0.8.3/lib
> -rw-r--r--   2 hduser supergroup   42821282 2016-05-20 09:03
> /usr/lib/tez-0.8.3/tez-0.8.3.tar.gz
>
>
> Tez is installed in
>
> cd $TEZ_HOME
> hduser@rhes564: /usr/lib/tez-0.8.3>
>
> tez-site.xml is soft linked in $TEZ_HOME as follows
>
> hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml
> lrwxrwxrwx 1 hduser hadoop 49 May 20 19:22 tez-site.xml ->
> /home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml
>
> This is error
>
> Application application_1463758195355_0004 failed 2 times due to Error
> launching appattempt_1463758195355_0004_02. Got exception:
> org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
> at
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
> at
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
> at
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
> at
> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
> at javax.security.auth.Subject$ClassSet.(Subject.java:1369)
> at javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
> at
> org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
> at
> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> at org.apache.hadoop.ipc.Client.call(Client.java:1468)
> at org.apache.hadoop.ipc.Client.call(Client.java:1399)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy80.startContainers(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
> at
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119)
> at
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:254)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> . Failing the application.
> Thanks
>
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * 
> https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> *
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 20 May 2016 at 19:06, Hitesh Shah  wrote:
>
>> Logs from `bin/yarn logs -applicationId application_1463758195355_0002`
>> would be more useful to debug your setup issue. The RM logs usually do not
>> shed much light on why an application failed.
>> Can you confirm that you configured tez.lib.uris 

Re: My first TEZ job fails

[Mich Talebzadeh]

Hi Hitesh,

This is the content of tez-site.xml]

hduser@rhes564: /home/hduser/hadoop-2.6.0/etc/hadoop> cat tez-site.xml

  
tez.version
0.8.3
  
  
tez.lib.uris
/usr/lib/tez-0.8.3,/usr/lib/tez-0.8.3/lib/
  




This is what is in hdfs

drwxr-xr-x   - hduser supergroup  0 2016-05-20 19:25
/usr/lib/tez-0.8.3/lib
-rw-r--r--   2 hduser supergroup   42821282 2016-05-20 09:03
/usr/lib/tez-0.8.3/tez-0.8.3.tar.gz


Tez is installed in

cd $TEZ_HOME
hduser@rhes564: /usr/lib/tez-0.8.3>

tez-site.xml is soft linked in $TEZ_HOME as follows

hduser@rhes564: /usr/lib/tez-0.8.3> ls -l tez-site.xml
lrwxrwxrwx 1 hduser hadoop 49 May 20 19:22 tez-site.xml ->
/home/hduser/hadoop-2.6.0/etc/hadoop/tez-site.xml

This is error

Application application_1463758195355_0004 failed 2 times due to Error
launching appattempt_1463758195355_0004_02. Got exception:
org.apache.hadoop.ipc.RemoteException(java.lang.NoSuchMethodError):
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
at
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
at
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
at
org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
at java.util.HashMap.hash(HashMap.java:338)
at java.util.HashMap.put(HashMap.java:611)
at java.util.HashSet.add(HashSet.java:219)
at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
at javax.security.auth.Subject$ClassSet.(Subject.java:1369)
at javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
at
org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
at org.apache.hadoop.ipc.Client.call(Client.java:1468)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy80.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
at
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119)
at
org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:254)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
. Failing the application.
Thanks





Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com



On 20 May 2016 at 19:06, Hitesh Shah  wrote:

> Logs from `bin/yarn logs -applicationId application_1463758195355_0002`
> would be more useful to debug your setup issue. The RM logs usually do not
> shed much light on why an application failed.
> Can you confirm that you configured tez.lib.uris correctly to point to the
> tez tarball on HDFS (tez tar should be the one obtained from
> tez-dist/target/tez-0.8.3.tar.gz) ?
>
> — Hitesh
>
> > On May 20, 2016, at 10:24 AM, Mich Talebzadeh 
> wrote:
> >
> > Hi,
> >
> > I have just compiled and installed TEZ, trying to do a test with
> >
> > hadoop jar ./tez-examples-0.8.3.jar orderedwordcount /tmp/input/test.txt
> /tmp/out
> >
> > The job fails as follows. This is from yarn log
> >
> > 2016-05-20 18:19:26,945 INFO
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> appattempt_1463758195355_0002_01 (auth:SIMPLE)
> > 2016-05-20 18:19:26,950 WARN org.apache.had

Re: My first TEZ job fails

[Hitesh Shah]

Logs from `bin/yarn logs -applicationId application_1463758195355_0002` would 
be more useful to debug your setup issue. The RM logs usually do not shed much 
light on why an application failed.
Can you confirm that you configured tez.lib.uris correctly to point to the tez 
tarball on HDFS (tez tar should be the one obtained from 
tez-dist/target/tez-0.8.3.tar.gz) ? 

— Hitesh

> On May 20, 2016, at 10:24 AM, Mich Talebzadeh  
> wrote:
> 
> Hi,
> 
> I have just compiled and installed TEZ, trying to do a test with
> 
> hadoop jar ./tez-examples-0.8.3.jar orderedwordcount /tmp/input/test.txt 
> /tmp/out
> 
> The job fails as follows. This is from yarn log
> 
> 2016-05-20 18:19:26,945 INFO SecurityLogger.org.apache.hadoop.ipc.Server: 
> Auth successful for appattempt_1463758195355_0002_01 (auth:SIMPLE)
> 2016-05-20 18:19:26,950 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 0 on 59093, call 
> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContainers from 
> 50.140.197.217:46784 Call#2 Retry#0
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
> at 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
> at 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
> at 
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
> at 
> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
> at javax.security.auth.Subject$ClassSet.(Subject.java:1369)
> at javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
> at 
> org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
> at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
> at 
> org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
> at 
> org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> 2016-05-20 18:19:27,929 WARN 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
>  Event EventType: KILL_CONTAINER sent to absent container 
> container_1463758195355_0002_01_01
> 2016-05-20 18:19:27,944 INFO SecurityLogger.org.apache.hadoop.ipc.Server: 
> Auth successful for appattempt_1463758195355_0002_02 (auth:SIMPLE)
> 2016-05-20 18:19:27,949 WARN org.apache.hadoop.ipc.Server: IPC Server handler 
> 0 on 59093, call 
> org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContainers from 
> 50.140.197.217:46785 Call#3 Retry#0
> java.lang.NoSuchMethodError: 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
> at 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
> at 
> org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
> at 
> org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
> at 
> org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
> at java.util.HashMap.hash(HashMap.java:338)
> at java.util.HashMap.put(HashMap.java:611)
> at java.util.HashSet.add(HashSet.java:219)
> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
> at javax.security.auth.Subject$ClassSet.(Subject.java:1369)
> at javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
> at 
> org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGr

My first TEZ job fails

[Mich Talebzadeh]

Hi,

I have just compiled and installed TEZ, trying to do a test with

*hadoop jar ./tez-examples-0.8.3.jar orderedwordcount /tmp/input/test.txt
/tmp/out*

The job fails as follows. This is from yarn log

2016-05-20 18:19:26,945 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
Auth successful for appattempt_1463758195355_0002_01 (auth:SIMPLE)
2016-05-20 18:19:26,950 WARN org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 59093, call
org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContainers
from 50.140.197.217:46784 Call#2 Retry#0
java.lang.NoSuchMethodError:
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
at
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
at
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
at
org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
at java.util.HashMap.hash(HashMap.java:338)
at java.util.HashMap.put(HashMap.java:611)
at java.util.HashSet.add(HashSet.java:219)
at
javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
at javax.security.auth.Subject$ClassSet.(Subject.java:1369)
at
javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
at
org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:95)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
2016-05-20 18:19:27,929 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl:
Event EventType: KILL_CONTAINER sent to absent container
container_1463758195355_0002_01_01
2016-05-20 18:19:27,944 INFO SecurityLogger.org.apache.hadoop.ipc.Server:
Auth successful for appattempt_1463758195355_0002_02 (auth:SIMPLE)
2016-05-20 18:19:27,949 WARN org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 59093, call
org.apache.hadoop.yarn.api.ContainerManagementProtocolPB.startContainers
from 50.140.197.217:46785 Call#3 Retry#0
java.lang.NoSuchMethodError:
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashLong(J)I
at
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationIdProto.hashCode(YarnProtos.java:2616)
at
org.apache.hadoop.yarn.proto.YarnProtos$ApplicationAttemptIdProto.hashCode(YarnProtos.java:3154)
at
org.apache.hadoop.yarn.proto.YarnSecurityTokenProtos$NMTokenIdentifierProto.hashCode(YarnSecurityTokenProtos.java:410)
at
org.apache.hadoop.yarn.security.NMTokenIdentifier.hashCode(NMTokenIdentifier.java:126)
at java.util.HashMap.hash(HashMap.java:338)
at java.util.HashMap.put(HashMap.java:611)
at java.util.HashSet.add(HashSet.java:219)
at
javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1409)
at javax.security.auth.Subject$ClassSet.(Subject.java:1369)
at
javax.security.auth.Subject.getPublicCredentials(Subject.java:720)
at
org.apache.hadoop.security.UserGroupInformation.getTokenIdentifiers(UserGroupInformation.java:1400)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.selectNMTokenIdentifier(ContainerManagerImpl.java:618)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.startContainers(ContainerManagerImpl.java:699)
at
org.apache.hadoop.yarn.api.impl.pb.service.ContainerManagementProtocolPBServiceImpl.startContainers(ContainerManagementProtocolPBServiceImpl.java:60)
at
org.apache.hadoop.yarn.proto.ContainerManagementProtocol$ContainerManagementProtocolService$2.callBlockingMethod(ContainerManagementProtocol.java:9