Re: Running Tez with Tachyon
mpl.VertexImpl.handle(VertexImpl.java:1975) > > at > org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203) > > at > org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2096) > > at > org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2082) > > at > org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183) > > at > org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114) > > at java.lang.Thread.run(Thread.java:745) > > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > Class tachyon.hadoop.TFS not found > > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195) > > at > org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638) > > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) > > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) > > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) > > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) > > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:105) > > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:80) > > at > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:309) > > at > org.apache.tez.mapreduce.committer.MROutputCommitter.getOutputCommitter(MROutputCommitter.java:137) > > ... 24 more > > Caused by: java.lang.ClassNotFoundException: Class tachyon.hadoop.TFS > not found > > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) > > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) > > ... 35 more > > , Vertex vertex_1447296197811_0003_1_02 [Sorter] killed/failed due > to:INIT_FAILURE], Vertex killed, vertexName=Summation, > vertexId=vertex_1447296197811_0003_1_01, diagnostics=[Vertex received Kill > in INITED state., Vertex vertex_1447296197811_0003_1_01 [Summation] > killed/failed due to:OTHER_VERTEX_FAILURE], DAG did not succeed due to > VERTEX_FAILURE. failedVertices:2 killedVertices:1] > > > > Best, > > > > -- > > Jiří Šimša > > > > On Thu, Nov 12, 2015 at 8:52 AM, Hitesh Shah wrote: > > The general approach for add-on jars requires 2 steps: > > > > 1) On the client host, where the job is submitted, you need to ensure > that the add-on jars are in the local classpath. This is usually done by > adding them to HADOOP_CLASSPATH. Please do pay attention to adding the jars > via "/*” instead of just "” > > 2) Next, "tez.aux.uris”. This controls additional files/jars needed in > the runtime on the cluster. Upload the tachyon jar to HDFS and ensure that > you provide the path to either the dir on HDFS or the full path to the file > and specify that in tez.aux.uris. > > > > The last thing to note is that you may need to pull additional > transitive dependencies of tachyon if it is not self-contained jar. > > > > thanks > > — HItesh > > > > On Nov 12, 2015, at 1:06 AM, Bikas Saha wrote: > > > > > Can you provide the full stack trace? > > > > > > Are you getting the exception on the client (while submitting the job) > or in the cluster (after the job started to run)? > > > > > > For the client side, the fix would be to add tachyon jars to the > client classpath. Looks like you tried some client side classpath fixes. > You could run ‘hadoop classpath’ to print the classpath being picked up by > the ‘hadoop jar’ command. And scan its output to check if your tachyon jars > are being picked up correctly or not. > > > > > > Bikas > > > > > > From: Jiří Šimša [mailto:jiri.si...@gmail.com] > > > Sent: Wednesday, November 11, 2015 6:54 PM > > > To: user@tez.apache.org > > > Subject: Running Tez with Tachyon > > > > > > Hello, > > > > > > I have followed the Tez installation instructions ( > https://tez.apache.org/install.html) and was able to successfully run the > ordered word count example: > > > > > > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar > orderedwordcount /input.txt /output.txt > > > > > > Next, I wanted to see if I can do the same, this time reading from and > writing to Tachyon (http://tachyon-project.org/) using: > > > > > > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar > orderedwordcount tachyon://localhost:19998/input.txt > tachyon://localhost:19998/output.txt > > > > > > Unsurprisingly, this resulted in the "Class tachyon.hadoop.TFS not > found" error because Tez needs the Tachyon client jar that defines the > tachyon.hadoop.TFS class. To that end, I have tried several options (listed > below) to provide this jar to Tez, none of which seems to have worked: > > > > > > 1) Adding the Tachyon client jar to HADOOP_CLASSPATH > > > 2) Specifying the Tachyon client jar with the -libjars flag for the > above command. > > > 3) Copying the Tachyon client jar into the > $HADOOP_HOME/share/hadoop/common/lib directory of my HADOOP installation. > > > 4) Copying the Tachyon client jar into HDFS and specifying a path to > it through the tez.aux.uris property in the tez-site.xml file (in a similar > fashion the tez.lib.uris property specifies the path to the Tez tarball). > > > 5) I modified the source code of the ordered word count example, > adding a call to TezClient#addAppMasterLocalFiles(...), providing a URI for > the Tachyon client jar uploaded to HDFS. > > > > > > Any advice on how to pass the Tachyon client jar to Tez to resolve > this issue would be greatly appreciated. Thank you. > > > > > > Best, > > > > > > -- > > > Jiří Šimša > > > > > > > >
Re: Running Tez with Tachyon
at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:105) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:80) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.getOutputCommitter(FileOutputFormat.java:309) > at > org.apache.tez.mapreduce.committer.MROutputCommitter.getOutputCommitter(MROutputCommitter.java:137) > ... 24 more > Caused by: java.lang.ClassNotFoundException: Class tachyon.hadoop.TFS not > found > at > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193) > ... 35 more > , Vertex vertex_1447296197811_0003_1_02 [Sorter] killed/failed due > to:INIT_FAILURE], Vertex killed, vertexName=Summation, > vertexId=vertex_1447296197811_0003_1_01, diagnostics=[Vertex received Kill in > INITED state., Vertex vertex_1447296197811_0003_1_01 [Summation] > killed/failed due to:OTHER_VERTEX_FAILURE], DAG did not succeed due to > VERTEX_FAILURE. failedVertices:2 killedVertices:1] > > Best, > > -- > Jiří Šimša > > On Thu, Nov 12, 2015 at 8:52 AM, Hitesh Shah wrote: > The general approach for add-on jars requires 2 steps: > > 1) On the client host, where the job is submitted, you need to ensure that > the add-on jars are in the local classpath. This is usually done by adding > them to HADOOP_CLASSPATH. Please do pay attention to adding the jars via > "/*” instead of just "” > 2) Next, "tez.aux.uris”. This controls additional files/jars needed in the > runtime on the cluster. Upload the tachyon jar to HDFS and ensure that you > provide the path to either the dir on HDFS or the full path to the file and > specify that in tez.aux.uris. > > The last thing to note is that you may need to pull additional transitive > dependencies of tachyon if it is not self-contained jar. > > thanks > — HItesh > > On Nov 12, 2015, at 1:06 AM, Bikas Saha wrote: > > > Can you provide the full stack trace? > > > > Are you getting the exception on the client (while submitting the job) or > > in the cluster (after the job started to run)? > > > > For the client side, the fix would be to add tachyon jars to the client > > classpath. Looks like you tried some client side classpath fixes. You could > > run ‘hadoop classpath’ to print the classpath being picked up by the > > ‘hadoop jar’ command. And scan its output to check if your tachyon jars are > > being picked up correctly or not. > > > > Bikas > > > > From: Jiří Šimša [mailto:jiri.si...@gmail.com] > > Sent: Wednesday, November 11, 2015 6:54 PM > > To: user@tez.apache.org > > Subject: Running Tez with Tachyon > > > > Hello, > > > > I have followed the Tez installation instructions > > (https://tez.apache.org/install.html) and was able to successfully run the > > ordered word count example: > > > > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar > > orderedwordcount /input.txt /output.txt > > > > Next, I wanted to see if I can do the same, this time reading from and > > writing to Tachyon (http://tachyon-project.org/) using: > > > > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar > > orderedwordcount tachyon://localhost:19998/input.txt > > tachyon://localhost:19998/output.txt > > > > Unsurprisingly, this resulted in the "Class tachyon.hadoop.TFS not found" > > error because Tez needs the Tachyon client jar that defines the > > tachyon.hadoop.TFS class. To that end, I have tried several options (listed > > below) to provide this jar to Tez, none of which seems to have worked: > > > > 1) Adding the Tachyon client jar to HADOOP_CLASSPATH > > 2) Specifying the Tachyon client jar with the -libjars flag for the above > > command. > > 3) Copying the Tachyon client jar into the > > $HADOOP_HOME/share/hadoop/common/lib directory of my HADOOP installation. > > 4) Copying the Tachyon client jar into HDFS and specifying a path to it > > through the tez.aux.uris property in the tez-site.xml file (in a similar > > fashion the tez.lib.uris property specifies the path to the Tez tarball). > > 5) I modified the source code of the ordered word count example, adding a > > call to TezClient#addAppMasterLocalFiles(...), providing a URI for the > > Tachyon client jar uploaded to HDFS. > > > > Any advice on how to pass the Tachyon client jar to Tez to resolve this > > issue would be greatly appreciated. Thank you. > > > > Best, > > > > -- > > Jiří Šimša > > >
Re: Running Tez with Tachyon
- > Jiří Šimša > > On Thu, Nov 12, 2015 at 8:52 AM, Hitesh Shah wrote: > >> The general approach for add-on jars requires 2 steps: >> >> 1) On the client host, where the job is submitted, you need to ensure >> that the add-on jars are in the local classpath. This is usually done by >> adding them to HADOOP_CLASSPATH. Please do pay attention to adding the jars >> via "/*” instead of just "” >> 2) Next, "tez.aux.uris”. This controls additional files/jars needed in >> the runtime on the cluster. Upload the tachyon jar to HDFS and ensure that >> you provide the path to either the dir on HDFS or the full path to the file >> and specify that in tez.aux.uris. >> >> The last thing to note is that you may need to pull additional transitive >> dependencies of tachyon if it is not self-contained jar. >> >> thanks >> — HItesh >> >> On Nov 12, 2015, at 1:06 AM, Bikas Saha wrote: >> >> > Can you provide the full stack trace? >> > >> > Are you getting the exception on the client (while submitting the job) >> or in the cluster (after the job started to run)? >> > >> > For the client side, the fix would be to add tachyon jars to the client >> classpath. Looks like you tried some client side classpath fixes. You could >> run ‘hadoop classpath’ to print the classpath being picked up by the >> ‘hadoop jar’ command. And scan its output to check if your tachyon jars are >> being picked up correctly or not. >> > >> > Bikas >> > >> > From: Jiří Šimša [mailto:jiri.si...@gmail.com] >> > Sent: Wednesday, November 11, 2015 6:54 PM >> > To: user@tez.apache.org >> > Subject: Running Tez with Tachyon >> > >> > Hello, >> > >> > I have followed the Tez installation instructions ( >> https://tez.apache.org/install.html) and was able to successfully run >> the ordered word count example: >> > >> > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar >> orderedwordcount /input.txt /output.txt >> > >> > Next, I wanted to see if I can do the same, this time reading from and >> writing to Tachyon (http://tachyon-project.org/) using: >> > >> > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar >> orderedwordcount tachyon://localhost:19998/input.txt >> tachyon://localhost:19998/output.txt >> > >> > Unsurprisingly, this resulted in the "Class tachyon.hadoop.TFS not >> found" error because Tez needs the Tachyon client jar that defines the >> tachyon.hadoop.TFS class. To that end, I have tried several options (listed >> below) to provide this jar to Tez, none of which seems to have worked: >> > >> > 1) Adding the Tachyon client jar to HADOOP_CLASSPATH >> > 2) Specifying the Tachyon client jar with the -libjars flag for the >> above command. >> > 3) Copying the Tachyon client jar into the >> $HADOOP_HOME/share/hadoop/common/lib directory of my HADOOP installation. >> > 4) Copying the Tachyon client jar into HDFS and specifying a path to it >> through the tez.aux.uris property in the tez-site.xml file (in a similar >> fashion the tez.lib.uris property specifies the path to the Tez tarball). >> > 5) I modified the source code of the ordered word count example, adding >> a call to TezClient#addAppMasterLocalFiles(...), providing a URI for the >> Tachyon client jar uploaded to HDFS. >> > >> > Any advice on how to pass the Tachyon client jar to Tez to resolve this >> issue would be greatly appreciated. Thank you. >> > >> > Best, >> > >> > -- >> > Jiří Šimša >> >> >
Re: Running Tez with Tachyon
p correctly or not. > > > > Bikas > > > > From: Jiří Šimša [mailto:jiri.si...@gmail.com] > > Sent: Wednesday, November 11, 2015 6:54 PM > > To: user@tez.apache.org > > Subject: Running Tez with Tachyon > > > > Hello, > > > > I have followed the Tez installation instructions ( > https://tez.apache.org/install.html) and was able to successfully run the > ordered word count example: > > > > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar > orderedwordcount /input.txt /output.txt > > > > Next, I wanted to see if I can do the same, this time reading from and > writing to Tachyon (http://tachyon-project.org/) using: > > > > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar > orderedwordcount tachyon://localhost:19998/input.txt > tachyon://localhost:19998/output.txt > > > > Unsurprisingly, this resulted in the "Class tachyon.hadoop.TFS not > found" error because Tez needs the Tachyon client jar that defines the > tachyon.hadoop.TFS class. To that end, I have tried several options (listed > below) to provide this jar to Tez, none of which seems to have worked: > > > > 1) Adding the Tachyon client jar to HADOOP_CLASSPATH > > 2) Specifying the Tachyon client jar with the -libjars flag for the > above command. > > 3) Copying the Tachyon client jar into the > $HADOOP_HOME/share/hadoop/common/lib directory of my HADOOP installation. > > 4) Copying the Tachyon client jar into HDFS and specifying a path to it > through the tez.aux.uris property in the tez-site.xml file (in a similar > fashion the tez.lib.uris property specifies the path to the Tez tarball). > > 5) I modified the source code of the ordered word count example, adding > a call to TezClient#addAppMasterLocalFiles(...), providing a URI for the > Tachyon client jar uploaded to HDFS. > > > > Any advice on how to pass the Tachyon client jar to Tez to resolve this > issue would be greatly appreciated. Thank you. > > > > Best, > > > > -- > > Jiří Šimša > >
Re: Running Tez with Tachyon
The general approach for add-on jars requires 2 steps: 1) On the client host, where the job is submitted, you need to ensure that the add-on jars are in the local classpath. This is usually done by adding them to HADOOP_CLASSPATH. Please do pay attention to adding the jars via "/*” instead of just "” 2) Next, "tez.aux.uris”. This controls additional files/jars needed in the runtime on the cluster. Upload the tachyon jar to HDFS and ensure that you provide the path to either the dir on HDFS or the full path to the file and specify that in tez.aux.uris. The last thing to note is that you may need to pull additional transitive dependencies of tachyon if it is not self-contained jar. thanks — HItesh On Nov 12, 2015, at 1:06 AM, Bikas Saha wrote: > Can you provide the full stack trace? > > Are you getting the exception on the client (while submitting the job) or in > the cluster (after the job started to run)? > > For the client side, the fix would be to add tachyon jars to the client > classpath. Looks like you tried some client side classpath fixes. You could > run ‘hadoop classpath’ to print the classpath being picked up by the ‘hadoop > jar’ command. And scan its output to check if your tachyon jars are being > picked up correctly or not. > > Bikas > > From: Jiří Šimša [mailto:jiri.si...@gmail.com] > Sent: Wednesday, November 11, 2015 6:54 PM > To: user@tez.apache.org > Subject: Running Tez with Tachyon > > Hello, > > I have followed the Tez installation instructions > (https://tez.apache.org/install.html) and was able to successfully run the > ordered word count example: > > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar > orderedwordcount /input.txt /output.txt > > Next, I wanted to see if I can do the same, this time reading from and > writing to Tachyon (http://tachyon-project.org/) using: > > $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar > orderedwordcount tachyon://localhost:19998/input.txt > tachyon://localhost:19998/output.txt > > Unsurprisingly, this resulted in the "Class tachyon.hadoop.TFS not found" > error because Tez needs the Tachyon client jar that defines the > tachyon.hadoop.TFS class. To that end, I have tried several options (listed > below) to provide this jar to Tez, none of which seems to have worked: > > 1) Adding the Tachyon client jar to HADOOP_CLASSPATH > 2) Specifying the Tachyon client jar with the -libjars flag for the above > command. > 3) Copying the Tachyon client jar into the > $HADOOP_HOME/share/hadoop/common/lib directory of my HADOOP installation. > 4) Copying the Tachyon client jar into HDFS and specifying a path to it > through the tez.aux.uris property in the tez-site.xml file (in a similar > fashion the tez.lib.uris property specifies the path to the Tez tarball). > 5) I modified the source code of the ordered word count example, adding a > call to TezClient#addAppMasterLocalFiles(...), providing a URI for the > Tachyon client jar uploaded to HDFS. > > Any advice on how to pass the Tachyon client jar to Tez to resolve this issue > would be greatly appreciated. Thank you. > > Best, > > -- > Jiří Šimša
RE: Running Tez with Tachyon
Can you provide the full stack trace? Are you getting the exception on the client (while submitting the job) or in the cluster (after the job started to run)? For the client side, the fix would be to add tachyon jars to the client classpath. Looks like you tried some client side classpath fixes. You could run ‘hadoop classpath’ to print the classpath being picked up by the ‘hadoop jar’ command. And scan its output to check if your tachyon jars are being picked up correctly or not. Bikas From: Jiří Šimša [mailto:jiri.si...@gmail.com] Sent: Wednesday, November 11, 2015 6:54 PM To: user@tez.apache.org Subject: Running Tez with Tachyon Hello, I have followed the Tez installation instructions (https://tez.apache.org/install.html) and was able to successfully run the ordered word count example: $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar orderedwordcount /input.txt /output.txt Next, I wanted to see if I can do the same, this time reading from and writing to Tachyon (http://tachyon-project.org/) using: $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar orderedwordcount tachyon://localhost:19998/input.txt tachyon://localhost:19998/output.txt Unsurprisingly, this resulted in the "Class tachyon.hadoop.TFS not found" error because Tez needs the Tachyon client jar that defines the tachyon.hadoop.TFS class. To that end, I have tried several options (listed below) to provide this jar to Tez, none of which seems to have worked: 1) Adding the Tachyon client jar to HADOOP_CLASSPATH 2) Specifying the Tachyon client jar with the -libjars flag for the above command. 3) Copying the Tachyon client jar into the $HADOOP_HOME/share/hadoop/common/lib directory of my HADOOP installation. 4) Copying the Tachyon client jar into HDFS and specifying a path to it through the tez.aux.uris property in the tez-site.xml file (in a similar fashion the tez.lib.uris property specifies the path to the Tez tarball). 5) I modified the source code of the ordered word count example, adding a call to TezClient#addAppMasterLocalFiles(...), providing a URI for the Tachyon client jar uploaded to HDFS. Any advice on how to pass the Tachyon client jar to Tez to resolve this issue would be greatly appreciated. Thank you. Best, -- Jiří Šimša
Running Tez with Tachyon
Hello, I have followed the Tez installation instructions ( https://tez.apache.org/install.html) and was able to successfully run the ordered word count example: $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar orderedwordcount /input.txt /output.txt Next, I wanted to see if I can do the same, this time reading from and writing to Tachyon (http://tachyon-project.org/) using: $ hadoop jar ./tez-examples/target/tez-examples-0.8.2-SNAPSHOT.jar orderedwordcount tachyon://localhost:19998/input.txt tachyon://localhost:19998/output.txt Unsurprisingly, this resulted in the "Class tachyon.hadoop.TFS not found" error because Tez needs the Tachyon client jar that defines the tachyon.hadoop.TFS class. To that end, I have tried several options (listed below) to provide this jar to Tez, none of which seems to have worked: 1) Adding the Tachyon client jar to HADOOP_CLASSPATH 2) Specifying the Tachyon client jar with the -libjars flag for the above command. 3) Copying the Tachyon client jar into the $HADOOP_HOME/share/hadoop/common/lib directory of my HADOOP installation. 4) Copying the Tachyon client jar into HDFS and specifying a path to it through the tez.aux.uris property in the tez-site.xml file (in a similar fashion the tez.lib.uris property specifies the path to the Tez tarball). 5) I modified the source code of the ordered word count example, adding a call to TezClient#addAppMasterLocalFiles(...), providing a URI for the Tachyon client jar uploaded to HDFS. Any advice on how to pass the Tachyon client jar to Tez to resolve this issue would be greatly appreciated. Thank you. Best, -- Jiří Šimša
