Re: Getting dot files for DAGs
I don’t believe the binary should need changing at all unless you need enhancements from recent commits. It should just be setting up the UI and configuring Tez for using YARN Timeline. The instructions that you can follow: http://tez.apache.org/tez-ui.html http://tez.apache.org/tez_yarn_timeline.html thanks — Hitesh On Oct 1, 2015, at 11:07 AM, James Pirz wrote: > Thanks for suggesting, I never used Tez UI before, and learned about it > yesterday. > I am trying to find out how I can enable/use it. Apparently it needs some > changes in the binary that I am using (I had built the binary for tez 0.7 > almost 2 months ago). > > > > > On Wed, Sep 30, 2015 at 10:27 PM, Jörn Franke wrote: > Why not use tez ui? > > Le jeu. 1 oct. 2015 à 2:29, James Pirz a écrit : > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. > I am interested in checking DAGs for my queries visually, and I realized that > I can do that by graphviz once I can get "dot" files of my DAGs. My issue is > I can not find those files, they are not in the log directory of Yarn or > Hadoop or under /tmp . > > Any hint as where I can find those files would be great. Do I need to add any > settings to my tez-site.xml in-order to enable generating them ? > > Thanks. >
Re: Getting dot files for DAGs
Thanks for suggesting, I never used Tez UI before, and learned about it yesterday. I am trying to find out how I can enable/use it. Apparently it needs some changes in the binary that I am using (I had built the binary for tez 0.7 almost 2 months ago). On Wed, Sep 30, 2015 at 10:27 PM, Jörn Franke wrote: > Why not use tez ui? > > Le jeu. 1 oct. 2015 à 2:29, James Pirz a écrit : > >> I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. >> I am interested in checking DAGs for my queries visually, and I realized >> that I can do that by graphviz once I can get "dot" files of my DAGs. My >> issue is I can not find those files, they are not in the log directory of >> Yarn or Hadoop or under /tmp . >> >> Any hint as where I can find those files would be great. Do I need to add >> any settings to my tez-site.xml in-order to enable generating them ? >> >> Thanks. >> >
Re: Getting dot files for DAGs
Oh, that is good to know. I never really looked into the Tez UI. Maybe I should.. :) - André On Thu, Oct 1, 2015 at 5:08 PM, Hitesh Shah wrote: > Hello Andre, > > For ATS, in the TEZ_DAG_ID entity, the dagPlan is already serialized and > available in the otherInfo section. It is not in the .dot format but it is > used by the Tez UI to come up with the graphical view of the dag plan. > > thanks > — Hitesh > > > On Oct 1, 2015, at 1:28 AM, Andre Kelpe wrote: > > > Maybe it would be a good idea to send the dot file to the ATS along with > the other information you are sending. I too wanted to look at a dot file > the other day and had problem finding it back. > > > > - André > > > > On Thu, Oct 1, 2015 at 4:00 AM, Hitesh Shah wrote: > > The .dot file is generated into the Tez Application Master’s container > log dir. Firstly, you need to figure out the yarn application in which the > query/Tez DAG ran. Once you have the applicationId, you can use one of > these 2 approaches: > > > > 1) Go to the YARN ResourceManager UI, find the application and click > through to the Application Master logs. The .dot file for the dag should be > visible there. > > 2) Using the application Id ( if the application has completed), get the > yarn logs using “bin/yarn logs -applicationId ” - once you have the > logs, you will be able to find the contents of the .dot file within them. > This approach only works if you have YARN log aggregation enabled. > > > > thanks > > — Hitesh > > > > > > On Sep 30, 2015, at 5:29 PM, James Pirz wrote: > > > > > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. > > > I am interested in checking DAGs for my queries visually, and I > realized that I can do that by graphviz once I can get "dot" files of my > DAGs. My issue is I can not find those files, they are not in the log > directory of Yarn or Hadoop or under /tmp . > > > > > > Any hint as where I can find those files would be great. Do I need to > add any settings to my tez-site.xml in-order to enable generating them ? > > > > > > Thanks. > > > > > > > > > > -- > > André Kelpe > > an...@concurrentinc.com > > http://concurrentinc.com > > -- André Kelpe an...@concurrentinc.com http://concurrentinc.com
Re: Getting dot files for DAGs
Hello Andre, For ATS, in the TEZ_DAG_ID entity, the dagPlan is already serialized and available in the otherInfo section. It is not in the .dot format but it is used by the Tez UI to come up with the graphical view of the dag plan. thanks — Hitesh On Oct 1, 2015, at 1:28 AM, Andre Kelpe wrote: > Maybe it would be a good idea to send the dot file to the ATS along with the > other information you are sending. I too wanted to look at a dot file the > other day and had problem finding it back. > > - André > > On Thu, Oct 1, 2015 at 4:00 AM, Hitesh Shah wrote: > The .dot file is generated into the Tez Application Master’s container log > dir. Firstly, you need to figure out the yarn application in which the > query/Tez DAG ran. Once you have the applicationId, you can use one of these > 2 approaches: > > 1) Go to the YARN ResourceManager UI, find the application and click through > to the Application Master logs. The .dot file for the dag should be visible > there. > 2) Using the application Id ( if the application has completed), get the yarn > logs using “bin/yarn logs -applicationId ” - once you have the logs, > you will be able to find the contents of the .dot file within them. This > approach only works if you have YARN log aggregation enabled. > > thanks > — Hitesh > > > On Sep 30, 2015, at 5:29 PM, James Pirz wrote: > > > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. > > I am interested in checking DAGs for my queries visually, and I realized > > that I can do that by graphviz once I can get "dot" files of my DAGs. My > > issue is I can not find those files, they are not in the log directory of > > Yarn or Hadoop or under /tmp . > > > > Any hint as where I can find those files would be great. Do I need to add > > any settings to my tez-site.xml in-order to enable generating them ? > > > > Thanks. > > > > > -- > André Kelpe > an...@concurrentinc.com > http://concurrentinc.com
Re: Getting dot files for DAGs
Maybe it would be a good idea to send the dot file to the ATS along with the other information you are sending. I too wanted to look at a dot file the other day and had problem finding it back. - André On Thu, Oct 1, 2015 at 4:00 AM, Hitesh Shah wrote: > The .dot file is generated into the Tez Application Master’s container log > dir. Firstly, you need to figure out the yarn application in which the > query/Tez DAG ran. Once you have the applicationId, you can use one of > these 2 approaches: > > 1) Go to the YARN ResourceManager UI, find the application and click > through to the Application Master logs. The .dot file for the dag should be > visible there. > 2) Using the application Id ( if the application has completed), get the > yarn logs using “bin/yarn logs -applicationId ” - once you have the > logs, you will be able to find the contents of the .dot file within them. > This approach only works if you have YARN log aggregation enabled. > > thanks > — Hitesh > > > On Sep 30, 2015, at 5:29 PM, James Pirz wrote: > > > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. > > I am interested in checking DAGs for my queries visually, and I realized > that I can do that by graphviz once I can get "dot" files of my DAGs. My > issue is I can not find those files, they are not in the log directory of > Yarn or Hadoop or under /tmp . > > > > Any hint as where I can find those files would be great. Do I need to > add any settings to my tez-site.xml in-order to enable generating them ? > > > > Thanks. > > -- André Kelpe an...@concurrentinc.com http://concurrentinc.com
Re: Getting dot files for DAGs
Thanks. I could locate them in the proper container's log directory and visualize them. I was at the wrong node, assuming that they would be available on any of the node, but they are really dumped in one of the nodes. On Wed, Sep 30, 2015 at 7:00 PM, Hitesh Shah wrote: > The .dot file is generated into the Tez Application Master’s container log > dir. Firstly, you need to figure out the yarn application in which the > query/Tez DAG ran. Once you have the applicationId, you can use one of > these 2 approaches: > > 1) Go to the YARN ResourceManager UI, find the application and click > through to the Application Master logs. The .dot file for the dag should be > visible there. > 2) Using the application Id ( if the application has completed), get the > yarn logs using “bin/yarn logs -applicationId ” - once you have the > logs, you will be able to find the contents of the .dot file within them. > This approach only works if you have YARN log aggregation enabled. > > thanks > — Hitesh > > > On Sep 30, 2015, at 5:29 PM, James Pirz wrote: > > > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. > > I am interested in checking DAGs for my queries visually, and I realized > that I can do that by graphviz once I can get "dot" files of my DAGs. My > issue is I can not find those files, they are not in the log directory of > Yarn or Hadoop or under /tmp . > > > > Any hint as where I can find those files would be great. Do I need to > add any settings to my tez-site.xml in-order to enable generating them ? > > > > Thanks. > >
Re: Getting dot files for DAGs
The .dot file is generated into the Tez Application Master’s container log dir. Firstly, you need to figure out the yarn application in which the query/Tez DAG ran. Once you have the applicationId, you can use one of these 2 approaches: 1) Go to the YARN ResourceManager UI, find the application and click through to the Application Master logs. The .dot file for the dag should be visible there. 2) Using the application Id ( if the application has completed), get the yarn logs using “bin/yarn logs -applicationId ” - once you have the logs, you will be able to find the contents of the .dot file within them. This approach only works if you have YARN log aggregation enabled. thanks — Hitesh On Sep 30, 2015, at 5:29 PM, James Pirz wrote: > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. > I am interested in checking DAGs for my queries visually, and I realized that > I can do that by graphviz once I can get "dot" files of my DAGs. My issue is > I can not find those files, they are not in the log directory of Yarn or > Hadoop or under /tmp . > > Any hint as where I can find those files would be great. Do I need to add any > settings to my tez-site.xml in-order to enable generating them ? > > Thanks.
Re: Getting dot files for DAGs
Hi James, It is under the working directory of the yarn container (should be the first container which is AM) Best Regard, Jeff Zhang From: James Pirz mailto:james.p...@gmail.com>> Reply-To: "user@tez.apache.org<mailto:user@tez.apache.org>" mailto:user@tez.apache.org>> Date: Thursday, October 1, 2015 at 8:29 AM To: "user@tez.apache.org<mailto:user@tez.apache.org>" mailto:user@tez.apache.org>> Cc: "u...@hive.apache.org<mailto:u...@hive.apache.org>" mailto:u...@hive.apache.org>> Subject: Getting dot files for DAGs I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. I am interested in checking DAGs for my queries visually, and I realized that I can do that by graphviz once I can get "dot" files of my DAGs. My issue is I can not find those files, they are not in the log directory of Yarn or Hadoop or under /tmp . Any hint as where I can find those files would be great. Do I need to add any settings to my tez-site.xml in-order to enable generating them ? Thanks.
Getting dot files for DAGs
I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. I am interested in checking DAGs for my queries visually, and I realized that I can do that by graphviz once I can get "dot" files of my DAGs. My issue is I can not find those files, they are not in the log directory of Yarn or Hadoop or under /tmp . Any hint as where I can find those files would be great. Do I need to add any settings to my tez-site.xml in-order to enable generating them ? Thanks.
