RE: Getting DAG Id from Hive on tez

[Dharmesh Kakadia]

Thanks Hitesh.
I am able to filter out the particular dag, now the only problem is how to get 
the DAG name. I see from a previous mail on the list[1] that Tez uses Hive 
queryId + counter as the dag name. I have access to the hive query Id, but how 
do I know the name from it? Is there a way to get the name/Id of the DAG 
directly ? Just trying out counter=1..N will be pretty bad for me in pre-hook.
[1] 
http://mail-archives.apache.org/mod_mbox/hive-user/201408.mbox/%3c1409015953.23241.yahoomail...@web161603.mail.bf1.yahoo.com%3E
 


Thanks,
Dharmesh

The Hive query id maps to the Tez dag name. You can try the following call 
against timeline:


/ws/v1/timeline/TEZ_DAG_ID?primaryFilter=dagName:{tezDagName}

thanks
— Hitesh

On Sep 14, 2015, at 10:45 PM, Dharmesh Kakadia 

wrote:

> Hi,
> 
> I am running Hive on Tez, with timeline server. We have a pre-hook in hive to 
> maintain
the statistics of what jobs ran by whom and how much resource it tool etc. that 
we had been
using it with Hive-MR. Now I am trying to port this pre-hook to work with Hive 
on Tez. 
> 
> I plan to using timeline server for querying the stat, but I am not able to 
> get the DAG
ID in the hook. How to get the DAG Id from Tez? Any help will be great.
> 
> Thanks,
> Dharmesh

Re: Getting DAG Id from Hive on tez

[Hitesh Shah]

This is a question that is probably meant for the Hive mailing list. I believe 
either the Hive query Id or the information from the query plan ( as used in 
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java
 ) should probably be able to give you that info but the Hive devs 
(d...@hive.apache.org) will likely have a better answer for this. 

— HItesh 


On Sep 15, 2015, at 1:56 AM, Dharmesh Kakadia 
 wrote:

> Thanks Hitesh.
> I am able to filter out the particular dag, now the only problem is how to 
> get the DAG name. I see from a previous mail on the list[1] that Tez uses 
> Hive queryId + counter as the dag name. I have access to the hive query Id, 
> but how do I know the name from it? Is there a way to get the name/Id of the 
> DAG directly ? Just trying out counter=1..N will be pretty bad for me in 
> pre-hook.
> [1] 
> http://mail-archives.apache.org/mod_mbox/hive-user/201408.mbox/%3c1409015953.23241.yahoomail...@web161603.mail.bf1.yahoo.com%3E
> 
> Thanks,
> Dharmesh
> 
> The Hive query id maps to the Tez dag name. You can try the following call 
> against timeline:
> 
> 
> /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=dagName:{tezDagName}
> 
> thanks
> — Hitesh
> 
> On Sep 14, 2015, at 10:45 PM, Dharmesh Kakadia <
> dharmesh.kaka...@research.iiit.ac.in
> >
> wrote:
> 
> > Hi,
> > 
> > I am running Hive on Tez, with timeline server. We have a pre-hook in hive 
> > to maintain
> the statistics of what jobs ran by whom and how much resource it tool etc. 
> that we had been
> using it with Hive-MR. Now I am trying to port this pre-hook to work with 
> Hive on Tez. 
> > 
> > I plan to using timeline server for querying the stat, but I am not able to 
> > get the DAG
> ID in the hook. How to get the DAG Id from Tez? Any help will be great.
> > 
> > Thanks,
> > Dharmesh
> 
> 

Getting DAG Id from Hive on tez

[Dharmesh Kakadia]

Hi,

I am running Hive on Tez, with timeline server. We have a pre-hook in hive to 
maintain the statistics of what jobs ran by whom and how much resource it tool 
etc. that we had been using it with Hive-MR. Now I am trying to port this 
pre-hook to work with Hive on Tez. 

I plan to using timeline server for querying the stat, but I am not able to get 
the DAG ID in the hook. How to get the DAG Id from Tez? Any help will be great.

Thanks,
Dharmesh

Re: Getting DAG Id from Hive on tez

[Hitesh Shah]

The Hive query id maps to the Tez dag name. You can try the following call 
against timeline: 

/ws/v1/timeline/TEZ_DAG_ID?primaryFilter=dagName:{tezDagName}

thanks
— Hitesh

On Sep 14, 2015, at 10:45 PM, Dharmesh Kakadia 
 wrote:

> Hi,
> 
> I am running Hive on Tez, with timeline server. We have a pre-hook in hive to 
> maintain the statistics of what jobs ran by whom and how much resource it 
> tool etc. that we had been using it with Hive-MR. Now I am trying to port 
> this pre-hook to work with Hive on Tez. 
> 
> I plan to using timeline server for querying the stat, but I am not able to 
> get the DAG ID in the hook. How to get the DAG Id from Tez? Any help will be 
> great.
> 
> Thanks,
> Dharmesh