It could be used by DAGClient internally but essentially it’s an additional library that can be used to get detailed information about any dag - running or completed (vs DAGClient that interacts with the dag is submitted). To be clear, I am calling it TezATSClient in provide clarity as to what its doing for the context of this thread. It will hide the details of ATS from users and provide a Tez-centric view (independent of internal details). Hence the actual name of the client will not have ATS in it.
If that makes sense, should we create jiras following that path? Bikas -----Original Message----- From: Siddharth Seth [mailto:[email protected]] Sent: Thursday, December 10, 2015 8:11 PM To: [email protected] Subject: Re: api compatibility within a minor release Agree with Andre's point - users do not need to bother about how the data is stored and retrieved, as long as it's accessible. Users should not need to know details of ATS - or even about it's existence ideally. The DAGClient monitors a dag for execution, and provides vertex progress. In addition, it should provide task progress and counters. Users should not need to worry about where this information comes from - the AM / ATS / some other source. @Bikas, the TezATSClient that you mention - is that meant to be used internally by the DAGClient, or is that a replacement to the DAGClient or is it an additional library that users will have to use or something else ? On Thu, Dec 10, 2015 at 12:59 PM, Bikas Saha <[email protected]> wrote: > Yes. That is what I was trying to convey with a TezATSClient - it > gives a Tez facade that provides expected semantics like > getFooForVertex() etc. and hides the actual ATS details from the user. > > -----Original Message----- > From: Andre Kelpe [mailto:[email protected]] > Sent: Thursday, December 10, 2015 2:09 AM > To: [email protected] > Subject: Re: api compatibility within a minor release > > From our point of view ATS is an implementation detail. As you said in > Budapest, it changes all the time anyway, so it is not the API we > would like to talk to. We care about the information stored in the ATS > that is relevant for the current DAG, but we don't really care about > the fact that the ATS is used. In MapReduce you also don't know, that > the information is stored int he JobHistoryServer. There is an API on > the RunningJob class to get to that information. We thing something > along those lines would be good. Then you can change the DAGClient and > the storage for the information all you want, we as consumers never have to > think about it. > > - André > > On Wed, Dec 9, 2015 at 11:39 PM, Bikas Saha <[email protected]> wrote: > > > Are we basically looking for detailed data that’s stored in the ATS? > > Then should we consider creating a TezATSClient that understand Tez > > semantics and fetches this information (in general for any DAG) > > instead of augmenting DAGClient (that is responsible for submitting > > a Tez DAG and monitoring a single DAG)? > > > > Bikas > > > > -----Original Message----- > > From: Andre Kelpe [mailto:[email protected]] > > Sent: Wednesday, December 9, 2015 10:32 AM > > To: Hitesh Shah <[email protected]> > > Cc: [email protected] > > Subject: Re: api compatibility within a minor release > > > > Hi, > > > > (undusting this thread after a while) > > > > I have opened https://issues.apache.org/jira/browse/TEZ-2981 to work > > around the issue. We believe that would be the best way forward. > > > > - André > > > > On Tue, Sep 15, 2015 at 11:04 PM, Hitesh Shah <[email protected]> wrote: > > > > > Hi Andre, > > > > > > I don’t believe I saw a JIRA created for this? Has this issue been > > > resolved? > > > > > > Also, it would be good if you can collate any particular private > > > APIs that you are using so that we can be aware of potential > > > issues with such > > APIs. > > > > > > thanks > > > — Hitesh > > > > > > On Aug 20, 2015, at 2:10 AM, Andre Kelpe > > > <[email protected]> > > wrote: > > > > > > > Thanks for the answer. We have a work-around for now. I am going > > > > to make that inventory and submit a patch that changes the > > > > annotations from @Private to @LimitedPrivate. > > > > > > > > From a semantic versioning point of view, I would still expect > > > > no > > > breaking > > > > changes in a bug fix release, but if the Tez community at large > > > > can work with that, we have to accept that, I guess. > > > > > > > > - André > > > > > > > > On Tue, Aug 18, 2015 at 8:34 PM, Hitesh Shah <[email protected]> > > wrote: > > > > > > > >> Hello Andre, > > > >> > > > >> For the most part, @Private is considered internal > > > >> implementation and subject to change at any point. In this > > > >> case, even more so as it is an *Impl class. > > > >> > > > >> What we can do is try the following: > > > >> - Look at all the various @Private classes being used by > Cascading. > > > >> - See which ones should not be used at all and which ones > > > >> can be considered to be a @LimitedPrivate for Cascading. > > > >> - For LimitedPrivate apis, we would then try to be a bit > > > >> more careful with respect to changing/breaking these APIs. I > > > >> would probably not say > > > that > > > >> they have will have the same guarantees as @Public/@Stable but > > > >> we can > > > work > > > >> with the Cascading community to handle changes to these APIs in > > > >> a > > > workable > > > >> manner on an ongoing basis. > > > >> > > > >> As for this API in question, for the short term fix, I guess a > > > >> simple approach might be to introduce a backward compatible API > > > >> ( I believe one which does not need the timeout param passed in )? > > > >> Would you mind > > > filing a > > > >> jira and hopefully provide a patch too? > > > >> > > > >> thanks > > > >> — Hitesh > > > >> > > > >> On Aug 18, 2015, at 8:01 AM, Andre Kelpe > > > >> <[email protected]> > > > wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> I have found a small API incompatibility in the Tez 0.6.2 release. > > > >>> The DAGClientTimelineImpl constructor got a new parameter for > > time-outs. > > > This > > > >>> was not present in 0.6.1 and from a semantic versioning point > > > >>> of view, > > > >> that > > > >>> is problematic. I know that the class is marked with the > > > >>> @Private annotation, but it would be great if such > > > >>> incompatible things aren't introduced in a bug-fix release. It > > > >>> would be easier for downsteam > > > >> projects, > > > >>> if you just added a second constructor. > > > >>> > > > >>> Is that an oversight or are classes with the @Private > > > >>> annotation > > > subject > > > >> to > > > >>> change and dowstream projects simply have to deal with it? > > > >>> > > > >>> Thanks! > > > >>> > > > >>> - André > > > >>> > > > >>> -- > > > >>> André Kelpe > > > >>> [email protected] > > > >>> http://concurrentinc.com > > > >> > > > >> > > > > > > > > > > > > -- > > > > André Kelpe > > > > [email protected] > > > > http://concurrentinc.com > > > > > > > > > > > > -- > > André Kelpe > > [email protected] > > http://concurrentinc.com > > > > > > > -- > André Kelpe > [email protected] > http://concurrentinc.com > >
