Agree with Andre's point - users do not need to bother about how the data is stored and retrieved, as long as it's accessible. Users should not need to know details of ATS - or even about it's existence ideally. The DAGClient monitors a dag for execution, and provides vertex progress. In addition, it should provide task progress and counters. Users should not need to worry about where this information comes from - the AM / ATS / some other source.
@Bikas, the TezATSClient that you mention - is that meant to be used internally by the DAGClient, or is that a replacement to the DAGClient or is it an additional library that users will have to use or something else ? On Thu, Dec 10, 2015 at 12:59 PM, Bikas Saha <[email protected]> wrote: > Yes. That is what I was trying to convey with a TezATSClient - it gives a > Tez facade that provides expected semantics like getFooForVertex() etc. and > hides the actual ATS details from the user. > > -----Original Message----- > From: Andre Kelpe [mailto:[email protected]] > Sent: Thursday, December 10, 2015 2:09 AM > To: [email protected] > Subject: Re: api compatibility within a minor release > > From our point of view ATS is an implementation detail. As you said in > Budapest, it changes all the time anyway, so it is not the API we would > like to talk to. We care about the information stored in the ATS that is > relevant for the current DAG, but we don't really care about the fact that > the ATS is used. In MapReduce you also don't know, that the information is > stored int he JobHistoryServer. There is an API on the RunningJob class to > get to that information. We thing something along those lines would be > good. Then you can change the DAGClient and the storage for the information > all you want, we as consumers never have to think about it. > > - André > > On Wed, Dec 9, 2015 at 11:39 PM, Bikas Saha <[email protected]> wrote: > > > Are we basically looking for detailed data that’s stored in the ATS? > > Then should we consider creating a TezATSClient that understand Tez > > semantics and fetches this information (in general for any DAG) > > instead of augmenting DAGClient (that is responsible for submitting a > > Tez DAG and monitoring a single DAG)? > > > > Bikas > > > > -----Original Message----- > > From: Andre Kelpe [mailto:[email protected]] > > Sent: Wednesday, December 9, 2015 10:32 AM > > To: Hitesh Shah <[email protected]> > > Cc: [email protected] > > Subject: Re: api compatibility within a minor release > > > > Hi, > > > > (undusting this thread after a while) > > > > I have opened https://issues.apache.org/jira/browse/TEZ-2981 to work > > around the issue. We believe that would be the best way forward. > > > > - André > > > > On Tue, Sep 15, 2015 at 11:04 PM, Hitesh Shah <[email protected]> wrote: > > > > > Hi Andre, > > > > > > I don’t believe I saw a JIRA created for this? Has this issue been > > > resolved? > > > > > > Also, it would be good if you can collate any particular private > > > APIs that you are using so that we can be aware of potential issues > > > with such > > APIs. > > > > > > thanks > > > — Hitesh > > > > > > On Aug 20, 2015, at 2:10 AM, Andre Kelpe <[email protected]> > > wrote: > > > > > > > Thanks for the answer. We have a work-around for now. I am going > > > > to make that inventory and submit a patch that changes the > > > > annotations from @Private to @LimitedPrivate. > > > > > > > > From a semantic versioning point of view, I would still expect no > > > breaking > > > > changes in a bug fix release, but if the Tez community at large > > > > can work with that, we have to accept that, I guess. > > > > > > > > - André > > > > > > > > On Tue, Aug 18, 2015 at 8:34 PM, Hitesh Shah <[email protected]> > > wrote: > > > > > > > >> Hello Andre, > > > >> > > > >> For the most part, @Private is considered internal implementation > > > >> and subject to change at any point. In this case, even more so as > > > >> it is an *Impl class. > > > >> > > > >> What we can do is try the following: > > > >> - Look at all the various @Private classes being used by > Cascading. > > > >> - See which ones should not be used at all and which ones can > > > >> be considered to be a @LimitedPrivate for Cascading. > > > >> - For LimitedPrivate apis, we would then try to be a bit more > > > >> careful with respect to changing/breaking these APIs. I would > > > >> probably not say > > > that > > > >> they have will have the same guarantees as @Public/@Stable but we > > > >> can > > > work > > > >> with the Cascading community to handle changes to these APIs in a > > > workable > > > >> manner on an ongoing basis. > > > >> > > > >> As for this API in question, for the short term fix, I guess a > > > >> simple approach might be to introduce a backward compatible API ( > > > >> I believe one which does not need the timeout param passed in )? > > > >> Would you mind > > > filing a > > > >> jira and hopefully provide a patch too? > > > >> > > > >> thanks > > > >> — Hitesh > > > >> > > > >> On Aug 18, 2015, at 8:01 AM, Andre Kelpe > > > >> <[email protected]> > > > wrote: > > > >> > > > >>> Hi, > > > >>> > > > >>> I have found a small API incompatibility in the Tez 0.6.2 release. > > > >>> The DAGClientTimelineImpl constructor got a new parameter for > > time-outs. > > > This > > > >>> was not present in 0.6.1 and from a semantic versioning point of > > > >>> view, > > > >> that > > > >>> is problematic. I know that the class is marked with the > > > >>> @Private annotation, but it would be great if such incompatible > > > >>> things aren't introduced in a bug-fix release. It would be > > > >>> easier for downsteam > > > >> projects, > > > >>> if you just added a second constructor. > > > >>> > > > >>> Is that an oversight or are classes with the @Private annotation > > > subject > > > >> to > > > >>> change and dowstream projects simply have to deal with it? > > > >>> > > > >>> Thanks! > > > >>> > > > >>> - André > > > >>> > > > >>> -- > > > >>> André Kelpe > > > >>> [email protected] > > > >>> http://concurrentinc.com > > > >> > > > >> > > > > > > > > > > > > -- > > > > André Kelpe > > > > [email protected] > > > > http://concurrentinc.com > > > > > > > > > > > > -- > > André Kelpe > > [email protected] > > http://concurrentinc.com > > > > > > > -- > André Kelpe > [email protected] > http://concurrentinc.com > >
