Thanks Aniket. I'll revert the aforementioned commits in 0.12 tonight. I will leave them in trunk until we decide what to do.
On Mon, Sep 30, 2013 at 3:37 PM, Aniket Mokashi <[email protected]> wrote: > +1 on reverting PIG-3419 and applying it to tez branch if its blocking > pig-0.12 release. > > On Mon, Sep 30, 2013 at 2:42 PM, Cheolsoo Park <[email protected]> > wrote: > > > I am waiting for +1 from Twitter. > > > > Like Alan suggested, let's revert PIG-3419 et al in 0.12 first. Then, we > > can decide what to do in trunk. I volunteer to do grunt work since I am > the > > one who committed them. > > > > > > On Mon, Sep 30, 2013 at 2:26 PM, Rohini Palaniswamy < > > [email protected] > > > wrote: > > > > > +1. I was already asking for keeping the new API changes only in Tez > > branch > > > till it evolves and is finalized, so I have no objections to reverting > > it. > > > > > > Regards, > > > Rohini > > > > > > > > > On Mon, Sep 30, 2013 at 1:28 PM, Alan Gates <[email protected]> > > wrote: > > > > > > > We should separate out two separate concerns. If I understand > > correctly > > > > we don't need any of these changes in 0.12. So we should revert > these > > > > patches from the 12 branch so that we can get it released quickly in > a > > > > backwards compatible way. > > > > > > > > We will then have plenty of time to discuss the separate question of > > how > > > > we proceed going forward (deprecated APIs or new APIs). > > > > > > > > Alan. > > > > > > > > On Sep 30, 2013, at 11:45 AM, Cheolsoo Park wrote: > > > > > > > > > Hi Jeremy, > > > > > > > > > > What you're saying makes sense, and patch is welcome. ;-) But > > > complexity > > > > > comes from that there are many classes that are associated with one > > > > > another, and it seems necessary to bring back all of them together > in > > > > order > > > > > to provide full backward compatibility. > > > > > > > > > > After spending many hours on the weekend, I concluded that adding > > more > > > > > workarounds (classes, methods, packages, etc) to the current code > > makes > > > > it > > > > > only less maintainable and readable. So I prefer a simpler > approach. > > > > > > > > > > For eg, we can just publish two jars - pig.jar w/ old API and > > > pig-new.jar > > > > > w/ new API - maybe not in 0.12 but in 0.13. Since we already have a > > > > > tez-branch, we can use it to manage the new version of classes. > Then, > > > > users > > > > > can switch to pig-new.jar gradually in 0.13 and 0.14. When we > finally > > > > merge > > > > > tez-branch into trunk, we can publish a single jar again. > > > > > > > > > > Of course, this is not trivial either because we have to maintain > two > > > > > branches. But I feel that managing two branches independently is > > easier > > > > > than maintaining all sorts of workarounds for backward > compatibility > > in > > > > the > > > > > source code. In addition, we will have more flexibility in terms of > > > > > designing new API because we will be completely free from backward > > > > > compatibility. No? > > > > > > > > > > Thanks, > > > > > Cheolsoo > > > > > > > > > > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn < > [email protected]> > > > > wrote: > > > > > > > > > >> What about the option of leaving all of the MR specific logic in > the > > > > >> original classes but marking those methods as deprecated and > telling > > > > people > > > > >> to switch to using a MR specific object that extends the original > > > class. > > > > >> So for example: > > > > >> > > > > >> JobStats - Reverted to being as it was before PIG-3419 but with > all > > MR > > > > >> specific logic deprecated. > > > > >> MRJobStats - Would just extend JobStats. > > > > >> > > > > >> If we did this, external software could switch their code from > using > > > > >> JobStats to MRJobStats at their own pace and without breaking > > against > > > > any > > > > >> specific version of Pig. After a few versions the MR specific > logic > > > > could > > > > >> be removed from JobStats and pushed into MRJobStats and it > shouldn't > > > > break > > > > >> anything for people that had made that change. > > > > >> > > > > >> I'm not familiar with all of the changes in PIG-3419 so this might > > not > > > > work > > > > >> everywhere. > > > > >> > > > > >> > > > > >> On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park < > > [email protected]> > > > > >> wrote: > > > > >> > > > > >>> To be specific, we will need to revert all the following commits > in > > > > >> order: > > > > >>> > > > > >>> > > > > >>> commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982 > > > > >>> Author: Cheolsoo Park <[email protected]> > > > > >>> Date: Fri Sep 20 22:47:29 2013 +0000 > > > > >>> > > > > >>> PIG-3471: Add a base abstract class for ExecutionEngine > > (cheolsoo) > > > > >>> > > > > >>> git-svn-id: > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68 > > > > >>> > > > > >>> commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1 > > > > >>> Author: Jianyong Dai <[email protected]> > > > > >>> Date: Wed Sep 18 19:09:49 2013 +0000 > > > > >>> > > > > >>> PIG-3457: Provide backward compatibility for PigStatsUtil and > > > > >> JobStats > > > > >>> > > > > >>> git-svn-id: > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68 > > > > >>> > > > > >>> commit e85cf34c92713aa697a1cda7a9c2b3db139350f7 > > > > >>> Author: Cheolsoo Park <[email protected]> > > > > >>> Date: Wed Sep 18 15:37:58 2013 +0000 > > > > >>> > > > > >>> PIG-3464: Mark ExecType and ExecutionEngine interfaces as > > evolving > > > > >>> (cheolsoo) > > > > >>> > > > > >>> commit fd8b7cdf9292b305f02386d560c25298ab492a0b > > > > >>> Author: Cheolsoo Park <[email protected]> > > > > >>> Date: Fri Aug 30 20:04:29 2013 +0000 > > > > >>> > > > > >>> PIG-3419: Pluggable Execution Engine (achalsoni81 via > cheolsoo) > > > > >>> > > > > >>> git-svn-id: > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68 > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai < > > [email protected]> > > > > >>> wrote: > > > > >>> > > > > >>>> Thanks Cheolsoo! My opinion is provide backward compatibility > for > > > > >>> PigStats > > > > >>>> is a must, otherwise it could be a havoc. I imagine PigStats is > > > widely > > > > >>> used > > > > >>>> by Pig users via PigRunner and PPNL interface. People use > PigStats > > > to > > > > >>>> collect MR job details of the Pig job. Though PigStats is marked > > for > > > > >>>> Evolving, this is mostly for extending PigStats, not limiting > > > PigStats > > > > >> as > > > > >>>> PIG-3419 did. Even if we really need to change, we need to very > > well > > > > >>>> communicate with users over time, Pig 0.12 is not an option. > > > > >>>> > > > > >>>> PIG-3457 is trying to provide a backward compatibility way for > > > > >> PigStats, > > > > >>>> but just like Cheolsoo said, it is far from ideal. I now tend to > > > agree > > > > >>>> Rohini's suggestion on PIG-3419, rollback PIG-3419, until we > find > > a > > > > >>> better > > > > >>>> way. Seems PIG-3419 is a little premature. Besides the above > > > mentioned > > > > >>>> PigStats issue, I've already found 2 additional issues: > > > > >>>> 1. "explain" shows unoptimized logical plan instead of optimized > > one > > > > >>>> 2. HangingJobKiller is removed > > > > >>>> > > > > >>>> How does others think? > > > > >>>> > > > > >>>> Thanks, > > > > >>>> Daniel > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park < > > > [email protected]> > > > > >>>> wrote: > > > > >>>> > > > > >>>>> Hi devs, > > > > >>>>> > > > > >>>>> PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419> > broke > > > > >>> backward > > > > >>>>> compatibility for downstream applications such as Oozie, and > > > > >>>>> PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is > > > > >>>>> trying to fix it. > > > > >>>>> > > > > >>>>> In summary, we need to keep the old MR-specific JobStats and > > > PigStats > > > > >>> for > > > > >>>>> backward compatibility sadly because they're currently exposed > in > > > > >>> several > > > > >>>>> user-facing API including PigRunner, PigServer, etc. However, > > this > > > > >>>> defeats > > > > >>>>> the purpose of PIG-3419 > > > > >>>>> <https://issues.apache.org/jira/browse/PIG-3419> because > > > > >>>>> we cannot implement non-MR execution engines in the back-end if > > the > > > > >>>>> front-end API is tied to MR. > > > > >>>>> > > > > >>>>> Daniel and I were having a discussion in the jira, but it seems > > > more > > > > >>>>> complicated than I thought. I am wondering whether anyone has a > > > good > > > > >>>>> suggestion on how to solve it. This is blocking the 0.12 > release. > > > > >>>>> > > > > >>>>> Thanks, > > > > >>>>> Cheolsoo > > > > >>>>> > > > > >>>> > > > > >>>> -- > > > > >>>> CONFIDENTIALITY NOTICE > > > > >>>> NOTICE: This message is intended for the use of the individual > or > > > > >> entity > > > > >>> to > > > > >>>> which it is addressed and may contain information that is > > > > confidential, > > > > >>>> privileged and exempt from disclosure under applicable law. If > the > > > > >> reader > > > > >>>> of this message is not the intended recipient, you are hereby > > > notified > > > > >>> that > > > > >>>> any printing, copying, dissemination, distribution, disclosure > or > > > > >>>> forwarding of this communication is strictly prohibited. If you > > have > > > > >>>> received this communication in error, please contact the sender > > > > >>> immediately > > > > >>>> and delete it from your system. Thank You. > > > > >>>> > > > > >>> > > > > >> > > > > >> > > > > >> > > > > >> -- > > > > >> > > > > >> Jeremy Karn / Lead Developer > > > > >> MORTAR DATA / 519 277 4391 / www.mortardata.com > > > > >> > > > > > > > > > > > > -- > > > > CONFIDENTIALITY NOTICE > > > > NOTICE: This message is intended for the use of the individual or > > entity > > > to > > > > which it is addressed and may contain information that is > confidential, > > > > privileged and exempt from disclosure under applicable law. If the > > reader > > > > of this message is not the intended recipient, you are hereby > notified > > > that > > > > any printing, copying, dissemination, distribution, disclosure or > > > > forwarding of this communication is strictly prohibited. If you have > > > > received this communication in error, please contact the sender > > > immediately > > > > and delete it from your system. Thank You. > > > > > > > > > > > > > -- > "...:::Aniket:::... Quetzalco@tl" >
