OK, I reverted PIG-3471, 3457, 3464, and 3419 in 0.12: http://svn.apache.org/viewvc?view=revision&revision=1527867
You can also view each revert here: https://github.com/piaozhexiu/apache-pig/commits/revert I had to manually resolve some conflicts particularly due to PIG-3430. I believe I didn't break anything, but please let me know if I made any mistake. Test-commit passes. Thank you, Cheolsoo On Mon, Sep 30, 2013 at 3:52 PM, Cheolsoo Park <[email protected]> wrote: > Thanks Aniket. > > I'll revert the aforementioned commits in 0.12 tonight. I will leave them > in trunk until we decide what to do. > > > On Mon, Sep 30, 2013 at 3:37 PM, Aniket Mokashi <[email protected]>wrote: > >> +1 on reverting PIG-3419 and applying it to tez branch if its blocking >> pig-0.12 release. >> >> On Mon, Sep 30, 2013 at 2:42 PM, Cheolsoo Park <[email protected]> >> wrote: >> >> > I am waiting for +1 from Twitter. >> > >> > Like Alan suggested, let's revert PIG-3419 et al in 0.12 first. Then, we >> > can decide what to do in trunk. I volunteer to do grunt work since I am >> the >> > one who committed them. >> > >> > >> > On Mon, Sep 30, 2013 at 2:26 PM, Rohini Palaniswamy < >> > [email protected] >> > > wrote: >> > >> > > +1. I was already asking for keeping the new API changes only in Tez >> > branch >> > > till it evolves and is finalized, so I have no objections to reverting >> > it. >> > > >> > > Regards, >> > > Rohini >> > > >> > > >> > > On Mon, Sep 30, 2013 at 1:28 PM, Alan Gates <[email protected]> >> > wrote: >> > > >> > > > We should separate out two separate concerns. If I understand >> > correctly >> > > > we don't need any of these changes in 0.12. So we should revert >> these >> > > > patches from the 12 branch so that we can get it released quickly >> in a >> > > > backwards compatible way. >> > > > >> > > > We will then have plenty of time to discuss the separate question of >> > how >> > > > we proceed going forward (deprecated APIs or new APIs). >> > > > >> > > > Alan. >> > > > >> > > > On Sep 30, 2013, at 11:45 AM, Cheolsoo Park wrote: >> > > > >> > > > > Hi Jeremy, >> > > > > >> > > > > What you're saying makes sense, and patch is welcome. ;-) But >> > > complexity >> > > > > comes from that there are many classes that are associated with >> one >> > > > > another, and it seems necessary to bring back all of them >> together in >> > > > order >> > > > > to provide full backward compatibility. >> > > > > >> > > > > After spending many hours on the weekend, I concluded that adding >> > more >> > > > > workarounds (classes, methods, packages, etc) to the current code >> > makes >> > > > it >> > > > > only less maintainable and readable. So I prefer a simpler >> approach. >> > > > > >> > > > > For eg, we can just publish two jars - pig.jar w/ old API and >> > > pig-new.jar >> > > > > w/ new API - maybe not in 0.12 but in 0.13. Since we already have >> a >> > > > > tez-branch, we can use it to manage the new version of classes. >> Then, >> > > > users >> > > > > can switch to pig-new.jar gradually in 0.13 and 0.14. When we >> finally >> > > > merge >> > > > > tez-branch into trunk, we can publish a single jar again. >> > > > > >> > > > > Of course, this is not trivial either because we have to maintain >> two >> > > > > branches. But I feel that managing two branches independently is >> > easier >> > > > > than maintaining all sorts of workarounds for backward >> compatibility >> > in >> > > > the >> > > > > source code. In addition, we will have more flexibility in terms >> of >> > > > > designing new API because we will be completely free from backward >> > > > > compatibility. No? >> > > > > >> > > > > Thanks, >> > > > > Cheolsoo >> > > > > >> > > > > On Mon, Sep 30, 2013 at 11:12 AM, Jeremy Karn < >> [email protected]> >> > > > wrote: >> > > > > >> > > > >> What about the option of leaving all of the MR specific logic in >> the >> > > > >> original classes but marking those methods as deprecated and >> telling >> > > > people >> > > > >> to switch to using a MR specific object that extends the original >> > > class. >> > > > >> So for example: >> > > > >> >> > > > >> JobStats - Reverted to being as it was before PIG-3419 but with >> all >> > MR >> > > > >> specific logic deprecated. >> > > > >> MRJobStats - Would just extend JobStats. >> > > > >> >> > > > >> If we did this, external software could switch their code from >> using >> > > > >> JobStats to MRJobStats at their own pace and without breaking >> > against >> > > > any >> > > > >> specific version of Pig. After a few versions the MR specific >> logic >> > > > could >> > > > >> be removed from JobStats and pushed into MRJobStats and it >> shouldn't >> > > > break >> > > > >> anything for people that had made that change. >> > > > >> >> > > > >> I'm not familiar with all of the changes in PIG-3419 so this >> might >> > not >> > > > work >> > > > >> everywhere. >> > > > >> >> > > > >> >> > > > >> On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park < >> > [email protected]> >> > > > >> wrote: >> > > > >> >> > > > >>> To be specific, we will need to revert all the following >> commits in >> > > > >> order: >> > > > >>> >> > > > >>> >> > > > >>> commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982 >> > > > >>> Author: Cheolsoo Park <[email protected]> >> > > > >>> Date: Fri Sep 20 22:47:29 2013 +0000 >> > > > >>> >> > > > >>> PIG-3471: Add a base abstract class for ExecutionEngine >> > (cheolsoo) >> > > > >>> >> > > > >>> git-svn-id: >> > > > >>> >> > > > >>> >> > > > >> >> > > > >> > > >> > >> https://svn.apache.org/repos/asf/pig/trunk@152516513f79535-47bb-0310-9956-ffa450edef68 >> > > > >>> >> > > > >>> commit 4305a6f4737d07396ae13fd95d7c1da7933b38a1 >> > > > >>> Author: Jianyong Dai <[email protected]> >> > > > >>> Date: Wed Sep 18 19:09:49 2013 +0000 >> > > > >>> >> > > > >>> PIG-3457: Provide backward compatibility for PigStatsUtil and >> > > > >> JobStats >> > > > >>> >> > > > >>> git-svn-id: >> > > > >>> >> > > > >>> >> > > > >> >> > > > >> > > >> > >> https://svn.apache.org/repos/asf/pig/trunk@152453213f79535-47bb-0310-9956-ffa450edef68 >> > > > >>> >> > > > >>> commit e85cf34c92713aa697a1cda7a9c2b3db139350f7 >> > > > >>> Author: Cheolsoo Park <[email protected]> >> > > > >>> Date: Wed Sep 18 15:37:58 2013 +0000 >> > > > >>> >> > > > >>> PIG-3464: Mark ExecType and ExecutionEngine interfaces as >> > evolving >> > > > >>> (cheolsoo) >> > > > >>> >> > > > >>> commit fd8b7cdf9292b305f02386d560c25298ab492a0b >> > > > >>> Author: Cheolsoo Park <[email protected]> >> > > > >>> Date: Fri Aug 30 20:04:29 2013 +0000 >> > > > >>> >> > > > >>> PIG-3419: Pluggable Execution Engine (achalsoni81 via >> cheolsoo) >> > > > >>> >> > > > >>> git-svn-id: >> > > > >>> >> > > > >>> >> > > > >> >> > > > >> > > >> > >> https://svn.apache.org/repos/asf/pig/trunk@151906213f79535-47bb-0310-9956-ffa450edef68 >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> On Mon, Sep 30, 2013 at 10:33 AM, Daniel Dai < >> > [email protected]> >> > > > >>> wrote: >> > > > >>> >> > > > >>>> Thanks Cheolsoo! My opinion is provide backward compatibility >> for >> > > > >>> PigStats >> > > > >>>> is a must, otherwise it could be a havoc. I imagine PigStats is >> > > widely >> > > > >>> used >> > > > >>>> by Pig users via PigRunner and PPNL interface. People use >> PigStats >> > > to >> > > > >>>> collect MR job details of the Pig job. Though PigStats is >> marked >> > for >> > > > >>>> Evolving, this is mostly for extending PigStats, not limiting >> > > PigStats >> > > > >> as >> > > > >>>> PIG-3419 did. Even if we really need to change, we need to very >> > well >> > > > >>>> communicate with users over time, Pig 0.12 is not an option. >> > > > >>>> >> > > > >>>> PIG-3457 is trying to provide a backward compatibility way for >> > > > >> PigStats, >> > > > >>>> but just like Cheolsoo said, it is far from ideal. I now tend >> to >> > > agree >> > > > >>>> Rohini's suggestion on PIG-3419, rollback PIG-3419, until we >> find >> > a >> > > > >>> better >> > > > >>>> way. Seems PIG-3419 is a little premature. Besides the above >> > > mentioned >> > > > >>>> PigStats issue, I've already found 2 additional issues: >> > > > >>>> 1. "explain" shows unoptimized logical plan instead of >> optimized >> > one >> > > > >>>> 2. HangingJobKiller is removed >> > > > >>>> >> > > > >>>> How does others think? >> > > > >>>> >> > > > >>>> Thanks, >> > > > >>>> Daniel >> > > > >>>> >> > > > >>>> >> > > > >>>> >> > > > >>>> >> > > > >>>> On Mon, Sep 30, 2013 at 9:51 AM, Cheolsoo Park < >> > > [email protected]> >> > > > >>>> wrote: >> > > > >>>> >> > > > >>>>> Hi devs, >> > > > >>>>> >> > > > >>>>> PIG-3419 <https://issues.apache.org/jira/browse/PIG-3419> >> broke >> > > > >>> backward >> > > > >>>>> compatibility for downstream applications such as Oozie, and >> > > > >>>>> PIG-3457<https://issues.apache.org/jira/browse/PIG-3457> is >> > > > >>>>> trying to fix it. >> > > > >>>>> >> > > > >>>>> In summary, we need to keep the old MR-specific JobStats and >> > > PigStats >> > > > >>> for >> > > > >>>>> backward compatibility sadly because they're currently >> exposed in >> > > > >>> several >> > > > >>>>> user-facing API including PigRunner, PigServer, etc. However, >> > this >> > > > >>>> defeats >> > > > >>>>> the purpose of PIG-3419 >> > > > >>>>> <https://issues.apache.org/jira/browse/PIG-3419> because >> > > > >>>>> we cannot implement non-MR execution engines in the back-end >> if >> > the >> > > > >>>>> front-end API is tied to MR. >> > > > >>>>> >> > > > >>>>> Daniel and I were having a discussion in the jira, but it >> seems >> > > more >> > > > >>>>> complicated than I thought. I am wondering whether anyone has >> a >> > > good >> > > > >>>>> suggestion on how to solve it. This is blocking the 0.12 >> release. >> > > > >>>>> >> > > > >>>>> Thanks, >> > > > >>>>> Cheolsoo >> > > > >>>>> >> > > > >>>> >> > > > >>>> -- >> > > > >>>> CONFIDENTIALITY NOTICE >> > > > >>>> NOTICE: This message is intended for the use of the individual >> or >> > > > >> entity >> > > > >>> to >> > > > >>>> which it is addressed and may contain information that is >> > > > confidential, >> > > > >>>> privileged and exempt from disclosure under applicable law. If >> the >> > > > >> reader >> > > > >>>> of this message is not the intended recipient, you are hereby >> > > notified >> > > > >>> that >> > > > >>>> any printing, copying, dissemination, distribution, disclosure >> or >> > > > >>>> forwarding of this communication is strictly prohibited. If you >> > have >> > > > >>>> received this communication in error, please contact the sender >> > > > >>> immediately >> > > > >>>> and delete it from your system. Thank You. >> > > > >>>> >> > > > >>> >> > > > >> >> > > > >> >> > > > >> >> > > > >> -- >> > > > >> >> > > > >> Jeremy Karn / Lead Developer >> > > > >> MORTAR DATA / 519 277 4391 / www.mortardata.com >> > > > >> >> > > > >> > > > >> > > > -- >> > > > CONFIDENTIALITY NOTICE >> > > > NOTICE: This message is intended for the use of the individual or >> > entity >> > > to >> > > > which it is addressed and may contain information that is >> confidential, >> > > > privileged and exempt from disclosure under applicable law. If the >> > reader >> > > > of this message is not the intended recipient, you are hereby >> notified >> > > that >> > > > any printing, copying, dissemination, distribution, disclosure or >> > > > forwarding of this communication is strictly prohibited. If you have >> > > > received this communication in error, please contact the sender >> > > immediately >> > > > and delete it from your system. Thank You. >> > > > >> > > >> > >> >> >> >> -- >> "...:::Aniket:::... Quetzalco@tl" >> > >
