To close the loop, here is the current status of issues that PIG-3419 introduced-
1) Backward incompatibility: Daniel provided a clever solution (PIG-3520). In summary, JobStats and PigStatsUtil will keep MR-specific methods as abstract methods. All the sub-classes that extend them (e.g. SimpleJobStats, TezJobStats, etc) must implement those methods. Non-MR classes will simply throw a "not implemented" exception for those methods. 2) HangingJobKiller: PIG-3524 brings it back to the launcher. Pending review. 3) Explain shows unoptimized plan: PIG-3508 keeps track of the issue. To be fixed. 4) PigStats.get() and ScriptState.get() return MR-specific objects: PIG-3525 keeps track of the issue. To be fixed. Given the current status, I think we can keep PIG-3419 in trunk. Please let me know if anyone thinks otherwise. Thanks, Cheolsoo On Tue, Oct 15, 2013 at 1:54 PM, Daniel Dai <[email protected]> wrote: > I still want to try a little bit. Give me one more day. > > Thanks, > Daniel > > > On Tue, Oct 15, 2013 at 12:40 PM, Jeremy Karn <[email protected]> > wrote: > > > I'm +1 on reverting PIG-3419 on trunk and then trying to redo it in a > > better way. Either way it would be good to get this resolved soon. > > > > > > On Mon, Oct 7, 2013 at 5:49 PM, Daniel Dai <[email protected]> > wrote: > > > > > Let me first revert PIG-3457, that complicates things. I will try to > > > explore for a solution in next the few days. But if we cannot settle > > > quickly, I would suggest to revert PIG-3419 on trunk first, and redo it > > > later in a better way. Otherwise, it will be very hard to revert. > > > > > > Thanks, > > > Daniel > > > > > > > > > On Mon, Oct 7, 2013 at 12:05 PM, Cheolsoo Park <[email protected]> > > > wrote: > > > > > > > Hi devs, > > > > > > > > This is a follow-up discussion about how to resolve the backward > > > > incompatibility of PIG-3419 (Pluggable execution engine). Per the > > > previous > > > > discussion <http://search-hadoop.com/m/wYz6hz9CoE>, I reverted it in > > > 0.12 > > > > but kept it in trunk. As we keep committing more changes into trunk, > it > > > > gets harder to back out PIG-3419 cleanly. So I suggest we should > make a > > > > decision sooner rather than later. > > > > > > > > The crux of the problem is as follows: > > > > > > > > PIG-3419 removes all the MR-specific things from JobStats. However, > > > > PigRunner and PigServer returns PigStats that in turn exposes > JobStats > > to > > > > end users. So changing JobStats breaks backward compatibility for > > > > downstream projects such as Oozie. While changing the semantics of > > > JobStats > > > > is acceptable, we must provide a deprecation path so that end users > can > > > > upgrade their applications smoothly. > > > > > > > > Proposed solutions: > > > > > > > > 1) Provide backward compatibility in source code: Maybe possible, but > > no > > > > one has come up with a clean solution. For eg, I failed. :( > > > > > > > > 2) Publish two jars: We keep PIG-3419 only in tez-branch and publish > > two > > > > jars (one for old API and one for new API) for a couple of future > > > releases. > > > > If we do this, we're going to revert PIG-3419 in trunk. > > > > > > > > What do you think? > > > > > > > > Thanks, > > > > Cheolsoo > > > > > > > > > > -- > > > CONFIDENTIALITY NOTICE > > > NOTICE: This message is intended for the use of the individual or > entity > > to > > > which it is addressed and may contain information that is confidential, > > > privileged and exempt from disclosure under applicable law. If the > reader > > > of this message is not the intended recipient, you are hereby notified > > that > > > any printing, copying, dissemination, distribution, disclosure or > > > forwarding of this communication is strictly prohibited. If you have > > > received this communication in error, please contact the sender > > immediately > > > and delete it from your system. Thank You. > > > > > > > > > > > -- > > > > Jeremy Karn / Lead Developer > > MORTAR DATA / 519 277 4391 / www.mortardata.com > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
