sounds good to me too Julien On Mar 1, 2013, at 11:33 AM, Bill Graham wrote:
> +1 to releasing Pig 0.11.1 when this is addressed. I should be able to help > with the release again. > > > > On Fri, Mar 1, 2013 at 11:25 AM, Prashant Kommireddi > <prash1...@gmail.com>wrote: > >> Hey Guys, >> >> I wanted to start a conversation on this again. If Kai is not looking at >> PIG-3194 I can start working on it to get 0.11 compatible with 20.2. If >> everyone agrees, we should roll out 0.11.1 sooner than usual and I >> volunteer to help with it in anyway possible. >> >> Any objections to getting 0.11.1 out soon after 3194 is fixed? >> >> -Prashant >> >> On Wed, Feb 20, 2013 at 3:34 PM, Russell Jurney <russell.jur...@gmail.com >>> wrote: >> >>> I stand corrected. Cool, 0.11 is good! >>> >>> >>> On Wed, Feb 20, 2013 at 1:15 PM, Jarek Jarcec Cecho <jar...@apache.org >>>> wrote: >>> >>>> Just a unrelated note: The CDH3 is more closer to Hadoop 1.x than to >>> 0.20. >>>> >>>> Jarcec >>>> >>>> On Wed, Feb 20, 2013 at 12:04:51PM -0800, Dmitriy Ryaboy wrote: >>>>> I agree -- this is a good release. The bugs Kai pointed out should be >>>>> fixed, but as they are not critical regressions, we can fix them in >>>> 0.11.1 >>>>> (if someone wants to roll 0.11.1 the minute these fixes are >> committed, >>> I >>>>> won't mind and will dutifully vote for the release). >>>>> >>>>> I think the Hadoop 20.2 incompatibility is unfortunate but iirc this >> is >>>>> fixable by setting HADOOP_USER_CLASSPATH_FIRST=true (was that in >> 20.2?) >>>>> >>>>> FWIW Twitter's running CDH3 and this release works in our >> environment. >>>>> >>>>> At this point things that block a release are critical regressions in >>>>> performance or correctness. >>>>> >>>>> D >>>>> >>>>> >>>>> On Wed, Feb 20, 2013 at 11:52 AM, Alan Gates <ga...@hortonworks.com> >>>> wrote: >>>>> >>>>>> No. Bugs like these are supposed to be found and fixed after we >>> branch >>>>>> from trunk (which happened several months ago in the case of 0.11). >>>> The >>>>>> point of RCs are to check that it's a good build, licenses are >> right, >>>> etc. >>>>>> Any bugs found this late in the game have to be seen as failures >> of >>>>>> earlier testing. >>>>>> >>>>>> Alan. >>>>>> >>>>>> On Feb 20, 2013, at 11:33 AM, Russell Jurney wrote: >>>>>> >>>>>>> Isn't the point of an RC to find and fix bugs like these> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 20, 2013 at 11:31 AM, Bill Graham < >>> billgra...@gmail.com> >>>>>> wrote: >>>>>>> >>>>>>>> Regarding Pig 11 rc2, I propose we continue with the current >> vote >>>> as is >>>>>>>> (which closes today EOD). Patches for 0.20.2 issues can be >> rolled >>>> into a >>>>>>>> Pig 0.11.1 release whenever they're available and tested. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Feb 20, 2013 at 9:24 AM, Olga Natkovich < >>>> onatkov...@yahoo.com >>>>>>>>> wrote: >>>>>>>> >>>>>>>>> I agree that supporting as much as we can is a good goal. The >>>> issue is >>>>>>>> who >>>>>>>>> is going to be testing against all these versions? We found the >>>> issues >>>>>>>>> under discussion because of a customer report, not because we >>>>>>>> consistently >>>>>>>>> test against all versions. Perhaps when we decide which >> versions >>> to >>>>>>>> support >>>>>>>>> for next release we need also to agree who is going to be >> testing >>>> and >>>>>>>>> maintaining compatibility with a particular version. >>>>>>>>> >>>>>>>>> For instance since Hadoop 23 compatibility is important for us >> at >>>> Yahoo >>>>>>>> we >>>>>>>>> have been maintaining compatibility with this version for 0.9, >>>> 0.10 and >>>>>>>>> will do the same for 0.11 and going forward. I think we would >>> need >>>>>> others >>>>>>>>> to step in and claim the versions of their interest. >>>>>>>>> >>>>>>>>> Olga >>>>>>>>> >>>>>>>>> >>>>>>>>> ________________________________ >>>>>>>>> From: Kai Londenberg <kai.londenb...@googlemail.com> >>>>>>>>> To: dev@pig.apache.org >>>>>>>>> Sent: Wednesday, February 20, 2013 1:51 AM >>>>>>>>> Subject: Re: pig 0.11 candidate 2 feedback: Several problems >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I stronly agree with Jonathan here. If there are good reasons >> why >>>> you >>>>>>>>> can't support an older version of Hadoop any more, that's one >>>> thing. >>>>>>>>> But having to change 2 lines of code doesn't really qualify as >>>> such in >>>>>>>>> my point of view ;) >>>>>>>>> >>>>>>>>> At least for me, pig support for 0.20.2 is essential - without >>> it, >>>> I >>>>>>>>> can't use it. If it doesn't support it, I'll have to branch pig >>> and >>>>>>>>> hack it myself, or stop using it. >>>>>>>>> >>>>>>>>> I guess, there are a lot of people still running 0.20.2 >> Clusters. >>>> If >>>>>>>>> you really have lots of data stored on HDFS and a continuously >>> busy >>>>>>>>> cluster, an upgrade is nothing you do "just because". >>>>>>>>> >>>>>>>>> >>>>>>>>> 2013/2/20 Jonathan Coveney <jcove...@gmail.com>: >>>>>>>>>> I agree that we shouldn't have to support old versions >> forever. >>>> That >>>>>>>>> said, >>>>>>>>>> I also don't think we should be too blase about supporting >> older >>>>>>>> versions >>>>>>>>>> where it is not odious to do so. We have a lot of competition >> in >>>> the >>>>>>>>>> language space and the broader the versions we can support, >> the >>>> better >>>>>>>>>> (assuming it isn't too odious to do so). In this case, I don't >>>> think >>>>>> it >>>>>>>>>> should be too hard to change ObjectSerializer so that the >>>>>> commons-codec >>>>>>>>>> code used is compatible with both versions...we could just >>> in-line >>>>>> some >>>>>>>>> of >>>>>>>>>> the Base64 code, and comment accordingly. >>>>>>>>>> >>>>>>>>>> That said, we also should be clear about what versions we >>>> support, but >>>>>>>>> 6-12 >>>>>>>>>> months seems short. The upgrade cycles on Hadoop are really, >>>> really >>>>>>>> long. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2013/2/20 Prashant Kommireddi <prash1...@gmail.com> >>>>>>>>>> >>>>>>>>>>> Agreed, that makes sense. Probably supporting older hadoop >>>> version >>>>>> for >>>>>>>>> a 1 >>>>>>>>>>> or 2 pig releases before moving to a newer/stable version? >>>>>>>>>>> >>>>>>>>>>> Having said that, should we use 0.11 period to communicate >> the >>>> same >>>>>> to >>>>>>>>> the >>>>>>>>>>> community and start moving on 0.12 onwards? I know we are way >>>> past >>>>>>>> 6-12 >>>>>>>>>>> months (1-2 release) time frame with 0.20.2, but we also need >>> to >>>> make >>>>>>>>> sure >>>>>>>>>>> users are aware and plan accordingly. >>>>>>>>>>> >>>>>>>>>>> I'd also be interested to hear how other projects (Hive, >> Oozie) >>>> are >>>>>>>>>>> handling this. >>>>>>>>>>> >>>>>>>>>>> -Prashant >>>>>>>>>>> >>>>>>>>>>> On Tue, Feb 19, 2013 at 3:22 PM, Olga Natkovich < >>>>>> onatkov...@yahoo.com >>>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> It seems that for each Pig release we need to agree and >>> clearly >>>>>>>> state >>>>>>>>>>>> which Hadoop versions it will support. I guess the main >>>> question is >>>>>>>>> how >>>>>>>>>>> we >>>>>>>>>>>> decide on this. Perhaps we should say that Pig no longer >>>> supports >>>>>>>>> older >>>>>>>>>>>> Hadoop versions once the newer one is out for at least 6-12 >>>> month to >>>>>>>>> make >>>>>>>>>>>> sure it is stable. I don't think we can support old versions >>>>>>>>>>> indefinitely. >>>>>>>>>>>> It is in everybody's interest to keep moving forward. >>>>>>>>>>>> >>>>>>>>>>>> Olga >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ________________________________ >>>>>>>>>>>> From: Prashant Kommireddi <prash1...@gmail.com> >>>>>>>>>>>> To: dev@pig.apache.org >>>>>>>>>>>> Sent: Tuesday, February 19, 2013 10:57 AM >>>>>>>>>>>> Subject: Re: pig 0.11 candidate 2 feedback: Several problems >>>>>>>>>>>> >>>>>>>>>>>> What do you guys feel about the JIRA to do with 0.20.2 >>>> compatibility >>>>>>>>>>>> (PIG-3194)? I am interested in discussing the strategy >> around >>>>>>>> backward >>>>>>>>>>>> compatibility as this is something that would haunt us each >>>> time we >>>>>>>>> move >>>>>>>>>>> to >>>>>>>>>>>> the next hadoop version. For eg, we might be in a similar >>>> situation >>>>>>>>> while >>>>>>>>>>>> moving to Hadoop 2.0, when some of the stuff might break for >>>> 1.0. >>>>>>>>>>>> >>>>>>>>>>>> I feel it would be good to get this JIRA fix in for 0.11, as >>>> 0.20.2 >>>>>>>>> users >>>>>>>>>>>> might be caught unaware. Of course, I must admit there is >>>> selfish >>>>>>>>>>> interest >>>>>>>>>>>> here and it's probably easier for us to have a workaround on >>> Pig >>>>>>>>> rather >>>>>>>>>>>> than upgrade hadoop in all our production DCs. >>>>>>>>>>>> >>>>>>>>>>>> -Prashant >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Feb 19, 2013 at 9:54 AM, Russell Jurney < >>>>>>>>>>> russell.jur...@gmail.com >>>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I think someone should step up and fix the easy ones, if >>>> possible. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Feb 19, 2013 at 9:51 AM, Bill Graham < >>>>>>>> billgra...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks Kai for reporting these. >>>>>>>>>>>>>> >>>>>>>>>>>>>> What do people think about the severity of these issues >>> w.r.t. >>>>>>>> Pig >>>>>>>>>>> 11? >>>>>>>>>>>> I >>>>>>>>>>>>>> see a few possible options: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. We include some or all of these patches in a new Pig 11 >>> rc. >>>>>>>>> We'd >>>>>>>>>>>> want >>>>>>>>>>>>> to >>>>>>>>>>>>>> make sure that they don't destabilize the current branch. >>> This >>>>>>>>>>> approach >>>>>>>>>>>>>> makes sense if we think Pig 11 wouldn't be a good release >>>>>>>> without >>>>>>>>> one >>>>>>>>>>>> or >>>>>>>>>>>>>> more of these included. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. We continue with the Pig 11 release without these, but >>> then >>>>>>>>>>> include >>>>>>>>>>>>> one >>>>>>>>>>>>>> or more in a 0.11.1 release. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3. We continue with the Pig 11 release without these, but >>> then >>>>>>>>>>> include >>>>>>>>>>>>> them >>>>>>>>>>>>>> in a 0.12 release. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Jon has a patch for the MAP issue >>>>>>>>>>>>>> (PIG-3144<https://issues.apache.org/jira/browse/PIG-3144 >>> ) >>>>>>>>>>>>>> ready, which seems like the most pressing of the three to >>> me. >>>>>>>>>>>>>> >>>>>>>>>>>>>> thanks, >>>>>>>>>>>>>> Bill >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Feb 18, 2013 at 2:27 AM, Kai Londenberg < >>>>>>>>>>>>>> kai.londenb...@googlemail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I just subscribed to the dev mailing list in order to >> give >>>> you >>>>>>>>> some >>>>>>>>>>>>>>> feedback on pig 0.11 candidate 2. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The following three issues are currently present in 0.11 >>>>>>>>> candidate >>>>>>>>>>> 2: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/PIG-3144 - >>> 'Erroneous >>>>>>>> map >>>>>>>>>>>> entry >>>>>>>>>>>>>>> alias resolution leading to "Duplicate schema alias" >>> errors' >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/PIG-3194 - Changes >>> to >>>>>>>>>>>>>>> ObjectSerializer.java break compatibility with Hadoop >>> 0.20.2 >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/PIG-3195 - Race >>>>>>>>> Condition in >>>>>>>>>>>>>>> PhysicalOperator leads to ExecException "Error while >> trying >>>> to >>>>>>>>> get >>>>>>>>>>>>>>> next result in POStream" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The last two of these are easily solveable (see the >> tickets >>>>>>>> for >>>>>>>>>>>>>>> details on that). The first one is a bit trickier I >> think, >>>> but >>>>>>>>> at >>>>>>>>>>>>>>> least there is a workaround for it (pass Map fields >> through >>>> an >>>>>>>>> UDF) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In my personal opinion, each of these problems is pretty >>>>>>>> severe, >>>>>>>>>>> but >>>>>>>>>>>>>>> opinions about the importance of the MAP Datatype and >>> STREAM >>>>>>>>>>>> Operator, >>>>>>>>>>>>>>> as well as Hadoop 0.20.2 compatibility might differ. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> so far .. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kai Londenberg >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> *Note that I'm no longer using my Yahoo! email address. >>> Please >>>>>>>>> email >>>>>>>>>>> me >>>>>>>>>>>>> at >>>>>>>>>>>>>> billgra...@gmail.com going forward.* >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Russell Jurney twitter.com/rjurney >> russell.jur...@gmail.com >>>>>>>>>>>>> datasyndrome.com >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Note that I'm no longer using my Yahoo! email address. Please >>>> email me >>>>>> at >>>>>>>> billgra...@gmail.com going forward.* >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>>>>> datasyndrome.com >>>>>> >>>>>> >>>> >>> >>> >>> >>> -- >>> Russell Jurney twitter.com/rjurney russell.jur...@gmail.com >>> datasyndrome.com >>> >>
