I looked into one NPE on CSV. Turns out it is already fixed by previous 1.11 PRs, so we’re good.
There is another reported by a user on the user list, but I’ve not heard back about getting sample data. Let’s not hold the release for that. - Paul > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <[email protected]> > wrote: > > *Blockers* > DRILL-5659 - Parth (done) > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days) > > *Possible blockers* > NPE on CSV - Paul can you please provide an update? > > Kind regards > Arina > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva < > [email protected]> wrote: > >> *Blockers* >> DRILL-5659 - problem is identified, Parth is working on the fix. >> >> *Not blockers* >> DRILL-5660 - Vitalii >> DRILL-5468 - Jinfeng >> >> *Possible blockers* >> DRILL-5669 - Boaz Ben-Zvi / Paul >> NPE on CSV - Paul >> >> Kind regards >> Arina >> >> >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <[email protected]> wrote: >> >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us to >>> track down the incompatibility the issue internally, but production users >>> probably don’t attempt to use one build of Drill with metadata files >>> created from another. We can worry about the issue later only if a user >>> actually runs into it. >>> >>> As suggested earlier, let’s just document this obscure case in a release >>> note. >>> >>> - Paul >>> >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <[email protected]> wrote: >>>> >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet metadata >>> cache >>>> file is involved. >>>> >>>> I might be wrong, but I thought that if someone is running on 1.10.0, >>> and >>>> he wants to go back to 1.0.0, where the first version of parquet >>> meatadata >>>> cache was introduced, can we guarantee that Drillbit would work >>> correctly? >>>> I'm trying to understand what DRILL-5660 asks for is a new requirement, >>> or >>>> an requirement being there starting from Drill 1.0.0? >>>> >>>> Regression failures are normally regarded as release blocker. >>>> >>>> >>>> >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <[email protected]> wrote: >>>> >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at all, >>> so >>>>> that is a time savings. We could handle this via a release note item >>> that >>>>> simply states that an upgrade from 1.10 to 1.11 is irreversible: we do >>> not >>>>> support the ability to fall back to the old version. Everyone OK with >>> that? >>>>> >>>>> We are seeing some stress test regression failures that, I’m told, will >>>>> show up as JIRA entries shortly. >>>>> >>>>> - Paul >>>>> >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <[email protected]> wrote: >>>>>> >>>>>> I put some analysis in DRILL-5468. >>>>>> >>>>>> Also, IMHO, DRILL-5660 is not a release blocker. >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <[email protected]> >>>>> wrote: >>>>>> >>>>>>> Re DRILL-5634: >>>>>>> From the Apache commons web page [1] - >>>>>>> >>>>>>> Please note that Apache Commons Crypto doesn't implement the >>>>> cryptographic >>>>>>> algorithm such as AES directly.* It wraps to Openssl or JCE which >>>>> implement >>>>>>> the algorithms.* >>>>>>> >>>>>>> >>>>>>> Since we are cleared for using Openssl and JCE, we are OK to include >>> the >>>>>>> AES code in the UDFs. >>>>>>> >>>>>>> >>>>>>> Parth >>>>>>> >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/ >>>>>>> >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> During today hangouts we agreed on approximate cut-off date - 14 >>> July, >>>>>>> 2017 >>>>>>>> (date may be shifted due to the blockers). >>>>>>>> I'll start preparing test RC on Thursday to solve any release >>>>> preparation >>>>>>>> issues beforehand. >>>>>>>> >>>>>>>> *Blockers* >>>>>>>> DRILL-5660 - Vitalii >>>>>>>> >>>>>>>> *Possible blockers* >>>>>>>> DRILL-5468 - Jinfeng >>>>>>>> NPE on CSV source - Paul >>>>>>>> DRILL-5659 - Parth >>>>>>>> >>>>>>>> *Will be included if done before the cut-off date* >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons implementation >>>>> and >>>>>>>> update. >>>>>>>> >>>>>>>> Kind regards >>>>>>>> Arina >>>>>>>> >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <[email protected]> >>>>> wrote: >>>>>>>> >>>>>>>>> Hi Arina, >>>>>>>>> I can’t make the hangout, but we have a few options. Basically, >>> since >>>>>>>> all >>>>>>>>> the hashing algorithms have no export restrictions, we could: >>>>>>>>> 1. Remove the AES encrypt/decrypt until I can figure out what we >>> have >>>>>>> to >>>>>>>>> do (and do it) >>>>>>>>> 2. I can try to research it today or tomorrow to figure out >>> exactly >>>>>>> what >>>>>>>>> we have to do to include AES. >>>>>>>>> 3. Remove the entire package >>>>>>>>> >>>>>>>>> I’d like to see the hashing functions at least included because >>> we’ve >>>>>>> had >>>>>>>>> a lot of requests for that functionality. I also don’t want this >>> to >>>>>>>> hold >>>>>>>>> things up, so please let me know your preference. Regardless, I’m >>>>>>> going >>>>>>>> to >>>>>>>>> start trying to figure out the restrictions so we’ll know for the >>>>>>> future. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> — C >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva < >>>>>>>> [email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> @Jinfeng, >>>>>>>>>> Let's discuss today on Drill hangout final cut-off date. >>>>>>>>>> >>>>>>>>>> @Charles >>>>>>>>>> Since there were questions about AES in DRILL-5634, should we >>> exclude >>>>>>>> it >>>>>>>>>> from the candidates for 1.11.0 release? >>>>>>>>>> >>>>>>>>>> @Boaz >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass code >>>>>>>> review? >>>>>>>>>> >>>>>>>>>> Also I believe we should include DRILL-5660 where we need to bump >>> up >>>>>>>>>> parquet metadata version. >>>>>>>>>> @Vitalii >>>>>>>>>> Do you have any ETA for the fix? >>>>>>>>>> >>>>>>>>>> Kind regards >>>>>>>>>> Arina >>>>>>>>>> >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <[email protected]> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Arina, >>>>>>>>>>> >>>>>>>>>>> Two PR have just been submitted, relating to the Hash Aggr >>>>>>> Spill: >>>>>>>>>>> DRILL-5616 and DRILL-5665 . >>>>>>>>>>> Can these be considered for 1.11 as well ? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Boaz >>>>>>>>>>> >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll defer that >>>>>>> until >>>>>>>>>>> after the release to allow others more time to work on other >>>>>>> features >>>>>>>>>>> rather than doing code reviews. >>>>>>>>>>> >>>>>>>>>>> - Paul >>>>>>>>>>> >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <[email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Arina, >>>>>>>>>>>> >>>>>>>>>>>> It has been a while since we discussed 1.11.0 release. Any >>> update >>>>>>>>>>> on the >>>>>>>>>>>> new release plan? Do we have a cut-off date ? >>>>>>>>>>>> >>>>>>>>>>>> In the past, I believe Drill release is a time-bounded, not >>> feature >>>>>>>>>>>> bounded. If certain fixes/features could not make it within a >>>>>>> certain >>>>>>>>>>>> time-frame, we had better defer them into future release, unless >>>>>>>>>>> those >>>>>>>>>>>> fixes/features are regarded as blocking issues for a new >>> release. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva < >>>>>>>>>>> [email protected] >>>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Latest update: >>>>>>>>>>>>> >>>>>>>>>>>>> We wait for the following Jiras before final cut off. >>>>>>>>>>>>> 1. DRILL-5601 - Paul >>>>>>>>>>>>> 2. DRILL-5420 - Kunal >>>>>>>>>>>>> 3. DRILL-5634 - Charles >>>>>>>>>>>>> >>>>>>>>>>>>> Kind regards >>>>>>>>>>>>> Arina >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua < >>> [email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Parth's already +1'd the process with a minor update. So, >>> we're >>>>>>>>>>> good to >>>>>>>>>>>>> go >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to >>> ready-to-commit. >>>>>>>>>>>>>> >>>>>>>>>>>>>> <http://www.mapr.com/> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ________________________________ >>>>>>>>>>>>>> From: Parth Chandra <[email protected]> >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM >>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are also >>>>>>>>>>> addressed. >>>>>>>>>>>>> If >>>>>>>>>>>>>> the changes are done before your cutoff deadline,we can merge >>> it >>>>>>>>>>> in. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> here is the list of Jira we are waiting before making final >>> cut >>>>>>>>>>> off: >>>>>>>>>>>>>>> 1. DRILL-5601 - Paul >>>>>>>>>>>>>>> 2. DRILL-3640 - Kunal >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please let me know if there are any others Jiras we should >>> wait >>>>>>>>>>> for. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> @Charles, >>>>>>>>>>>>>>> you have mentioned that you want to include some UDFs in >>> Drill >>>>>>>>>>> 1.11, do >>>>>>>>>>>>>> you >>>>>>>>>>>>>>> have any ETA for the PR? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> @Paul / Parth >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do we want to >>>>>>>>>>> include it >>>>>>>>>>>>> in >>>>>>>>>>>>>>> this release as well? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kind regards >>>>>>>>>>>>>>> Arina >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra < >>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from gists written >>> by >>>>>>>>>>>>> previous >>>>>>>>>>>>>>>>> release managers into an updated gist here [1] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> HTH >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Parth >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill- >>>>>>>>>>>>> Release-Process >>>>>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill- >>>>>>>>>>> Release-Process> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers < >>>>>>> [email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Arina, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a candidate for >>> 1.11, >>>>>>>>>>> the >>>>>>>>>>>>>>> other >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>> not. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The “managed external sort” is the good candidate for >>> 1.11: >>>>>>>>>>> The PR >>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will allow >>> us >>>>>>> to >>>>>>>>>>>>>> enable >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> “new” external sort by default. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory >>> fragmentation >>>>>>>>>>> issue >>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open for a >>>>>>> while. >>>>>>>>>>>>> They >>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to review. So, >>> the >>>>>>>>>>>>> project >>>>>>>>>>>>>>> does >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release >>> instead. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> - Paul >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>> >>> >>
