Thanks, Boaz! Unfortunately, I still see the failures connected to DRILL-5669. I'll share the details.
Kind regards Arina On Wed, Jul 19, 2017 at 2:00 AM, Boaz Ben-Zvi <[email protected]> wrote: > Hi Arina, > > The three Jira’s are ready (+1, squashed, rebased, etc.): > > • DRILL-5616: Add memory checks, plus minor metrics changes > (Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a ) > • DRILL-5665: Add the option planner.force_2phase_aggr to override small > inputs > (Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e ) > • DRILL-5669: Add a configurable option for minimum memory allocation to > buffered ops > (Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 ) > > Thanks, > > Boaz > > On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <[email protected]> > wrote: > > Hi Parth, > > no problem, I did not start preparing the release since there is one > blocker has left: > > *DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)* > > > Kind regards > Arina > > On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <[email protected]> > wrote: > > > Hi Arina, > > > > I just realized that I did not push the C++ client fix ( > DRILL-5659) to > > master. I have just done so. > > > > Sorry if I messed things up for the release, but we should get > this fix > > in. > > > > Parth > > > > On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <[email protected]> > wrote: > > > > > I looked into one NPE on CSV. Turns out it is already fixed by > previous > > > 1.11 PRs, so we’re good. > > > > > > There is another reported by a user on the user list, but I’ve not > heard > > > back about getting sample data. Let’s not hold the release for > that. > > > > > > - Paul > > > > > > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva < > > > [email protected]> wrote: > > > > > > > > *Blockers* > > > > DRILL-5659 - Parth (done) > > > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days) > > > > > > > > *Possible blockers* > > > > NPE on CSV - Paul can you please provide an update? > > > > > > > > Kind regards > > > > Arina > > > > > > > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva < > > > > [email protected]> wrote: > > > > > > > >> *Blockers* > > > >> DRILL-5659 - problem is identified, Parth is working on the fix. > > > >> > > > >> *Not blockers* > > > >> DRILL-5660 - Vitalii > > > >> DRILL-5468 - Jinfeng > > > >> > > > >> *Possible blockers* > > > >> DRILL-5669 - Boaz Ben-Zvi / Paul > > > >> NPE on CSV - Paul > > > >> > > > >> Kind regards > > > >> Arina > > > >> > > > >> > > > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <[email protected]> > > wrote: > > > >> > > > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle > for us > > to > > > >>> track down the incompatibility the issue internally, but > production > > > users > > > >>> probably don’t attempt to use one build of Drill with metadata > files > > > >>> created from another. We can worry about the issue later only > if a > > user > > > >>> actually runs into it. > > > >>> > > > >>> As suggested earlier, let’s just document this obscure case in > a > > > release > > > >>> note. > > > >>> > > > >>> - Paul > > > >>> > > > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <[email protected]> > wrote: > > > >>>> > > > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet > metadata > > > >>> cache > > > >>>> file is involved. > > > >>>> > > > >>>> I might be wrong, but I thought that if someone is running on > > 1.10.0, > > > >>> and > > > >>>> he wants to go back to 1.0.0, where the first version of > parquet > > > >>> meatadata > > > >>>> cache was introduced, can we guarantee that Drillbit would > work > > > >>> correctly? > > > >>>> I'm trying to understand what DRILL-5660 asks for is a new > > > requirement, > > > >>> or > > > >>>> an requirement being there starting from Drill 1.0.0? > > > >>>> > > > >>>> Regression failures are normally regarded as release blocker. > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers < > [email protected]> > > > wrote: > > > >>>> > > > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do > it at > > > all, > > > >>> so > > > >>>>> that is a time savings. We could handle this via a release > note > > item > > > >>> that > > > >>>>> simply states that an upgrade from 1.10 to 1.11 is > irreversible: we > > > do > > > >>> not > > > >>>>> support the ability to fall back to the old version. > Everyone OK > > with > > > >>> that? > > > >>>>> > > > >>>>> We are seeing some stress test regression failures that, I’m > told, > > > will > > > >>>>> show up as JIRA entries shortly. > > > >>>>> > > > >>>>> - Paul > > > >>>>> > > > >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <[email protected]> > wrote: > > > >>>>>> > > > >>>>>> I put some analysis in DRILL-5468. > > > >>>>>> > > > >>>>>> Also, IMHO, DRILL-5660 is not a release blocker. > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra < > > [email protected]> > > > >>>>> wrote: > > > >>>>>> > > > >>>>>>> Re DRILL-5634: > > > >>>>>>> From the Apache commons web page [1] - > > > >>>>>>> > > > >>>>>>> Please note that Apache Commons Crypto doesn't implement > the > > > >>>>> cryptographic > > > >>>>>>> algorithm such as AES directly.* It wraps to Openssl or > JCE which > > > >>>>> implement > > > >>>>>>> the algorithms.* > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> Since we are cleared for using Openssl and JCE, we are OK > to > > > include > > > >>> the > > > >>>>>>> AES code in the UDFs. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> Parth > > > >>>>>>> > > > >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/ > > > >>>>>>> > > > >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva < > > > >>>>>>> [email protected]> wrote: > > > >>>>>>> > > > >>>>>>>> During today hangouts we agreed on approximate cut-off > date - 14 > > > >>> July, > > > >>>>>>> 2017 > > > >>>>>>>> (date may be shifted due to the blockers). > > > >>>>>>>> I'll start preparing test RC on Thursday to solve any > release > > > >>>>> preparation > > > >>>>>>>> issues beforehand. > > > >>>>>>>> > > > >>>>>>>> *Blockers* > > > >>>>>>>> DRILL-5660 - Vitalii > > > >>>>>>>> > > > >>>>>>>> *Possible blockers* > > > >>>>>>>> DRILL-5468 - Jinfeng > > > >>>>>>>> NPE on CSV source - Paul > > > >>>>>>>> DRILL-5659 - Parth > > > >>>>>>>> > > > >>>>>>>> *Will be included if done before the cut-off date* > > > >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz > > > >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons > > > implementation > > > >>>>> and > > > >>>>>>>> update. > > > >>>>>>>> > > > >>>>>>>> Kind regards > > > >>>>>>>> Arina > > > >>>>>>>> > > > >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre < > > [email protected]> > > > >>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> Hi Arina, > > > >>>>>>>>> I can’t make the hangout, but we have a few options. > > Basically, > > > >>> since > > > >>>>>>>> all > > > >>>>>>>>> the hashing algorithms have no export restrictions, we > could: > > > >>>>>>>>> 1. Remove the AES encrypt/decrypt until I can figure > out what > > we > > > >>> have > > > >>>>>>> to > > > >>>>>>>>> do (and do it) > > > >>>>>>>>> 2. I can try to research it today or tomorrow to figure > out > > > >>> exactly > > > >>>>>>> what > > > >>>>>>>>> we have to do to include AES. > > > >>>>>>>>> 3. Remove the entire package > > > >>>>>>>>> > > > >>>>>>>>> I’d like to see the hashing functions at least included > because > > > >>> we’ve > > > >>>>>>> had > > > >>>>>>>>> a lot of requests for that functionality. I also don’t > want > > > this > > > >>> to > > > >>>>>>>> hold > > > >>>>>>>>> things up, so please let me know your preference. > Regardless, > > > I’m > > > >>>>>>> going > > > >>>>>>>> to > > > >>>>>>>>> start trying to figure out the restrictions so we’ll > know for > > the > > > >>>>>>> future. > > > >>>>>>>>> > > > >>>>>>>>> Best, > > > >>>>>>>>> — C > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva < > > > >>>>>>>> [email protected]> > > > >>>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>> @Jinfeng, > > > >>>>>>>>>> Let's discuss today on Drill hangout final cut-off date. > > > >>>>>>>>>> > > > >>>>>>>>>> @Charles > > > >>>>>>>>>> Since there were questions about AES in DRILL-5634, > should we > > > >>> exclude > > > >>>>>>>> it > > > >>>>>>>>>> from the candidates for 1.11.0 release? > > > >>>>>>>>>> > > > >>>>>>>>>> @Boaz > > > >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will > pass > > code > > > >>>>>>>> review? > > > >>>>>>>>>> > > > >>>>>>>>>> Also I believe we should include DRILL-5660 where we > need to > > > bump > > > >>> up > > > >>>>>>>>>> parquet metadata version. > > > >>>>>>>>>> @Vitalii > > > >>>>>>>>>> Do you have any ETA for the fix? > > > >>>>>>>>>> > > > >>>>>>>>>> Kind regards > > > >>>>>>>>>> Arina > > > >>>>>>>>>> > > > >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi < > > > [email protected]> > > > >>>>>>>> wrote: > > > >>>>>>>>>> > > > >>>>>>>>>>> Hi Arina, > > > >>>>>>>>>>> > > > >>>>>>>>>>> Two PR have just been submitted, relating to the Hash > Aggr > > > >>>>>>> Spill: > > > >>>>>>>>>>> DRILL-5616 and DRILL-5665 . > > > >>>>>>>>>>> Can these be considered for 1.11 as well ? > > > >>>>>>>>>>> > > > >>>>>>>>>>> Thanks, > > > >>>>>>>>>>> > > > >>>>>>>>>>> Boaz > > > >>>>>>>>>>> > > > >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <[email protected]> > wrote: > > > >>>>>>>>>>> > > > >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll > defer that > > > >>>>>>> until > > > >>>>>>>>>>> after the release to allow others more time to work on > other > > > >>>>>>> features > > > >>>>>>>>>>> rather than doing code reviews. > > > >>>>>>>>>>> > > > >>>>>>>>>>> - Paul > > > >>>>>>>>>>> > > > >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni < > [email protected]> > > > wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Hi Arina, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> It has been a while since we discussed 1.11.0 > release. Any > > > >>> update > > > >>>>>>>>>>> on the > > > >>>>>>>>>>>> new release plan? Do we have a cut-off date ? > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> In the past, I believe Drill release is a > time-bounded, not > > > >>> feature > > > >>>>>>>>>>>> bounded. If certain fixes/features could not make it > within > > a > > > >>>>>>> certain > > > >>>>>>>>>>>> time-frame, we had better defer them into future > release, > > > unless > > > >>>>>>>>>>> those > > > >>>>>>>>>>>> fixes/features are regarded as blocking issues for a > new > > > >>> release. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva < > > > >>>>>>>>>>> [email protected] > > > >>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>>> Latest update: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> We wait for the following Jiras before final cut off. > > > >>>>>>>>>>>>> 1. DRILL-5601 - Paul > > > >>>>>>>>>>>>> 2. DRILL-5420 - Kunal > > > >>>>>>>>>>>>> 3. DRILL-5634 - Charles > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> Kind regards > > > >>>>>>>>>>>>> Arina > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua < > > > >>> [email protected]> > > > >>>>>>>>>>> wrote: > > > >>>>>>>>>>>>> > > > >>>>>>>>>>>>>> Parth's already +1'd the process with a minor > update. So, > > > >>> we're > > > >>>>>>>>>>> good to > > > >>>>>>>>>>>>> go > > > >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to > > > >>> ready-to-commit. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> <http://www.mapr.com/> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> ________________________________ > > > >>>>>>>>>>>>>> From: Parth Chandra <[email protected]> > > > >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM > > > >>>>>>>>>>>>>> To: [email protected] > > > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls > comments are > > > also > > > >>>>>>>>>>> addressed. > > > >>>>>>>>>>>>> If > > > >>>>>>>>>>>>>> the changes are done before your cutoff deadline,we > can > > > merge > > > >>> it > > > >>>>>>>>>>> in. > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva < > > > >>>>>>>>>>>>>> [email protected]> wrote: > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Hi all, > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> here is the list of Jira we are waiting before > making > > final > > > >>> cut > > > >>>>>>>>>>> off: > > > >>>>>>>>>>>>>>> 1. DRILL-5601 - Paul > > > >>>>>>>>>>>>>>> 2. DRILL-3640 - Kunal > > > >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Please let me know if there are any others Jiras we > > should > > > >>> wait > > > >>>>>>>>>>> for. > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> @Charles, > > > >>>>>>>>>>>>>>> you have mentioned that you want to include some > UDFs in > > > >>> Drill > > > >>>>>>>>>>> 1.11, do > > > >>>>>>>>>>>>>> you > > > >>>>>>>>>>>>>>> have any ETA for the PR? > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> @Paul / Parth > > > >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do we > want > > to > > > >>>>>>>>>>> include it > > > >>>>>>>>>>>>> in > > > >>>>>>>>>>>>>>> this release as well? > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> Kind regards > > > >>>>>>>>>>>>>>> Arina > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva > < > > > >>>>>>>>>>>>>>> [email protected]> wrote: > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions. > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra < > > > >>>>>>>>>>> [email protected]> > > > >>>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from > gists > > > written > > > >>> by > > > >>>>>>>>>>>>> previous > > > >>>>>>>>>>>>>>>>> release managers into an updated gist here [1] > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> HTH > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> Parth > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> [1] https://github.com/ > parthchandra/drill/wiki/Drill- > > > >>>>>>>>>>>>> Release-Process > > > >>>>>>>>>>>>>>>>> <https://github.com/ > parthchandra/drill/wiki/Drill- > > > >>>>>>>>>>> Release-Process> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers < > > > >>>>>>> [email protected]> > > > >>>>>>>>>>>>>> wrote: > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Hi Arina, > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a > candidate > > for > > > >>> 1.11, > > > >>>>>>>>>>> the > > > >>>>>>>>>>>>>>> other > > > >>>>>>>>>>>>>>>>> is > > > >>>>>>>>>>>>>>>>>> not. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> The “managed external sort” is the good > candidate for > > > >>> 1.11: > > > >>>>>>>>>>> The PR > > > >>>>>>>>>>>>>> for > > > >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in > will > > allow > > > >>> us > > > >>>>>>> to > > > >>>>>>>>>>>>>> enable > > > >>>>>>>>>>>>>>>>> the > > > >>>>>>>>>>>>>>>>>> “new” external sort by default. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory > > > >>> fragmentation > > > >>>>>>>>>>> issue > > > >>>>>>>>>>>>>>> with > > > >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been > open > > for a > > > >>>>>>> while. > > > >>>>>>>>>>>>> They > > > >>>>>>>>>>>>>>> are > > > >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to > review. > > So, > > > >>> the > > > >>>>>>>>>>>>> project > > > >>>>>>>>>>>>>>> does > > > >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next > release > > > >>> instead. > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> Thanks, > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> - Paul > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>>> > > > >>>>>>>>>>>>>> > > > >>>>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>> > > > >>>>>>> > > > >>>>> > > > >>>>> > > > >>> > > > >>> > > > >> > > > > > > > > > > >
