Hi Arina,
The three Jira’s are ready (+1, squashed, rebased, etc.):
• DRILL-5616: Add memory checks, plus minor metrics changes
(Commit ID: c7ad56b7e075e84761896c9ce660b18b09d9e49a )
• DRILL-5665: Add the option planner.force_2phase_aggr to override small inputs
(Commit ID: 1917ec084bdef74962370fbe5b253172c634c86e )
• DRILL-5669: Add a configurable option for minimum memory allocation to
buffered ops
(Commit ID: 4d7a3fe3e7ae52db57b0fd1f0619c6d1d64308c8 )
Thanks,
Boaz
On 7/18/17, 3:04 AM, "Arina Yelchiyeva" <[email protected]> wrote:
Hi Parth,
no problem, I did not start preparing the release since there is one
blocker has left:
*DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)*
Kind regards
Arina
On Mon, Jul 17, 2017 at 11:26 PM, Parth Chandra <[email protected]> wrote:
> Hi Arina,
>
> I just realized that I did not push the C++ client fix ( DRILL-5659) to
> master. I have just done so.
>
> Sorry if I messed things up for the release, but we should get this fix
> in.
>
> Parth
>
> On Fri, Jul 14, 2017 at 8:15 AM, Paul Rogers <[email protected]> wrote:
>
> > I looked into one NPE on CSV. Turns out it is already fixed by previous
> > 1.11 PRs, so we’re good.
> >
> > There is another reported by a user on the user list, but I’ve not heard
> > back about getting sample data. Let’s not hold the release for that.
> >
> > - Paul
> >
> > > On Jul 14, 2017, at 2:38 AM, Arina Yelchiyeva <
> > [email protected]> wrote:
> > >
> > > *Blockers*
> > > DRILL-5659 - Parth (done)
> > > DRILL-5669 - Boaz Ben-Zvi (fix will be ready in a couple of days)
> > >
> > > *Possible blockers*
> > > NPE on CSV - Paul can you please provide an update?
> > >
> > > Kind regards
> > > Arina
> > >
> > > On Thu, Jul 13, 2017 at 2:46 PM, Arina Yelchiyeva <
> > > [email protected]> wrote:
> > >
> > >> *Blockers*
> > >> DRILL-5659 - problem is identified, Parth is working on the fix.
> > >>
> > >> *Not blockers*
> > >> DRILL-5660 - Vitalii
> > >> DRILL-5468 - Jinfeng
> > >>
> > >> *Possible blockers*
> > >> DRILL-5669 - Boaz Ben-Zvi / Paul
> > >> NPE on CSV - Paul
> > >>
> > >> Kind regards
> > >> Arina
> > >>
> > >>
> > >> On Thu, Jul 13, 2017 at 1:28 AM, Paul Rogers <[email protected]>
> wrote:
> > >>
> > >>> I withdraw my suggestion to fix DRILL-5660. Was a big hassle for us
> to
> > >>> track down the incompatibility the issue internally, but production
> > users
> > >>> probably don’t attempt to use one build of Drill with metadata files
> > >>> created from another. We can worry about the issue later only if a
> user
> > >>> actually runs into it.
> > >>>
> > >>> As suggested earlier, let’s just document this obscure case in a
> > release
> > >>> note.
> > >>>
> > >>> - Paul
> > >>>
> > >>>> On Jul 12, 2017, at 3:04 PM, Jinfeng Ni <[email protected]> wrote:
> > >>>>
> > >>>> Upgrade from 1.10 to 1.11 is irreversible only when parquet
metadata
> > >>> cache
> > >>>> file is involved.
> > >>>>
> > >>>> I might be wrong, but I thought that if someone is running on
> 1.10.0,
> > >>> and
> > >>>> he wants to go back to 1.0.0, where the first version of parquet
> > >>> meatadata
> > >>>> cache was introduced, can we guarantee that Drillbit would work
> > >>> correctly?
> > >>>> I'm trying to understand what DRILL-5660 asks for is a new
> > requirement,
> > >>> or
> > >>>> an requirement being there starting from Drill 1.0.0?
> > >>>>
> > >>>> Regression failures are normally regarded as release blocker.
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Wed, Jul 12, 2017 at 2:54 PM, Paul Rogers <[email protected]>
> > wrote:
> > >>>>
> > >>>>> If we don’t do DRILL-5660 for 1.11 then we don’t need to do it at
> > all,
> > >>> so
> > >>>>> that is a time savings. We could handle this via a release note
> item
> > >>> that
> > >>>>> simply states that an upgrade from 1.10 to 1.11 is irreversible:
we
> > do
> > >>> not
> > >>>>> support the ability to fall back to the old version. Everyone OK
> with
> > >>> that?
> > >>>>>
> > >>>>> We are seeing some stress test regression failures that, I’m told,
> > will
> > >>>>> show up as JIRA entries shortly.
> > >>>>>
> > >>>>> - Paul
> > >>>>>
> > >>>>>> On Jul 12, 2017, at 11:05 AM, Jinfeng Ni <[email protected]> wrote:
> > >>>>>>
> > >>>>>> I put some analysis in DRILL-5468.
> > >>>>>>
> > >>>>>> Also, IMHO, DRILL-5660 is not a release blocker.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Tue, Jul 11, 2017 at 11:27 AM, Parth Chandra <
> [email protected]>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>>> Re DRILL-5634:
> > >>>>>>> From the Apache commons web page [1] -
> > >>>>>>>
> > >>>>>>> Please note that Apache Commons Crypto doesn't implement the
> > >>>>> cryptographic
> > >>>>>>> algorithm such as AES directly.* It wraps to Openssl or JCE
which
> > >>>>> implement
> > >>>>>>> the algorithms.*
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Since we are cleared for using Openssl and JCE, we are OK to
> > include
> > >>> the
> > >>>>>>> AES code in the UDFs.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Parth
> > >>>>>>>
> > >>>>>>> [1] https://commons.apache.org/proper/commons-crypto/
> > >>>>>>>
> > >>>>>>> On Tue, Jul 11, 2017 at 10:56 AM, Arina Yelchiyeva <
> > >>>>>>> [email protected]> wrote:
> > >>>>>>>
> > >>>>>>>> During today hangouts we agreed on approximate cut-off date -
14
> > >>> July,
> > >>>>>>> 2017
> > >>>>>>>> (date may be shifted due to the blockers).
> > >>>>>>>> I'll start preparing test RC on Thursday to solve any release
> > >>>>> preparation
> > >>>>>>>> issues beforehand.
> > >>>>>>>>
> > >>>>>>>> *Blockers*
> > >>>>>>>> DRILL-5660 - Vitalii
> > >>>>>>>>
> > >>>>>>>> *Possible blockers*
> > >>>>>>>> DRILL-5468 - Jinfeng
> > >>>>>>>> NPE on CSV source - Paul
> > >>>>>>>> DRILL-5659 - Parth
> > >>>>>>>>
> > >>>>>>>> *Will be included if done before the cut-off date*
> > >>>>>>>> DRILL-5616 / DRILL-5665 - Boaz
> > >>>>>>>> DRILL-5634 - Parth will take a look at Apache commons
> > implementation
> > >>>>> and
> > >>>>>>>> update.
> > >>>>>>>>
> > >>>>>>>> Kind regards
> > >>>>>>>> Arina
> > >>>>>>>>
> > >>>>>>>> On Tue, Jul 11, 2017 at 3:14 PM, Charles Givre <
> [email protected]>
> > >>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi Arina,
> > >>>>>>>>> I can’t make the hangout, but we have a few options.
> Basically,
> > >>> since
> > >>>>>>>> all
> > >>>>>>>>> the hashing algorithms have no export restrictions, we could:
> > >>>>>>>>> 1. Remove the AES encrypt/decrypt until I can figure out what
> we
> > >>> have
> > >>>>>>> to
> > >>>>>>>>> do (and do it)
> > >>>>>>>>> 2. I can try to research it today or tomorrow to figure out
> > >>> exactly
> > >>>>>>> what
> > >>>>>>>>> we have to do to include AES.
> > >>>>>>>>> 3. Remove the entire package
> > >>>>>>>>>
> > >>>>>>>>> I’d like to see the hashing functions at least included
because
> > >>> we’ve
> > >>>>>>> had
> > >>>>>>>>> a lot of requests for that functionality. I also don’t want
> > this
> > >>> to
> > >>>>>>>> hold
> > >>>>>>>>> things up, so please let me know your preference. Regardless,
> > I’m
> > >>>>>>> going
> > >>>>>>>> to
> > >>>>>>>>> start trying to figure out the restrictions so we’ll know for
> the
> > >>>>>>> future.
> > >>>>>>>>>
> > >>>>>>>>> Best,
> > >>>>>>>>> — C
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> On Jul 11, 2017, at 05:58, Arina Yelchiyeva <
> > >>>>>>>> [email protected]>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> @Jinfeng,
> > >>>>>>>>>> Let's discuss today on Drill hangout final cut-off date.
> > >>>>>>>>>>
> > >>>>>>>>>> @Charles
> > >>>>>>>>>> Since there were questions about AES in DRILL-5634, should we
> > >>> exclude
> > >>>>>>>> it
> > >>>>>>>>>> from the candidates for 1.11.0 release?
> > >>>>>>>>>>
> > >>>>>>>>>> @Boaz
> > >>>>>>>>>> Do we have any ETA when DRILL-5616 and DRILL-5665 will pass
> code
> > >>>>>>>> review?
> > >>>>>>>>>>
> > >>>>>>>>>> Also I believe we should include DRILL-5660 where we need to
> > bump
> > >>> up
> > >>>>>>>>>> parquet metadata version.
> > >>>>>>>>>> @Vitalii
> > >>>>>>>>>> Do you have any ETA for the fix?
> > >>>>>>>>>>
> > >>>>>>>>>> Kind regards
> > >>>>>>>>>> Arina
> > >>>>>>>>>>
> > >>>>>>>>>> On Tue, Jul 11, 2017 at 6:02 AM, Boaz Ben-Zvi <
> > [email protected]>
> > >>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Hi Arina,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Two PR have just been submitted, relating to the Hash Aggr
> > >>>>>>> Spill:
> > >>>>>>>>>>> DRILL-5616 and DRILL-5665 .
> > >>>>>>>>>>> Can these be considered for 1.11 as well ?
> > >>>>>>>>>>>
> > >>>>>>>>>>> Thanks,
> > >>>>>>>>>>>
> > >>>>>>>>>>> Boaz
> > >>>>>>>>>>>
> > >>>>>>>>>>> On 7/10/17, 1:49 PM, "Paul Rogers" <[email protected]> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> DRILL-5601 is no longer a blocker for 1.11; we’ll defer that
> > >>>>>>> until
> > >>>>>>>>>>> after the release to allow others more time to work on other
> > >>>>>>> features
> > >>>>>>>>>>> rather than doing code reviews.
> > >>>>>>>>>>>
> > >>>>>>>>>>> - Paul
> > >>>>>>>>>>>
> > >>>>>>>>>>>> On Jul 10, 2017, at 1:43 PM, Jinfeng Ni <[email protected]>
> > wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Hi Arina,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> It has been a while since we discussed 1.11.0 release. Any
> > >>> update
> > >>>>>>>>>>> on the
> > >>>>>>>>>>>> new release plan? Do we have a cut-off date ?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> In the past, I believe Drill release is a time-bounded, not
> > >>> feature
> > >>>>>>>>>>>> bounded. If certain fixes/features could not make it within
> a
> > >>>>>>> certain
> > >>>>>>>>>>>> time-frame, we had better defer them into future release,
> > unless
> > >>>>>>>>>>> those
> > >>>>>>>>>>>> fixes/features are regarded as blocking issues for a new
> > >>> release.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Tue, Jul 4, 2017 at 5:14 AM, Arina Yelchiyeva <
> > >>>>>>>>>>> [email protected]
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>> Latest update:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> We wait for the following Jiras before final cut off.
> > >>>>>>>>>>>>> 1. DRILL-5601 - Paul
> > >>>>>>>>>>>>> 2. DRILL-5420 - Kunal
> > >>>>>>>>>>>>> 3. DRILL-5634 - Charles
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Kind regards
> > >>>>>>>>>>>>> Arina
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Fri, Jun 30, 2017 at 9:29 PM, Kunal Khatua <
> > >>> [email protected]>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Parth's already +1'd the process with a minor update. So,
> > >>> we're
> > >>>>>>>>>>> good to
> > >>>>>>>>>>>>> go
> > >>>>>>>>>>>>>> on DRILL-5420. I'll change the label on that to
> > >>> ready-to-commit.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> <http://www.mapr.com/>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ________________________________
> > >>>>>>>>>>>>>> From: Parth Chandra <[email protected]>
> > >>>>>>>>>>>>>> Sent: Thursday, June 29, 2017 9:44:39 AM
> > >>>>>>>>>>>>>> To: [email protected]
> > >>>>>>>>>>>>>> Subject: Re: [DISCUSS] Drill 1.11.0 release
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> I'm Ok with the pcap format plugin if Pauls comments are
> > also
> > >>>>>>>>>>> addressed.
> > >>>>>>>>>>>>> If
> > >>>>>>>>>>>>>> the changes are done before your cutoff deadline,we can
> > merge
> > >>> it
> > >>>>>>>>>>> in.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 12:26 AM, Arina Yelchiyeva <
> > >>>>>>>>>>>>>> [email protected]> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Hi all,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> here is the list of Jira we are waiting before making
> final
> > >>> cut
> > >>>>>>>>>>> off:
> > >>>>>>>>>>>>>>> 1. DRILL-5601 - Paul
> > >>>>>>>>>>>>>>> 2. DRILL-3640 - Kunal
> > >>>>>>>>>>>>>>> 3. DRILL-5420 - Kunal
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Please let me know if there are any others Jiras we
> should
> > >>> wait
> > >>>>>>>>>>> for.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> @Charles,
> > >>>>>>>>>>>>>>> you have mentioned that you want to include some UDFs in
> > >>> Drill
> > >>>>>>>>>>> 1.11, do
> > >>>>>>>>>>>>>> you
> > >>>>>>>>>>>>>>> have any ETA for the PR?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> @Paul / Parth
> > >>>>>>>>>>>>>>> I see there is active CR for pcap format PR, do we want
> to
> > >>>>>>>>>>> include it
> > >>>>>>>>>>>>> in
> > >>>>>>>>>>>>>>> this release as well?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Kind regards
> > >>>>>>>>>>>>>>> Arina
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On Thu, Jun 29, 2017 at 10:05 AM, Arina Yelchiyeva <
> > >>>>>>>>>>>>>>> [email protected]> wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Parth, thanks a lot the instructions.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> On Wed, Jun 28, 2017 at 8:24 PM, Parth Chandra <
> > >>>>>>>>>>> [email protected]>
> > >>>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> FWIW, I put release instructions culled from gists
> > written
> > >>> by
> > >>>>>>>>>>>>> previous
> > >>>>>>>>>>>>>>>>> release managers into an updated gist here [1]
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> HTH
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> Parth
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> [1] https://github.com/parthchandra/drill/wiki/Drill-
> > >>>>>>>>>>>>> Release-Process
> > >>>>>>>>>>>>>>>>> <https://github.com/parthchandra/drill/wiki/Drill-
> > >>>>>>>>>>> Release-Process>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>> On Tue, Jun 27, 2017 at 5:24 PM, Paul Rogers <
> > >>>>>>> [email protected]>
> > >>>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Hi Arina,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> I have two projects in progress: one is a candidate
> for
> > >>> 1.11,
> > >>>>>>>>>>> the
> > >>>>>>>>>>>>>>> other
> > >>>>>>>>>>>>>>>>> is
> > >>>>>>>>>>>>>>>>>> not.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The “managed external sort” is the good candidate for
> > >>> 1.11:
> > >>>>>>>>>>> The PR
> > >>>>>>>>>>>>>> for
> > >>>>>>>>>>>>>>>>>> DRILL-5601 is being reviewed. Getting that in will
> allow
> > >>> us
> > >>>>>>> to
> > >>>>>>>>>>>>>> enable
> > >>>>>>>>>>>>>>>>> the
> > >>>>>>>>>>>>>>>>>> “new” external sort by default.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> The other project is to address Drill’s memory
> > >>> fragmentation
> > >>>>>>>>>>> issue
> > >>>>>>>>>>>>>>> with
> > >>>>>>>>>>>>>>>>>> vectors over 16 MB in size. Two PRs have been open
> for a
> > >>>>>>> while.
> > >>>>>>>>>>>>> They
> > >>>>>>>>>>>>>>> are
> > >>>>>>>>>>>>>>>>>> the first of several. Each takes a while to review.
> So,
> > >>> the
> > >>>>>>>>>>>>> project
> > >>>>>>>>>>>>>>> does
> > >>>>>>>>>>>>>>>>>> not fit 1.11. The work can go into the next release
> > >>> instead.
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>> - Paul
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>
> > >>>
> > >>
> >
> >
>
