The S3 python bindings would be a nice addition to the release. I don't think we should block on this but the PR is ready. Opinions? https://github.com/apache/arrow/pull/5423
On Wed, Sep 25, 2019 at 5:28 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > OK, I'll start the process today. I'll send up e-mail updates as I make > progress. > > On Wed, Sep 25, 2019 at 8:22 AM Wes McKinney <wesmck...@gmail.com> wrote: > >> Yes, all systems go as far as I'm concerned. >> >> On Wed, Sep 25, 2019 at 9:56 AM Neal Richardson >> <neal.p.richard...@gmail.com> wrote: >> > >> > Andy's DataFusion issue and Wes's Parquet one have both been merged, >> > and it looks like the LICENSE issue is being resolved as I type. So >> > are we good to go now? >> > >> > Neal >> > >> > >> > On Tue, Sep 24, 2019 at 10:30 PM Andy Grove <andygrov...@gmail.com> >> wrote: >> > > >> > > I found a last minute issue with DataFusion (Rust) and would >> appreciate it >> > > if we could merge ARROW-6086 (PR is >> > > https://github.com/apache/arrow/pull/5494) before cutting the RC. >> > > >> > > Thanks, >> > > >> > > Andy. >> > > >> > > >> > > On Tue, Sep 24, 2019 at 6:19 PM Micah Kornfield < >> emkornfi...@gmail.com> >> > > wrote: >> > > >> > > > OK, I'm going to postpone cutting a release until tomorrow (hoping >> we can >> > > > issues resolved by then).. I'll also try to review the third-party >> > > > additions since 14.x. >> > > > >> > > > On Tue, Sep 24, 2019 at 4:20 PM Wes McKinney <wesmck...@gmail.com> >> wrote: >> > > > >> > > > > I found a licensing issue >> > > > > >> > > > > https://issues.apache.org/jira/browse/ARROW-6679 >> > > > > >> > > > > It might be worth examining third party code added to the project >> > > > > since 0.14.x to make sure there are no other such issues. >> > > > > >> > > > > On Tue, Sep 24, 2019 at 6:10 PM Wes McKinney <wesmck...@gmail.com >> > >> > > > wrote: >> > > > > > >> > > > > > I have diagnosed the problem (Thrift "string" data must be >> UTF-8, >> > > > > > cannot be arbitrary binary) and am working on a patch right now >> > > > > > >> > > > > > On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney < >> wesmck...@gmail.com> >> > > > > wrote: >> > > > > > > >> > > > > > > I just opened >> > > > > > > >> > > > > > > https://issues.apache.org/jira/browse/ARROW-6678 >> > > > > > > >> > > > > > > Please don't cut an RC until I have an opportunity to >> diagnose this, >> > > > > > > will report back. >> > > > > > > >> > > > > > > >> > > > > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney < >> wesmck...@gmail.com> >> > > > > wrote: >> > > > > > > > >> > > > > > > > I'm investigating a possible Parquet-related compatibility >> bug >> > > > that I >> > > > > > > > encountered through some routine testing / benchmarking. >> I'll >> > > > report >> > > > > > > > back once I figure out what is going on (if anything) >> > > > > > > > >> > > > > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield < >> > > > > emkornfi...@gmail.com> wrote: >> > > > > > > > >> >> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e. >> you can >> > > > > get it >> > > > > > > > >> signed by another PMC member), but is not 100% essential. >> > > > > > > > > >> > > > > > > > > That won't be an option for me this week (it seems like I >> would >> > > > > need to meet one face-to-face). I'll try to get the GPG checked >> in and >> > > > the >> > > > > rest of the pre-requisites done tomorrow (Monday) to hopefully >> start the >> > > > > release on Tuesday (hopefully we can solve the last >> blocker/integration >> > > > > tests by then). >> > > > > > > > > >> > > > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney < >> > > > wesmck...@gmail.com> >> > > > > wrote: >> > > > > > > > >> >> > > > > > > > >> It's ideal if your GPG key is in the web of trust (i.e. >> you can >> > > > > get it >> > > > > > > > >> signed by another PMC member), but is not 100% essential. >> > > > > > > > >> >> > > > > > > > >> Speaking of the release, there are at least 2 code >> changes I >> > > > still >> > > > > > > > >> want to get in >> > > > > > > > >> >> > > > > > > > >> ARROW-5717 >> > > > > > > > >> ARROW-6353 >> > > > > > > > >> >> > > > > > > > >> I just pushed updates to ARROW-5717, will merge once the >> build >> > > > is >> > > > > green. >> > > > > > > > >> >> > > > > > > > >> There are a couple of Rust patches still marked for >> 0.15. The >> > > > rest >> > > > > > > > >> seems to be documentation and a couple of integration >> test >> > > > > failures we >> > > > > > > > >> should see about fixing in time. >> > > > > > > > >> >> > > > > > > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield < >> > > > > emkornfi...@gmail.com> wrote: >> > > > > > > > >> > >> > > > > > > > >> > Thanks Krisztián and Wes, >> > > > > > > > >> > I've gone ahead and started registering myself on all >> the >> > > > > packaging sites. >> > > > > > > > >> > >> > > > > > > > >> > Is there any review process when adding my GPG key to >> the SVN >> > > > > file? [1] >> > > > > > > > >> > doesn't seem to mention explicitly. >> > > > > > > > >> > >> > > > > > > > >> > Thanks, >> > > > > > > > >> > Micah >> > > > > > > > >> > >> > > > > > > > >> > [1] >> https://www.apache.org/dev/version-control.html#https-svn >> > > > > > > > >> > >> > > > > > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs < >> > > > > szucs.kriszt...@gmail.com> >> > > > > > > > >> > wrote: >> > > > > > > > >> > >> > > > > > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney < >> > > > > wesmck...@gmail.com> wrote: >> > > > > > > > >> > > >> > > > > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield < >> > > > > emkornfi...@gmail.com> >> > > > > > > > >> > >> wrote: >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> The process should be well documented at this >> point but >> > > > > there are a >> > > > > > > > >> > >> >> number of steps. >> > > > > > > > >> > >> > >> > > > > > > > >> > >> > Is [1] the up-to-date documentation for the >> release? >> > > > Are >> > > > > there >> > > > > > > > >> > >> instructions for the adding the code signing Key to >> SVN? >> > > > > > > > >> > >> > >> > > > > > > > >> > >> > I will make a go of it. i will try to mitigate >> any >> > > > > internet issues by >> > > > > > > > >> > >> doing the process for a cloud instance (I assume >> that isn't >> > > > > a problem?). >> > > > > > > > >> > >> > >> > > > > > > > >> > >> >> > > > > > > > >> > >> Setting up a new cloud environment suitable for >> producing >> > > > an >> > > > > RC may be >> > > > > > > > >> > >> time consuming, but you are welcome to try. >> Krisztian -- >> > > > are >> > > > > you >> > > > > > > > >> > >> available next week to help Micah and potentially >> take over >> > > > > producing >> > > > > > > > >> > >> the RC if there are issues? >> > > > > > > > >> > >> >> > > > > > > > >> > > Sure, I'll be available next week. We can also grant >> access >> > > > to >> > > > > > > > >> > > https://github.com/ursa-labs/crossbow because >> configuring >> > > > all >> > > > > > > > >> > > the CI backends can be time consuming. >> > > > > > > > >> > > >> > > > > > > > >> > >> >> > > > > > > > >> > >> > Thanks, >> > > > > > > > >> > >> > Micah >> > > > > > > > >> > >> > >> > > > > > > > >> > >> > [1] >> > > > > > > > >> > >> >> > > > > >> > > > >> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide >> > > > > > > > >> > >> > >> > > > > > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney < >> > > > > wesmck...@gmail.com> >> > > > > > > > >> > >> wrote: >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> The process should be well documented at this >> point but >> > > > > there are a >> > > > > > > > >> > >> >> number of steps. Note that you need to add your >> code >> > > > > signing key to >> > > > > > > > >> > >> >> the KEYS file in SVN (that's not very hard to >> do). I >> > > > > think it's fine >> > > > > > > > >> > >> >> to hand off the process to others after the VOTE >> but it >> > > > > would be >> > > > > > > > >> > >> >> tricky to have multiple RMs involved with >> producing the >> > > > > source and >> > > > > > > > >> > >> >> binary artifacts for the vote >> > > > > > > > >> > >> >> >> > > > > > > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield >> < >> > > > > > > > >> > >> emkornfi...@gmail.com> wrote: >> > > > > > > > >> > >> >> > >> > > > > > > > >> > >> >> > SGTM, as well. >> > > > > > > > >> > >> >> > >> > > > > > > > >> > >> >> > I should have a little bit of time next week >> if I can >> > > > > help as RM but >> > > > > > > > >> > >> I have >> > > > > > > > >> > >> >> > a couple of concerns: >> > > > > > > > >> > >> >> > 1. In the past I've had trouble downloading >> and >> > > > > validating >> > > > > > > > >> > >> releases. I'm a >> > > > > > > > >> > >> >> > bit worried, that I might have similar >> problems doing >> > > > > the necessary >> > > > > > > > >> > >> uploads. >> > > > > > > > >> > >> >> > 2. My internet connection will likely be not >> great, I >> > > > > don't know if >> > > > > > > > >> > >> this >> > > > > > > > >> > >> >> > would make it even less likely to be >> successful. >> > > > > > > > >> > >> >> > >> > > > > > > > >> > >> >> > Does it become problematic if somehow I would >> have to >> > > > > abandon the >> > > > > > > > >> > >> process >> > > > > > > > >> > >> >> > mid-release? Is there anyone who could serve >> as a >> > > > > backup? Are the >> > > > > > > > >> > >> steps >> > > > > > > > >> > >> >> > well documented? >> > > > > > > > >> > >> >> > >> > > > > > > > >> > >> >> > Thanks, >> > > > > > > > >> > >> >> > Micah >> > > > > > > > >> > >> >> > >> > > > > > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal >> Richardson < >> > > > > > > > >> > >> neal.p.richard...@gmail.com> >> > > > > > > > >> > >> >> > wrote: >> > > > > > > > >> > >> >> > >> > > > > > > > >> > >> >> > > Sounds good to me. >> > > > > > > > >> > >> >> > > >> > > > > > > > >> > >> >> > > Do we have a release manager yet? Any >> volunteers? >> > > > > > > > >> > >> >> > > >> > > > > > > > >> > >> >> > > Neal >> > > > > > > > >> > >> >> > > >> > > > > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney >> < >> > > > > wesmck...@gmail.com> >> > > > > > > > >> > >> wrote: >> > > > > > > > >> > >> >> > > >> > > > > > > > >> > >> >> > > > hi all, >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > > It looks like we're drawing close to be >> able to >> > > > > make the 0.15.0 >> > > > > > > > >> > >> >> > > > release. I would suggest "pencils down" at >> the end >> > > > > of this week >> > > > > > > > >> > >> and >> > > > > > > > >> > >> >> > > > see if a release candidate can be produced >> next >> > > > > Monday September >> > > > > > > > >> > >> 23. >> > > > > > > > >> > >> >> > > > Any thoughts or objections? >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > > Thanks, >> > > > > > > > >> > >> >> > > > Wes >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes >> McKinney < >> > > > > > > > >> > >> wesmck...@gmail.com> >> > > > > > > > >> > >> >> > > wrote: >> > > > > > > > >> > >> >> > > > > >> > > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm >> planning to >> > > > > amend the >> > > > > > > > >> > >> Format docs >> > > > > > > > >> > >> >> > > > > today regarding the EOS issue and also >> update >> > > > the >> > > > > C++ library >> > > > > > > > >> > >> >> > > > > >> > > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric >> Erhardt >> > > > > > > > >> > >> >> > > > > <eric.erha...@microsoft.com> wrote: >> > > > > > > > >> > >> >> > > > > > >> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the >> > > > > > > > >> > >> ARROW-6313-flatbuffer-alignment >> > > > > > > > >> > >> >> > > > branch into master before the 0.15 release, >> > > > correct? >> > > > > > > > >> > >> >> > > > > > >> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment >> changes are >> > > > > ready to be >> > > > > > > > >> > >> merged into >> > > > > > > > >> > >> >> > > > the alignment branch - >> > > > > > > > >> > >> https://github.com/apache/arrow/pull/5280/ >> > > > > > > > >> > >> >> > > > > > >> > > > > > > > >> > >> >> > > > > > Eric >> > > > > > > > >> > >> >> > > > > > >> > > > > > > > >> > >> >> > > > > > -----Original Message----- >> > > > > > > > >> > >> >> > > > > > From: Micah Kornfield < >> emkornfi...@gmail.com> >> > > > > > > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 >> 10:24 PM >> > > > > > > > >> > >> >> > > > > > To: Wes McKinney <wesmck...@gmail.com> >> > > > > > > > >> > >> >> > > > > > Cc: dev <dev@arrow.apache.org>; >> niki.lj < >> > > > > niki...@aliyun.com> >> > > > > > > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 >> release >> > > > > > > > >> > >> >> > > > > > >> > > > > > > > >> > >> >> > > > > > I should have a little more bandwidth >> to help >> > > > > with some of >> > > > > > > > >> > >> the >> > > > > > > > >> > >> >> > > > packaging starting tomorrow and going into >> the >> > > > > weekend. >> > > > > > > > >> > >> >> > > > > > >> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes >> McKinney < >> > > > > > > > >> > >> wesmck...@gmail.com> >> > > > > > > > >> > >> >> > > > wrote: >> > > > > > > > >> > >> >> > > > > > >> > > > > > > > >> > >> >> > > > > > > Hi folks, >> > > > > > > > >> > >> >> > > > > > > >> > > > > > > > >> > >> >> > > > > > > With the state of nightly packaging >> and >> > > > > integration builds >> > > > > > > > >> > >> things >> > > > > > > > >> > >> >> > > > > > > aren't looking too good for being in >> release >> > > > > readiness by >> > > > > > > > >> > >> the end >> > > > > > > > >> > >> >> > > of >> > > > > > > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm >> planning >> > > > > to be working >> > > > > > > > >> > >> to close >> > > > > > > > >> > >> >> > > as >> > > > > > > > >> > >> >> > > > > > > many issues as I can and also to >> help with >> > > > > the ongoing >> > > > > > > > >> > >> alignment >> > > > > > > > >> > >> >> > > > fixes. >> > > > > > > > >> > >> >> > > > > > > >> > > > > > > > >> > >> >> > > > > > > Wes >> > > > > > > > >> > >> >> > > > > > > >> > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah >> > > > Kornfield >> > > > > < >> > > > > > > > >> > >> >> > > emkornfi...@gmail.com >> > > > > > > > >> > >> >> > > > > >> > > > > > > > >> > >> >> > > > > > > wrote: >> > > > > > > > >> > >> >> > > > > > > >> > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a >> dashboard of >> > > > > the current >> > > > > > > > >> > >> issues: >> > > > > > > > >> > >> >> > > > > > >> >> > > > > > > > >> > >> >> > > > > > >> >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > > > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi >> > > > > > > > >> > >> >> > > > > > >> ki.apache.org >> > > > > > > > >> > >> >> > > > >> > > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea >> > > > > > > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt% >> > > > > 40microsoft.com >> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034 >> > > > > > > > >> > >> >> > > > > > >> >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > > > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376 >> > > > > > > > >> > >> >> > > > > > >> >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > > > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4 >> > > > > > > > >> > >> >> > > > > > >> %3D&reserved=0 >> > > > > > > > >> > >> >> > > > > > >> >> > > > > > > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes >> > > > McKinney < >> > > > > > > > >> > >> wesmck...@gmail.com> >> > > > > > > > >> > >> >> > > > wrote: >> > > > > > > > >> > >> >> > > > > > >> >> > > > > > > > >> > >> >> > > > > > >>> hi all, >> > > > > > > > >> > >> >> > > > > > >>> >> > > > > > > > >> > >> >> > > > > > >>> It doesn't seem like we're going >> to be in >> > > > a >> > > > > position to >> > > > > > > > >> > >> release >> > > > > > > > >> > >> >> > > at >> > > > > > > > >> > >> >> > > > > > >>> the beginning of next week. I hope >> that >> > > > one >> > > > > more week of >> > > > > > > > >> > >> work (or >> > > > > > > > >> > >> >> > > > > > >>> less) will be enough to get us >> there. >> > > > Aside >> > > > > from merging >> > > > > > > > >> > >> the >> > > > > > > > >> > >> >> > > > > > >>> alignment changes, we need to make >> sure >> > > > > that our >> > > > > > > > >> > >> packaging jobs >> > > > > > > > >> > >> >> > > > > > >>> required for the release candidate >> are all >> > > > > working. >> > > > > > > > >> > >> >> > > > > > >>> >> > > > > > > > >> > >> >> > > > > > >>> If folks could remove issues from >> the >> > > > > 0.15.0 backlog >> > > > > > > > >> > >> that they >> > > > > > > > >> > >> >> > > > don't >> > > > > > > > >> > >> >> > > > > > >>> think they will finish by end of >> next week >> > > > > that would >> > > > > > > > >> > >> help focus >> > > > > > > > >> > >> >> > > > > > >>> efforts (there are currently 78 >> issues in >> > > > > 0.15.0 still). >> > > > > > > > >> > >> I am >> > > > > > > > >> > >> >> > > > > > >>> looking to tackle a few small >> features >> > > > > related to >> > > > > > > > >> > >> dictionaries >> > > > > > > > >> > >> >> > > > while >> > > > > > > > >> > >> >> > > > > > >>> the release window is still open. >> > > > > > > > >> > >> >> > > > > > >>> >> > > > > > > > >> > >> >> > > > > > >>> - Wes >> > > > > > > > >> > >> >> > > > > > >>> >> > > > > > > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes >> > > > > McKinney < >> > > > > > > > >> > >> >> > > wesmck...@gmail.com> >> > > > > > > > >> > >> >> > > > > > >>> wrote: >> > > > > > > > >> > >> >> > > > > > >>> > >> > > > > > > > >> > >> >> > > > > > >>> > hi, >> > > > > > > > >> > >> >> > > > > > >>> > >> > > > > > > > >> > >> >> > > > > > >>> > I think we should try to release >> the >> > > > week >> > > > > of September >> > > > > > > > >> > >> 9, so >> > > > > > > > >> > >> >> > > > > > >>> > development work should be >> completed by >> > > > > end of next >> > > > > > > > >> > >> week. >> > > > > > > > >> > >> >> > > > > > >>> > >> > > > > > > > >> > >> >> > > > > > >>> > Does that seem reasonable? >> > > > > > > > >> > >> >> > > > > > >>> > >> > > > > > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the >> > > > protocol >> > > > > alignment >> > > > > > > > >> > >> changes for >> > > > > > > > >> > >> >> > > > > > >>> > C++ in the next couple of days >> -- I >> > > > think >> > > > > that getting >> > > > > > > > >> > >> the >> > > > > > > > >> > >> >> > > > > > >>> > alignment work done is the main >> barrier >> > > > > to releasing. >> > > > > > > > >> > >> >> > > > > > >>> > >> > > > > > > > >> > >> >> > > > > > >>> > Thanks >> > > > > > > > >> > >> >> > > > > > >>> > Wes >> > > > > > > > >> > >> >> > > > > > >>> > >> > > > > > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM >> Ji Liu >> > > > > > > > >> > >> >> > > > > > >>> > <niki...@aliyun.com.invalid> >> > > > > > > > >> > >> >> > > > > > >>> wrote: >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I >> can think >> > > > > of several >> > > > > > > > >> > >> bugs that >> > > > > > > > >> > >> >> > > > need >> > > > > > > > >> > >> >> > > > > > >>> > > to >> > > > > > > > >> > >> >> > > > > > >>> be fixed or reminded. >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary >> entries are >> > > > > required in >> > > > > > > > >> > >> IPC streams >> > > > > > > > >> > >> >> > > > > > >>> > > even >> > > > > > > > >> > >> >> > > > > > >>> when empty[1] >> > > > > > > > >> > >> >> > > > > > >>> > > This one is under review now, >> however >> > > > > through this >> > > > > > > > >> > >> PR we find >> > > > > > > > >> > >> >> > > > > > >>> > > that >> > > > > > > > >> > >> >> > > > > > >>> there seems a bug in java reading >> and >> > > > > writing >> > > > > > > > >> > >> dictionaries in IPC >> > > > > > > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] >> since >> > > > it >> > > > > assumes all >> > > > > > > > >> > >> >> > > > dictionaries >> > > > > > > > >> > >> >> > > > > > >>> are at the start of stream (see >> details in >> > > > > PR comments, >> > > > > > > > >> > >> and this >> > > > > > > > >> > >> >> > > > > > >>> fix may not catch up with version >> 0.15). >> > > > > @Micah Kornfield >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit >> ints as >> > > > > strings in >> > > > > > > > >> > >> integration >> > > > > > > > >> > >> >> > > > test >> > > > > > > > >> > >> >> > > > > > >>> JSON files[3] >> > > > > > > > >> > >> >> > > > > > >>> > > Java side code already checked >> in, >> > > > other >> > > > > > > > >> > >> implementations >> > > > > > > > >> > >> >> > > seems >> > > > > > > > >> > >> >> > > > not. >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in >> > > > > JdbcAdapter[4] >> > > > > > > > >> > >> Caused by >> > > > > > > > >> > >> >> > > trying >> > > > > > > > >> > >> >> > > > > > >>> > > to load all records in one >> contiguous >> > > > > batch, fixed >> > > > > > > > >> > >> >> > > > > > >>> by providing iterator API for >> iteratively >> > > > > reading in >> > > > > > > > >> > >> >> > > ARROW-6219[5]. >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > Thanks, >> > > > > > > > >> > >> >> > > > > > >>> > > Ji Liu >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > [1] >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE >> > > > > > > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com >> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7 >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D& >> > > > > > > > >> > >> >> > > > > > >>> > > reserved=0 [2] >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% >> > > > > > > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org >> > > > > > > > >> > >> >> > > > >> %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh >> > > > > > > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com >> > > > > > > > >> > >> >> > > > >> %7Ccbead81a42104034a4f308d736678a45%7C72f988 >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve >> > > > > > > > >> > >> >> > > > > > >>> > > d=0 [3] >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% >> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org >> > > > > > > > >> > >> >> > > > >> %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0 >> > > > > > > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt% >> 40microsoft.com >> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678 >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216 >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D& >> > > > > > > > >> > >> >> > > > > > >>> > > ;reserved=0 [4] >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% >> > > > > > > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org >> > > > > > > > >> > >> >> > > > >> %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02 >> > > > > > > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt% >> 40microsoft.com >> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73 >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064 >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D >> > > > > > > > >> > >> >> > > > > > >>> > > &reserved=0] >> > > > > > > > >> > >> >> > > > > > >>> >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > > > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis >> > > > > > > > >> > >> >> > > > > > >>> sues.apache.org >> > > > > > > > >> > >> >> > > > >> > > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric >> > > > > > > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com >> > > > > > > > >> > >> >> > > > >> %7Ccbead81a42104034a4f308d736678a45%7C72f988 >> > > > > > > > >> > >> >> > > > > > >>> >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > > > >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3 >> > > > > > > > >> > >> >> > > > > > >>> >> > > > > > > > >> > >> >> > > > > LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0 >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > >> > > > > ---------------------------------------------------------------- >> > > > > > > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney < >> > > > > wesmck...@gmail.com> Send >> > > > > > > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 >> To:dev < >> > > > > > > > >> > >> dev@arrow.apache.org> >> > > > > > > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 >> > > > release >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > I'm going to work some on >> organizing >> > > > > the 0.15.0 >> > > > > > > > >> > >> backlog some >> > > > > > > > >> > >> >> > > > > > >>> > > this week, if anyone wants to >> help >> > > > with >> > > > > grooming >> > > > > > > > >> > >> >> > > (particularly >> > > > > > > > >> > >> >> > > > > > >>> > > for languages other than >> C++/Python >> > > > > where I'm >> > > > > > > > >> > >> focusing) that >> > > > > > > > >> > >> >> > > > > > >>> > > would be helpful. There have >> been >> > > > > almost 500 JIRA >> > > > > > > > >> > >> issues >> > > > > > > > >> > >> >> > > opened >> > > > > > > > >> > >> >> > > > > > >>> > > since the >> > > > > > > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should >> make sure >> > > > > to check >> > > > > > > > >> > >> whether >> > > > > > > > >> > >> >> > > there's >> > > > > > > > >> > >> >> > > > > > >>> > > any regressions or other >> serious bugs >> > > > > that we should >> > > > > > > > >> > >> try to >> > > > > > > > >> > >> >> > > fix >> > > > > > > > >> > >> >> > > > > > >>> > > for 0.15.0. >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 >> PM Wes >> > > > > McKinney >> > > > > > > > >> > >> >> > > > > > >>> > > <wesmck...@gmail.com> >> > > > > > > > >> > >> >> > > > > > >>> wrote: >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in >> 0.14.1 >> > > > > seems to be >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2 >> > > > > > > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org >> > > > > > > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02 >> > > > > > > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt% >> > > > 40microsoft.com >> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376 >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u >> > > > > > > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0 >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > I think the root cause could >> be the >> > > > > Windows >> > > > > > > > >> > >> changes in >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2 >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7 >> > > > > > > > >> > >> >> > > > > > >>> > > > CEric.Erhardt% >> 40microsoft.com >> > > > > > > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821 >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl >> > > > > > > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0 >> > > > > > > > >> > >> >> > > > > > >>> >> 223ae744cc2a12c60cecb5db593263a03c13f85a >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a >> > > > > volunteer would look >> > > > > > > > >> > >> into what >> > > > > > > > >> > >> >> > > > > > >>> > > > was >> > > > > > > > >> > >> >> > > > > > >>> wrong >> > > > > > > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on >> Windows. >> > > > > Otherwise >> > > > > > > > >> > >> 0.15.0 Windows >> > > > > > > > >> > >> >> > > > > > >>> > > > wheels will be broken, too >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found >> at >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2 >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252 >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > F0.14.1&data=02%7C01%7CEric.Erhardt% >> > > > > > > > >> > >> 40microsoft.com >> > > > > > > > >> > >> >> > > > %7Ccbea >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4 >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > >> > > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0 >> > > > > > > > >> > >> >> > > > > > >>> > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 >> PM >> > > > > Antoine Pitrou < >> > > > > > > > >> > >> >> > > > > > >>> solip...@pitrou.net> wrote: >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 >> 11:17:07 -0700 >> > > > > Micah >> > > > > > > > >> > >> Kornfield >> > > > > > > > >> > >> >> > > > > > >>> > > > > <emkornfi...@gmail.com> >> wrote: >> > > > > > > > >> > >> >> > > > > > >>> > > > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are >> > > > > > > > >> > >> >> > > > > > >>> > > > > > > independent, we could >> have >> > > > > 32-bit array >> > > > > > > > >> > >> lengths and >> > > > > > > > >> > >> >> > > > > > >>> variable-length >> > > > > > > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit >> offsets if >> > > > we >> > > > > wanted (we >> > > > > > > > >> > >> just >> > > > > > > > >> > >> >> > > > wouldn't >> > > > > > > > >> > >> >> > > > > > >>> > > > > > > be >> > > > > > > > >> > >> >> > > > > > >>> able to >> > > > > > > > >> > >> >> > > > > > >>> > > > > > > have a List child with >> more >> > > > > than INT32_MAX >> > > > > > > > >> > >> elements). >> > > > > > > > >> > >> >> > > > > > >>> > > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we >> could do >> > > > > this in C++ >> > > > > > > > >> > >> but we >> > > > > > > > >> > >> >> > > > don't. >> > > > > > > > >> > >> >> > > > > > >>> I'm not sure we >> > > > > > > > >> > >> >> > > > > > >>> > > > > > would have introduced the >> > > > "Large" >> > > > > types if we >> > > > > > > > >> > >> did. >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice >> as much >> > > > > space as 32-bit >> > > > > > > > >> > >> >> > > offsets, >> > > > > > > > >> > >> >> > > > > > >>> > > > > so if >> > > > > > > > >> > >> >> > > > > > >>> you're >> > > > > > > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish >> lists or >> > > > > strings, >> > > > > > > > >> > >> 32-bit >> > > > > > > > >> > >> >> > > offsets >> > > > > > > > >> > >> >> > > > > > >>> > > > > are preferrable. So even >> with >> > > > > 64-bit array >> > > > > > > > >> > >> lengths from >> > > > > > > > >> > >> >> > > > the >> > > > > > > > >> > >> >> > > > > > >>> > > > > start >> > > > > > > > >> > >> >> > > > > > >>> it would >> > > > > > > > >> > >> >> > > > > > >>> > > > > still be beneficial to >> have types >> > > > > with 32-bit >> > > > > > > > >> > >> offsets. >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited >> address >> > > > > space in Java >> > > > > > > > >> > >> and >> > > > > > > > >> > >> >> > > calling >> > > > > > > > >> > >> >> > > > > > >>> > > > > > it a >> > > > > > > > >> > >> >> > > > > > >>> reference >> > > > > > > > >> > >> >> > > > > > >>> > > > > > implementation seems >> suboptimal. >> > > > > If a consumer >> > > > > > > > >> > >> uses a >> > > > > > > > >> > >> >> > > > "Large" >> > > > > > > > >> > >> >> > > > > > >>> type >> > > > > > > > >> > >> >> > > > > > >>> > > > > > presumably it is because >> they >> > > > > need the ability >> > > > > > > > >> > >> to store >> > > > > > > > >> > >> >> > > > > > >>> > > > > > more >> > > > > > > > >> > >> >> > > > > > >>> than INT32_MAX >> > > > > > > > >> > >> >> > > > > > >>> > > > > > child elements in a >> column, >> > > > > otherwise it is >> > > > > > > > >> > >> just >> > > > > > > > >> > >> >> > > wasting >> > > > > > > > >> > >> >> > > > > > >>> > > > > > space >> > > > > > > > >> > >> >> > > > > > >>> [1]. >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the >> individual >> > > > > elements >> > > > > > > > >> > >> (lists or >> > > > > > > > >> > >> >> > > > > > >>> > > > > strings) >> > > > > > > > >> > >> >> > > > > > >>> are >> > > > > > > > >> > >> >> > > > > > >>> > > > > large, not much space is >> wasted in >> > > > > proportion, >> > > > > > > > >> > >> so it may >> > > > > > > > >> > >> >> > > be >> > > > > > > > >> > >> >> > > > > > >>> simpler in >> > > > > > > > >> > >> >> > > > > > >>> > > > > such a case to always >> create a >> > > > > "Large" type >> > > > > > > > >> > >> array. >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose >> theoretically >> > > > there >> > > > > might be some >> > > > > > > > >> > >> >> > > > > > >>> > > > > > performance >> > > > > > > > >> > >> >> > > > > > >>> benefits on >> > > > > > > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to >> using >> > > > the >> > > > > native word >> > > > > > > > >> > >> sizes. >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit >> > > > > architectures don't do >> > > > > > > > >> > >> that, as >> > > > > > > > >> > >> >> > > > > > >>> > > > > 32-bit >> > > > > > > > >> > >> >> > > > > > >>> is an >> > > > > > > > >> > >> >> > > > > > >>> > > > > extremely common integer >> size even >> > > > > in >> > > > > > > > >> > >> high-performance >> > > > > > > > >> > >> >> > > > code. >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > Regards >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > Antoine. >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > > > >> > > > > > > > >> > >> >> > > > > > >>> > > >> > > > > > > > >> > >> >> > > > > > >>> >> > > > > > > > >> > >> >> > > > > > >> >> > > > > > > > >> > >> >> > > > >> > > > > > > > >> > >> >> > > >> > > > > > > > >> > >> >> > > > > > > > >> > > >> > > > > >> > > > >> >
