I have diagnosed the problem (Thrift "string" data must be UTF-8, cannot be arbitrary binary) and am working on a patch right now
On Tue, Sep 24, 2019 at 6:02 PM Wes McKinney <wesmck...@gmail.com> wrote: > > I just opened > > https://issues.apache.org/jira/browse/ARROW-6678 > > Please don't cut an RC until I have an opportunity to diagnose this, > will report back. > > > On Tue, Sep 24, 2019 at 5:51 PM Wes McKinney <wesmck...@gmail.com> wrote: > > > > I'm investigating a possible Parquet-related compatibility bug that I > > encountered through some routine testing / benchmarking. I'll report > > back once I figure out what is going on (if anything) > > > > On Sun, Sep 22, 2019 at 11:51 PM Micah Kornfield <emkornfi...@gmail.com> > > wrote: > > >> > > >> It's ideal if your GPG key is in the web of trust (i.e. you can get it > > >> signed by another PMC member), but is not 100% essential. > > > > > > That won't be an option for me this week (it seems like I would need to > > > meet one face-to-face). I'll try to get the GPG checked in and the rest > > > of the pre-requisites done tomorrow (Monday) to hopefully start the > > > release on Tuesday (hopefully we can solve the last blocker/integration > > > tests by then). > > > > > > On Sat, Sep 21, 2019 at 7:12 PM Wes McKinney <wesmck...@gmail.com> wrote: > > >> > > >> It's ideal if your GPG key is in the web of trust (i.e. you can get it > > >> signed by another PMC member), but is not 100% essential. > > >> > > >> Speaking of the release, there are at least 2 code changes I still > > >> want to get in > > >> > > >> ARROW-5717 > > >> ARROW-6353 > > >> > > >> I just pushed updates to ARROW-5717, will merge once the build is green. > > >> > > >> There are a couple of Rust patches still marked for 0.15. The rest > > >> seems to be documentation and a couple of integration test failures we > > >> should see about fixing in time. > > >> > > >> On Fri, Sep 20, 2019 at 11:26 PM Micah Kornfield <emkornfi...@gmail.com> > > >> wrote: > > >> > > > >> > Thanks Krisztián and Wes, > > >> > I've gone ahead and started registering myself on all the packaging > > >> > sites. > > >> > > > >> > Is there any review process when adding my GPG key to the SVN file? [1] > > >> > doesn't seem to mention explicitly. > > >> > > > >> > Thanks, > > >> > Micah > > >> > > > >> > [1] https://www.apache.org/dev/version-control.html#https-svn > > >> > > > >> > On Fri, Sep 20, 2019 at 5:01 PM Krisztián Szűcs > > >> > <szucs.kriszt...@gmail.com> > > >> > wrote: > > >> > > > >> > > On Thu, Sep 19, 2019 at 5:52 PM Wes McKinney <wesmck...@gmail.com> > > >> > > wrote: > > >> > > > > >> > >> On Thu, Sep 19, 2019 at 12:13 AM Micah Kornfield > > >> > >> <emkornfi...@gmail.com> > > >> > >> wrote: > > >> > >> >> > > >> > >> >> The process should be well documented at this point but there > > >> > >> >> are a > > >> > >> >> number of steps. > > >> > >> > > > >> > >> > Is [1] the up-to-date documentation for the release? Are there > > >> > >> instructions for the adding the code signing Key to SVN? > > >> > >> > > > >> > >> > I will make a go of it. i will try to mitigate any internet > > >> > >> > issues by > > >> > >> doing the process for a cloud instance (I assume that isn't a > > >> > >> problem?). > > >> > >> > > > >> > >> > > >> > >> Setting up a new cloud environment suitable for producing an RC may > > >> > >> be > > >> > >> time consuming, but you are welcome to try. Krisztian -- are you > > >> > >> available next week to help Micah and potentially take over > > >> > >> producing > > >> > >> the RC if there are issues? > > >> > >> > > >> > > Sure, I'll be available next week. We can also grant access to > > >> > > https://github.com/ursa-labs/crossbow because configuring all > > >> > > the CI backends can be time consuming. > > >> > > > > >> > >> > > >> > >> > Thanks, > > >> > >> > Micah > > >> > >> > > > >> > >> > [1] > > >> > >> https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide > > >> > >> > > > >> > >> > On Wed, Sep 18, 2019 at 8:29 AM Wes McKinney <wesmck...@gmail.com> > > >> > >> wrote: > > >> > >> >> > > >> > >> >> The process should be well documented at this point but there > > >> > >> >> are a > > >> > >> >> number of steps. Note that you need to add your code signing key > > >> > >> >> to > > >> > >> >> the KEYS file in SVN (that's not very hard to do). I think it's > > >> > >> >> fine > > >> > >> >> to hand off the process to others after the VOTE but it would be > > >> > >> >> tricky to have multiple RMs involved with producing the source > > >> > >> >> and > > >> > >> >> binary artifacts for the vote > > >> > >> >> > > >> > >> >> On Tue, Sep 17, 2019 at 10:55 PM Micah Kornfield < > > >> > >> emkornfi...@gmail.com> wrote: > > >> > >> >> > > > >> > >> >> > SGTM, as well. > > >> > >> >> > > > >> > >> >> > I should have a little bit of time next week if I can help as > > >> > >> >> > RM but > > >> > >> I have > > >> > >> >> > a couple of concerns: > > >> > >> >> > 1. In the past I've had trouble downloading and validating > > >> > >> releases. I'm a > > >> > >> >> > bit worried, that I might have similar problems doing the > > >> > >> >> > necessary > > >> > >> uploads. > > >> > >> >> > 2. My internet connection will likely be not great, I don't > > >> > >> >> > know if > > >> > >> this > > >> > >> >> > would make it even less likely to be successful. > > >> > >> >> > > > >> > >> >> > Does it become problematic if somehow I would have to abandon > > >> > >> >> > the > > >> > >> process > > >> > >> >> > mid-release? Is there anyone who could serve as a backup? > > >> > >> >> > Are the > > >> > >> steps > > >> > >> >> > well documented? > > >> > >> >> > > > >> > >> >> > Thanks, > > >> > >> >> > Micah > > >> > >> >> > > > >> > >> >> > On Tue, Sep 17, 2019 at 4:25 PM Neal Richardson < > > >> > >> neal.p.richard...@gmail.com> > > >> > >> >> > wrote: > > >> > >> >> > > > >> > >> >> > > Sounds good to me. > > >> > >> >> > > > > >> > >> >> > > Do we have a release manager yet? Any volunteers? > > >> > >> >> > > > > >> > >> >> > > Neal > > >> > >> >> > > > > >> > >> >> > > On Tue, Sep 17, 2019 at 4:06 PM Wes McKinney > > >> > >> >> > > <wesmck...@gmail.com> > > >> > >> wrote: > > >> > >> >> > > > > >> > >> >> > > > hi all, > > >> > >> >> > > > > > >> > >> >> > > > It looks like we're drawing close to be able to make the > > >> > >> >> > > > 0.15.0 > > >> > >> >> > > > release. I would suggest "pencils down" at the end of this > > >> > >> >> > > > week > > >> > >> and > > >> > >> >> > > > see if a release candidate can be produced next Monday > > >> > >> >> > > > September > > >> > >> 23. > > >> > >> >> > > > Any thoughts or objections? > > >> > >> >> > > > > > >> > >> >> > > > Thanks, > > >> > >> >> > > > Wes > > >> > >> >> > > > > > >> > >> >> > > > On Wed, Sep 11, 2019 at 11:23 AM Wes McKinney < > > >> > >> wesmck...@gmail.com> > > >> > >> >> > > wrote: > > >> > >> >> > > > > > > >> > >> >> > > > > hi Eric -- yes, that's correct. I'm planning to amend the > > >> > >> Format docs > > >> > >> >> > > > > today regarding the EOS issue and also update the C++ > > >> > >> >> > > > > library > > >> > >> >> > > > > > > >> > >> >> > > > > On Wed, Sep 11, 2019 at 11:21 AM Eric Erhardt > > >> > >> >> > > > > <eric.erha...@microsoft.com> wrote: > > >> > >> >> > > > > > > > >> > >> >> > > > > > I assume the plan is to merge the > > >> > >> ARROW-6313-flatbuffer-alignment > > >> > >> >> > > > branch into master before the 0.15 release, correct? > > >> > >> >> > > > > > > > >> > >> >> > > > > > BTW - I believe the C# alignment changes are ready to > > >> > >> >> > > > > > be > > >> > >> merged into > > >> > >> >> > > > the alignment branch - > > >> > >> https://github.com/apache/arrow/pull/5280/ > > >> > >> >> > > > > > > > >> > >> >> > > > > > Eric > > >> > >> >> > > > > > > > >> > >> >> > > > > > -----Original Message----- > > >> > >> >> > > > > > From: Micah Kornfield <emkornfi...@gmail.com> > > >> > >> >> > > > > > Sent: Tuesday, September 10, 2019 10:24 PM > > >> > >> >> > > > > > To: Wes McKinney <wesmck...@gmail.com> > > >> > >> >> > > > > > Cc: dev <dev@arrow.apache.org>; niki.lj > > >> > >> >> > > > > > <niki...@aliyun.com> > > >> > >> >> > > > > > Subject: Re: Timeline for 0.15.0 release > > >> > >> >> > > > > > > > >> > >> >> > > > > > I should have a little more bandwidth to help with > > >> > >> >> > > > > > some of > > >> > >> the > > >> > >> >> > > > packaging starting tomorrow and going into the weekend. > > >> > >> >> > > > > > > > >> > >> >> > > > > > On Tuesday, September 10, 2019, Wes McKinney < > > >> > >> wesmck...@gmail.com> > > >> > >> >> > > > wrote: > > >> > >> >> > > > > > > > >> > >> >> > > > > > > Hi folks, > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > With the state of nightly packaging and integration > > >> > >> >> > > > > > > builds > > >> > >> things > > >> > >> >> > > > > > > aren't looking too good for being in release > > >> > >> >> > > > > > > readiness by > > >> > >> the end > > >> > >> >> > > of > > >> > >> >> > > > > > > this week but maybe I'm wrong. I'm planning to be > > >> > >> >> > > > > > > working > > >> > >> to close > > >> > >> >> > > as > > >> > >> >> > > > > > > many issues as I can and also to help with the > > >> > >> >> > > > > > > ongoing > > >> > >> alignment > > >> > >> >> > > > fixes. > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > Wes > > >> > >> >> > > > > > > > > >> > >> >> > > > > > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield < > > >> > >> >> > > emkornfi...@gmail.com > > >> > >> >> > > > > > > >> > >> >> > > > > > > wrote: > > >> > >> >> > > > > > > > > >> > >> >> > > > > > >> Just for reference [1] has a dashboard of the > > >> > >> >> > > > > > >> current > > >> > >> issues: > > >> > >> >> > > > > > >> > > >> > >> >> > > > > > >> > > >> > >> >> > > > > > >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwi > > >> > >> >> > > > > > >> ki.apache.org > > >> > >> >> > > > %2Fconfluence%2Fdisplay%2FARROW%2FArrow%2B0.15.0%2BRelea > > >> > >> >> > > > > > >> se&data=02%7C01%7CEric.Erhardt%40microsoft.com > > >> > >> >> > > > %7Ccbead81a42104034 > > >> > >> >> > > > > > >> > > >> > >> >> > > > > > >> > >> a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376 > > >> > >> >> > > > > > >> > > >> > >> >> > > > > > >> > >> 90648216338&sdata=0Upux3i%2B9X6f8uanGKSGM5VYxR6c2ADWrxSPi1%2FgbH4 > > >> > >> >> > > > > > >> %3D&reserved=0 > > >> > >> >> > > > > > >> > > >> > >> >> > > > > > >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney < > > >> > >> wesmck...@gmail.com> > > >> > >> >> > > > wrote: > > >> > >> >> > > > > > >> > > >> > >> >> > > > > > >>> hi all, > > >> > >> >> > > > > > >>> > > >> > >> >> > > > > > >>> It doesn't seem like we're going to be in a > > >> > >> >> > > > > > >>> position to > > >> > >> release > > >> > >> >> > > at > > >> > >> >> > > > > > >>> the beginning of next week. I hope that one more > > >> > >> >> > > > > > >>> week of > > >> > >> work (or > > >> > >> >> > > > > > >>> less) will be enough to get us there. Aside from > > >> > >> >> > > > > > >>> merging > > >> > >> the > > >> > >> >> > > > > > >>> alignment changes, we need to make sure that our > > >> > >> packaging jobs > > >> > >> >> > > > > > >>> required for the release candidate are all working. > > >> > >> >> > > > > > >>> > > >> > >> >> > > > > > >>> If folks could remove issues from the 0.15.0 > > >> > >> >> > > > > > >>> backlog > > >> > >> that they > > >> > >> >> > > > don't > > >> > >> >> > > > > > >>> think they will finish by end of next week that > > >> > >> >> > > > > > >>> would > > >> > >> help focus > > >> > >> >> > > > > > >>> efforts (there are currently 78 issues in 0.15.0 > > >> > >> >> > > > > > >>> still). > > >> > >> I am > > >> > >> >> > > > > > >>> looking to tackle a few small features related to > > >> > >> dictionaries > > >> > >> >> > > > while > > >> > >> >> > > > > > >>> the release window is still open. > > >> > >> >> > > > > > >>> > > >> > >> >> > > > > > >>> - Wes > > >> > >> >> > > > > > >>> > > >> > >> >> > > > > > >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney < > > >> > >> >> > > wesmck...@gmail.com> > > >> > >> >> > > > > > >>> wrote: > > >> > >> >> > > > > > >>> > > > >> > >> >> > > > > > >>> > hi, > > >> > >> >> > > > > > >>> > > > >> > >> >> > > > > > >>> > I think we should try to release the week of > > >> > >> >> > > > > > >>> > September > > >> > >> 9, so > > >> > >> >> > > > > > >>> > development work should be completed by end of > > >> > >> >> > > > > > >>> > next > > >> > >> week. > > >> > >> >> > > > > > >>> > > > >> > >> >> > > > > > >>> > Does that seem reasonable? > > >> > >> >> > > > > > >>> > > > >> > >> >> > > > > > >>> > I plan to get up a patch for the protocol > > >> > >> >> > > > > > >>> > alignment > > >> > >> changes for > > >> > >> >> > > > > > >>> > C++ in the next couple of days -- I think that > > >> > >> >> > > > > > >>> > getting > > >> > >> the > > >> > >> >> > > > > > >>> > alignment work done is the main barrier to > > >> > >> >> > > > > > >>> > releasing. > > >> > >> >> > > > > > >>> > > > >> > >> >> > > > > > >>> > Thanks > > >> > >> >> > > > > > >>> > Wes > > >> > >> >> > > > > > >>> > > > >> > >> >> > > > > > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu > > >> > >> >> > > > > > >>> > <niki...@aliyun.com.invalid> > > >> > >> >> > > > > > >>> wrote: > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > Hi, Wes, on the java side, I can think of > > >> > >> >> > > > > > >>> > > several > > >> > >> bugs that > > >> > >> >> > > > need > > >> > >> >> > > > > > >>> > > to > > >> > >> >> > > > > > >>> be fixed or reminded. > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > i. ARROW-6040: Dictionary entries are required > > >> > >> >> > > > > > >>> > > in > > >> > >> IPC streams > > >> > >> >> > > > > > >>> > > even > > >> > >> >> > > > > > >>> when empty[1] > > >> > >> >> > > > > > >>> > > This one is under review now, however through > > >> > >> >> > > > > > >>> > > this > > >> > >> PR we find > > >> > >> >> > > > > > >>> > > that > > >> > >> >> > > > > > >>> there seems a bug in java reading and writing > > >> > >> dictionaries in IPC > > >> > >> >> > > > > > >>> which is Inconsistent with spec[2] since it > > >> > >> >> > > > > > >>> assumes all > > >> > >> >> > > > dictionaries > > >> > >> >> > > > > > >>> are at the start of stream (see details in PR > > >> > >> >> > > > > > >>> comments, > > >> > >> and this > > >> > >> >> > > > > > >>> fix may not catch up with version 0.15). @Micah > > >> > >> >> > > > > > >>> Kornfield > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in > > >> > >> integration > > >> > >> >> > > > test > > >> > >> >> > > > > > >>> JSON files[3] > > >> > >> >> > > > > > >>> > > Java side code already checked in, other > > >> > >> implementations > > >> > >> >> > > seems > > >> > >> >> > > > not. > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] > > >> > >> Caused by > > >> > >> >> > > trying > > >> > >> >> > > > > > >>> > > to load all records in one contiguous batch, > > >> > >> >> > > > > > >>> > > fixed > > >> > >> >> > > > > > >>> by providing iterator API for iteratively reading > > >> > >> >> > > > > > >>> in > > >> > >> >> > > ARROW-6219[5]. > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > Thanks, > > >> > >> >> > > > > > >>> > > Ji Liu > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > [1] > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > 2Fgithub.com%2Fapache%2Farrow%2Fpull%2F4960&data=02%7C01%7CE > > >> > >> >> > > > > > >>> > > ric.Erhardt%40microsoft.com > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7 > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&a > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > mp;sdata=eDF%2FAsJmVs7WjfEuNBYo%2F1TypIN44xx1TTlK6kQHZVg%3D& > > >> > >> >> > > > > > >>> > > reserved=0 [2] > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% > > >> > >> >> > > > > > >>> > > 2Farrow.apache.org > > >> > >> >> > > > %2Fdocs%2Fipc.html&data=02%7C01%7CEric.Erh > > >> > >> >> > > > > > >>> > > ardt%40microsoft.com > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988 > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdat > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > a=H0pM8bVKsOyeORDhHxLlS%2BpaS%2F5meT52wxTKmNssuMk%3D&reserve > > >> > >> >> > > > > > >>> > > d=0 [3] > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-1875&data=02%7C0 > > >> > >> >> > > > > > >>> > > 1%7CEric.Erhardt%40microsoft.com > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678 > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216 > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > 338&sdata=coTpuoEGhfjyOSBTagdlohOTX24DQZmtbWC0gYsDmkM%3D& > > >> > >> >> > > > > > >>> > > ;reserved=0 [4] > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F% > > >> > >> >> > > > > > >>> > > 2Fissues.apache.org > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6202%5B5&data=02 > > >> > >> >> > > > > > >>> > > %7C01%7CEric.Erhardt%40microsoft.com > > >> > >> >> > > > %7Ccbead81a42104034a4f308d73 > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > 6678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064 > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > 8216338&sdata=gnyUMk8cUgwc802QBLF3eAp3mznYwonlbF0qmGyzgmY%3D > > >> > >> >> > > > > > >>> > > &reserved=0] > > >> > >> >> > > > > > >>> > > >> > >> >> > > > > > >> > >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fis > > >> > >> >> > > > > > >>> sues.apache.org > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6219&data=02%7C01%7CEric > > >> > >> >> > > > > > >>> .Erhardt%40microsoft.com > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a45%7C72f988 > > >> > >> >> > > > > > >>> > > >> > >> >> > > > > > >> > >> bf86f141af91ab2d7cd011db47%7C1%7C0%7C637037690648216338&sdata=d3 > > >> > >> >> > > > > > >>> > > >> > >> LF%2BTeWSprASqO%2ByE4LywlsULHGcb1Iq%2F2byHrEPkY%3D&reserved=0 > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > ---------------------------------------------------------------- > > >> > >> >> > > > > > >>> > > -- From:Wes McKinney <wesmck...@gmail.com> Send > > >> > >> >> > > > > > >>> > > Time:2019年8月19日(星期一) 23:03 To:dev < > > >> > >> dev@arrow.apache.org> > > >> > >> >> > > > > > >>> > > Subject:Re: Timeline for 0.15.0 release > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > I'm going to work some on organizing the 0.15.0 > > >> > >> backlog some > > >> > >> >> > > > > > >>> > > this week, if anyone wants to help with > > >> > >> >> > > > > > >>> > > grooming > > >> > >> >> > > (particularly > > >> > >> >> > > > > > >>> > > for languages other than C++/Python where I'm > > >> > >> focusing) that > > >> > >> >> > > > > > >>> > > would be helpful. There have been almost 500 > > >> > >> >> > > > > > >>> > > JIRA > > >> > >> issues > > >> > >> >> > > opened > > >> > >> >> > > > > > >>> > > since the > > >> > >> >> > > > > > >>> > > 0.14.0 release, so we should make sure to check > > >> > >> whether > > >> > >> >> > > there's > > >> > >> >> > > > > > >>> > > any regressions or other serious bugs that we > > >> > >> >> > > > > > >>> > > should > > >> > >> try to > > >> > >> >> > > fix > > >> > >> >> > > > > > >>> > > for 0.15.0. > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney > > >> > >> >> > > > > > >>> > > <wesmck...@gmail.com> > > >> > >> >> > > > > > >>> wrote: > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2 > > >> > >> >> > > > > > >>> > > > F%2Fissues.apache.org > > >> > >> >> > > > %2Fjira%2Fbrowse%2FARROW-6015&data=02 > > >> > >> >> > > > > > >>> > > > %7C01%7CEric.Erhardt%40microsoft.com > > >> > >> >> > > > %7Ccbead81a42104034a4f308d > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > 736678a45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6370376 > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > 90648216338&sdata=D9lqHR16oRAFlPaIrcXq3UtW%2BLuJQW1u0Gom2u > > >> > >> >> > > > > > >>> > > > WEWg0%3D&reserved=0 > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > > > >>> > > > I think the root cause could be the Windows > > >> > >> changes in > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2 > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > F%2Fgithub.com%2Fapache%2Farrow%2Fcommit%2F&data=02%7C01%7 > > >> > >> >> > > > > > >>> > > > CEric.Erhardt%40microsoft.com > > >> > >> >> > > > %7Ccbead81a42104034a4f308d736678a > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > 45%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C63703769064821 > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > 6338&sdata=iPmFB%2BncIbmvp5D31vjB4A2KyuMP%2B83Vp7%2BDiOxvl > > >> > >> >> > > > > > >>> > > > bs%3D&reserved=0 > > >> > >> >> > > > > > >>> 223ae744cc2a12c60cecb5db593263a03c13f85a > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > > > >>> > > > I would be appreciative if a volunteer would > > >> > >> >> > > > > > >>> > > > look > > >> > >> into what > > >> > >> >> > > > > > >>> > > > was > > >> > >> >> > > > > > >>> wrong > > >> > >> >> > > > > > >>> > > > with the 0.14.1 wheels on Windows. Otherwise > > >> > >> 0.15.0 Windows > > >> > >> >> > > > > > >>> > > > wheels will be broken, too > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > > > >>> > > > The bad wheels can be found at > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2 > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > F%2Fbintray.com%2Fapache%2Farrow%2Fpython%23files%2Fpython%252 > > >> > >> >> > > > > > >>> > > > F0.14.1&data=02%7C01%7CEric.Erhardt% > > >> > >> 40microsoft.com > > >> > >> >> > > > %7Ccbea > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > d81a42104034a4f308d736678a45%7C72f988bf86f141af91ab2d7cd011db4 > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > 7%7C1%7C0%7C637037690648216338&sdata=vZzx4HNS9qp2UWhFagqfJ > > >> > >> >> > > > > > >>> > > > zbY%2BGzwspH1TO3wdfrbA6Y%3D&reserved=0 > > >> > >> >> > > > > > >>> > > > > > >> > >> >> > > > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine > > >> > >> >> > > > > > >>> > > > Pitrou < > > >> > >> >> > > > > > >>> solip...@pitrou.net> wrote: > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 Micah > > >> > >> Kornfield > > >> > >> >> > > > > > >>> > > > > <emkornfi...@gmail.com> wrote: > > >> > >> >> > > > > > >>> > > > > > > > > >> > >> >> > > > > > >>> > > > > > > In C++ they are > > >> > >> >> > > > > > >>> > > > > > > independent, we could have 32-bit array > > >> > >> lengths and > > >> > >> >> > > > > > >>> variable-length > > >> > >> >> > > > > > >>> > > > > > > types with 64-bit offsets if we wanted > > >> > >> >> > > > > > >>> > > > > > > (we > > >> > >> just > > >> > >> >> > > > wouldn't > > >> > >> >> > > > > > >>> > > > > > > be > > >> > >> >> > > > > > >>> able to > > >> > >> >> > > > > > >>> > > > > > > have a List child with more than > > >> > >> >> > > > > > >>> > > > > > > INT32_MAX > > >> > >> elements). > > >> > >> >> > > > > > >>> > > > > > > > >> > >> >> > > > > > >>> > > > > > I think the point is we could do this in > > >> > >> >> > > > > > >>> > > > > > C++ > > >> > >> but we > > >> > >> >> > > > don't. > > >> > >> >> > > > > > >>> I'm not sure we > > >> > >> >> > > > > > >>> > > > > > would have introduced the "Large" types > > >> > >> >> > > > > > >>> > > > > > if we > > >> > >> did. > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > 64-bit offsets take twice as much space as > > >> > >> >> > > > > > >>> > > > > 32-bit > > >> > >> >> > > offsets, > > >> > >> >> > > > > > >>> > > > > so if > > >> > >> >> > > > > > >>> you're > > >> > >> >> > > > > > >>> > > > > storing lots of small-ish lists or strings, > > >> > >> 32-bit > > >> > >> >> > > offsets > > >> > >> >> > > > > > >>> > > > > are preferrable. So even with 64-bit array > > >> > >> lengths from > > >> > >> >> > > > the > > >> > >> >> > > > > > >>> > > > > start > > >> > >> >> > > > > > >>> it would > > >> > >> >> > > > > > >>> > > > > still be beneficial to have types with > > >> > >> >> > > > > > >>> > > > > 32-bit > > >> > >> offsets. > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > > Going with the limited address space in > > >> > >> >> > > > > > >>> > > > > > Java > > >> > >> and > > >> > >> >> > > calling > > >> > >> >> > > > > > >>> > > > > > it a > > >> > >> >> > > > > > >>> reference > > >> > >> >> > > > > > >>> > > > > > implementation seems suboptimal. If a > > >> > >> >> > > > > > >>> > > > > > consumer > > >> > >> uses a > > >> > >> >> > > > "Large" > > >> > >> >> > > > > > >>> type > > >> > >> >> > > > > > >>> > > > > > presumably it is because they need the > > >> > >> >> > > > > > >>> > > > > > ability > > >> > >> to store > > >> > >> >> > > > > > >>> > > > > > more > > >> > >> >> > > > > > >>> than INT32_MAX > > >> > >> >> > > > > > >>> > > > > > child elements in a column, otherwise it > > >> > >> >> > > > > > >>> > > > > > is > > >> > >> just > > >> > >> >> > > wasting > > >> > >> >> > > > > > >>> > > > > > space > > >> > >> >> > > > > > >>> [1]. > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > Probably. Though if the individual elements > > >> > >> (lists or > > >> > >> >> > > > > > >>> > > > > strings) > > >> > >> >> > > > > > >>> are > > >> > >> >> > > > > > >>> > > > > large, not much space is wasted in > > >> > >> >> > > > > > >>> > > > > proportion, > > >> > >> so it may > > >> > >> >> > > be > > >> > >> >> > > > > > >>> simpler in > > >> > >> >> > > > > > >>> > > > > such a case to always create a "Large" type > > >> > >> array. > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > > [1] I suppose theoretically there might > > >> > >> >> > > > > > >>> > > > > > be some > > >> > >> >> > > > > > >>> > > > > > performance > > >> > >> >> > > > > > >>> benefits on > > >> > >> >> > > > > > >>> > > > > > 64-bit architectures to using the native > > >> > >> >> > > > > > >>> > > > > > word > > >> > >> sizes. > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > Concretely, common 64-bit architectures > > >> > >> >> > > > > > >>> > > > > don't do > > >> > >> that, as > > >> > >> >> > > > > > >>> > > > > 32-bit > > >> > >> >> > > > > > >>> is an > > >> > >> >> > > > > > >>> > > > > extremely common integer size even in > > >> > >> high-performance > > >> > >> >> > > > code. > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > Regards > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > Antoine. > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > > > >> > >> >> > > > > > >>> > > > > >> > >> >> > > > > > >>> > > >> > >> >> > > > > > >> > > >> > >> >> > > > > > >> > >> >> > > > > >> > >> > > >> > >
