I should have a little more bandwidth to help with some of the packaging starting tomorrow and going into the weekend.
On Tuesday, September 10, 2019, Wes McKinney <wesmck...@gmail.com> wrote: > Hi folks, > > With the state of nightly packaging and integration builds things aren't > looking too good for being in release readiness by the end of this week but > maybe I'm wrong. I'm planning to be working to close as many issues as I > can and also to help with the ongoing alignment fixes. > > Wes > > On Thu, Sep 5, 2019, 11:07 PM Micah Kornfield <emkornfi...@gmail.com> > wrote: > >> Just for reference [1] has a dashboard of the current issues: >> >> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.15.0+Release >> >> On Thu, Sep 5, 2019 at 3:43 PM Wes McKinney <wesmck...@gmail.com> wrote: >> >>> hi all, >>> >>> It doesn't seem like we're going to be in a position to release at the >>> beginning of next week. I hope that one more week of work (or less) >>> will be enough to get us there. Aside from merging the alignment >>> changes, we need to make sure that our packaging jobs required for the >>> release candidate are all working. >>> >>> If folks could remove issues from the 0.15.0 backlog that they don't >>> think they will finish by end of next week that would help focus >>> efforts (there are currently 78 issues in 0.15.0 still). I am looking >>> to tackle a few small features related to dictionaries while the >>> release window is still open. >>> >>> - Wes >>> >>> On Tue, Aug 27, 2019 at 3:48 PM Wes McKinney <wesmck...@gmail.com> >>> wrote: >>> > >>> > hi, >>> > >>> > I think we should try to release the week of September 9, so >>> > development work should be completed by end of next week. >>> > >>> > Does that seem reasonable? >>> > >>> > I plan to get up a patch for the protocol alignment changes for C++ in >>> > the next couple of days -- I think that getting the alignment work >>> > done is the main barrier to releasing. >>> > >>> > Thanks >>> > Wes >>> > >>> > On Mon, Aug 19, 2019 at 12:25 PM Ji Liu <niki...@aliyun.com.invalid> >>> wrote: >>> > > >>> > > Hi, Wes, on the java side, I can think of several bugs that need to >>> be fixed or reminded. >>> > > >>> > > i. ARROW-6040: Dictionary entries are required in IPC streams even >>> when empty[1] >>> > > This one is under review now, however through this PR we find that >>> there seems a bug in java reading and writing dictionaries in IPC which is >>> Inconsistent with spec[2] since it assumes all dictionaries are at the >>> start of stream (see details in PR comments, and this fix may not catch up >>> with version 0.15). @Micah Kornfield >>> > > >>> > > ii. ARROW-1875: Write 64-bit ints as strings in integration test >>> JSON files[3] >>> > > Java side code already checked in, other implementations seems not. >>> > > >>> > > iii. ARROW-6202: OutOfMemory in JdbcAdapter[4] >>> > > Caused by trying to load all records in one contiguous batch, fixed >>> by providing iterator API for iteratively reading in ARROW-6219[5]. >>> > > >>> > > Thanks, >>> > > Ji Liu >>> > > >>> > > [1] https://github.com/apache/arrow/pull/4960 >>> > > [2] https://arrow.apache.org/docs/ipc.html >>> > > [3] https://issues.apache.org/jira/browse/ARROW-1875 >>> > > [4] https://issues.apache.org/jira/browse/ARROW-6202[5] >>> https://issues.apache.org/jira/browse/ARROW-6219 >>> > > >>> > > >>> > > >>> > > ------------------------------------------------------------------ >>> > > From:Wes McKinney <wesmck...@gmail.com> >>> > > Send Time:2019年8月19日(星期一) 23:03 >>> > > To:dev <dev@arrow.apache.org> >>> > > Subject:Re: Timeline for 0.15.0 release >>> > > >>> > > I'm going to work some on organizing the 0.15.0 backlog some this >>> > > week, if anyone wants to help with grooming (particularly for >>> > > languages other than C++/Python where I'm focusing) that would be >>> > > helpful. There have been almost 500 JIRA issues opened since the >>> > > 0.14.0 release, so we should make sure to check whether there's any >>> > > regressions or other serious bugs that we should try to fix for >>> > > 0.15.0. >>> > > >>> > > On Thu, Aug 15, 2019 at 6:23 PM Wes McKinney <wesmck...@gmail.com> >>> wrote: >>> > > > >>> > > > The Windows wheel issue in 0.14.1 seems to be >>> > > > >>> > > > https://issues.apache.org/jira/browse/ARROW-6015 >>> > > > >>> > > > I think the root cause could be the Windows changes in >>> > > > >>> > > > https://github.com/apache/arrow/commit/ >>> 223ae744cc2a12c60cecb5db593263a03c13f85a >>> > > > >>> > > > I would be appreciative if a volunteer would look into what was >>> wrong >>> > > > with the 0.14.1 wheels on Windows. Otherwise 0.15.0 Windows wheels >>> > > > will be broken, too >>> > > > >>> > > > The bad wheels can be found at >>> > > > >>> > > > https://bintray.com/apache/arrow/python#files/python%2F0.14.1 >>> > > > >>> > > > On Thu, Aug 15, 2019 at 1:28 PM Antoine Pitrou < >>> solip...@pitrou.net> wrote: >>> > > > > >>> > > > > On Thu, 15 Aug 2019 11:17:07 -0700 >>> > > > > Micah Kornfield <emkornfi...@gmail.com> wrote: >>> > > > > > > >>> > > > > > > In C++ they are >>> > > > > > > independent, we could have 32-bit array lengths and >>> variable-length >>> > > > > > > types with 64-bit offsets if we wanted (we just wouldn't be >>> able to >>> > > > > > > have a List child with more than INT32_MAX elements). >>> > > > > > >>> > > > > > I think the point is we could do this in C++ but we don't. >>> I'm not sure we >>> > > > > > would have introduced the "Large" types if we did. >>> > > > > >>> > > > > 64-bit offsets take twice as much space as 32-bit offsets, so if >>> you're >>> > > > > storing lots of small-ish lists or strings, 32-bit offsets are >>> > > > > preferrable. So even with 64-bit array lengths from the start >>> it would >>> > > > > still be beneficial to have types with 32-bit offsets. >>> > > > > >>> > > > > > Going with the limited address space in Java and calling it a >>> reference >>> > > > > > implementation seems suboptimal. If a consumer uses a "Large" >>> type >>> > > > > > presumably it is because they need the ability to store more >>> than INT32_MAX >>> > > > > > child elements in a column, otherwise it is just wasting space >>> [1]. >>> > > > > >>> > > > > Probably. Though if the individual elements (lists or strings) >>> are >>> > > > > large, not much space is wasted in proportion, so it may be >>> simpler in >>> > > > > such a case to always create a "Large" type array. >>> > > > > >>> > > > > > [1] I suppose theoretically there might be some performance >>> benefits on >>> > > > > > 64-bit architectures to using the native word sizes. >>> > > > > >>> > > > > Concretely, common 64-bit architectures don't do that, as 32-bit >>> is an >>> > > > > extremely common integer size even in high-performance code. >>> > > > > >>> > > > > Regards >>> > > > > >>> > > > > Antoine. >>> > > > > >>> > > > > >>> > > >>> >>
