Hi Micah, Thanks a lot for doing this.
I am a little concerned about if there is any negative performance impact on the current 32-bit-length based applications. Can we do some performance comparison on our existing benchmarks? Best, Liya Fan On Tue, Aug 6, 2019 at 3:35 PM Micah Kornfield <emkornfi...@gmail.com> wrote: > There have been some previous discussions on the mailing about supporting > 64-bit lengths for Java ValueVectors (this is what the IPC specification > and C++ support). I created a PR [1] that changes all APIs that I could > find that take an index to take an "long" instead of an "int" (and > similarly change "size/rowcount" APIs). > > It is a big change, so I think it is worth discussing if it is something we > still want to move forward with. It would be nice to come to a conclusion > quickly, ideally in the next few days, to avoid a lot of merge conflicts. > > The reason I did this work now is the C++ implementation has added support > for LargeList, LargeBinary and LargeString arrays and based on prior > discussions we need to have similar support in Java before our next > release. Support 64-bit indexes means we can have full compatibility and > make the most use of the types in Java. > > Look forward to hearing feedback. > > Thanks, > Micah > > [1] https://github.com/apache/arrow/pull/5020 >
