> > Hi Jacques, I hope you had a good rest.
I did, thanks! On Fri, Aug 23, 2019 at 9:25 AM Jacques Nadeau <jacq...@apache.org> wrote: > I don't think we should couple this discussion with the implementation of > large list, etc since I think those two concepts are independent. > > I've asked some others on my team their opinions on the risk here. I think > we should probably review some our more complex vector interactions and see > how the jvm's assembly changes with this kind of change. Using > microbenchmarking is good but I think we also need to see whether we're > constantly inserting additional instructions or if in most cases, this > actually doesn't impact instruction count. > > > > On Wed, Aug 21, 2019 at 12:18 PM Micah Kornfield <emkornfi...@gmail.com> > wrote: > >> >>> With regards to the reference implementation point. It is a good point. >>> I'm on vacation this week. Unless you're pushing hard on this, can we pick >>> this up and discuss more next week? >> >> >> Hi Jacques, I hope you had a good rest. Any more thoughts on the >> reference implementation aspect of this? >> >> >>> To copy the sentiments from the 0.15.0 release thread, I think it >>> would be best to decouple this discussion from the release timeline >>> given how many people we have relying on regular releases coming out. >>> We can keep continue making major 0.x releases until we're ready to >>> release 1.0.0. >> >> >> I'm OK with it as long as other stakeholders are. Timed releases are the >> way to go. As stated on the release thread [1] we need a better mechanism >> to avoid this type of issue arising again. The release thread also had >> some more discussion on compatibility. >> >> Thanks, >> Micah >> >> [1] >> https://lists.apache.org/thread.html/d70feeceaf2570906ade117030b29887af7c77ca5c4a976e6d555920@%3Cdev.arrow.apache.org%3E >> >> >> On Wed, Aug 14, 2019 at 3:23 PM Wes McKinney <wesmck...@gmail.com> wrote: >> >>> On Sun, Aug 11, 2019 at 9:40 PM Micah Kornfield <emkornfi...@gmail.com> >>> wrote: >>> > >>> > Hi Wes and Jacques, >>> > See responses below. >>> > >>> > With regards to the reference implementation point. It is a good >>> point. I'm >>> > > on vacation this week. Unless you're pushing hard on this, can we >>> pick this >>> > > up and discuss more next week? >>> > >>> > >>> > Sure thing, enjoy your vacation. I think the only practical >>> implications >>> > are it delays choices around implementing LargeList, LargeBinary, >>> > LargeString in Java, which in turn might push out the 0.15.0 release. >>> > >>> >>> To copy the sentiments from the 0.15.0 release thread, I think it >>> would be best to decouple this discussion from the release timeline >>> given how many people we have relying on regular releases coming out. >>> We can keep continue making major 0.x releases until we're ready to >>> release 1.0.0. >>> >>> > My stance on this is that I don't know how important it is for Java to >>> > > support vectors over INT32_MAX elements. The use cases enabled by >>> > > having very large arrays seem to be concentrated in the native code >>> > > world (e.g. C/C++/Rust) -- that could just be implementation-centrism >>> > > on my part, though. >>> > >>> > >>> > A data point against this view is Spark has done work to eliminate 2GB >>> > memory limits on its block sizes [1]. I don't claim to understand the >>> > implications of this. Bryan might you have any thoughts here? I'm OK >>> with >>> > INT32_MAX, as well, I think we should think about what this means for >>> > adding Large types to Java and implications for reference >>> implementations. >>> > >>> > Thanks, >>> > Micah >>> > >>> > [1] https://issues.apache.org/jira/browse/SPARK-6235 >>> > >>> > On Sun, Aug 11, 2019 at 6:31 PM Jacques Nadeau <jacq...@apache.org> >>> wrote: >>> > >>> > > Hey Micah, >>> > > >>> > > Appreciate the offer on the compiling. The reality is I'm more >>> concerned >>> > > about the unknowns than the compiling issue itself. Any time you've >>> been >>> > > tuning for a while, changing something like this could be totally >>> fine or >>> > > cause a couple of major issues. For example, we've done a very large >>> amount >>> > > of work reducing heap memory footprint of the vectors. Are target is >>> to >>> > > actually get it down to 24 bytes per ArrowBuf and 24 bytes heap per >>> vector >>> > > (not including arrow bufs). >>> > > >>> > > With regards to the reference implementation point. It is a good >>> point. >>> > > I'm on vacation this week. Unless you're pushing hard on this, can >>> we pick >>> > > this up and discuss more next week? >>> > > >>> > > thanks, >>> > > Jacques >>> > > >>> > > On Sat, Aug 10, 2019 at 7:39 PM Micah Kornfield < >>> emkornfi...@gmail.com> >>> > > wrote: >>> > > >>> > >> Hi Jacques, >>> > >> I definitely understand these concerns and this change is risky >>> because it >>> > >> is so large. Perhaps, creating a new hierarchy, might be the >>> cleanest way >>> > >> of dealing with this. This could have other benefits like cleaning >>> up >>> > >> some >>> > >> cruft around dictionary encode and "orphaned" method. Per past >>> e-mail >>> > >> threads I agree it is beneficial to have 2 separate reference >>> > >> implementations that can communicate fully, and my intent here was >>> to >>> > >> close >>> > >> that gap. >>> > >> >>> > >> Trying to >>> > >> > determine the ramifications of these changes would be challenging >>> and >>> > >> time >>> > >> > consuming against all the different ways we interact with the >>> Arrow Java >>> > >> > library. >>> > >> >>> > >> >>> > >> Understood. I took a quick look at Dremio-OSS it seems like it has >>> a >>> > >> simple java build system? If it is helpful, I can try to get a fork >>> > >> running that at least compiles against this PR. My plan would be >>> to cast >>> > >> any place that was changed to return a long back to an int, so in >>> essence >>> > >> the Dremio algorithms would reman 32-bit implementations. >>> > >> >>> > >> I don't have the infrastructure to test this change properly from a >>> > >> distributed systems perspective, so it would still take some time >>> from >>> > >> Dremio to validate for regressions. >>> > >> >>> > >> I'm not saying I'm against this but want to make sure we've >>> > >> > explored all less disruptive options before considering changing >>> > >> something >>> > >> > this fundamental (especially when I generally hold the view that >>> large >>> > >> cell >>> > >> > counts against massive contiguous memory is an anti pattern to >>> scalable >>> > >> > analytical processing--purely subjective of course). >>> > >> >>> > >> >>> > >> I'm open to other ideas here, as well. I don't think it is out of >>> the >>> > >> question to leave the Java implementation as 32-bit, but if we do, >>> then I >>> > >> think we should consider a different strategy for reference >>> > >> implementations. >>> > >> >>> > >> Thanks, >>> > >> Micah >>> > >> >>> > >> On Sat, Aug 10, 2019 at 5:09 PM Jacques Nadeau <jacq...@apache.org> >>> > >> wrote: >>> > >> >>> > >> > Hey Micah, I didn't have a particular path in mind. Was thinking >>> more >>> > >> along >>> > >> > the lines of extra methods as opposed to separate classes. >>> > >> > >>> > >> > Arrow hasn't historically been a place where we're writing >>> algorithms in >>> > >> > Java so the fact that they aren't there doesn't mean they don't >>> exist. >>> > >> We >>> > >> > have a large amount of code that depends on the current behavior >>> that is >>> > >> > deployed in hundreds of customer clusters (you can peruse our >>> dremio >>> > >> repo >>> > >> > to see how extensively we leverage Arrow if interested). Trying to >>> > >> > determine the ramifications of these changes would be challenging >>> and >>> > >> time >>> > >> > consuming against all the different ways we interact with the >>> Arrow Java >>> > >> > library. I'm not saying I'm against this but want to make sure >>> we've >>> > >> > explored all less disruptive options before considering changing >>> > >> something >>> > >> > this fundamental (especially when I generally hold the view that >>> large >>> > >> cell >>> > >> > counts against massive contiguous memory is an anti pattern to >>> scalable >>> > >> > analytical processing--purely subjective of course). >>> > >> > >>> > >> > On Sat, Aug 10, 2019, 4:17 PM Micah Kornfield < >>> emkornfi...@gmail.com> >>> > >> > wrote: >>> > >> > >>> > >> > > Hi Jacques, >>> > >> > > What avenue were you thinking for supporting both paths? I >>> didn't >>> > >> want >>> > >> > > to pursue a different class hierarchy, because I felt like that >>> would >>> > >> > > effectively fork the code base, but that is potentially an >>> option that >>> > >> > > would allow us to have a complete reference implementation in >>> Java >>> > >> that >>> > >> > can >>> > >> > > fully interact with C++, without major changes to this code. >>> > >> > > >>> > >> > > For supporting both APIs on the same classes/interfaces, I >>> think they >>> > >> > > roughly fall into three categories, changes to input parameters, >>> > >> changes >>> > >> > to >>> > >> > > output parameters and algorithm changes. >>> > >> > > >>> > >> > > For inputs, changing from int to long is essentially a no-op >>> from the >>> > >> > > compiler perspective. From the limited micro-benchmarking this >>> also >>> > >> > > doesn't seem to have a performance impact. So we could keep two >>> > >> versions >>> > >> > > of the methods that only differ on inputs, but it is not clear >>> what >>> > >> the >>> > >> > > value of that would be. >>> > >> > > >>> > >> > > For outputs, we can't support methods "long getLength()" and >>> "int >>> > >> > > getLength()" in the same class, so we would be forced into >>> something >>> > >> like >>> > >> > > "long getLength(boolean dummy)" which I think is a less >>> desirable. >>> > >> > > >>> > >> > > For algorithm changes, there did not appear to be too many >>> places >>> > >> where >>> > >> > we >>> > >> > > actually loop over all elements (it is quite possible I missed >>> > >> something >>> > >> > > here), the ones that I did find I was able to mitigate >>> performance >>> > >> > > penalties as noted above. Some of the current implementation >>> will >>> > >> get a >>> > >> > > lot slower for "large arrays", but we can likely fix those >>> later or in >>> > >> > this >>> > >> > > PR with a nested while loop instead of 2 for loops. >>> > >> > > >>> > >> > > Thanks, >>> > >> > > Micah >>> > >> > > >>> > >> > > >>> > >> > > On Saturday, August 10, 2019, Jacques Nadeau < >>> jacq...@apache.org> >>> > >> wrote: >>> > >> > > >>> > >> > >> This is a pretty massive change to the apis. I wonder how >>> nasty it >>> > >> would >>> > >> > >> be to just support both paths. Have you evaluated how complex >>> that >>> > >> > would be? >>> > >> > >> >>> > >> > >> On Wed, Aug 7, 2019 at 11:08 PM Micah Kornfield < >>> > >> emkornfi...@gmail.com> >>> > >> > >> wrote: >>> > >> > >> >>> > >> > >>> After more investigation, it looks like Float8Benchmarks at >>> least >>> > >> on my >>> > >> > >>> machine are within the range of noise. >>> > >> > >>> >>> > >> > >>> For BitVectorHelper I pushed a new commit [1], seems to bring >>> the >>> > >> > >>> BitVectorHelper benchmarks back inline (and even with some >>> > >> improvement >>> > >> > >>> for >>> > >> > >>> getNullCountBenchmark). >>> > >> > >>> >>> > >> > >>> Benchmark Mode Cnt >>> Score >>> > >> > >>> Error >>> > >> > >>> Units >>> > >> > >>> BitVectorHelperBenchmarks.allBitsNullBenchmark avgt 5 >>> 3.821 ± >>> > >> > >>> 0.031 >>> > >> > >>> ns/op >>> > >> > >>> BitVectorHelperBenchmarks.getNullCountBenchmark avgt 5 >>> 14.884 ± >>> > >> > >>> 0.141 >>> > >> > >>> ns/op >>> > >> > >>> >>> > >> > >>> I applied the same pattern to other loops that I could find, >>> and for >>> > >> > any >>> > >> > >>> "for (long" loop on the critical path, I broke it up into two >>> loops. >>> > >> > the >>> > >> > >>> first loop does iteration by integer, the second finishes off >>> for >>> > >> any >>> > >> > >>> long >>> > >> > >>> values. As a side note it seems like optimization for loops >>> using >>> > >> long >>> > >> > >>> counters at least have a semi-recent open bug for the JVM [2] >>> > >> > >>> >>> > >> > >>> Thanks, >>> > >> > >>> Micah >>> > >> > >>> >>> > >> > >>> [1] >>> > >> > >>> >>> > >> > >>> >>> > >> > >>> > >> >>> https://github.com/apache/arrow/pull/5020/commits/2ea2c1ae83e3baa7b9a99a6d06276d968df41797 >>> > >> > >>> [2] https://bugs.openjdk.java.net/browse/JDK-8223051 >>> > >> > >>> >>> > >> > >>> On Wed, Aug 7, 2019 at 8:11 PM Micah Kornfield < >>> > >> emkornfi...@gmail.com> >>> > >> > >>> wrote: >>> > >> > >>> >>> > >> > >>> > Indeed, the BoundChecking and CheckNullForGet variables can >>> make a >>> > >> > big >>> > >> > >>> > difference. I didn't initially run the benchmarks with >>> these >>> > >> turned >>> > >> > on >>> > >> > >>> > (you can see the result from above with Float8Benchmarks). >>> Here >>> > >> are >>> > >> > >>> new >>> > >> > >>> > numbers including with the flags enabled. It looks like >>> using >>> > >> longs >>> > >> > >>> might >>> > >> > >>> > be a little bit slower, I'll see what I can do to mitigate >>> this. >>> > >> > >>> > >>> > >> > >>> > Ravindra also volunteered to try to benchmark the changes >>> with >>> > >> > Dremio's >>> > >> > >>> > code on today's sync call. >>> > >> > >>> > >>> > >> > >>> > New >>> > >> > >>> > >>> > >> > >>> > Benchmark Mode Cnt >>> Score >>> > >> > >>> Error >>> > >> > >>> > Units >>> > >> > >>> > >>> > >> > >>> > BitVectorHelperBenchmarks.allBitsNullBenchmark avgt 5 >>> > >> 4.176 ± >>> > >> > >>> 1.292 >>> > >> > >>> > ns/op >>> > >> > >>> > >>> > >> > >>> > BitVectorHelperBenchmarks.getNullCountBenchmark avgt 5 >>> > >> 26.102 ± >>> > >> > >>> 0.700 >>> > >> > >>> > ns/op >>> > >> > >>> > >>> > >> > >>> > Float8Benchmarks.copyFromBenchmark avgt 5 7.398 ± >>> 0.084 >>> > >> us/op >>> > >> > >>> > >>> > >> > >>> > Float8Benchmarks.readWriteBenchmark avgt 5 2.711 ± >>> 0.057 >>> > >> us/op >>> > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > >>> > >> > >>> > old >>> > >> > >>> > >>> > >> > >>> > BitVectorHelperBenchmarks.allBitsNullBenchmark avgt 5 >>> > >> 3.828 ± >>> > >> > >>> 0.030 >>> > >> > >>> > ns/op >>> > >> > >>> > >>> > >> > >>> > BitVectorHelperBenchmarks.getNullCountBenchmark avgt 5 >>> > >> 20.611 ± >>> > >> > >>> 0.188 >>> > >> > >>> > ns/op >>> > >> > >>> > >>> > >> > >>> > Float8Benchmarks.copyFromBenchmark avgt 5 6.597 ± >>> 0.462 >>> > >> us/op >>> > >> > >>> > >>> > >> > >>> > Float8Benchmarks.readWriteBenchmark avgt 5 2.615 ± >>> 0.027 >>> > >> us/op >>> > >> > >>> > >>> > >> > >>> > On Wed, Aug 7, 2019 at 7:13 PM Fan Liya < >>> liya.fa...@gmail.com> >>> > >> > wrote: >>> > >> > >>> > >>> > >> > >>> >> Hi Gonzalo, >>> > >> > >>> >> >>> > >> > >>> >> Thanks for sharing the performance results. >>> > >> > >>> >> I am wondering if you have turned off the flag >>> > >> > >>> >> BoundsChecking#BOUNDS_CHECKING_ENABLED. >>> > >> > >>> >> If not, the lower throughput should be expected. >>> > >> > >>> >> >>> > >> > >>> >> Best, >>> > >> > >>> >> Liya Fan >>> > >> > >>> >> >>> > >> > >>> >> On Wed, Aug 7, 2019 at 10:23 PM Micah Kornfield < >>> > >> > >>> emkornfi...@gmail.com> >>> > >> > >>> >> wrote: >>> > >> > >>> >> >>> > >> > >>> >>> Hi Gonzalo, >>> > >> > >>> >>> Thank you for the feedback. I wasn't aware of the JIT >>> > >> > >>> implications. At >>> > >> > >>> >>> least on the benchmark run they don't seem to have an >>> impact. >>> > >> > >>> >>> >>> > >> > >>> >>> If there are other benchmarks that people have that can >>> > >> validate if >>> > >> > >>> this >>> > >> > >>> >>> change will be problematic I would appreciate trying to >>> run them >>> > >> > >>> with the >>> > >> > >>> >>> PR. I will try to run the ones for zeroing/popcnt >>> tonight to >>> > >> see >>> > >> > if >>> > >> > >>> >>> there >>> > >> > >>> >>> is a change in those. >>> > >> > >>> >>> >>> > >> > >>> >>> -Micah >>> > >> > >>> >>> >>> > >> > >>> >>> >>> > >> > >>> >>> >>> > >> > >>> >>> On Wednesday, August 7, 2019, Gonzalo Ortiz Jaureguizar < >>> > >> > >>> >>> golthir...@gmail.com> wrote: >>> > >> > >>> >>> >>> > >> > >>> >>> > I would recommend to take care with this kind of >>> changes. >>> > >> > >>> >>> > >>> > >> > >>> >>> > I didn't try Arrow in more than one year, but by then >>> the >>> > >> > >>> performance >>> > >> > >>> >>> was >>> > >> > >>> >>> > quite bad in comparison with plain byte buffer access >>> > >> > >>> >>> > (see >>> http://git.net/apache-arrow-development/msg02353.html *) >>> > >> > and >>> > >> > >>> >>> > there are several optimizations that the JVM >>> (specifically, >>> > >> C2) >>> > >> > >>> does >>> > >> > >>> >>> not >>> > >> > >>> >>> > apply when dealing with int instead of longs. One of the >>> > >> > >>> >>> > most commons is the loop unrolling and vectorization. >>> > >> > >>> >>> > >>> > >> > >>> >>> > * It doesn't seem the best way to reference an old >>> email on >>> > >> the >>> > >> > >>> list, >>> > >> > >>> >>> but >>> > >> > >>> >>> > it is the only result shown by Google >>> > >> > >>> >>> > >>> > >> > >>> >>> > El mié., 7 ago. 2019 a las 11:42, Fan Liya (< >>> > >> > liya.fa...@gmail.com >>> > >> > >>> >) >>> > >> > >>> >>> > escribió: >>> > >> > >>> >>> > >>> > >> > >>> >>> >> Hi Micah, >>> > >> > >>> >>> >> >>> > >> > >>> >>> >> Thanks for your effort. The performance result looks >>> good. >>> > >> > >>> >>> >> >>> > >> > >>> >>> >> As you indicated, ArrowBuf will take additional 12 >>> bytes (4 >>> > >> > bytes >>> > >> > >>> for >>> > >> > >>> >>> each >>> > >> > >>> >>> >> of length, write index, and read index). >>> > >> > >>> >>> >> Similar overheads also exist for vectors like >>> > >> > >>> BaseFixedWidthVector, >>> > >> > >>> >>> >> BaseVariableWidthVector, etc. >>> > >> > >>> >>> >> >>> > >> > >>> >>> >> IMO, such overheads are small enough to justify the >>> change. >>> > >> > >>> >>> >> Let's check if there are other overheads. >>> > >> > >>> >>> >> >>> > >> > >>> >>> >> Best, >>> > >> > >>> >>> >> Liya Fan >>> > >> > >>> >>> >> >>> > >> > >>> >>> >> On Wed, Aug 7, 2019 at 3:30 PM Micah Kornfield < >>> > >> > >>> emkornfi...@gmail.com >>> > >> > >>> >>> > >>> > >> > >>> >>> >> wrote: >>> > >> > >>> >>> >> >>> > >> > >>> >>> >> > Hi Liya Fan, >>> > >> > >>> >>> >> > Based on the Float8Benchmark there does not seem to >>> be any >>> > >> > >>> >>> meaningful >>> > >> > >>> >>> >> > performance difference on my machine. At least for >>> me, the >>> > >> > >>> >>> benchmarks >>> > >> > >>> >>> >> are >>> > >> > >>> >>> >> > not stable enough to say one is faster than the >>> other (I've >>> > >> > >>> pasted >>> > >> > >>> >>> >> results >>> > >> > >>> >>> >> > below). That being said my machine isn't >>> necessarily the >>> > >> most >>> > >> > >>> >>> reliable >>> > >> > >>> >>> >> for >>> > >> > >>> >>> >> > benchmarking. >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> > On an intuitive level, this makes sense to me, for >>> the >>> > >> most >>> > >> > >>> part it >>> > >> > >>> >>> >> seems >>> > >> > >>> >>> >> > like the change just moves casting from "int" to >>> "long" >>> > >> > further >>> > >> > >>> up >>> > >> > >>> >>> the >>> > >> > >>> >>> >> > stack for "PlatformDepdendent" operations. If >>> there are >>> > >> > other >>> > >> > >>> >>> >> benchmarks >>> > >> > >>> >>> >> > that you think are worth running let me know. >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> > One downside performance wise I think for his change >>> is it >>> > >> > >>> >>> increases the >>> > >> > >>> >>> >> > size of ArrowBuf objects, which I suppose could >>> influence >>> > >> > cache >>> > >> > >>> >>> misses >>> > >> > >>> >>> >> at >>> > >> > >>> >>> >> > some level or increase the size of call-stacks, but >>> this >>> > >> > doesn't >>> > >> > >>> >>> seem to >>> > >> > >>> >>> >> > show up in the benchmark.. >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> > Thanks, >>> > >> > >>> >>> >> > Micah >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> > Sample benchmark numbers: >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> > [New Code] >>> > >> > >>> >>> >> > Benchmark Mode Cnt >>> Score >>> > >> Error >>> > >> > >>> >>> Units >>> > >> > >>> >>> >> > Float8Benchmarks.copyFromBenchmark avgt 5 >>> 15.441 ± >>> > >> 0.469 >>> > >> > >>> >>> us/op >>> > >> > >>> >>> >> > Float8Benchmarks.readWriteBenchmark avgt 5 >>> 14.057 ± >>> > >> 0.115 >>> > >> > >>> >>> us/op >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> > [Old code] >>> > >> > >>> >>> >> > Benchmark Mode Cnt >>> Score >>> > >> Error >>> > >> > >>> >>> Units >>> > >> > >>> >>> >> > Float8Benchmarks.copyFromBenchmark avgt 5 >>> 16.248 ± >>> > >> 1.409 >>> > >> > >>> >>> us/op >>> > >> > >>> >>> >> > Float8Benchmarks.readWriteBenchmark avgt 5 >>> 14.150 ± >>> > >> 0.084 >>> > >> > >>> >>> us/op >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> > On Tue, Aug 6, 2019 at 1:18 AM Fan Liya < >>> > >> liya.fa...@gmail.com >>> > >> > > >>> > >> > >>> >>> wrote: >>> > >> > >>> >>> >> > >>> > >> > >>> >>> >> >> Hi Micah, >>> > >> > >>> >>> >> >> >>> > >> > >>> >>> >> >> Thanks a lot for doing this. >>> > >> > >>> >>> >> >> >>> > >> > >>> >>> >> >> I am a little concerned about if there is any >>> negative >>> > >> > >>> performance >>> > >> > >>> >>> >> impact >>> > >> > >>> >>> >> >> on the current 32-bit-length based applications. >>> > >> > >>> >>> >> >> Can we do some performance comparison on our >>> existing >>> > >> > >>> benchmarks? >>> > >> > >>> >>> >> >> >>> > >> > >>> >>> >> >> Best, >>> > >> > >>> >>> >> >> Liya Fan >>> > >> > >>> >>> >> >> >>> > >> > >>> >>> >> >> >>> > >> > >>> >>> >> >> On Tue, Aug 6, 2019 at 3:35 PM Micah Kornfield < >>> > >> > >>> >>> emkornfi...@gmail.com> >>> > >> > >>> >>> >> >> wrote: >>> > >> > >>> >>> >> >> >>> > >> > >>> >>> >> >>> There have been some previous discussions on the >>> mailing >>> > >> > about >>> > >> > >>> >>> >> supporting >>> > >> > >>> >>> >> >>> 64-bit lengths for Java ValueVectors (this is >>> what the >>> > >> IPC >>> > >> > >>> >>> >> specification >>> > >> > >>> >>> >> >>> and C++ support). I created a PR [1] that changes >>> all >>> > >> APIs >>> > >> > >>> that I >>> > >> > >>> >>> >> could >>> > >> > >>> >>> >> >>> find that take an index to take an "long" instead >>> of an >>> > >> > "int" >>> > >> > >>> (and >>> > >> > >>> >>> >> >>> similarly change "size/rowcount" APIs). >>> > >> > >>> >>> >> >>> >>> > >> > >>> >>> >> >>> It is a big change, so I think it is worth >>> discussing if >>> > >> it >>> > >> > is >>> > >> > >>> >>> >> something >>> > >> > >>> >>> >> >>> we >>> > >> > >>> >>> >> >>> still want to move forward with. It would be nice >>> to >>> > >> come >>> > >> > to >>> > >> > >>> a >>> > >> > >>> >>> >> >>> conclusion >>> > >> > >>> >>> >> >>> quickly, ideally in the next few days, to avoid a >>> lot of >>> > >> > merge >>> > >> > >>> >>> >> conflicts. >>> > >> > >>> >>> >> >>> >>> > >> > >>> >>> >> >>> The reason I did this work now is the C++ >>> implementation >>> > >> has >>> > >> > >>> added >>> > >> > >>> >>> >> >>> support >>> > >> > >>> >>> >> >>> for LargeList, LargeBinary and LargeString arrays >>> and >>> > >> based >>> > >> > on >>> > >> > >>> >>> prior >>> > >> > >>> >>> >> >>> discussions we need to have similar support in Java >>> > >> before >>> > >> > our >>> > >> > >>> >>> next >>> > >> > >>> >>> >> >>> release. Support 64-bit indexes means we can have >>> full >>> > >> > >>> >>> compatibility >>> > >> > >>> >>> >> and >>> > >> > >>> >>> >> >>> make the most use of the types in Java. >>> > >> > >>> >>> >> >>> >>> > >> > >>> >>> >> >>> Look forward to hearing feedback. >>> > >> > >>> >>> >> >>> >>> > >> > >>> >>> >> >>> Thanks, >>> > >> > >>> >>> >> >>> Micah >>> > >> > >>> >>> >> >>> >>> > >> > >>> >>> >> >>> [1] https://github.com/apache/arrow/pull/5020 >>> > >> > >>> >>> >> >>> >>> > >> > >>> >>> >> >> >>> > >> > >>> >>> >> >>> > >> > >>> >>> > >>> > >> > >>> >>> >>> > >> > >>> >> >>> > >> > >>> >>> > >> > >> >>> > >> > >>> > >> >>> > > >>> >>
