etseidl commented on PR #11475: URL: https://github.com/apache/arrow-rs/pull/11475#issuecomment-6617815990
> Looks like perf is flat (which is good). I guess we don't have a direct measure of memory consumption per benchmark. Thanks for bringing this up @adriangb! While trying to answer I found a pretty big issue with the PR. Everything is roses for sparse indexes. I had my agent whip up a quick test where the populated index entries were kept fixed (10 entries), but the declared dimensions of the table increase. | Declared dimensions (RG × columns) | Populated entries per kind | Main retained index bytes | PR retained index bytes | Main median lifecycle | PR median lifecycle | |---|---:|---:|---:|---:|---:| | 10 × 100 | 10 | 280,954 | 3,450 | 9.092 µs | 5.354 µs | | 100 × 100 | 10 | 2,805,274 | 3,450 | 87.288 µs | 5.409 µs | | 100 × 1,000 | 10 | 28,005,274 | 3,450 | 14.646 ms | 5.472 µs | | 1,000 × 1,000 | 10 | 280,048,474 | 3,450 | 151.662 ms | 5.451 µs | Pretty good stuff. But then I asked the obvious question, what about fully populated indexes? A comparison of fully populated 1000 x 1000 showed a significant slowdown for the new form (memory use was nearly flat, but total lifecycle time goes from 500ms to 1740ms), with most of the extra time going to populating the hash map in the builder, and then sorting the entries in the freeze stage. That got me thinking that for the normal case where we're building an index from scratch, we're already doing the traversal in row-major order, and we know the exact bounds (mod missing indexes) up front. I had my agent whip up a test of going straight to the boxed slice form (skipping the builder), and was able to almost reach parity. | Workload | Main | Original PR | Direct-vector PR | | :------- |-----:|-------------:| ----:| | Fully populated 1,000 × 1,000 | 0.979 s | 2.487 s | 1.084 s | | Sparse 10 × 100, 10 entries/kind | 20.9 us | 13.7 us | 9.8 us | | Sparse 1,000 × 1,000, same population | 205.8 ms | 13.8 us | 9.7 us | I'll polish that work up and push it soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
