Awesome06 commented on PR #55425: URL: https://github.com/apache/arrow/pull/55425#issuecomment-6681166238
@dmitry-chirkov-dremio Based on our prior conversation, these are some of the results when considering different approaches. ### Benchmark Notes & Alternative Approach Considered (Option B: Adaptive Batch Sampling) During investigation of GH-55254, we also evaluated an adaptive heuristic that samples the first 64 rows of each batch: - If >= 50% mismatch, it continues with the prefix-checking branch. - If matches dominate and strings are out-of-line (> 12 bytes), it switches to direct payload `memcmp` to skip redundant prefix checks. #### Comparative Benchmark Results: | Workload | Baseline (#55171) | Option A (Current PR) | Option B (Adaptive) | |---|---|---|---| | 131,072 * 8 B, equal | 539.12 us | 366.16 us (1.47x) | 492.86 us (1.09x) | | 131,072 * 64 B, equal | 629.59 us | 735.81 us (0.86x) | 798.38 us (0.79x) | | 131,072 * 8 B, mismatch | 286.29 us | 161.75 us (1.77x) | 154.31 us (1.86x) | | 131,072 * 64 B, mismatch | 657.29 us | 307.06 us (2.14x) | 327.67 us (2.01x) | | Array vs Scalar (1% match) | 400.32 us | 85.72 us (4.67x) | 85.72 us (4.67x) | Option B introduces sampling state and branch overhead that offsets gains on small/medium batches without improving mismatch rejection speed. Option A provides consistent speedups (1.8x - 2.1x on array mismatches, 4.67x on array-scalar filters) with cleaner, stateless logic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
