hhhizzz commented on issue #10776: URL: https://github.com/apache/arrow-rs/issues/10776#issuecomment-5386833107
Thanks for the ping. I did a focused experiment on this today to quantify the Boolean filter -> selectors -> mask overhead described in this issue. For the experiment, I generated deterministic `BooleanArray` inputs with several selection shapes and measured the selection construction + lowering path separately from predicate evaluation and Parquet decoding. I included a Q25-like case with ~15% scattered short runs, an alternating run-1 case, the run-32 Auto boundary, a clustered run-128 case, a sparse long-run case, and all-selected. I also ran the candidate through the existing async Parquet reader path using paired measurements against a clean baseline. The results suggest this round trip is indeed quite expensive for fragmented predicate results: | Shape | Current construction + lowering | Experimental result | Paired async reader change | |---|---:|---:|---:| | Q25-like, 15% scattered | 14.675 ms | 506.9 µs (28.95x faster) | 26.02% faster | | Fragmented, run 1 | 48.595 ms | 513.8 µs (94.57x faster) | 48.91% faster | | Auto boundary, run 32 | 431.4 µs | 445.9 µs (3.35% slower) | 1.87% faster | | Clustered, run 128 | 165.8 µs | 154.5 µs (6.77% faster) | 0.29% faster | | Sparse | 66.2 µs | 65.5 µs (1.04% faster) | 0.30% faster | | All selected | no-op | no-op | 1.45% slower | I also tried an unconditional mask-first variant as an experiment. It improved the fragmented cases, but regressed the lowering path by 30.74% for run-128 and 107.55% for the sparse case. So simply constructing a mask first for every predicate result does not look like a good general solution. For correctness, I compared 50,000 randomized inputs against the current behavior, including the selected strategy/backing, logical selection, selected-row count, and total length, and all 50,000 matched. I also covered LIMIT/padding, non-byte-aligned buffers, runs crossing filter boundaries, empty filters, and the run-31/run-32 Auto boundary. The async exact-payload preflight passed all 9 cases as well. I haven't run a full DataFusion ClickBench Q25 comparison yet because current DataFusion/Arrow main have some unrelated compatibility differences that would require additional porting. For now I kept this experiment focused on the specific selection-construction cost identified here. These results look promising, but this is still a focused experiment rather than a production-ready change. I plan to spend the next few days working through the implementation details and testing a broader set of cases to make sure an optimization here doesn't introduce regressions elsewhere. @haohuaijin, since you're also looking at this area, happy to coordinate so we don't duplicate work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
