yinli-systems commented on PR #11062: URL: https://github.com/apache/arrow-rs/pull/11062#issuecomment-5712195769
Thanks — I rechecked the consumer side and now have a concrete DataFusion A/B case for this PR. DataFusion main currently has an explicit planning guard for finite-offset `RANGE` frames over an REE ordering column. That guard was retained in [DataFusion #24565](https://github.com/apache/datafusion/pull/24565) specifically because the frame implementation computes `current_value +/- offset`, while Arrow arithmetic could not handle the run-end-wrapped scalar. The review there also concluded that this should be implemented in the Arrow kernels rather than special-cased in DataFusion. I tested DataFusion main at `b300cea226` (Arrow 59.3) with a minimal removal of that guard and an SLT query whose expected IDs are `2, 3`: ```sql SELECT id FROM ( SELECT ROW_NUMBER() OVER (ORDER BY temperature) AS id, SUM(temperature) OVER ( ORDER BY arrow_cast( temperature, 'RunEndEncoded("run_ends": non-null Int32, "values": Int64)' ) RANGE BETWEEN 1 PRECEDING AND 1 FOLLOWING ) AS windowed_sum FROM sensor_readings ) WHERE windowed_sum IN (45, 69) ORDER BY id; ``` The A/B result was: - Current DataFusion main: rejected at planning by the existing guard. - Guard removed, unmodified Arrow 59.3: the four frame sums are incorrectly `89, 89, 89, 89`, and the query returns no rows. - Guard removed, this PR backported to Arrow 59.3: the frame sums are `20, 45, 69, 47`, the query returns `2, 3`, and the focused DataFusion sqllogictest passes. This is a pure DataFusion execution path, so there is no Spark fallback involved. It also shows that the guard is preventing a real correctness problem: removing it without the Arrow change produces silent wrong frame boundaries, not just a missing optimization. I separately checked compression preservation with `year * 100 + month` over 50,000,000 logical rows. Both intermediate results remained one run with one physical value. In three release-mode processes (two warmups and seven samples each), the one-run REE path took 3.29–4.29 microseconds versus 68.21–68.60 milliseconds for already-resident dense arrays. Inputs plus both live results were 880 bytes by Arrow's array accounting versus 800,000,384 bytes dense; one `/usr/bin/time -l` run measured maximum RSS of 6,815,744 versus 1,206,681,600 bytes. This is intentionally an ideal one-run kernel microbenchmark, not a full-query speedup claim. I also investigated an Iceberg/Comet partition-arithmetic candidate, but did not find it valid evidence: in the current Comet path the relevant operands are normalized to dense Spark-compatible integers before arithmetic. I am therefore not using that claim. If this DataFusion case meets the use-case bar, the follow-up after Arrow updates would be small: remove the guard and add the end-to-end `2, 3` regression above. I reviewed the code path and test output myself; Codex assisted with the repository search, test harness, and drafting this summary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
