hemanthboyina opened a new pull request, #58520: URL: https://github.com/apache/spark/pull/58520
**What changes were proposed in this pull request?**
OFFSET 0 produces a logical Offset(0, child) node. The EliminateOffsets
optimizer rule normally removes it (offset 0 is a no-op), so physical planning
never sees it. But EliminateOffsets is an excludable
rule, so when it is disabled via spark.sql.optimizer.excludedRules, the
Offset(0) node survives into planning, where it becomes CollectLimitExec(limit
= -1, offset = 0) (or GlobalLimitExec(limit = -1, offset
= 0)), which fails the operator's assertion.
This PR makes the physical planner handle a leftover Offset(0) directly:
since it is a no-op, it is planned as its child. This is added in both offset
planning paths — SpecialLimits (terminal) and
BasicOperators (non-terminal) — so the resulting plan is identical to the
plan produced when EliminateOffsets removes the node.
**Why are the changes needed?**
Excluding an optimization rule should only affect performance, never break
correctness. Today, excluding EliminateOffsets and running a query with OFFSET
0 fails during physical planning:
$ spark-sql --conf
spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.EliminateOffsets
> SELECT 1 AS x OFFSET 0;
java.lang.AssertionError: assertion failed
at
org.apache.spark.sql.execution.CollectLimitExec.<init>(limit.scala:49)
EliminateOffsets is not in nonExcludableRules, so it is legitimately
excludable and the planner must tolerate the un-optimized node.
**Does this PR introduce any user-facing change?**
Yes. Previously, running a query containing OFFSET 0 with EliminateOffsets
excluded threw an AssertionError during physical planning. Now the query
succeeds and returns the correct result (OFFSET 0 is a
no-op). With default settings there is no behavior change.
**How was this patch tested?**
Added a unit test in SQLQuerySuite (SPARK-59044: OFFSET 0 succeeds when
EliminateOffsets is in excludedRules) that excludes the rule and verifies both
planning paths: SELECT 1 AS x OFFSET 0 (terminal /
CollectLimitExec) and a non-terminal OFFSET 0 subquery (GlobalLimitExec).
Both return the expected rows. The test fails before the fix and passes after.
**Was this patch authored or co-authored using generative AI tooling?**
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
