SEPURI-SAI-KRISHNA opened a new pull request, #58047:
URL: https://github.com/apache/spark/pull/58047
### What changes were proposed in this pull request?
`Sequence.sequenceLength` computes the sequence length in `Long` and falls
back to `BigInt` when that arithmetic overflows. The fallback recomputes the
length exactly, raises `COLLECTION_SIZE_LIMIT_EXCEEDED` if it is too large to
allocate, and then throws `internalError("Unreachable code reached.")` on the
assumption that no other outcome is possible.
That assumption does not hold. `Math.subtractExact(stop, start)` overflows
whenever `stop - start` exceeds the `Long` range, which says nothing about the
length, because the length also depends on `step`. For a large step the exact
length is small, no limit error is raised, and control reaches the
`internalError` with the correct length sitting unused in `safeLen`.
This PR returns `safeLen.toInt` instead of throwing. The check immediately
above already bounds `safeLen` by `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`,
and the caller has already rejected boundaries whose step points the wrong way,
so the value is positive and the narrowing conversion is exact. The now-unused
`SparkException.internalError` import is removed.
No other behaviour changes: an overflow with a length that really is too
large still raises `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` from the same
fallback.
### Why are the changes needed?
`sequence()` over BIGINT raises an `INTERNAL_ERROR` (SQLSTATE XX000) for
queries that have a small, well-defined result. `INTERNAL_ERROR` is reserved
for conditions that indicate a bug in Spark, so surfacing it for ordinary user
input is wrong on both counts: the query should have succeeded, and the error
tells the user to report a bug rather than fix their query.
```sql
SELECT sequence(-9223372036854775808L, 9223372036854775807L,
9223372036854775807L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
```
The element-filling code already handles steps of this magnitude correctly,
so only the length computation was at fault: `sequence(-4611686018427387904L,
4611686018427387903L, 4611686018427387903L)`, whose endpoints are close enough
together to avoid the overflow, returns `[-4611686018427387904, -1,
4611686018427387902]` on master today.
`sequenceLength` was introduced in this shape by SPARK-43393
(`afc4c49927cb`), which fixed a real correctness bug where the `Long` length
computation overflowed silently and `sequence` returned an empty array. The
overflow cases that fix was written against all had huge lengths, which is why
the tail looked unreachable. The method has not changed since, so every release
containing SPARK-43393 is affected.
### Does this PR introduce _any_ user-facing change?
Yes. Queries that failed with `INTERNAL_ERROR` now return their result. Both
the interpreted and the codegen path are affected, since both call
`Sequence.sequenceLength`.
Before:
```sql
SELECT sequence(-9223372036854775808L, 9223372036854775807L,
9223372036854775807L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
SELECT sequence(-9223372036854775808L, 0L, 4611686018427387904L);
-- [INTERNAL_ERROR] Unreachable code reached. SQLSTATE: XX000
```
After:
```sql
SELECT sequence(-9223372036854775808L, 9223372036854775807L,
9223372036854775807L);
-- [-9223372036854775808, -1, 9223372036854775806]
SELECT sequence(-9223372036854775808L, 0L, 4611686018427387904L);
-- [-9223372036854775808, -4611686018427387904, 0]
```
No query that previously succeeded changes its result, and no query that
previously raised `COLLECTION_SIZE_LIMIT_EXCEEDED` stops raising it. Since the
previous behaviour was an internal error, no migration guide entry is needed.
### How was this patch tested?
Three cases added to the `Sequence of numbers` test in
`CollectionExpressionsSuite`, next to the existing SPARK-43393 overflow cases.
`checkEvaluation` exercises both the interpreted and the codegen path:
* a positive step at both `Long` extremes,
* a positive step where only `stop - start` overflows,
* the negative-step mirror.
`build/sbt 'catalyst/testOnly *CollectionExpressionsSuite'` passes (62
tests). The existing SPARK-43393 cases, which cover the fallback still raising
the limit error, pass unchanged.
Also verified end to end against a local `SparkSession`, including a
non-foldable form reading the arguments from a temp view, to confirm the fix
applies at runtime and not only through constant folding.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
