SEPURI-SAI-KRISHNA opened a new pull request, #57832:
URL: https://github.com/apache/spark/pull/57832
### What changes were proposed in this pull request?
`array_repeat` and `array_insert` raise a different error for an array that
exceeds the maximum array length depending on whether the expression runs
through the interpreted path or through generated code. This PR makes the
codegen paths report the same error the interpreted paths already report.
**`array_repeat`** — `ArrayRepeat.eval` checks `count` against
`ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH` and raises
`COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER`, but `genCodeForNumberOfElements`
performs no such check, so the generated code proceeds to the array allocation
and fails with the internal `_LEGACY_ERROR_TEMP_2176`. The same bound check is
added to the generated code. The interpreted path was migrated to the named
condition by SPARK-45710 (4.0.0); the codegen path was not.
**`array_insert`** — the codegen branch taken when `pos` is not foldable
called `createArrayWithElementsExceedLimitError`, which hardcodes `"parameter"
-> toSQLId("count")`. `array_insert(array, pos, value)` has no `count`
parameter. It now calls `arrayFunctionWithElementsExceedLimitError`, matching
all three interpreted sites and both sibling codegen sites in the same
expression. Introduced by SPARK-56567 (4.2.0).
`Sequence` also reports `COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER` naming a
`count` parameter it does not have, but it does so from `sequenceLength`, which
both execution paths share. There is no interpreted-vs-codegen inconsistency
there, so it is left alone.
### Why are the changes needed?
The same query produces a different error class depending on the execution
mode, and codegen is the default path, so the wrong error is the one users
normally see.
`array_repeat` leaks an internal `_LEGACY_ERROR_TEMP_*` condition, which
carries no SQLSTATE and is not meant to reach users. `array_insert` produces a
message naming a parameter the function does not have, which is actively
misleading:
```
the value of parameter(s) `count` in the function `array_insert` is invalid
```
Neither divergence was caught because the existing coverage does not reach
the generated code. The existing test `"Elements exceed limit for
array_repeat()"` in `QueryExecutionErrorsSuite` runs `select array_repeat(1,
2147483647)`, which is constant-folded and therefore only exercises the
interpreted path. The `array_insert` branch is reachable only with a
non-foldable `pos`, because a foldable positive `pos` takes a different codegen
branch.
### Does this PR introduce _any_ user-facing change?
Yes, in error reporting only. Both functions raise an error under exactly
the same conditions as before, and neither produces different results.
Before, `array_repeat` with a count above the limit, executed by codegen:
```
_LEGACY_ERROR_TEMP_2176
Cannot create array with 2147483647 elements of data due to exceeding the
limit
2147483632 elements for ArrayData. array_repeat failed.
```
After, matching what the interpreted path already produced:
```
[COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] Can't create array with
2147483647 elements
which exceeding the array size limit 2147483632, the value of parameter(s)
`count`
in the function `array_repeat` is invalid. SQLSTATE: 54000
```
Before, `array_insert` with a non-foldable `pos` above the limit, executed
by codegen:
```
[COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER] ... the value of parameter(s)
`count`
in the function `array_insert` is invalid. SQLSTATE: 54000
```
After, matching the interpreted path:
```
[COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION] ... unsuccessful try to create
arrays
in the function `array_insert`. SQLSTATE: 54000
```
This is a change compared to released versions: the `array_repeat`
divergence is present since 4.0.0 and the `array_insert` one since 4.2.0.
### How was this patch tested?
Two cases added to `CollectionExpressionsSuite`, both using
`checkErrorInExpression`, which runs the expression under `CODEGEN_ONLY` and
`NO_CODEGEN` and so asserts the two paths agree. The `array_insert` case binds
`pos` to a `BoundReference` so that the non-foldable branch is the one
exercised. Both cases fail on the unpatched code.
- `catalyst/testOnly *CollectionExpressionsSuite` — 63/63 pass
- `sql/testOnly *QueryExecutionErrorsSuite -- -z "exceed limit"` — 3/3 pass,
covering the existing `concat`, `flatten` and `array_repeat` limit tests
The behaviour change was also confirmed end to end by evaluating each
expression directly through both `eval` and `GenerateMutableProjection`, before
and after the patch. Before, the two paths disagreed for both functions; after,
both produce byte-identical messages. `sequence`, which this PR does not touch,
still diverges, confirming the check was live.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
