sepuri sai krishna created SPARK-58631:
------------------------------------------
Summary: array_repeat and array_insert raise inconsistent array
size limit errors in codegen and interpreted execution
Key: SPARK-58631
URL: https://issues.apache.org/jira/browse/SPARK-58631
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: sepuri sai krishna
{{array_repeat}} and {{array_insert}} raise a different error for an array that
exceeds the maximum array length depending on whether the expression runs
through the interpreted path or through generated code.
h3. array_repeat
{code}
eval -> COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER (SQLSTATE 54000)
codegen -> _LEGACY_ERROR_TEMP_2176
{code}
{{ArrayRepeat.eval}} checks {{count}} against
{{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}} and raises
{{COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER}}.
{{ArrayRepeat.genCodeForNumberOfElements}} performs no such check, so the
generated code proceeds to the array allocation, which fails with the internal
{{_LEGACY_ERROR_TEMP_2176}}. Codegen is the default execution path, so the
internal error class is what users normally see. The interpreted path was
migrated to the named condition by SPARK-45710 (4.0.0); the codegen path was
not.
h3. array_insert
{code}
eval -> COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION
codegen -> COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER
{code}
The codegen branch taken when {{pos}} is not foldable calls
{{createArrayWithElementsExceedLimitError}}, which hardcodes {{"parameter" ->
toSQLId("count")}}. The resulting message reads "the value of parameter(s)
`count` in the function `array_insert` is invalid", but {{array_insert(array,
pos, value)}} has no {{count}} parameter. Every other site in the expression --
all three interpreted ones and the two sibling codegen ones -- uses
{{arrayFunctionWithElementsExceedLimitError}}
({{COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION}}). Introduced by SPARK-56567
(4.2.0).
h3. Why this was not caught
The existing test "Elements exceed limit for array_repeat()" in
{{QueryExecutionErrorsSuite}} runs {{select array_repeat(1, 2147483647)}},
which is constant-folded and therefore only exercises the interpreted path. The
{{array_insert}} branch is reachable only with a non-foldable {{pos}}, because
a foldable positive {{pos}} takes a different codegen branch.
h3. Scope
Error reporting only -- neither function produces wrong results, and the
conditions under which each raises an error are unchanged.
{{Sequence}} also reports {{COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETER}} naming a
{{count}} parameter it does not have, but it does so from {{sequenceLength}},
which both execution paths share. There is no interpreted-vs-codegen
inconsistency there, so it is left alone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
