sepuri sai krishna created SPARK-59374:
------------------------------------------
Summary: concat raises an internal error instead of
COLLECTION_SIZE_LIMIT_EXCEEDED in codegen
Key: SPARK-59374
URL: https://issues.apache.org/jira/browse/SPARK-59374
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.4, 4.1.3, 4.2.0, 4.3.0
Reporter: sepuri sai krishna
{{Concat.genCodeForNumberOfElements}} does not check the result element count
against {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}}, while the interpreted
path
does. The two paths therefore report different error conditions for the same
input: the interpreted path raises {{COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION}},
and codegen reaches {{ArrayData.allocateArrayData}} and raises the internal
{{_LEGACY_ERROR_TEMP_2176}}.
Same pattern as SPARK-58631, which fixed it for {{array_repeat}}.
Priority is Minor because the limit is not cheaply reachable here: {{concat}}
needs its inputs to hold that many elements for real, so an executor will
normally run out of memory first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
