sepuri sai krishna created SPARK-59375:
------------------------------------------
Summary: flatten raises an internal error instead of
COLLECTION_SIZE_LIMIT_EXCEEDED in codegen
Key: SPARK-59375
URL: https://issues.apache.org/jira/browse/SPARK-59375
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.0.4, 4.1.3, 4.2.0, 4.3.0
Reporter: sepuri sai krishna
{{Flatten.genCodeForNumberOfElements}} does not check the result element count
against {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}}, while the interpreted
path
in {{Flatten.nullSafeEval}} does. The two paths therefore report different error
conditions for the same input: the interpreted path raises
{{COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION}}, and codegen reaches
{{ArrayData.allocateArrayData}} and raises the internal
{{_LEGACY_ERROR_TEMP_2176}}.
Same pattern as SPARK-58631, which fixed it for {{array_repeat}}.
Priority is Minor because the limit is not cheaply reachable here: {{flatten}}
needs its input arrays to hold that many elements for real, so an executor will
normally run out of memory first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
