[
https://issues.apache.org/jira/browse/SPARK-59374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59374:
-----------------------------------
Labels: pull-request-available (was: )
> concat raises an internal error instead of COLLECTION_SIZE_LIMIT_EXCEEDED in
> codegen
> ------------------------------------------------------------------------------------
>
> Key: SPARK-59374
> URL: https://issues.apache.org/jira/browse/SPARK-59374
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.3.0, 4.1.3, 4.0.4
> Reporter: sepuri sai krishna
> Priority: Minor
> Labels: pull-request-available
>
> {{Concat.genCodeForNumberOfElements}} does not check the result element count
> against {{ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH}}, while the interpreted
> path
> does. The two paths therefore report different error conditions for the same
> input: the interpreted path raises
> {{COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTION}},
> and codegen reaches {{ArrayData.allocateArrayData}} and raises the internal
> {{_LEGACY_ERROR_TEMP_2176}}.
> Same pattern as SPARK-58631, which fixed it for {{array_repeat}}.
> Priority is Minor because the limit is not cheaply reachable here: {{concat}}
> needs its inputs to hold that many elements for real, so an executor will
> normally run out of memory first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
