[
https://issues.apache.org/jira/browse/SPARK-57515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-57515:
-----------------------------------
Labels: pull-request-available (was: )
> Surface MALFORMED_CSV_RECORD instead of ArrayIndexOutOfBoundsException when
> CSV header exceeds maxColumns
> ---------------------------------------------------------------------------------------------------------
>
> Key: SPARK-57515
> URL: https://issues.apache.org/jira/browse/SPARK-57515
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.3
> Reporter: Jubin Soni
> Priority: Minor
> Labels: pull-request-available
>
> When reading a CSV file with {{header=true}} and the header row contains more
> columns than {{maxColumns}} (default: {{{}20480{}}}, user-configurable),
> Spark throws an internal {{java.lang.ArrayIndexOutOfBoundsException}} instead
> of returning a structured {{MALFORMED_CSV_RECORD}} error.
> This occurs because CSV header validation paths invoke Univocity parsing APIs
> directly without the malformed-record handling introduced for data rows.
> *Affected Code Paths*
> The issue affects all CSV read paths that validate headers:
> # *Non-multiLine file read*
> ** {{CSVHeaderChecker}} calls {{tokenizer.parseLine(header)}} directly.
> # *MultiLine file read*
> ** {{CSVHeaderChecker}} calls {{tokenizer.parseNext()}} directly.
> # *Dataset[String] csv()*
> ** {{CSVHeaderChecker}} creates a new {{CsvParser}} and calls
> {{parser.parseLine(line)}} directly.
> In all three cases, a header exceeding {{maxColumns}} surfaces a raw
> {{{}ArrayIndexOutOfBoundsException{}}}.
> *Background*
> SPARK-57195 (merged 2026-06-14) fixed the same
> {{ArrayIndexOutOfBoundsException}} issue for CSV data rows by converting
> parser failures into {{MALFORMED_CSV_RECORD}} errors.
> The SPARK-57195 discussion explicitly noted:
> {quote}Header rows are out of scope from this PR. A header over maxColumns
> still surfaces the raw AIOOBE (CSVHeaderChecker), a pre-existing gap.
> {quote}
> As a result, header parsing remains inconsistent with data row parsing.
> *Expected Behavior*
> When the header row exceeds {{{}maxColumns{}}}, Spark should fail with a
> structured {{MALFORMED_CSV_RECORD}} error, consistent with data row handling.
> *Actual Behavior*
> Spark throws:
>
> {{java.lang.ArrayIndexOutOfBoundsException}}
> originating from Univocity parser internals.
> *Steps to Reproduce*
>
> {{import java.nio.file.\{Files, Paths}
> import java.nio.charset.StandardCharsets
> val path = "/tmp/test_header.csv"
> Files.write(
> Paths.get(path),
> "a,b,c\n1,2,3\n".getBytes(StandardCharsets.UTF_8)
> )
> spark.read
> .option("header", "true")
> .option("maxColumns", "2")
> .csv(path)
> .collect()}}
> *Result*
>
> {{java.lang.ArrayIndexOutOfBoundsException}}
> *Expected Result*
> A Spark {{MALFORMED_CSV_RECORD}} error indicating that the CSV record exceeds
> the configured {{maxColumns}} limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
