Jubin Soni created SPARK-61727:
----------------------------------
Summary: Surface MALFORMED_CSV_RECORD instead of
TextParsingException when schema_of_csv input exceeds maxColumns or
maxCharsPerColumn
Key: SPARK-61727
URL: https://issues.apache.org/jira/browse/SPARK-61727
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.3.0, 4.4.0
Reporter: Jubin Soni
h2. Bug / gap
schema_of_csv surfaces a raw com.univocity.parsers.common.TextParsingException
(caused by java.lang.ArrayIndexOutOfBoundsException) when its input exceeds
maxColumns or maxCharsPerColumn. SchemaOfCsvEvaluator.evaluate
(sql/catalyst/.../expressions/csv/CsvExpressionEvalUtils.scala) calls
csvParser.parseLine directly instead of the UnivocityParser.parseLine helper
that SPARK-57195 added for exactly this failure. from_csv and CSV data source
reads/inference on the same input already raise MALFORMED_CSV_RECORD.
h2. Reproduction
{code}
SELECT schema_of_csv('a,b,c', map('maxColumns', '2'));
-- com.univocity.parsers.common.TextParsingException:
java.lang.ArrayIndexOutOfBoundsException - Index 2 out of bounds for length 2
SELECT schema_of_csv('abcdef', map('maxCharsPerColumn', '3'));
-- com.univocity.parsers.common.TextParsingException: Length of parsed input
(4) exceeds the maximum number of characters defined in your parser settings
(3).
-- Same input, other paths:
SELECT from_csv('a,b,c', 'a STRING', map('maxColumns', '2'));
-- [MALFORMED_CSV_RECORD] Malformed CSV record: a,b,c SQLSTATE: KD000
{code}
Expected:
{code}
SELECT schema_of_csv('a,b,c', map('maxColumns', '2'));
-- [MALFORMED_CSV_RECORD] Malformed CSV record: a,b,c SQLSTATE: KD000
{code}
h2. Why it matters
Users get a raw third-party parser exception with no error condition or
SQLSTATE, inconsistent with from_csv and CSV reads, which already report
MALFORMED_CSV_RECORD for the same input. Also reachable with default options
for a sample row of more than 20480 columns.
h2. Related work
SPARK-57195 added UnivocityParser.parseLine to translate this exception into
MALFORMED_CSV_RECORD for CSV schema inference and multiLine reads; SPARK-57515
closed the header-row gap it left. The schema_of_csv expression path was not
covered by either. Searched JIRA for schema_of_csv, SchemaOfCsv, maxColumns,
maxCharsPerColumn, TextParsingException, and GitHub PRs for schema_of_csv /
SchemaOfCsvEvaluator / CsvExpressionEvalUtils / MALFORMED_CSV_RECORD on
2026-11-14.
h2. Proposed fix
Replace csvParser.parseLine(csv.toString) in SchemaOfCsvEvaluator.evaluate with
UnivocityParser.parseLine(csvParser, csv.toString), reusing the existing
translation (including its bounded bad-record content).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]