Jubin Soni created SPARK-61727:
----------------------------------

             Summary: Surface MALFORMED_CSV_RECORD instead of 
TextParsingException when schema_of_csv input exceeds maxColumns or 
maxCharsPerColumn
                 Key: SPARK-61727
                 URL: https://issues.apache.org/jira/browse/SPARK-61727
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.3.0, 4.4.0
            Reporter: Jubin Soni


h2. Bug / gap
schema_of_csv surfaces a raw com.univocity.parsers.common.TextParsingException 
(caused by java.lang.ArrayIndexOutOfBoundsException) when its input exceeds 
maxColumns or maxCharsPerColumn. SchemaOfCsvEvaluator.evaluate 
(sql/catalyst/.../expressions/csv/CsvExpressionEvalUtils.scala) calls 
csvParser.parseLine directly instead of the UnivocityParser.parseLine helper 
that SPARK-57195 added for exactly this failure. from_csv and CSV data source 
reads/inference on the same input already raise MALFORMED_CSV_RECORD.
h2. Reproduction
{code}
SELECT schema_of_csv('a,b,c', map('maxColumns', '2'));
-- com.univocity.parsers.common.TextParsingException: 
java.lang.ArrayIndexOutOfBoundsException - Index 2 out of bounds for length 2
SELECT schema_of_csv('abcdef', map('maxCharsPerColumn', '3'));
-- com.univocity.parsers.common.TextParsingException: Length of parsed input 
(4) exceeds the maximum number of characters defined in your parser settings 
(3).
-- Same input, other paths:
SELECT from_csv('a,b,c', 'a STRING', map('maxColumns', '2'));
-- [MALFORMED_CSV_RECORD] Malformed CSV record: a,b,c SQLSTATE: KD000
{code}
Expected:
{code}
SELECT schema_of_csv('a,b,c', map('maxColumns', '2'));
-- [MALFORMED_CSV_RECORD] Malformed CSV record: a,b,c SQLSTATE: KD000
{code}
h2. Why it matters
Users get a raw third-party parser exception with no error condition or 
SQLSTATE, inconsistent with from_csv and CSV reads, which already report 
MALFORMED_CSV_RECORD for the same input. Also reachable with default options 
for a sample row of more than 20480 columns.
h2. Related work
SPARK-57195 added UnivocityParser.parseLine to translate this exception into 
MALFORMED_CSV_RECORD for CSV schema inference and multiLine reads; SPARK-57515 
closed the header-row gap it left. The schema_of_csv expression path was not 
covered by either. Searched JIRA for schema_of_csv, SchemaOfCsv, maxColumns, 
maxCharsPerColumn, TextParsingException, and GitHub PRs for schema_of_csv / 
SchemaOfCsvEvaluator / CsvExpressionEvalUtils / MALFORMED_CSV_RECORD on 
2026-11-14.
h2. Proposed fix
Replace csvParser.parseLine(csv.toString) in SchemaOfCsvEvaluator.evaluate with 
UnivocityParser.parseLine(csvParser, csv.toString), reusing the existing 
translation (including its bounded bad-record content).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to