Jorge Molina created SPARK-58458:
------------------------------------
Summary: No test covers the CSV record split caused by a quoted
line break when multiLine is disabled
Key: SPARK-58458
URL: https://issues.apache.org/jira/browse/SPARK-58458
Project: Spark
Issue Type: Test
Components: SQL
Affects Versions: 5.0.0
Reporter: Jorge Molina
A line break inside a quoted value is valid CSV (RFC 4180 §2.6). With the
default
`multiLine=false`, the CSV reader cuts such a record at the break: the head
keeps its
leading fields with the rest of the schema nulled, and the remainder of the
quoted
value starts a new record. The row count rises by one.
`CSVSuite` does not cover this. The suite contains exactly one input literal
with a
newline inside quotes, in {{lineSep with 2 chars when multiLine set to ...}},
and it is
confined to the {{multiLine = true}} branch:
{code:scala}
val inputData = if (multiLine) {
s"""name,"i am the${newLine} column1"${newLine}jack,30${newLine}tom,18"""
} else {
s"name,age${newLine}jack,30${newLine}tom,18"
}
{code}
The {{multiLine = false}} branch drops the quoted break, so the default path is
never
exercised with the input that distinguishes it. `UnivocityParserSuite` does not
cover
it either.
h3. Why it is worth a test
The behaviour follows from the non-multiLine read path being line-oriented by
design:
`HadoopFileLinesReader` splits the input on the line separator before the
tokenizer
sees it, and `UnivocityParser.parseLine` then receives one physical line at a
time.
That is what makes the file splittable, and this ticket does *not* propose
changing it.
It proposes pinning what the current design produces, so a future change to
that path
cannot alter it silently.
Two consequences are worth locking down in particular:
* {{DROPMALFORMED}} discards *both* halves, so the whole source record is lost
rather
than the damaged half — the opposite of what the mode's name suggests for one
bad
line.
* {{PERMISSIVE}} without {{columnNameOfCorruptRecord}} — the default — retains
both
halves with no signal. The split *adds* a row while rejecting the fragment
*removes*
one, so a row-count reconciliation can pass over a record that lost its
trailing
fields.
h3. Measured behaviour to be asserted
Measured on `pyspark` 3.5.9 — three records with a quoted line break in the
second,
explicit three-column schema. *These are the values to be asserted, not yet
confirmed
on `master`* (5.0.0-SNAPSHOT); the fork's CI run is what confirms them, and the
PR will
not be opened before it is green.
|| read || rows ||
| default ({{multiLine=false}}) | 4: {{("1","ACME LTDA","SP")}},
{{("2","EMPRESA COM",null)}}, {{("QUEBRA DE LINHA\"","RJ",null)}},
{{("3","OUTRA EMPRESA","MG")}} |
| {{multiLine=true}} | 3, break preserved inside the value |
| {{mode=DROPMALFORMED}} | 2: records 1 and 3; record 2 lost entirely |
| {{columnNameOfCorruptRecord}} declared | 4, both halves carrying their raw
text |
PR to follow, test-only, one test added to `CSVSuite`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
