Joorgem opened a new pull request, #57671:
URL: https://github.com/apache/spark/pull/57671
### What changes were proposed in this pull request?
Adds a test to `CSVSuite` pinning what the CSV reader does with a line break
inside a
quoted value when `multiLine` is left at its default of `false`: the record
is cut at
the break into a NULL-tailed head plus a fragment, and the row count rises
by one.
The test also pins the three consequences that follow from it — the
`multiLine=true`
contrast, `DROPMALFORMED` discarding both halves, and the raw text captured
when
`columnNameOfCorruptRecord` is declared.
### Why are the changes needed?
The behaviour is not covered today. `CSVSuite` contains exactly one input
literal with
a newline inside quotes, in the `lineSep with 2 chars when multiLine set to
...` test,
and it is confined to the `multiLine = true` branch:
```scala
val inputData = if (multiLine) {
s"""name,"i am the${newLine} column1"${newLine}jack,30${newLine}tom,18"""
} else {
s"name,age${newLine}jack,30${newLine}tom,18"
}
```
The `multiLine = false` branch deliberately drops the quoted break, so the
default
path is never exercised with the input that distinguishes it.
`UnivocityParserSuite`
does not cover it either.
That leaves an untested behaviour that users do meet: a line break inside a
quoted
value is valid CSV (RFC 4180 §2.6) and appears in published datasets, so
reading such
a file with default options is an ordinary thing to do. The split is a
consequence of
the non-multiLine read path being line-oriented by design —
`HadoopFileLinesReader`
splits the input on the line separator before the tokenizer sees it, and
`UnivocityParser.parseLine` then receives one physical line at a time —
which is what
makes the file splittable. This PR does not propose changing that; it pins
what the
current design produces, so a future change to that path cannot alter it
silently.
`DROPMALFORMED` losing the whole source record rather than the damaged half
is the
part most worth having locked down, since it is the opposite of what the
mode's name
suggests for one bad line.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
This PR *is* the test. It was run in CI against this base before the PR was
opened:
`Build modules: sql - slow tests`, `sql - extended tests` and `sql - other
tests` all
passed, so the test holds in `CSVv1Suite`, `CSVv2Suite` and
`CSVLegacyTimeParserSuite`
alike.
The expected rows were derived before the Scala was written, from `pyspark`
3.5.9 with
the same three-record input and the same explicit three-column schema. 3.5.9
is two
majors behind this base, so those values were treated as a hypothesis to be
confirmed
by that CI run rather than as evidence in themselves:
| read | rows |
|---|---|
| default (`multiLine=false`) | 4 — `("1","ACME LTDA","SP")`, `("2","EMPRESA
COM",null)`, `("QUEBRA DE LINHA\"","RJ",null)`, `("3","OUTRA EMPRESA","MG")` |
| `multiLine=true` | 3, with the break preserved inside the value |
| `mode=DROPMALFORMED` | 2 — records 1 and 3; record 2 lost entirely |
| `columnNameOfCorruptRecord` declared | 4, both halves carrying raw text
`2,"EMPRESA COM` and `QUEBRA DE LINHA",RJ` |
Reproduction script: `docs/evidence/c5_expected_rows.py` in the authoring
repo (not
part of this patch).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
### Related
This is the third of three separate concerns in the same area, filed in
order:
1. [#57608](https://github.com/apache/spark/pull/57608) — documents what
`multiLine=false`
does to a line break inside a quoted value (open, reviewed, revision
applied).
2. [#57658](https://github.com/apache/spark/pull/57658) — corrects the
`PERMISSIVE`
"drops corrupt records" sentence in the CSV and JSON docs.
3. This PR — pins the behaviour #57608 documents, so it cannot change
silently.
Flagging the overlap explicitly since all three touch CSV: this one adds a
test and
changes no docs and no `main` code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
