This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new e1472df93869 [MINOR][DOC] Document CSV read behaviour when multiLine
is disabled
e1472df93869 is described below
commit e1472df93869194edd629619d0bb88c3a9eeb068
Author: Jorge Molina <[email protected]>
AuthorDate: Mon Aug 10 23:07:40 2026 -0700
[MINOR][DOC] Document CSV read behaviour when multiLine is disabled
### What changes were proposed in this pull request?
Extends the `multiLine` entry in the CSV data source options table to
describe what happens when the option is left at its default of `false` and a
quoted value contains a line break.
The entry currently documents only what enabling the option does:
> Allows a row to span multiple lines, by parsing line breaks within quoted
values as part of the value itself. CSV built-in functions ignore this option.
The added sentence states that the record is terminated at the line break,
that the remaining schema fields are set to `null`, that the rest of the value
begins a new record, and how each `mode` then behaves.
One table row changed; no other files touched.
### Why are the changes needed?
A line break inside a quoted field is valid CSV (RFC 4180 §2.6) and occurs
in real published datasets, so a reader can meet this without doing anything
unusual. The entry says what `multiLine=true` enables but never what the
default costs, and the behaviour is not guessable from it:
- the record is split in two, which **increases** the record count;
- the leading half keeps plausible values for its first fields and `null`
for the rest, so it can satisfy schema and not-null style checks;
- under the default `PERMISSIVE` mode there is no signal at all unless the
schema happens to declare `columnNameOfCorruptRecord`;
- `DROPMALFORMED` discards *both* halves, losing the entire source record
rather than a bad line;
- a bare `count()` reports the inflated number without surfacing a
malformed record, because column pruning means no column is parsed.
The last two were the surprises worth writing down, and neither is
inferable from the current text.
SPARK-21356 was closed as Invalid because "the workaround looks so easy".
That is true, and is exactly the argument for documenting the default: the
workaround only helps a reader who knows they need it.
### Does this PR introduce _any_ user-facing change?
No. Documentation only; no behaviour change.
### How was this patch tested?
Documentation-only, so no tests were added. The described behaviour was
verified against `pyspark` 3.5.9 using a three-record file whose second record
contains a quoted line break, read with an explicit three-column schema:
| read | result |
|---|---|
| default (`multiLine=false`) | 4 rows from 3 records: `2, 'EMPRESA COM',
null` plus a fragment `'QUEBRA DE LINHA"', 'RJ', null` |
| `multiLine=true` | 3 rows, values intact |
| `mode=PERMISSIVE`, `collect()` | 4 rows, no error |
| `mode=DROPMALFORMED`, `collect()` | 2 rows — both halves dropped |
| `mode=FAILFAST`, `collect()` | raises |
| any mode, bare `count()` | 4 rows, no error |
With `columnNameOfCorruptRecord` declared, both halves are captured as
corrupt, with raw text `2,"EMPRESA COM` and `QUEBRA DE LINHA",RJ`.
`git diff --check` is clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
Closes #57608 from Joorgem/docs-csv-multiline-disabled.
Authored-by: Jorge Molina <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
(cherry picked from commit bf76295bba7500407e7fdce948ef4c056cde078c)
Signed-off-by: Szehon Ho <[email protected]>
---
docs/sql-data-sources-csv.md | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md
index fcc70622262f..bf63fe2be5d3 100644
--- a/docs/sql-data-sources-csv.md
+++ b/docs/sql-data-sources-csv.md
@@ -207,7 +207,7 @@ Data source options of CSV can be set via:
<tr>
<td><code>mode</code></td>
<td>PERMISSIVE</td>
- <td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default).<br>
+ <td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default). In particular, when <code>multiLine</code> is disabled, a quoted
value containing a line separator [...]
<ul>
<li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To capture the malformed string, a user can set a string
type field named <code>columnNameOfCorruptRecord</code> in a user-defined
schema. If a schema does not have the field, the corrupt record is still
retained with its malformed fields set to <code>null</code>, but the malformed
[...]
<li><code>DROPMALFORMED</code>: ignores the whole corrupted records.
This mode is unsupported in the CSV built-in functions.</li>
@@ -231,7 +231,8 @@ Data source options of CSV can be set via:
<tr>
<td><code>multiLine</code></td>
<td>false</td>
- <td>Allows a row to span multiple lines, by parsing line breaks within
quoted values as part of the value itself. CSV built-in functions ignore this
option.</td>
+ <td>Allows a row to span multiple lines, by parsing line breaks within
quoted values as part of the value itself. CSV built-in functions ignore this
option.<br>
+ When this option is disabled (the default), a line break inside a quoted
value terminates the record at that break: the value is truncated, and the rest
of the value begins a new record. Either or both of the resulting records may
then be malformed (for example when the token count no longer matches the
schema); how they are handled is controlled by <code>mode</code>.</td>
<td>read</td>
</tr>
<tr>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
