This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 96931993a5bb [SPARK-58457][SQL][DOC] Correct the PERMISSIVE mode claim
that corrupt records are dropped
96931993a5bb is described below
commit 96931993a5bb85edd4d0e26e8f8c43bca8e517e0
Author: Jorge Molina <[email protected]>
AuthorDate: Mon Aug 10 23:02:52 2026 -0700
[SPARK-58457][SQL][DOC] Correct the PERMISSIVE mode claim that corrupt
records are dropped
### What changes were proposed in this pull request?
Both the CSV and JSON data source option tables describe `PERMISSIVE` mode
with a
character-identical pair of sentences:
> **To keep corrupt records**, an user can set a string type field named
> `columnNameOfCorruptRecord` in an user-defined schema. **If a schema does
not have
> the field, it drops corrupt records during parsing.**
Neither is correct for either data source. `columnNameOfCorruptRecord` does
not
control whether a corrupt record is kept — it is always kept, with its
malformed
fields set to `null`. The field controls whether the *malformed string
itself* is
available.
This PR replaces both sentences in both files:
> To capture the malformed string, an user can set a string type field named
> `columnNameOfCorruptRecord` in an user-defined schema. If a schema does
not have the
> field, the corrupt record is still retained with its malformed fields set
to `null`,
> but the malformed string is not available.
Both sentences move together on purpose. Correcting only the second one
would leave
the paragraph contradicting itself, since the first assigns retention to
the field
while the second would deny retention without it.
### Why are the changes needed?
The sentence is the one a reader consults when deciding whether they need
`columnNameOfCorruptRecord`, and it points them the wrong way. Read
literally, a
schema *without* the field is the safe choice, because bad records
disappear rather
than entering the result. The truth is the opposite: without the field,
malformed
records are still there, indistinguishable from records that were genuinely
null in
the source, and there is no signal that anything was wrong.
That matters most in exactly the situation the docs are meant to protect: a
pipeline whose schema does not declare the field will silently admit partial
records, and no error, count or mode setting reveals it.
### Does this PR introduce _any_ user-facing change?
No. Documentation only; no behaviour change. The behaviour being documented
is
already asserted by the test suite on `master` (see below), so this
corrects the
description rather than describing anything new.
### How was this patch tested?
Documentation-only change, so no tests were added. No new verification is
needed
either: **`master`'s own test suite already asserts the corrected
behaviour, in both
data sources.**
**CSV** — `CSVSuite`'s `SPARK-18699 put malformed records in a
'columnNameOfCorruptRecord' field` reads `test-data/value-malformed.csv`,
whose two
records are `0,2013-111_11 12:13:14` (an unparseable date) and
`1,1983-08-04`. With
schema `a INT, b DATE` and **no** `columnNameOfCorruptRecord` field it
asserts:
```scala
checkAnswer(df1, Row(0, null) :: Row(1,
java.sql.Date.valueOf("1983-08-04")) :: Nil)
```
Two rows. The corrupt record is kept with the unparseable field nulled.
Were the
documented sentence accurate, this assertion would expect one row.
**JSON** — `JsonSuite` is even more direct. The test is named
`Corrupt records: PERMISSIVE mode, without designated column for malformed
records`,
reads the `corruptRecords` fixture with a three-column user-defined schema
and no
corrupt-record column, and asserts five rows — four of them entirely null —
under
Spark's own comment:
```scala
// Corrupted records are replaced with null
Row(null, null, null),
```
Nothing is dropped in either data source, on `master`, by the project's own
assertions.
**Corroborated by direct measurement** across four kinds of bad record on
`pyspark` 3.5.9 — three source records per case, `PERMISSIVE`, user-defined
schema
`a INT, b STRING`:
| format | bad record | rows without the field | rows with the field |
flagged corrupt |
|---|---|---|---|---|
| JSON | not parseable as JSON at all | 3 | 3 | 1 |
| JSON | valid JSON, wrong type for schema | 3 | 3 | 1 |
| CSV | wrong type for schema | 3 | 3 | 1 |
| CSV | unterminated quote | 3 | 3 | 0 |
The counts are identical either way in every case — nothing is dropped.
Declaring
the field adds the ability to see the record, not the decision to keep it.
The JSON rows are worth calling out, since JSON is the data source where the
sentence would be most defensible: a whole JSON record genuinely can be
unparseable, and it is still retained.
Reproduction script: `docs/evidence/c4_repro_json_vs_csv_drop.py` in the
authoring
repo (not part of this patch). Note that this measurement was taken on
3.5.9, not on
`master` — it is corroboration, and the two `master` test assertions above
are the
load-bearing evidence.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
### Related
- [#57608](https://github.com/apache/spark/pull/57608) documents what
`multiLine=false`
does to a line break inside a quoted value. It touches the same CSV
options table, so
I am flagging it to avoid the appearance of a duplicate — the two changes
are
different entries (`multiLine` there, `mode` here) and do not overlap
textually.
- One thing deliberately left out of this PR: *why* the sentence is there —
whether it
was copied between the two files or was true in an older release. I did
not
investigate it, so I make no claim about it. This change only corrects
what the
behaviour is today.
Closes #57658 from Joorgem/docs-permissive-never-drops.
Authored-by: Jorge Molina <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
---
docs/sql-data-sources-csv.md | 2 +-
docs/sql-data-sources-json.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md
index 9dfe9739b7ac..fcc70622262f 100644
--- a/docs/sql-data-sources-csv.md
+++ b/docs/sql-data-sources-csv.md
@@ -209,7 +209,7 @@ Data source options of CSV can be set via:
<td>PERMISSIVE</td>
<td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default).<br>
<ul>
- <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To keep corrupt records, an user can set a string type field
named <code>columnNameOfCorruptRecord</code> in an user-defined schema. If a
schema does not have the field, it drops corrupt records during parsing. A
record with less/more tokens than schema is not a corrupted record to [...]
+ <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To capture the malformed string, a user can set a string
type field named <code>columnNameOfCorruptRecord</code> in a user-defined
schema. If a schema does not have the field, the corrupt record is still
retained with its malformed fields set to <code>null</code>, but the malformed
[...]
<li><code>DROPMALFORMED</code>: ignores the whole corrupted records.
This mode is unsupported in the CSV built-in functions.</li>
<li><code>FAILFAST</code>: throws an exception when it meets corrupted
records.</li>
</ul>
diff --git a/docs/sql-data-sources-json.md b/docs/sql-data-sources-json.md
index ba8691320f90..a30156b7a708 100644
--- a/docs/sql-data-sources-json.md
+++ b/docs/sql-data-sources-json.md
@@ -170,7 +170,7 @@ Data source options of JSON can be set via:
<td><code>PERMISSIVE</code></td>
<td>Allows a mode for dealing with corrupt records during parsing.<br>
<ul>
- <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To keep corrupt records, an user can set a string type field
named <code>columnNameOfCorruptRecord</code> in an user-defined schema. If a
schema does not have the field, it drops corrupt records during parsing. When
inferring a schema, it implicitly adds a <code>columnNameOfCorrupt [...]
+ <li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To capture the malformed string, a user can set a string
type field named <code>columnNameOfCorruptRecord</code> in a user-defined
schema. If a schema does not have the field, the corrupt record is still
retained with its malformed fields set to <code>null</code>, but the malformed
[...]
<li><code>DROPMALFORMED</code>: ignores the whole corrupted records.
This mode is unsupported in the JSON built-in functions.</li>
<li><code>FAILFAST</code>: throws an exception when it meets corrupted
records.</li>
</ul>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
