Jorge Molina created SPARK-58457:
------------------------------------
Summary: CSV and JSON docs claim PERMISSIVE drops corrupt records
when no columnNameOfCorruptRecord field is declared; it never does
Key: SPARK-58457
URL: https://issues.apache.org/jira/browse/SPARK-58457
Project: Spark
Issue Type: Documentation
Components: SQL
Affects Versions: 5.0.0
Reporter: Jorge Molina
`docs/sql-data-sources-csv.md` and `docs/sql-data-sources-json.md` both describe
`PERMISSIVE` mode with a character-identical pair of sentences:
{quote}
*To keep corrupt records*, an user can set a string type field named
`columnNameOfCorruptRecord` in an user-defined schema. *If a schema does not
have the
field, it drops corrupt records during parsing.*
{quote}
Neither sentence matches the behaviour of either data source.
`columnNameOfCorruptRecord` does not control whether a corrupt record is kept —
it is
always kept, with its malformed fields set to `null`. The field controls
whether the
*malformed string itself* is available.
This is the sentence a reader consults when deciding whether they need the
field, and
it points them the wrong way — it implies a schema *without* the field is the
safe
choice because bad records disappear. In fact they are admitted,
indistinguishable
from records that were genuinely null at the source, with no error and no
signal.
h3. The test suite on master already asserts the correct behaviour, in both
data sources
*CSV* — `CSVSuite`'s test {{SPARK-18699 put malformed records in a
`columnNameOfCorruptRecord` field}} reads `test-data/value-malformed.csv`,
whose two
records are `0,2013-111_11 12:13:14` (unparseable date) and `1,1983-08-04`. With
schema `a INT, b DATE` and *no* corrupt-record field it asserts:
{code:scala}
checkAnswer(df1, Row(0, null) :: Row(1, java.sql.Date.valueOf("1983-08-04")) ::
Nil)
{code}
Two rows — the corrupt record kept, its bad field nulled. Were the documented
sentence accurate, this assertion would expect one row.
*JSON* — `JsonSuite` is more direct still. The test is named {{Corrupt records:
PERMISSIVE mode, without designated column for malformed records}}, reads the
`corruptRecords` fixture with a three-column user-defined schema and no
corrupt-record
column, and asserts five rows — four entirely null — under Spark's own comment:
{code:scala}
// Corrupted records are replaced with null
Row(null, null, null),
{code}
So the correction is not a claim about behaviour that needs establishing; it is
alignment with what the project already asserts.
h3. Corroborated by direct measurement
`pyspark` 3.5.9, three source records per case, `PERMISSIVE`, user-defined
schema
`a INT, b STRING`:
|| format || bad record || rows without the field || rows with the field ||
flagged corrupt ||
| JSON | not parseable as JSON at all | 3 | 3 | 1 |
| JSON | valid JSON, wrong type for schema | 3 | 3 | 1 |
| CSV | wrong type for schema | 3 | 3 | 1 |
| CSV | unterminated quote | 3 | 3 | 0 |
Identical counts in every case. The JSON rows matter most: JSON is where the
sentence
would be most defensible, since a whole JSON record genuinely can be
unparseable —
and it is still retained.
Measured on `pyspark` 3.5.9 rather than on `master`, so this is corroboration;
the two
`master` assertions above are the load-bearing evidence.
h3. Proposed fix
Replace *both* sentences, in both files, with:
{quote}
To capture the malformed string, an user can set a string type field named
`columnNameOfCorruptRecord` in an user-defined schema. If a schema does not
have the
field, the corrupt record is still retained with its malformed fields set to
`null`,
but the malformed string is not available.
{quote}
Both sentences have to move together: correcting only the second would leave the
paragraph contradicting itself, since the first assigns retention to the field
while
the second would deny retention without it. One line per file. PR to follow.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
