Joorgem opened a new pull request, #57608: URL: https://github.com/apache/spark/pull/57608
### What changes were proposed in this pull request? Extends the `multiLine` entry in the CSV data source options table to describe what happens when the option is left at its default of `false` and a quoted value contains a line break. The entry currently documents only what enabling the option does: > Allows a row to span multiple lines, by parsing line breaks within quoted values as part of the value itself. CSV built-in functions ignore this option. The added sentence states that the record is terminated at the line break, that the remaining schema fields are set to `null`, that the rest of the value begins a new record, and how each `mode` then behaves. One table row changed; no other files touched. ### Why are the changes needed? A line break inside a quoted field is valid CSV (RFC 4180 §2.6) and occurs in real published datasets, so a reader can meet this without doing anything unusual. The entry says what `multiLine=true` enables but never what the default costs, and the behaviour is not guessable from it: - the record is split in two, which **increases** the record count; - the leading half keeps plausible values for its first fields and `null` for the rest, so it can satisfy schema and not-null style checks; - under the default `PERMISSIVE` mode there is no signal at all unless the schema happens to declare `columnNameOfCorruptRecord`; - `DROPMALFORMED` discards *both* halves, losing the entire source record rather than a bad line; - a bare `count()` reports the inflated number without surfacing a malformed record, because column pruning means no column is parsed. The last two were the surprises worth writing down, and neither is inferable from the current text. SPARK-21356 was closed as Invalid because "the workaround looks so easy". That is true, and is exactly the argument for documenting the default: the workaround only helps a reader who knows they need it. ### Does this PR introduce _any_ user-facing change? No. Documentation only; no behaviour change. ### How was this patch tested? Documentation-only, so no tests were added. The described behaviour was verified against `pyspark` 3.5.9 using a three-record file whose second record contains a quoted line break, read with an explicit three-column schema: | read | result | |---|---| | default (`multiLine=false`) | 4 rows from 3 records: `2, 'EMPRESA COM', null` plus a fragment `'QUEBRA DE LINHA"', 'RJ', null` | | `multiLine=true` | 3 rows, values intact | | `mode=PERMISSIVE`, `collect()` | 4 rows, no error | | `mode=DROPMALFORMED`, `collect()` | 2 rows — both halves dropped | | `mode=FAILFAST`, `collect()` | raises | | any mode, bare `count()` | 4 rows, no error | With `columnNameOfCorruptRecord` declared, both halves are captured as corrupt, with raw text `2,"EMPRESA COM` and `QUEBRA DE LINHA",RJ`. `git diff --check` is clean. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
