szehon-ho commented on code in PR #57608:
URL: https://github.com/apache/spark/pull/57608#discussion_r3694755018
##########
docs/sql-data-sources-csv.md:
##########
@@ -231,7 +231,8 @@ Data source options of CSV can be set via:
<tr>
<td><code>multiLine</code></td>
<td>false</td>
- <td>Allows a row to span multiple lines, by parsing line breaks within
quoted values as part of the value itself. CSV built-in functions ignore this
option.</td>
+ <td>Allows a row to span multiple lines, by parsing line breaks within
quoted values as part of the value itself. CSV built-in functions ignore this
option.<br>
+ When this option is disabled (the default), a line break inside a quoted
value terminates the record at that break: the value is truncated, the
remaining fields of the schema are set to <code>null</code>, and the rest of
the value begins a new record. Both resulting records are malformed; how they
are handled is controlled by <code>mode</code>.</td>
Review Comment:
The absolute claim that both halves are always malformed (and, in the `mode`
row, that `DROPMALFORMED` always discards both) looks a bit strong.
With `multiLine=false`, each physical line is parsed on its own. Univocity's
default `STOP_AT_DELIMITER` does not fail on an unclosed quote; Spark mainly
treats a record as corrupt when `tokens.length != schema.length`. So:
- 2-column schema, `1,"hello\nworld"` → first line parses as a valid `[1,
hello]` (silent truncation, no corrupt signal), second line is malformed
- 3-column schema, `1,"foo\nbar,baz,qux"` → first half malformed, second
half can be a valid 3-field row
Worth softening to: the record is split and the leading value is truncated;
either or both halves **may** be malformed (typically via token-count
mismatch), and `mode` then applies. Same for "remaining fields … set to `null`"
— that only holds when the leading half has fewer tokens than the schema.
Suggested direction for `multiLine`:
```html
When this option is disabled (the default), a line break inside a quoted
value terminates the record at that break: the value is truncated, and the rest
of the value begins a new record. Either or both of the resulting records may
then be malformed (for example when the token count no longer matches the
schema); how they are handled is controlled by <code>mode</code>.
```
And in the `mode` sentence, avoid "two malformed records" / "discards both
halves" as always-true — something like "may produce extra records, some or all
of which are malformed" would match the parser more closely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
