szehon-ho commented on code in PR #57608:
URL: https://github.com/apache/spark/pull/57608#discussion_r3668241473
##########
docs/sql-data-sources-csv.md:
##########
@@ -231,7 +231,8 @@ Data source options of CSV can be set via:
<tr>
<td><code>multiLine</code></td>
<td>false</td>
- <td>Allows a row to span multiple lines, by parsing line breaks within
quoted values as part of the value itself. CSV built-in functions ignore this
option.</td>
+ <td>Allows a row to span multiple lines, by parsing line breaks within
quoted values as part of the value itself. CSV built-in functions ignore this
option.<br>
+ When this option is disabled, a line break inside a quoted value
terminates the record at that break: the value is truncated, the remaining
fields of the schema are set to <code>null</code>, and the rest of the value
begins a new record. Both halves are malformed records, so what happens next
follows <code>mode</code>: <code>FAILFAST</code> raises an error,
<code>DROPMALFORMED</code> discards both halves and therefore loses the whole
source record, and <code>PERMISSIVE</code> retains both, without any signal
unless the schema declares <code>columnNameOfCorruptRecord</code>. Note also
that the split <em>increases</em> the number of records, and that an action
requiring no columns (a bare <code>count()</code>, for instance) may surface
none of this, because of parser column pruning.</td>
Review Comment:
Thanks for documenting this — the default-`multiLine` gotcha is worth
calling out. Suggestion on placement: keep the option-specific fact under
`multiLine`, and move the `mode` / column-pruning surprises next to `mode`,
which already documents corrupt handling and pruning.
**`multiLine`** — keep only the split behavior:
```html
<td>Allows a row to span multiple lines, by parsing line breaks within
quoted values as part of the value itself. CSV built-in functions ignore this
option.<br>
When this option is disabled (the default), a line break inside a quoted
value terminates the record at that break: the value is truncated, the
remaining fields of the schema are set to <code>null</code>, and the rest of
the value begins a new record. Both resulting records are malformed; how they
are handled is controlled by <code>mode</code>.</td>
```
**`mode`** — append after the existing column-pruning sentence (before the
`<ul>`):
```html
<td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default). In particular, when <code>multiLine</code> is disabled, a quoted
value that contains a line break is split into two malformed records, which
increases the record count: <code>DROPMALFORMED</code> discards both halves and
therefore loses the whole source record, <code>FAILFAST</code> raises an error,
and <code>PERMISSIVE</code> retains both without a signal unless the schema
declares <code>columnNameOfCorruptRecord</code>. An action that requires no
columns (for example a bare <code>count()</code>) may surface none of this
because of column pruning.<br>
<ul>
<li><code>PERMISSIVE</code>: when it meets a corrupted record, puts the
malformed string into a field configured by
<code>columnNameOfCorruptRecord</code>, and sets malformed fields to
<code>null</code>. To keep corrupt records, an user can set a string type field
named <code>columnNameOfCorruptRecord</code> in an user-defined schema. If a
schema does not have the field, it drops corrupt records during parsing. A
record with less/more tokens than schema is not a corrupted record to CSV. When
it meets a record having fewer tokens than the length of the schema, sets
<code>null</code> to extra fields. When the record has more tokens than the
length of the schema, it drops extra tokens.</li>
<li><code>DROPMALFORMED</code>: ignores the whole corrupted records. This
mode is unsupported in the CSV built-in functions.</li>
<li><code>FAILFAST</code>: throws an exception when it meets corrupted
records.</li>
</ul>
</td>
```
That keeps `multiLine` to what the default does to the record, and puts the
DROPMALFORMED / PERMISSIVE / `count()` caveats with the option that already
covers corrupt handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
