szehon-ho commented on code in PR #57608:
URL: https://github.com/apache/spark/pull/57608#discussion_r3715130544
##########
docs/sql-data-sources-csv.md:
##########
@@ -207,7 +207,7 @@ Data source options of CSV can be set via:
<tr>
<td><code>mode</code></td>
<td>PERMISSIVE</td>
- <td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default).<br>
+ <td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default). In particular, when <code>multiLine</code> is disabled, a quoted
value that contains a line break is split, which increases the record count,
and some or all of the resulting records may be malformed — typically when the
token count no longer matches the schema. <code>mode</code> then applies to
each record independently, so a half whose token count happens to match the
schema is retained as a valid row even though its value was truncated,
including under <code>DROPMALFORMED</code>, which drops only the malformed
halves. An action that requires no columns (for example a bare
<code>count()</code>
) may surface none of this because of column pruning.<br>
Review Comment:
Latest wording looks good to me. The only remaining concern is the same one
from this thread: until #57658 lands, this page still contradicts itself
because the new text correctly treats token-count mismatch as malformed, while
the existing `PERMISSIVE` bullet says the opposite. If #57658 is expected to
merge first, I am fine with this as-is; otherwise we should fold that bullet
correction into this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
