HyukjinKwon commented on code in PR #57608:
URL: https://github.com/apache/spark/pull/57608#discussion_r3716750277
##########
docs/sql-data-sources-csv.md:
##########
@@ -207,7 +207,7 @@ Data source options of CSV can be set via:
<tr>
<td><code>mode</code></td>
<td>PERMISSIVE</td>
- <td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default).<br>
+ <td>Allows a mode for dealing with corrupt records during parsing. It
supports the following case-insensitive modes. Note that Spark tries to parse
only required columns in CSV under column pruning. Therefore, corrupt records
can be different based on required set of fields. This behavior can be
controlled by <code>spark.sql.csv.parser.columnPruning.enabled</code> (enabled
by default). In particular, when <code>multiLine</code> is disabled, a quoted
value containing a line break splits one source record into two before
<code>mode</code> is applied. Each of the two is then evaluated on its own, so
a record whose token count happens to match the schema is retained as valid
even though its value was truncated. This holds under
<code>DROPMALFORMED</code> as well, since that mode drops only the records it
finds malformed. An action requiring no columns (a bare <code>count()</code>,
for example) may surface none of this because of column pruning.<br>
Review Comment:
`a line break` technically it is a `lineSep` option
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
