HyukjinKwon commented on a change in pull request #24894: [SPARK-28058][DOC]
Add a note to DROPMALFORMED mode of CSV for column pruning
URL: https://github.com/apache/spark/pull/24894#discussion_r294350158
##########
File path: python/pyspark/sql/readwriter.py
##########
@@ -441,7 +441,12 @@ def csv(self, path, schema=None, sep=None, encoding=None,
quote=None, escape=Non
When it meets a record having fewer tokens than the length
of the schema, \
sets ``null`` to extra fields. When the record has more
tokens than the \
length of the schema, it drops extra tokens.
- * ``DROPMALFORMED`` : ignores the whole corrupted records.
+ * ``DROPMALFORMED`` : ignores the whole corrupted records.
Note that when CSV \
+ parser column pruning
(``spark.sql.csv.parser.columnPruning.enabled``) is \
+ enabled (it is enabled by default), the malformed columns
can be ignored during \
+ parsing if they are pruned, resulting the corrupted records
are not dropped. \
+ Disabling the column pruning feature can drop corrupted
records even malformed \
+ columns are not read.
Review comment:
I think it's good to note. But for JIRA itself, do you know the cause? I
haven't taken a close look but it looked to me like Univocity parses the tokens
weirdly in a certain condition. It is possible that it's a bug.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
