Liang-Chi Hsieh created SPARK-28082:
---------------------------------------
Summary: Add a note to DROPMALFORMED mode of CSV for column pruning
Key: SPARK-28082
URL: https://issues.apache.org/jira/browse/SPARK-28082
Project: Spark
Issue Type: Documentation
Components: PySpark, SQL
Affects Versions: 3.0.0
Reporter: Liang-Chi Hsieh
This is inspired by SPARK-28058.
When using {{DROPMALFORMED}} mode, corrupted records aren't dropped if
malformed columns aren't read. This behavior is due to CSV parser column
pruning. Current doc of {{DROPMALFORMED}} doesn't mention the effect of column
pruning. Users will be confused by the fact that {{DROPMALFORMED}} mode doesn't
work as expected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
