Liang-Chi Hsieh created SPARK-28082: ---------------------------------------
Summary: Add a note to DROPMALFORMED mode of CSV for column pruning Key: SPARK-28082 URL: https://issues.apache.org/jira/browse/SPARK-28082 Project: Spark Issue Type: Documentation Components: PySpark, SQL Affects Versions: 3.0.0 Reporter: Liang-Chi Hsieh This is inspired by SPARK-28058. When using {{DROPMALFORMED}} mode, corrupted records aren't dropped if malformed columns aren't read. This behavior is due to CSV parser column pruning. Current doc of {{DROPMALFORMED}} doesn't mention the effect of column pruning. Users will be confused by the fact that {{DROPMALFORMED}} mode doesn't work as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org