[ https://issues.apache.org/jira/browse/SPARK-28082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-28082. ---------------------------------- Resolution: Duplicate > Add a note to DROPMALFORMED mode of CSV for column pruning > ---------------------------------------------------------- > > Key: SPARK-28082 > URL: https://issues.apache.org/jira/browse/SPARK-28082 > Project: Spark > Issue Type: Documentation > Components: Documentation, PySpark, SQL > Affects Versions: 3.0.0 > Reporter: Liang-Chi Hsieh > Priority: Trivial > > This is inspired by SPARK-28058. > When using {{DROPMALFORMED}} mode, corrupted records aren't dropped if > malformed columns aren't read. This behavior is due to CSV parser column > pruning. Current doc of {{DROPMALFORMED}} doesn't mention the effect of > column pruning. Users will be confused by the fact that {{DROPMALFORMED}} > mode doesn't work as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org