viirya opened a new pull request #24894: [SPARK-28058][DOC] Add a note to DROPMALFORMED mode of CSV for column pruning URL: https://github.com/apache/spark/pull/24894 ## What changes were proposed in this pull request? When using `DROPMALFORMED` mode, corrupted records aren't dropped if malformed columns aren't read. This behavior is due to CSV parser column pruning. Current doc of `DROPMALFORMED` doesn't mention the effect of column pruning. Users will be confused by the fact that `DROPMALFORMED` mode doesn't work as expected. This is a doc improvement to add a note to doc of `DROPMALFORMED` to explain it. ## How was this patch tested? N/A. This is just doc change.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
