cloud-fan commented on code in PR #44939:
URL: https://github.com/apache/spark/pull/44939#discussion_r1471206040
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala:
##########
@@ -286,6 +288,16 @@ class CSVOptions(
*/
val isColumnPruningEnabled: Boolean = getBool(COLUMN_PRUNING, !multiLine &&
columnPruning)
+ /**
+ * Returns true if column pruning is enabled and there are no existence
column default values in
+ * the [[schema]]. This is useful when we want to disable column pruning
when there are such
+ * defaults, instead preferring to reach in each row and then post-process
it to substitute the
+ * default values after.
Review Comment:
do we know why CSV column pruning doesn't work with default values?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
