Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r209726516
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
---
@@ -76,41 +76,43 @@ object DataSourceV2Strategy extends Strategy {
/**
* Applies column pruning to the data source, w.r.t. the references of
the given expressions.
*
- * @return new output attributes after column pruning.
+ * @return the created `ScanConfig`(since column pruning is the last
step of operator pushdown),
+ * and new output attributes after column pruning.
--- End diff --
Why is column pruning not "technically" operator pushdown? This is done by
pushing a Project operator to the source.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
