Github user rdblue commented on a diff in the pull request:
https://github.com/apache/spark/pull/22009#discussion_r209996219
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
---
@@ -76,41 +76,43 @@ object DataSourceV2Strategy extends Strategy {
/**
* Applies column pruning to the data source, w.r.t. the references of
the given expressions.
*
- * @return new output attributes after column pruning.
+ * @return the created `ScanConfig`(since column pruning is the last
step of operator pushdown),
+ * and new output attributes after column pruning.
--- End diff --
Okay, I see what you mean that we can't currently push down everything that
is done by a projection. I'm not sure how relevant that point is, though. The
implementation is still allowed to project more columns than Spark requests
based on the other operations that are pushed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
