cloud-fan commented on a change in pull request #26150: [SPARK-29501][SQL]
SupportsPushDownRequiredColumns should report pruned schema immediately
URL: https://github.com/apache/spark/pull/26150#discussion_r337462178
##########
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
##########
@@ -38,16 +36,10 @@
@Evolving
public interface Scan {
- /**
- * Returns the actual schema of this data source scan, which may be
different from the physical
- * schema of the underlying storage, as column pruning or other
optimizations may happen.
- */
- StructType readSchema();
-
/**
* A description string of this scan, which may includes information like:
what filters are
* configured for this scan, what's the value of some important options like
path, etc. The
- * description doesn't need to include {@link #readSchema()}, as Spark
already knows it.
+ * description doesn't need to include the schema, as Spark already knows it.
Review comment:
I'm not sure this is the right direction. If we add more pushdown in the
future (limit, aggregate, etc.), are we going to add methods to `Scan` every
time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
