rdblue commented on a change in pull request #26150: [SPARK-29501][SQL]
SupportsPushDownRequiredColumns should report pruned schema immediately
URL: https://github.com/apache/spark/pull/26150#discussion_r336663358
##########
File path:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java
##########
@@ -38,16 +36,10 @@
@Evolving
public interface Scan {
- /**
- * Returns the actual schema of this data source scan, which may be
different from the physical
- * schema of the underlying storage, as column pruning or other
optimizations may happen.
- */
- StructType readSchema();
-
/**
* A description string of this scan, which may includes information like:
what filters are
* configured for this scan, what's the value of some important options like
path, etc. The
- * description doesn't need to include {@link #readSchema()}, as Spark
already knows it.
+ * description doesn't need to include the schema, as Spark already knows it.
Review comment:
I would rather not remove `readSchema`. The scan should be self-describing
back to Spark, and the read schema is a key piece of information. In fact, I'd
like to add more methods to access other things, like pushed filters and
residual filters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
