cloud-fan commented on a change in pull request #25955: [SPARK-29277][SQL] Add
early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r333962848
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
##########
@@ -55,7 +56,52 @@ case class DataSourceV2Relation(
}
override def computeStats(): Statistics = {
- val scan = newScanBuilder().build()
+ if (Utils.isTesting) {
+ // when testing, throw an exception if this computeStats method is
called because stats should
+ // not be accessed before pushing the projection and filters to create a
scan. otherwise, the
+ // stats are not accurate because they are based on a full table scan of
all columns.
+ throw new UnsupportedOperationException(
+ s"BUG: computeStats called before pushdown on DSv2 relation: $name")
+ } else {
+ // when not testing, return stats because bad stats are better than
failing a query
+ newScanBuilder() match {
+ case r: SupportsReportStatistics =>
+ val statistics = r.estimateStatistics()
+ DataSourceV2Relation.transformV2Stats(statistics, None,
conf.defaultSizeInBytes)
+ case _ =>
+ Statistics(sizeInBytes = conf.defaultSizeInBytes)
+ }
+ }
+ }
+
+ override def newInstance(): DataSourceV2Relation = {
+ copy(output = output.map(_.newInstance()))
+ }
+}
+
+/**
+ * A logical plan for a DSv2 table with a scan already created.
+ *
+ * This is used in the optimizer to push filters and projection down before
conversion to physical
+ * plan. This ensures that the stats that are used by the optimizer account
for the filters and
+ * projection that will be pushed down.
+ *
+ * @param table a DSv2 [[Table]]
+ * @param scan a DSv2 [[Scan]]
+ * @param output the output attributes of this relation
+ */
+case class DataSourceV2ScanRelation(
Review comment:
Makes sense. But it's still better to reduce the maintenance cost a bit. How
about we clearly define the life cycle of a v2 relation as:
`UnresolvedRelation` (before analysis) -> `DataSourceV2Relation` (after
analysis) -> `DataSourceV2ScanRelation` (after optimization) -> the physical
scan node.
Then we only need to match one node:
1. if dealing with `analyzedPlan`, match `DataSourceV2Relation`
2. if dealing with `optimizedPlan`, match `DataSourceV2ScanRelation`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
