rdblue commented on a change in pull request #25955: [SPARK-29277][SQL] Add
early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r335211563
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
##########
@@ -55,7 +56,52 @@ case class DataSourceV2Relation(
}
override def computeStats(): Statistics = {
- val scan = newScanBuilder().build()
+ if (Utils.isTesting) {
+ // when testing, throw an exception if this computeStats method is
called because stats should
+ // not be accessed before pushing the projection and filters to create a
scan. otherwise, the
+ // stats are not accurate because they are based on a full table scan of
all columns.
+ throw new UnsupportedOperationException(
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
