aokolnychyi commented on code in PR #57487:
URL: https://github.com/apache/spark/pull/57487#discussion_r3686206016
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsReportStatistics.java:
##########
@@ -36,4 +38,36 @@ public interface SupportsReportStatistics extends Scan {
* Returns the estimated statistics of this data source scan.
*/
Statistics estimateStatistics();
+
+ /**
+ * Returns the estimated size in bytes of this scan without computing full
statistics.
+ * <p>
+ * When cost-based optimization or plan statistics are disabled, Spark
primarily needs the scan's
+ * size in bytes (for example, for broadcast-join thresholding). This method
lets connectors serve
+ * that size estimate cheaply and avoid computing the full statistics. The
default implementation
+ * delegates to {@link #estimateStatistics()} and returns its {@code
sizeInBytes()}, so connectors
+ * that already compute statistics cheaply do not need to override this
method.
+ *
+ * @since 4.3.0
+ */
+ default OptionalLong estimateSizeInBytes() {
+ Statistics statistics = estimateStatistics();
+ return statistics != null ? statistics.sizeInBytes() :
OptionalLong.empty();
+ }
+
+ /**
+ * Returns whether the statistics reported by this scan already reflect all
filters that were
+ * fully pushed down to the data source.
+ * <p>
+ * When {@code true} (the default), the reported statistics describe exactly
the data the scan
+ * will produce. When {@code false}, they do <em>not</em> account for the
fully pushed filters
+ * (for example, they describe the whole table), so Spark may use those
fully pushed filters to
Review Comment:
Very good call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
