JingsongLi commented on code in PR #11304:
URL: https://github.com/apache/paimon/pull/11304#discussion_r4642773107


##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonFormatTableScan.scala:
##########
@@ -55,7 +55,25 @@ case class PaimonFormatTableScan(
   }
 
   override def estimateStatistics: Statistics = {
-    val splits = inputSplits
-    PaimonStatistics(splits, readTableRowType, table.rowType(), 
table.statistics(), plannedRowCount)
+    // Spark can ask a fresh scan for statistics before filter and column 
pushdown.
+    // Avoid listing files in every catalog-managed partition just to estimate 
the table size.
+    if (
+      table.partitionManager() != null &&
+      pushedPartitionFilters.isEmpty && pushedDataFilters.isEmpty && 
pushedLimit.isEmpty &&
+      readTableRowType == tableRowType

Review Comment:
   Track Spark’s `pruneColumns` callback in `FormatTableScanBuilder` to scope 
the unknown estimate to requests before pushdown. Spark invokes that callback 
even when the query needs every column. `PaimonFormatTable.newScanBuilder` 
currently calls `pruneColumns(schema)` to set the initial schema, so separate 
that initialization from the callback. This lets the finalized scan keep its 
split-based estimates. [Spark’s scan builder 
flow](https://github.com/apache/spark/blob/v3.5.9/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala#L163-L188)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to