JingsongLi commented on code in PR #11304:
URL: https://github.com/apache/paimon/pull/11304#discussion_r4642703035


##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonFormatTableScan.scala:
##########
@@ -55,7 +55,25 @@ case class PaimonFormatTableScan(
   }
 
   override def estimateStatistics: Statistics = {
-    val splits = inputSplits
-    PaimonStatistics(splits, readTableRowType, table.rowType(), 
table.statistics(), plannedRowCount)
+    // Spark can ask a fresh scan for statistics before filter and column 
pushdown.
+    // Avoid listing files in every catalog-managed partition just to estimate 
the table size.
+    if (
+      table.partitionManager() != null &&
+      pushedPartitionFilters.isEmpty && pushedDataFilters.isEmpty && 
pushedLimit.isEmpty &&
+      readTableRowType == tableRowType

Review Comment:
   **[P2] Preserve useful estimates for finalized full-schema scans.** This 
condition also matches a fully planned scan that reads every column without 
filters. On Spark 3.5.9, with AQE disabled and automatic broadcast enabled at a 
1 MiB threshold, I inserted one row into a REST catalog-managed CSV table 
`dim(id INT, dt STRING)` partitioned by `dt`. `SELECT r.id, d.id, d.dt FROM 
range(1000000) r LEFT JOIN dim d ON r.id = coalesce(d.id, -1)` used 
`BroadcastHashJoin` on master and `SortMergeJoin` with an exchange on each side 
on this head. The full-schema `BatchScan` returned an unknown size even though 
the dimension was eligible for broadcast. Please keep the early listing 
avoidance while preserving a useful estimate for the finalized scan, for 
example from catalog partition sizes if available, and cover this unfiltered 
join plan in the regression tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to