JingsongLi commented on code in PR #11304:
URL: https://github.com/apache/paimon/pull/11304#discussion_r4642703035
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonFormatTableScan.scala:
##########
@@ -55,7 +55,25 @@ case class PaimonFormatTableScan(
}
override def estimateStatistics: Statistics = {
- val splits = inputSplits
- PaimonStatistics(splits, readTableRowType, table.rowType(),
table.statistics(), plannedRowCount)
+ // Spark can ask a fresh scan for statistics before filter and column
pushdown.
+ // Avoid listing files in every catalog-managed partition just to estimate
the table size.
+ if (
+ table.partitionManager() != null &&
+ pushedPartitionFilters.isEmpty && pushedDataFilters.isEmpty &&
pushedLimit.isEmpty &&
+ readTableRowType == tableRowType
Review Comment:
**[P2] Preserve useful estimates for finalized full-schema scans.** This
condition also matches a fully planned scan that reads every column without
filters. On Spark 3.5.9, with AQE disabled and automatic broadcast enabled at a
1 MiB threshold, I inserted one row into a REST catalog-managed CSV table
`dim(id INT, dt STRING)` partitioned by `dt`. `SELECT r.id, d.id, d.dt FROM
range(1000000) r LEFT JOIN dim d ON r.id = coalesce(d.id, -1)` used
`BroadcastHashJoin` on master and `SortMergeJoin` with an exchange on each side
on this head. The full-schema `BatchScan` returned an unknown size even though
the dimension was eligible for broadcast. Please keep the early listing
avoidance while preserving a useful estimate for the finalized scan, for
example from catalog partition sizes if available, and cover this unfiltered
join plan in the regression tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]