JingsongLi commented on code in PR #11304:
URL: https://github.com/apache/paimon/pull/11304#discussion_r4642773107
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/paimon/spark/PaimonFormatTableScan.scala:
##########
@@ -55,7 +55,25 @@ case class PaimonFormatTableScan(
}
override def estimateStatistics: Statistics = {
- val splits = inputSplits
- PaimonStatistics(splits, readTableRowType, table.rowType(),
table.statistics(), plannedRowCount)
+ // Spark can ask a fresh scan for statistics before filter and column
pushdown.
+ // Avoid listing files in every catalog-managed partition just to estimate
the table size.
+ if (
+ table.partitionManager() != null &&
+ pushedPartitionFilters.isEmpty && pushedDataFilters.isEmpty &&
pushedLimit.isEmpty &&
+ readTableRowType == tableRowType
Review Comment:
Track Spark’s `pruneColumns` callback in `FormatTableScanBuilder` to scope
the unknown estimate to requests before pushdown. Spark invokes that callback
even when the query needs every column. `PaimonFormatTable.newScanBuilder`
currently calls `pruneColumns(schema)` to set the initial schema, so separate
that initialization from the callback. This lets the finalized scan keep its
split-based estimates. [Spark’s scan builder
flow](https://github.com/apache/spark/blob/v3.5.9/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala#L163-L188)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]