liujiayi771 opened a new pull request, #11304: URL: https://github.com/apache/paimon/pull/11304
### Purpose Spark can ask a fresh DataSource V2 scan for statistics before filter and column pushdown. For a catalog-managed Format Table, the estimate currently plans input splits, which lists files in every partition. This can make a statistics request expensive on object stores even when the query will later prune most partitions. Return unknown size and row count for a full-schema scan that has no pushed filters or limit so Spark can use its default estimate. Scans with pushed partition predicates retain the existing split-based estimates and partition row counts. ### Tests - Added a regression in `CatalogManagedPartitionTest` that disables partition listing in the REST catalog and checks that an unfiltered scan can still report unknown statistics. - Extended the filtered-scan test to check that it still reports a size estimate. - Ran all four tests in `CatalogManagedPartitionTest` with the Spark 3 profile and ran the Maven formatting and static checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
