liujiayi771 opened a new pull request, #11304:
URL: https://github.com/apache/paimon/pull/11304

   ### Purpose
   
   Spark can ask a fresh DataSource V2 scan for statistics before filter and 
column pushdown. For a catalog-managed Format Table, the estimate currently 
plans input splits, which lists files in every partition. This can make a 
statistics request expensive on object stores even when the query will later 
prune most partitions.
   
   Return unknown size and row count for a full-schema scan that has no pushed 
filters or limit so Spark can use its default estimate. Scans with pushed 
partition predicates retain the existing split-based estimates and partition 
row counts.
   
   ### Tests
   
   - Added a regression in `CatalogManagedPartitionTest` that disables 
partition listing in the REST catalog and checks that an unfiltered scan can 
still report unknown statistics.
   - Extended the filtered-scan test to check that it still reports a size 
estimate.
   - Ran all four tests in `CatalogManagedPartitionTest` with the Spark 3 
profile and ran the Maven formatting and static checks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to