jerry-024 opened a new issue, #859: URL: https://github.com/apache/paimon-rust/issues/859
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-rust/issues) and found nothing similar. ### Motivation DataFusion queries that group by partition columns and calculate `COUNT(*)` currently plan every live file and read data files. On tables with very large file counts, planning full manifest entries and column statistics can exhaust memory even though the result is derivable from metadata. Data-evolution tables also cannot simply sum per-file row counts because multiple column-group or blob files may cover the same row-ID ranges, and deletion-vector rows must be excluded. ### Solution Add an exact manifest-only path for `COUNT(*)` grouped by partition columns: - stream only the required manifest fields and apply partition pruning; - net ADD/DELETE entries using complete file identity; - union row-ID ranges only when data evolution is enabled; - subtract known deletion-vector cardinalities; - fall back to the ordinary scan whenever metadata cannot provide an exact answer. The implementation should avoid retaining all live file metadata and statistics. Its aggregation state may still grow with live DELETE entries and fragmented, non-adjacent row-ID ranges. ### Anything else? Implementation: apache/paimon-rust#858 ### Willingness to contribute - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
