leaves12138 opened a new pull request, #9046: URL: https://github.com/apache/paimon/pull/9046
### Purpose `DataEvolutionFileStoreScan.evolutionStats` runs once per Row-ID group and repeatedly projects the same file schemas and rebuilds the same field-ID arrays. This becomes expensive for wide data-evolution tables with many groups, even when the snapshot contains only a few distinct physical column layouts. This change adds a scan-local, thread-safe cache keyed by `(schemaId, writeCols, valueStatsCols)`. The cache reuses: - the projected data-file schema; - the projected stats schema; - physical field-ID arrays; - stats field-ID arrays. The existing static `evolutionStats` test entry point still uses a fresh cache, while normal scan planning shares one cache across all Row-ID groups in that scan. On a real-world metadata snapshot with 1,027 fields, 19,856 active files, and 17,670 Row-ID groups, the exact predicate scan improved from a hot average of 45.71 seconds to 10.32 seconds (4.43x faster). Both versions produced zero candidate files. ### Tests - `mvn -pl paimon-core -DwildcardSuites=none -Dtest=DataEvolutionFileStoreScanTest test` - Replayed the wide-table metadata snapshot against `master` and this branch; both returned the same candidate files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
