leaves12138 opened a new pull request, #9045: URL: https://github.com/apache/paimon/pull/9045
### Purpose Optimize scan planning for data-evolution tables with wide schemas. On a production table (1027 columns, ~19856 active files in one snapshot forming ~17670 row-id groups), `DataEvolutionFileStoreScan.postFilterManifestEntries` is extremely slow. Each row-id group calls `filterByStats(group)` -> `evolutionStats()`, whose field matching is a triple nested linear scan: for every target field it linearly scans the file's field ids, and on a hit linearly scans the stats field ids again. For a group with K files covering F columns this is O(F * K * F) comparisons, repeated per group, single-threaded. ### Changes In `evolutionStats`: - Build a `fieldId -> index` HashMap per file for both the file schema and the stats schema, turning each lookup into O(1). - Track the number of unresolved fields and stop iterating files once every field is resolved. - Precompute target field types once instead of calling `schema.fields().get(j).type()` per lookup. Behavior is preserved: - Fields are still resolved from the highest-sequence-number file first (metas sorted desc, first hit wins). - `-2` (field present without stats) and type-mismatch semantics are unchanged, including the blob/vector `excludedFileFieldIds` fixup. ### Verification - `DataEvolutionFileStoreScanTest`: 8/8 passed - `DataEvolutionTableTest`: 42/42 passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
