sundapeng commented on PR #9046: URL: https://github.com/apache/paimon/pull/9046#issuecomment-5190896000
Nice improvement — the cache targets the right bottleneck: for wide data-evolution tables, `evolutionStats` re-projects the same file schemas for every row-id group, and that repeated `project()` work dominates scan planning. Two suggestions: 1. As a possible follow-up: layer predicate-driven field resolution on top of the cache. A quick benchmark (1027 cols × 2000 row-id groups, single layout): the cache alone gives ~50×, and additionally resolving only the fields referenced by `inputFilter` roughly doubles the gain again (~100× total). After the cache, the remaining cost is the O(F²) field-matching loop; stats for fields not referenced by the predicate are never read by `inputFilter.test(...)`, so skipping their resolution is safe (unresolved fields read as null → conservatively kept). If you add this, extract the referenced fields once per scan rather than per group — per-group extraction rebuilds the RowType name→field map and shows up in the profile. 2. A one-line comment on the `HashMap` cache's thread safety would help: it's only touched on the single-threaded `plan()` post-filter path, so it's safe, but the class calls out thread-safety requirements elsewhere, so making this explicit avoids reviewers wondering about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
