zhoulii opened a new issue, #9543: URL: https://github.com/apache/paimon/issues/9543
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation The existing data evolution self-merge shortcut avoids the join, but it still scans and rewrites all splits remaining after partition pruning, even when conditional WHEN MATCHED clauses affect only a small subset of rows. This can cause significant read and write amplification for large tables, especially when a self-merge updates or deletes only a small range of data. ### Solution Push eligible WHEN MATCHED conditions into Paimon's SnapshotReader as a conservative file-pruning predicate. For self-merges, the implementation: - Combines all matched-action conditions with OR. - Rewrites source attributes to the corresponding target attributes. - Combines the result with the existing merge partition predicate. - Uses partition and file statistics to exclude splits that cannot match any action. The original merge condition and action ordering remain unchanged and are still evaluated by MergeRows. If any matched action is unconditional, non-deterministic, or cannot be fully converted to a Paimon predicate, the optimization falls back to the existing behavior. This introduces no additional Spark job and can substantially reduce the number of files scanned and rewritten when a selective self-merge modifies only a small portion of a large table. ### Anything else? _No response_ ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
