JingsongLi opened a new pull request, #8410: URL: https://github.com/apache/paimon/pull/8410
## Summary Optimize Data Evolution UPDATE so filtered self-merge-on-row-id updates can use the global index to locate touched RowIDs and prune target files, while still falling back to full unindexed ranges for data written after index creation. ## Changes - Add a Spark UPDATE shortcut for non-join self-merge-on-row-id plans that converts the filter to a Paimon predicate and uses `GlobalIndexScanner` to find touched target files. - Make `GlobalIndexScanner` snapshot-aware for UPDATE and infer partition filters from the full predicate when callers pass `partitionFilter = null`. - Refactor `GlobalIndexCoverage` to expose explicit full/detail unindexed range APIs without depending on `GlobalIndexSearchMode`; callers now choose the search mode outside coverage. - Keep unindexed rows separate from indexed scan results, then combine them explicitly where full fallback is required. - Add regression coverage for BTree unindexed fallback, partition filter inference, and the Spark UPDATE shortcut avoiding a join. ## Testing - [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=BtreeGlobalIndexTableTest test` - [x] `mvn -pl paimon-spark/paimon-spark-common -am -Pspark3,fast-build -DskipTests compile` - [x] `mvn -pl paimon-spark/paimon-spark-3.5 -am -Pspark3,fast-build -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.RowTrackingTest -Dtest=none test` ## Notes Local Spark 4 compile was not completed because this machine is on Java 8 and `antlr4-maven-plugin:4.13.1` requires Java 11 bytecode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
