wangzhigang1999 opened a new issue, #9686: URL: https://github.com/apache/paimon/issues/9686
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. I found the related work below, but no issue or PR for the Python update-aware refresh path. ### Motivation In #8407, we gained incremental global-index building across Java/Flink/Spark/Python: repeated builds skip indexed row ranges and cover newly appended data. In #8818, the Java/Core/Flink/Spark path gained DEIX source snapshot metadata and refresh planning for updates within indexed ranges. I would like to bring that update-aware refresh behavior to PyPaimon for Data Evolution tables. For example, after building a BTree index on `label`, a user who opts into `global-index.column-update-action=IGNORE` can update `label` within an indexed row range. Python's next `create_global_index` call still treats that range as covered, so it does not rebuild the affected index. Python-built DE indexes also lack the DEIX source snapshot metadata needed by the Java refresh planner. The default update protection remains in place. This proposal concerns refreshing indexes after users opt into indexed-column updates, not changing that default or promising synchronous index maintenance. ### Solution I propose following the existing Java behavior in a few reviewable steps: 1. Record Java-compatible DEIX metadata for new Python-built DE indexes, using the snapshot scanned for the build rather than the latest snapshot at commit time. 2. Add update-aware planning in Python: identify affected indexed ranges from the source snapshot and indexed-column changes, and combine them with uncovered ranges. 3. Rebuild the selected ranges and commit old-index removals and new-index additions together, with conflict handling consistent with Java. For legacy indexes without trustworthy source metadata, retain Java's explicit-rebuild requirement. Keep the current update policy and supported-index restrictions, and account for the write-column metadata changes in #9574. ### Anything else? I have a local implementation of the first step and have tested DEIX byte compatibility, source snapshot selection, and Python/Java BTree and Bitmap cross-language read/write scenarios. I also checked that Java refresh planners select Python-built indexes after indexed-column updates. I have not implemented the full Python refresh path or validated a complete Spark/Flink refresh job. @JingsongLi @leaves12138, is anyone already working on the Python side of #8818? Does this scope make sense? I would be happy to contribute, starting with the source metadata PR. If the direction looks good, we can use this issue to track the follow-up work. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
