JingsongLi opened a new pull request, #11348: URL: https://github.com/apache/paimon/pull/11348
### Purpose An `ALTER TABLE` can change an indexed column's type while the latest snapshot still references the previous schema. A query can then use index keys encoded with the old type, and an incremental build can keep treating the affected ranges as indexed. Record the schema actually used to build each Data Evolution global index in the versioned `_SOURCE_META` payload. Compare indexed field IDs and types when planning queries and incremental builds, including the extra fields of composite indexes. Queries skip incompatible index files and incremental builds replace them. FULL and DETAIL full-text searches fall back to the underlying rows when all matching index files are incompatible. Centroid vector search can still use a compatible vector column when an attached Bitmap column changes. Older Data Evolution query readers treat `_SOURCE_META` as opaque bytes and can read the existing manifest schema with the v2 payload. The new codec also reads v1 metadata. Older index maintenance paths that decode the payload require the v2 codec for newly built indexes. ### Tests - Core index, manifest, and search suites (294 tests). - Flink BTree and Bitmap refresh integration tests (2 tests) and Spark global-index procedure suite (7 tests). - PyPaimon manifest, upsert, and execution boundary tests (30 tests) and the Rust 0.5.0 sorted-index integration suite (100 tests). - Cross-version smoke tests for indexed filters and TopN on an older Java reader and indexed filters on an older PyPaimon reader. The Rust integration suite also checks that BTree and Bitmap query plans use indexes with v2 source metadata. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
