JingsongLi opened a new pull request, #11348:
URL: https://github.com/apache/paimon/pull/11348

   ### Purpose
   
   An `ALTER TABLE` can change an indexed column's type while the latest 
snapshot still references the previous schema. A query can then use index keys 
encoded with the old type, and an incremental build can keep treating the 
affected ranges as indexed.
   
   Record the schema actually used to build each Data Evolution global index in 
the versioned `_SOURCE_META` payload. Compare indexed field IDs and types when 
planning queries and incremental builds, including the extra fields of 
composite indexes. Queries skip incompatible index files and incremental builds 
replace them. FULL and DETAIL full-text searches fall back to the underlying 
rows when all matching index files are incompatible. Centroid vector search can 
still use a compatible vector column when an attached Bitmap column changes.
   
   Older Data Evolution query readers treat `_SOURCE_META` as opaque bytes and 
can read the existing manifest schema with the v2 payload. The new codec also 
reads v1 metadata. Older index maintenance paths that decode the payload 
require the v2 codec for newly built indexes.
   
   ### Tests
   
   - Core index, manifest, and search suites (294 tests).
   - Flink BTree and Bitmap refresh integration tests (2 tests) and Spark 
global-index procedure suite (7 tests).
   - PyPaimon manifest, upsert, and execution boundary tests (30 tests) and the 
Rust 0.5.0 sorted-index integration suite (100 tests).
   - Cross-version smoke tests for indexed filters and TopN on an older Java 
reader and indexed filters on an older PyPaimon reader. The Rust integration 
suite also checks that BTree and Bitmap query plans use indexes with v2 source 
metadata.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to