Yohahaha opened a new issue, #8415: URL: https://github.com/apache/paimon/issues/8415
### Motivation Currently, when a user wants to rebuild or replace an existing global index in Paimon, they must execute two separate operations: 1. `CALL sys.drop_global_index(table => '...', index_column => '...', index_type => '...')` 2. `CALL sys.create_global_index(table => '...', index_column => '...', index_type => '...')` This two-step process has the following drawbacks: - **Non-atomic**: Between the drop and create, the table has no index, which may affect concurrent queries relying on the global index. - **Verbose**: Users must issue two commands for what is logically a single "rebuild" operation. - **Error-prone**: If the second step fails, the user is left with no index at all. ### Proposal Add an optional `replace` parameter (default: `false`) to the `create_global_index` procedure in both Spark and Flink. When `replace=true`: - If an index of the same type on the same column already exists, atomically replace it (delete old index files and commit new index files in a single transaction). - If no existing index is found, behave as normal create. When `replace=false` (default): - If an existing index of the same type on the same column already exists, either fail with an error or proceed with creating a new one (current behavior). ### Reference Lance (a columnar data format for ML) implements a similar pattern in their `CreateIndexBuilder`: - https://github.com/lance-format/lance/blob/v7.0.0/rust/lance/src/index/create.rs#L96-L210 Key snippet from Lance: ```rust pub fn replace(mut self, replace: bool) -> Self { self.replace = replace; self } // During execute: let removed_indices = if self.replace { self.dataset.load_indices().await? .iter() .filter(|idx| idx.name == new_idx.name) .cloned() .collect() } else { vec![] }; // Then commit with both new_indices and removed_indices atomically ``` ### Expected API **Spark:** ```sql CALL sys.create_global_index( table => 'db.my_table', index_column => 'user_id', index_type => 'btree', replace => true ) ``` **Flink:** ```sql CALL sys.create_global_index( `table` => 'db.my_table', index_column => 'user_id', index_type => 'btree', `replace` => true ) ``` ### Additional Context Related Paimon source files: - `paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/procedure/CreateGlobalIndexProcedure.java` - `paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/procedure/CreateGlobalIndexProcedure.java` - `paimon-core/src/main/java/org/apache/paimon/globalindex/btree/BTreeGlobalIndexBuilder.java` - `paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/globalindex/GenericIndexTopoBuilder.java` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
