JingsongLi opened a new pull request, #1261: URL: https://github.com/apache/paimon-rust/pull/1261
### Purpose Maintain declared global indexes synchronously after append/Data Evolution writes when `write-only=false`, so callers do not need a separate index build after each write. ### Brief change log - Add Java-style per-family column declarations (`global-index.<family>.index-columns`) and per-column native options. Reuse the existing scalar, full-text and vector builders. - Capture the current schema and a source snapshot, fill missing row-ID coverage, coalesce partial shards, and atomically publish additions with retired references. Vector algorithm or metric changes replace incompatible payloads across the column; historical files remain readable. - Replan after concurrent snapshot/schema changes. Use a stable identity derived from the data commit for idempotent recovery, and retain submitted files when publication is uncertain. Recovery also resumes an interrupted index phase after compaction has completed. - Share field configuration between automatic/manual builds, planned vector shards, and FULL/DETAIL raw fallback, preserving explicit option precedence. - Maintain declared indexes after manual compaction as well. Data, compaction and index publication are separate transactions because global row IDs must be assigned before indexing. An index failure leaves the data committed; retrying with the same commit user and identifier resumes maintenance. ### Tests - `cargo test --locked -p paimon --all-targets --features vortex` — passed; core unit tests: 4727 passed, 6 ignored. - `cargo +1.98.0 clippy --locked --all-targets --workspace --features vortex -- -D warnings` — passed. - `cargo +1.98.0 fmt --all -- --check` — passed. - Regression coverage includes CAS conflicts, uncertain publication, concurrent recovery, append/overwrite/manual compaction recovery, stale REST committers, FULL/DETAIL fallback with no or partial coverage, manual builder defaults/overrides, and vector algorithm/metric changes with interrupted publication and replay. ### API and Format New table options declare automatically maintained indexes. Maintenance runs with `write-only=false` (the default) only when indexes are declared; `write-only=true` skips automatic maintenance. Supported tables have row tracking, no primary key and `bucket=-1`. Existing global-index builders, payload formats and index manifests are reused. Primary-key index maintenance is outside this change. ### Documentation Update the compaction and SQL guides with declarations, option precedence, definition replacement, and recovery APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
