JingsongLi opened a new pull request, #8613: URL: https://github.com/apache/paimon/pull/8613
## What changed - add field-level `pk-index.compaction.level-fanout` and `pk-index.compaction.stale-ratio-threshold` options for primary-key indexes - share the LSM level planner between sorted and vector primary-key indexes, replacing the vector-only configuration - maintain sorted primary-key indexes incrementally with independent multi-source index compaction - merge stale or over-fanout sorted index sources without coupling the work to data compaction - fail fast when a selected index build plan fails instead of silently skipping it ## Why Sorted primary-key indexes previously produced per-write index files without an independent compaction policy. Over time, readers could accumulate too many sources and retain stale entries until unrelated data compaction happened. This change gives each primary-key index its own LSM maintenance policy so source count and stale data can be controlled independently from data compaction. ## Impact Users can tune primary-key index compaction per indexed field with: - `fields.<field>.pk-index.compaction.level-fanout` - `fields.<field>.pk-index.compaction.stale-ratio-threshold` The old vector-specific compaction options are replaced; backward compatibility is intentionally not retained. ## Validation - 100 focused primary-key index tests passed - `mvn -pl paimon-core -am -DskipTests compile` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
