JingsongLi opened a new pull request, #1244: URL: https://github.com/apache/paimon-rust/pull/1244
### Purpose Closes #1242. Scalar global-index builds currently retain every projected Arrow batch and every serialized sort key in a shard. Stream the source and spill sorted runs once the sort buffer reaches its budget, preserving existing Java-compatible index output. ### Brief change log - Extract scalar, composite and multivalue keys one batch at a time, retaining physical row-ID validation across batches. - Add `sorted-index.sort-memory` (default `64mb`) for serialized keys and row slots. Reject invalid budgets and individual keys that exceed the budget. - Sort and merge temporary runs on blocking workers, using the existing typed key comparator, NULL ordering and row-ID tie-break. Pairwise merges avoid opening every run at once. - Feed the existing BTree/Bitmap/Multivalue writers from memory or the final disk run, with temporary-file ownership retained through output and cleanup on errors. The setting bounds extraction/sort state, not total process RSS. Reader buffers, the active Arrow batch, bitmap writer state and per-key BTree postings retain separate memory costs. FM keeps its existing streaming path. Local scratch space is required when spilling. ### Tests - Core tests: `cargo +1.98.0 test --locked -p paimon --all-targets --features vortex` with test debug symbols disabled. - Workspace Clippy, all targets with Vortex, denying warnings. - Formatting and diff checks. - Regression coverage forces disk spills for BTree v1/v2, Bitmap, Multivalue and composite keys; covers cross-batch row IDs, numeric order differing from serialized-byte order, NULL/empty/duplicate keys, invalid budgets, oversized keys and temporary-file cleanup. - Independent static review of ordering, source-row counts and cleanup completed without actionable findings. Local validation ran on macOS. Linux/Windows remain for CI. ### API and Format Adds the Rust build option and its CoreOptions accessor. Index formats, shard boundaries, metadata and dependencies are unchanged. ### Documentation Document the option, spill lifecycle and memory/disk boundaries in Getting Started. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
