JingsongLi opened a new issue, #1242: URL: https://github.com/apache/paimon-rust/issues/1242
Scalar global-index extraction currently collects all projected Arrow batches for a shard, materializes every serialized key, and sorts the complete key vector. With the default 25-million-row sorted-index range, extraction and sorting retain memory proportional to shard size before index output begins. Stream the source batches and bound retained serialized sort keys with a configurable buffer. Spill sorted runs to local temporary files and merge them with the existing Paimon key comparator. Preserve NULL ordering, row-ID ties, composite keys, multivalue source-row counts, and existing Java-compatible index files. Reject oversized individual keys explicitly and clean temporary runs on failure. Scope: bound extraction/sort state, not total process RSS. Existing bitmap writer state and per-key BTree postings have separate memory costs; changing their file-writing strategy is outside this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
