JingsongLi opened a new pull request, #5: URL: https://github.com/apache/paimon-full-text/pull/5
## Summary Optimize full-text index reader open by batching archive file positional reads before loading them into Tantivy's `RamDirectory`. This follows the same `pread` batching shape used by `paimon-vector-index` while keeping existing Java `readAt` implementations compatible. ## Changes - Batch full-text archive file reads during reader open with bounded batches of ranges and bytes. - Add a `ReadRequest::new` helper and storage read helpers that can consume files batch-by-batch without retaining all archive payloads at once. - Add Java `FullTextIndexInput.pread(long[], byte[][])` as an optional override, with a default implementation backed by `readAt`. - Update JNI input handling to call Java `pread` once per batch instead of looping through `readAt` calls. - Add Rust and Java coverage to verify reader open issues multi-range reads. - Document the optional Java batched input hook. ## Testing - [x] `cargo test --workspace` - [x] `PYTHONPATH=python pytest -q python/tests` - [x] `PAIMON_FTINDEX_JNI_LIB_PATH=/Users/lijingsong/IdeaProjects/paimon-full-text/target/debug/libpaimon_ftindex_jni.dylib mvn -f java/pom.xml test` - [x] `cargo clippy --workspace --all-targets` ## Notes Existing Java implementations only need `readAt`; overriding `pread` is optional for storage clients that can batch, coalesce, or parallelize positional reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
