alexanderbianchi opened a new pull request, #3578: URL: https://github.com/apache/iceberg-rust/pull/3578
## Which issue does this PR close? No linked issue. This draft extends the runtime work in #3277 and #3406. Companion #3577 adds handle-based runtime construction; the two PRs are independent and can merge in either order. ## What changes are included in this PR? #3277 schedules Parquet opens and Bloom-filter work on the I/O runtime, but `FileIO` itself still polls storage operations on the caller's task. Likewise, `Table::with_runtime` preserves the manifest cache but does not rebind the FileIO used by the table or by cache misses. - Add `FileIO::with_runtime` and `FileIOBuilder::with_runtime`. - Dispatch asynchronous storage operations, reader/writer construction, range reads, writes, and close through a private I/O-runtime adapter. Preserve backend-returned storage and paths when constructing input/output handles. - Use the existing abort-on-drop handle so dropping a waiting operation also cancels its spawned task. - Bind table FileIO to the table runtime. When rebinding a table, update cache-miss I/O while retaining the shared parsed manifest/list cache and encryption configuration. - Pass the configured runtime into FileIO construction in Glue, HMS, S3 Tables, and SQL catalogs, including metadata I/O before table construction. - Keep runtime handles out of serialized FileIO; deserialization requires receiver-local rebinding. Standalone FileIO without an explicit runtime keeps its existing behavior. Adapted from the Apache-licensed [Datadog FileIO runtime patch](https://github.com/DataDog/iceberg-rust/commit/eb14abc0169338e2e1357c0c184f79d60af984f1), using upstream's existing cancellation helper. This does not add the fork's optional table-runtime fallback, REST convenience shim, or scan-reader accessor. Upstream's explicit table-runtime requirement is unchanged. ## Are these changes tested? - Thread-routing tests cover storage operations and returned reader/writer handles, including backend-specific child storage/path preservation and sharing the lazily built backend. - Serialization/rebinding and cancellation regression tests. - Cache tests verify retained entries after deleting their backing files, and successful uncached manifest reads after shutting down the old runtime and rebinding to a live runtime. - `cargo test -p iceberg --lib --all-features`: 2,091 passed, 1 existing ignored. - `cargo clippy -p iceberg --all-targets --all-features -- -D warnings`: passed. - Compile checks for all four affected catalogs passed; external-service catalog integration tests were not run. - Formatting and public API regeneration completed; two additive FileIO methods. ## AI Disclosure AI assistance was used to port the existing patch, add regression tests, and prepare this description. Local tests and Clippy were run. This remains a draft for author and maintainer review, particularly the scheduling/cancellation design. Per-operation spawning adds task hops even when a caller already runs on the I/O runtime; no performance claim is made, and benchmarking remains outstanding. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
