NoahKusaba opened a new issue, #3405: URL: https://github.com/apache/iceberg-rust/issues/3405
### Is your feature request related to a problem or challenge? A `Table` spawns its work, such as scan planning and delete file processing, on the `Runtime` it was built with, and that runtime can only be set through `TableBuilder::runtime`. Rebuilding a table to move it to another runtime also creates a new `ObjectCache`, so the rebuilt table starts with none of the manifests and manifest lists the original had loaded. So an application that caches `Table`s and wants each query's scans to run on the runtime executing that query has to choose between: - binding the cached table to whichever runtime built it. If that runtime shuts down while the table is still cached, later scans spawn onto it and their work never runs. - rebuilding the table per query and losing its manifest cache. With 16 concurrent scans of a table with 200 manifests, warm scans through a rebuilt table were about 20x slower than through the cached one. Today the only way around both is to bind every cached table to a dedicated runtime that lives as long as the process, so scan planning runs on a separate pool from the query. ### Describe the solution you'd like `Table::with_runtime`, which returns the table bound to another runtime and keeps its `ObjectCache`, in the style of the crate-private `with_metadata`: ```rust let table = cached_table.clone().with_runtime(Runtime::try_current()?); ``` ### Willingness to contribute I can contribute to this feature independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
