alexanderbianchi commented on PR #2671: URL: https://github.com/apache/iceberg-rust/pull/2671#issuecomment-6116801448
Clarification, ArrowReaderBuilder and Table::reader_builder() are already publicn [Example](https://github.com/alexanderbianchi/datafusion-distributed/blob/40212b7dc7a621efc739fca0fffb01e9b53d913c/iceberg/src/data_source.rs#L261). Exposing the builder type isn't the issue. The missing piece is a way to retain the reader settings chosen by a TableScan without retaining its manifest-planning state or binding the eventual reader to that scan's FileIO and runtime. This PR is being referenced in datafusion-iceberg now, and should largely be moved there. I can see the case for a lightweight reader-options value that travels with the scan plan. Ideally iceberg-rust would own the settings-to-builder mapping for both to_arrow() and external consumers, so the paths cannot drift. The eager DataFusion path could retain FileScanTasks and reader options, then drop TableScan. At execution it could create a reader from local resources and apply those options. The same boundary also works for distributed engines: DFD already plans tasks separately and builds readers on workers with worker-local FileIO and runtime (datafusion-contrib/datafusion-distributed#773). ```text Planning: scan.plan_files() → FileScanTasks scan.reader_options() → lightweight reader settings drop scan Execution, locally or on a worker: builder = ArrowReaderBuilder::new(local_file_io, local_runtime) reader = reader_options.apply_to(builder).build() reader.read(assigned_tasks) ``` For distributed execution, decoupling plan_files() from reading is an important supported use case: tasks are planned centrally, then workers construct readers with local storage/runtime resources and runtime predicate. Ideally we wouldn't need a `TableScan::arrow_reader_builder()`, and reader's wouldn't need to be transported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
