gabotechs opened a new pull request, #109:
URL: https://github.com/apache/datafusion-iceberg/pull/109
#67 adds eager file planning by introducing a second data source,
`EagerIcebergDataSource`, next to the existing one, plus a typestate builder to
pick between them. That works, but now there are two `DataSource` impls that
each carry their own reading, limit and metrics code, and a fairly heavy
builder whose main job is choosing which one you get.
This PR tries a smaller shape for the same idea. There's still a single
`IcebergDataSource`, and the only thing that varies is *where its file tasks
come from*. That is abstracted behind a `FileScanTaskProvider` trait, with two
implementations: a lazy one that discovers files when the stream is first
polled (today's behavior), and an eager one that holds file groups planned in
`TableProvider::scan` and exposes one partition per group.
```
IcebergDataSource
reader, limit, metrics, EXPLAIN prefix, partitioning
│
Arc<dyn FileScanTaskProvider>
┌─────────┴──────────┐
LazyFileScanTaskProvider EagerFileScanTaskProvider
discovers files on first file groups planned up front,
poll, 1 partition N partitions, memory reserved
```
The data source builds its `ArrowReader` in `open()` from the execution
`TaskContext`, so batch size and page-index settings keep coming from the
execution session for both paths. Providers are `Any`, so codecs and
distributed engines can downcast `file_scan_task_provider()` to inspect or ship
the planned tasks, and plug in their own provider with
`with_file_scan_task_provider`. Exposing `partitioning()` on the trait also
leaves room for a future provider that reports hash partitioning by Iceberg
partition values.
Eager planning is opt-in through a session option, off by default:
```sql
SET iceberg.execution.eager_file_planning = true;
```
Heads-up on API changes: `IcebergDataSource::builder` is replaced by
`IcebergDataSourceBuilder::new(&table, session, schema)`, and `snapshot_id()` /
`predicate()` moved from the data source to `LazyFileScanTaskProvider`, since
only lazy discovery uses them.
One known gap: an eager scan's `EXPLAIN` line only shows `planned_files=N`,
not the projection and predicate. That can be a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]