gabotechs opened a new pull request, #128:
URL: https://github.com/apache/datafusion-iceberg/pull/128
Closes #84.
A lazy Iceberg scan only exposes one output partition, so DataFusion can
schedule just one scan partition however many files the scan matches, and has
to add a round-robin repartition above it to get any parallelism. This makes
the lazy scan expose the session's `target_partitions` instead, without listing
files up front.
The partitions share one `TableScan::plan_files()` stream. A background task
drains it into a channel that every partition pulls from, so whichever
partition is free takes the next file, and the first file can be read while the
rest are still being planned. Once every partition has dropped its feed, for
example when a `LIMIT` is satisfied, discovery stops.
```
before after
DataSourceExec (1 partition) plan_files() ──► shared channel
plan_files() ──► p0 reads every file ├──► p0 reads
next file
RepartitionExec: RoundRobinBatch(N) ├──► p1 reads
next file
spreads batches over N partitions └──► pN-1 reads
next file
```
The shared discovery state is created per execution through DataFusion's
sibling state (`DataSource::create_sibling_state` / `open_with_args`), the same
hook `FileScanConfig` uses for its shared work queue. `FileScanTaskProvider`
gets a `create_sibling_state` hook that `IcebergDataSource` forwards, so
re-running a plan after `reset_state` starts a fresh discovery instead of
finding the previous one drained. The commits are split so the first one shows
the idea with the state kept on the provider, and the second moves it into
sibling state.
`FileScanTaskProvider::open` now yields DataFusion results; errors only get
converted to iceberg errors where the Arrow reader needs them.
One trade-off: a `LIMIT` without `ORDER BY` can now open up to one file per
partition before the limit stops the scan, rather than stopping after the first
file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]