gabotechs opened a new pull request, #128:
URL: https://github.com/apache/datafusion-iceberg/pull/128

   Closes #84.
   
   A lazy Iceberg scan only exposes one output partition, so DataFusion can 
schedule just one scan partition however many files the scan matches, and has 
to add a round-robin repartition above it to get any parallelism. This makes 
the lazy scan expose the session's `target_partitions` instead, without listing 
files up front.
   
   The partitions share one `TableScan::plan_files()` stream. A background task 
drains it into a channel that every partition pulls from, so whichever 
partition is free takes the next file, and the first file can be read while the 
rest are still being planned. Once every partition has dropped its feed, for 
example when a `LIMIT` is satisfied, discovery stops.
   
   ```
   before                                    after
   
   DataSourceExec (1 partition)              plan_files() ──► shared channel
     plan_files() ──► p0 reads every file                       ├──► p0 reads 
next file
   RepartitionExec: RoundRobinBatch(N)                          ├──► p1 reads 
next file
     spreads batches over N partitions                          └──► pN-1 reads 
next file
   ```
   
   The shared discovery state is created per execution through DataFusion's 
sibling state (`DataSource::create_sibling_state` / `open_with_args`), the same 
hook `FileScanConfig` uses for its shared work queue. `FileScanTaskProvider` 
gets a `create_sibling_state` hook that `IcebergDataSource` forwards, so 
re-running a plan after `reset_state` starts a fresh discovery instead of 
finding the previous one drained. The commits are split so the first one shows 
the idea with the state kept on the provider, and the second moves it into 
sibling state.
   
   `FileScanTaskProvider::open` now yields DataFusion results; errors only get 
converted to iceberg errors where the Arrow reader needs them.
   
   One trade-off: a `LIMIT` without `ORDER BY` can now open up to one file per 
partition before the limit stops the scan, rather than stopping after the first 
file.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to