gabotechs opened a new pull request, #109:
URL: https://github.com/apache/datafusion-iceberg/pull/109

   #67 adds eager file planning by introducing a second data source, 
`EagerIcebergDataSource`, next to the existing one, plus a typestate builder to 
pick between them. That works, but now there are two `DataSource` impls that 
each carry their own reading, limit and metrics code, and a fairly heavy 
builder whose main job is choosing which one you get.
   
   This PR tries a smaller shape for the same idea. There's still a single 
`IcebergDataSource`, and the only thing that varies is *where its file tasks 
come from*. That is abstracted behind a `FileScanTaskProvider` trait, with two 
implementations: a lazy one that discovers files when the stream is first 
polled (today's behavior), and an eager one that holds file groups planned in 
`TableProvider::scan` and exposes one partition per group.
   
   ```
                         IcebergDataSource
           reader, limit, metrics, EXPLAIN prefix, partitioning
                                 │
                    Arc<dyn FileScanTaskProvider>
                       ┌─────────┴──────────┐
        LazyFileScanTaskProvider      EagerFileScanTaskProvider
        discovers files on first      file groups planned up front,
        poll, 1 partition             N partitions, memory reserved
   ```
   
   The data source builds its `ArrowReader` in `open()` from the execution 
`TaskContext`, so batch size and page-index settings keep coming from the 
execution session for both paths. Providers are `Any`, so codecs and 
distributed engines can downcast `file_scan_task_provider()` to inspect or ship 
the planned tasks, and plug in their own provider with 
`with_file_scan_task_provider`. Exposing `partitioning()` on the trait also 
leaves room for a future provider that reports hash partitioning by Iceberg 
partition values.
   
   Eager planning is opt-in through a session option, off by default:
   
   ```sql
   SET iceberg.execution.eager_file_planning = true;
   ```
   
   Heads-up on API changes: `IcebergDataSource::builder` is replaced by 
`IcebergDataSourceBuilder::new(&table, session, schema)`, and `snapshot_id()` / 
`predicate()` moved from the data source to `LazyFileScanTaskProvider`, since 
only lazy discovery uses them.
   
   One known gap: an eager scan's `EXPLAIN` line only shows `planned_files=N`, 
not the projection and predicate. That can be a follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to