alexanderbianchi commented on code in PR #67:
URL: https://github.com/apache/datafusion-iceberg/pull/67#discussion_r4380041410
##########
crates/datafusion/README.md:
##########
@@ -20,3 +20,40 @@
# Apache Iceberg DataFusion Integration
This crate contains the integration of Apache DataFusion and Apache Iceberg.
+
+## Optional file planning before execution
+
+By default, the Iceberg reader discovers files when execution starts. To select
+`FileScanTask`s during `TableProvider::scan()` instead, register the planning
+options on the DataFusion session before planning a query:
+
+```rust
+use datafusion::prelude::SessionConfig;
+use iceberg_datafusion::IcebergPlanningConfig;
+
+let config = SessionConfig::new()
+ .with_option_extension(IcebergPlanningConfig::default())
+ .set_bool("iceberg.eager_file_planning", true);
+```
+
+### Behavior
+
+- Lazy planning is the default. Eager planning selects files before execution;
+ it reads metadata, not Parquet data.
+- Selected tasks are assigned round-robin to at most `target_partitions`.
+ An empty scan has one empty partition. Each eager partition reads one file
+ at a time when there are multiple output partitions.
+- Retained tasks have an estimated reservation in the query memory pool.
+ Exceeding the pool fails planning; there is no file-count cap or truncation.
+- Both modes use the same reader implementation. Reader clones share the
+ delete loader across partitions. Scan statistics remain unknown.
+
+### Trade-offs
+
+Eager planning completes file discovery and retains tasks even for queries
+that stop early. Repeated provider scans repeat planning. The planning caller
+must support metadata I/O: Iceberg's initial manifest-list read still runs on
+that runtime, even when the table has a separate I/O runtime.
+
+Register only one configuration extension under the `iceberg` prefix per
session.
+DataFusion Distributed currently uses that same prefix for its own options.
Review Comment:
Sorry this is entirely an oversight on my part failing to catch this in the
agent output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
