alexanderbianchi commented on code in PR #128:
URL:
https://github.com/apache/datafusion-iceberg/pull/128#discussion_r4646279387
##########
crates/datafusion/src/read/file_scan_task_provider/lazy.rs:
##########
Review Comment:
We need to keep the snapshot row estimate for the whole scan, but return
unknown stats for each reader when there’s more than one partition? The readers
claim files at runtime, so we don’t know either reader’s share during planning.
I tried this on the taxi fixture with `target_partitions = 2`:
```rust
let scan = harness.scan().await?;
assert_eq!(scan.output_partitioning().partition_count(), 2);
let stats = StatisticsContext::new();
let rows = |partition| {
stats.compute(scan.as_ref(),
&StatisticsArgs::new().with_partition(partition))
.map(|s| s.num_rows)
};
// rows(None) → Inexact(175000)
// rows(Some(0)) → Inexact(175000)
// rows(Some(1)) → error: Invalid Iceberg scan partition Some(1)
```
Partition 1 is valid, and partition 0 is getting the whole table’s estimate.
Changing only the bounds check would give both readers 175,000 rows. For this
fixture I’d expect `Inexact(175000)` globally and `Absent` for each reader.
##########
crates/datafusion/src/read/scan.rs:
##########
@@ -231,19 +233,36 @@ impl DataSource for IcebergDataSource {
partition: usize,
context: Arc<TaskContext>,
) -> Result<SendableRecordBatchStream> {
+ self.open_with_args(OpenArgs::new(partition, context))
Review Comment:
Is dropping support for direct `DataSource::open` on lazy scans intentional?
I tried the same call on the parent and on this PR, with `target_partitions =
1` so there’s no second reader to coordinate with:
```rust
let plan = harness.scan().await?;
let source = iceberg_source(&plan).unwrap();
let mut rows = source.open(0, harness.state().task_ctx())?;
assert!(rows.try_next().await?.is_some());
```
On the parent this returns a batch. On this PR, `open` returns `Missing
sibling state for LazyFileScanTaskProvider`. The normal `DataSourceExec` path
still works because it supplies sibling state; it’s the direct call that
changed.
We should preserve the single-partition case, or, if direct opens are no
longer supported for lazy scans, document that callers should execute the
`DataSourceExec`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]