alexanderbianchi commented on code in PR #128:
URL: 
https://github.com/apache/datafusion-iceberg/pull/128#discussion_r4646279387


##########
crates/datafusion/src/read/file_scan_task_provider/lazy.rs:
##########


Review Comment:
   We need to keep the snapshot row estimate for the whole scan, but return 
unknown stats for each reader when there’s more than one partition? The readers 
claim files at runtime, so we don’t know either reader’s share during planning.
   
   I tried this on the taxi fixture with `target_partitions = 2`:
   
   ```rust
   let scan = harness.scan().await?;
   assert_eq!(scan.output_partitioning().partition_count(), 2);
   
   let stats = StatisticsContext::new();
   let rows = |partition| {
       stats.compute(scan.as_ref(), 
&StatisticsArgs::new().with_partition(partition))
           .map(|s| s.num_rows)
   };
   
   // rows(None)    → Inexact(175000)
   // rows(Some(0)) → Inexact(175000)
   // rows(Some(1)) → error: Invalid Iceberg scan partition Some(1)
   ```
   
   Partition 1 is valid, and partition 0 is getting the whole table’s estimate. 
Changing only the bounds check would give both readers 175,000 rows. For this 
fixture I’d expect `Inexact(175000)` globally and `Absent` for each reader.



##########
crates/datafusion/src/read/scan.rs:
##########
@@ -231,19 +233,36 @@ impl DataSource for IcebergDataSource {
         partition: usize,
         context: Arc<TaskContext>,
     ) -> Result<SendableRecordBatchStream> {
+        self.open_with_args(OpenArgs::new(partition, context))

Review Comment:
   Is dropping support for direct `DataSource::open` on lazy scans intentional? 
I tried the same call on the parent and on this PR, with `target_partitions = 
1` so there’s no second reader to coordinate with:
   
   ```rust
   let plan = harness.scan().await?;
   let source = iceberg_source(&plan).unwrap();
   let mut rows = source.open(0, harness.state().task_ctx())?;
   assert!(rows.try_next().await?.is_some());
   ```
   
   On the parent this returns a batch. On this PR, `open` returns `Missing 
sibling state for LazyFileScanTaskProvider`. The normal `DataSourceExec` path 
still works because it supplies sibling state; it’s the direct call that 
changed.
   
   We should preserve the single-partition case, or, if direct opens are no 
longer supported for lazy scans, document that callers should execute the 
`DataSourceExec`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to