NoahKusaba opened a new pull request, #111:
URL: https://github.com/apache/datafusion-iceberg/pull/111

   ## Which issue does this PR close?
   
   - None.
   
   ## Rationale for this change
   
   `IcebergDataSource` applies its fetch with a hand-written stream. After the 
limit is reached it keeps reading the Iceberg scan to the end and discards 
every batch. The source also reports no metrics, so `EXPLAIN ANALYZE` shows 
nothing for the scan.
   
   ## What changes are included in this PR?
   
   - Applies the fetch with DataFusion's `LimitStream`, which stops reading 
once the limit is reached.
   - Gives `IcebergDataSource` an `ExecutionPlanMetricsSet`, so `EXPLAIN 
ANALYZE` shows the scan's output rows, batches and bytes, e.g. `DataSourceExec: 
format=iceberg, projection=[vendor_id], fetch=3, metrics=[output_rows=3, ...]`.
   
   ## Are there any user-facing changes?
   
   The scan's metrics now appear in `EXPLAIN ANALYZE`. There are no API changes.
   
   ## Are these changes tested?
   
   Yes. A new test in `limit_pushdown.rs` snapshots `EXPLAIN ANALYZE` for a 
limited scan, with `datafusion.explain.analyze_categories = 'rows'` so the 
output is deterministic. The existing limit tests cover the rows returned under 
a fetch.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to