mbutrovich commented on code in PR #111:
URL: 
https://github.com/apache/datafusion-iceberg/pull/111#discussion_r4605992581


##########
crates/datafusion/tests/limit_pushdown.rs:
##########
@@ -103,6 +103,30 @@ mod tests {
         Ok(())
     }
 
+    #[tokio::test]
+    async fn reports_the_rows_returned_by_a_limited_iceberg_scan() -> 
Result<()> {
+        let harness = IcebergTestHarness::new().await?;
+        // Only row counts, so the output doesn't vary between runs.
+        harness
+            .query("SET datafusion.explain.analyze_categories = 'rows'")
+            .await?;
+        let (_, explain) = harness
+            .query("EXPLAIN ANALYZE SELECT pickup_date FROM taxi LIMIT 3")
+            .await?;
+
+        insta::assert_snapshot!(explain, @r"
+    
+-------------------+---------------------------------------------------------------------------------------------------------------------------------+
+    | plan_type         | plan                                                 
                                                                           |
+    
+-------------------+---------------------------------------------------------------------------------------------------------------------------------+
+    | Plan with Metrics | CooperativeExec, metrics=[]                          
                                                                           |
+    |                   |   DataSourceExec: format=iceberg, 
projection=[pickup_date], fetch=3, metrics=[output_rows=3, output_batches=1, 
batches_split=0] |
+    |                   |                                                      
                                                                           |
+    
+-------------------+---------------------------------------------------------------------------------------------------------------------------------+
+    ");
+
+        Ok(())
+    }

Review Comment:
   This test checks the new metrics, but nothing checks the behavior in the PR 
title. The old `try_filter_map` stream also emitted 3 rows in 1 batch, so 
`output_rows=3, output_batches=1` would still show up if the scan went back to 
draining every file.
   
   Could we add a test that counts storage reads? 
[`FixtureStorage::reader`](https://github.com/apache/datafusion-iceberg/blob/86e8ce4339dd082c50f351f1ba1f715a23bfe89d/crates/datafusion/src/test_utils/harness.rs#L177-L179)
 could wrap the `FileRead` it returns for `.parquet` paths in a counter. The 
test could then assert that `SELECT * FROM taxi LIMIT 3` reads fewer ranges 
than `SELECT * FROM taxi`. I measured this with a temporary counter. On `main`, 
both queries read all 21 ranges (6,114,969 bytes). With this PR, the limited 
query reads 8 ranges (3,853,821 bytes).
   
   Two details would keep that test stable. The counter should live on the 
harness instead of in a `static`, because the tests in one file run in 
parallel. Comparing the two queries, instead of asserting exact counts, keeps 
the test independent of the runner's CPU count, because iceberg opens up to 
`num_cpus` data files at once.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to