alamb commented on code in PR #18370:
URL: https://github.com/apache/datafusion/pull/18370#discussion_r2478082945
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -123,6 +126,43 @@ async fn query_multi_csv_file() {
);
}
+#[tokio::test]
+async fn query_partitioned_csv_file() {
Review Comment:
Could you also please add a test with a query that applies predicates to the
three partition columns?
Something like
```sql
select * from csv_table_partitioned WHERE a = 2;
```
```sql
-- apply predicate to last in directory
select * from csv_table_partitioned WHERE c = 200;
```
```sql
-- apply predicate to both
select * from csv_table_partitioned WHERE a = 2 AND b = 20;
```
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -123,6 +126,43 @@ async fn query_multi_csv_file() {
);
}
+#[tokio::test]
+async fn query_partitioned_csv_file() {
+ let test = Test::new().with_partitioned_csv().await;
+ assert_snapshot!(
+ test.query("select * from csv_table_partitioned").await,
+ @r"
+ ------- Query Output (6 rows) -------
+ +---------+-------+-------+---+----+-----+
+ | d1 | d2 | d3 | a | b | c |
+ +---------+-------+-------+---+----+-----+
+ | 0.00001 | 1e-12 | true | 1 | 10 | 100 |
+ | 0.00003 | 5e-12 | false | 1 | 10 | 100 |
+ | 0.00002 | 2e-12 | true | 2 | 20 | 200 |
+ | 0.00003 | 5e-12 | false | 2 | 20 | 200 |
+ | 0.00003 | 3e-12 | true | 3 | 30 | 300 |
+ | 0.00003 | 5e-12 | false | 3 | 30 | 300 |
+ +---------+-------+-------+---+----+-----+
+ ------- Object Store Request Summary -------
+ RequestCountingObjectStore()
+ Total Requests: 13
+ - LIST (with delimiter) prefix=data
Review Comment:
This makes it super clear what is going on. It is a terrifying number of
LIST commands
##########
datafusion/core/tests/datasource/object_store_access.rs:
##########
@@ -375,6 +415,30 @@ impl Test {
self
}
+ /// Register a partitioned CSV table at the given path relative to the
[`datafusion_test_data`]
+ /// directory
Review Comment:
Yes, please do -- I think the comments are outdated (from an earlier
implementation that did in fact use the directory)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
