mbutrovich commented on code in PR #26943: URL: https://github.com/apache/datafusion/pull/26943#discussion_r4483017313
########## datafusion/sqllogictest/test_files/listing_table_partitions.slt: ########## @@ -127,3 +140,120 @@ select year, count(*) from root_file_test group by year order by year; ---- 2024 2 2025 1 + +# Test: with listing_table_strict_partitions, files outside the partition Review Comment: > Can we have a test like `SELECT * FROM table LIMIT 1` where the table has two files - with the correct partitioning and incorrect partitioning - want to check if we still fail if the query is completed by reading a subset of (valid)files I tried this on the head commit, and the answer depends on listing order. With `year=2024/data.parquet` (2 rows), `year=2025/data.parquet`, and a stray root file `zzz.parquet`, strict mode on, and the list files cache off, `select count(*) from t` fails as expected, but `select * from t limit 1` returns `1 alice 2024`. If the stray file is named `stale.parquet`, which lists before the `year=` directories, the same `LIMIT 1` query fails. With `datafusion.execution.collect_statistics = false`, `LIMIT 1` fails in both cases. The cause is the statistics-based file limit. When a scan has a limit and no non-partition filters, [`statistic_file_limit`](https://github.com/apache/datafusion/blob/b6b2e0e5d5f119f46b7fb6288827aee7041ab00d/datafusion/catalog-listing/src/table.rs#L646-L651) is passed down, and [`get_files_with_limit`](https://github.com/apache/datafusion/blob/b6b2e0e5d5f119f46b7fb6288827aee7041ab00d/datafusion/catalog-listing/src/table.rs#L1223-L1256) stops pulling from the listing once the exact row count passes it. An error that arrives after that point is dropped, and files after it are never listed. @xanderbailey, should strict mode turn off `statistic_file_limit`, so the check covers the whole listing? Otherwise, should the config doc say that a `LIMIT` can stop the listing before it reaches a mismatching file? Either way, a `LIMIT` test with the stray file listed last would cover what @hareshkh asked. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
