mbutrovich commented on code in PR #27151:
URL: https://github.com/apache/datafusion/pull/27151#discussion_r4616665402
##########
datafusion/catalog-listing/src/helpers.rs:
##########
@@ -594,6 +611,30 @@ mod tests {
vec!["mypartition"]
)
);
+ assert_eq!(
+ None,
+ parse_partitions_for_path(
+ &ListingTableUrl::parse("file:///bucket/mytable").unwrap(),
+ &Path::from("bucket/mytable/mypartition=v1.csv"),
+ vec!["mypartition"]
+ )
+ );
+ assert_eq!(
+ None,
+ parse_partitions_for_path(
+ &ListingTableUrl::parse("file:///bucket/mytable").unwrap(),
+
&Path::from("bucket/mytable/mypartition=v1/otherpartition=v2.csv"),
+ vec!["mypartition", "otherpartition"]
+ )
+ );
Review Comment:
Could we add a case where the table has more partition columns than the
stray file has directories? On `main` that's a different symptom from the wrong
value in #27134. The parser returns fewer values than there are columns, and
the query fails at planning. For a table `partitioned by (year varchar, month
varchar, day varchar)` with a valid `year=2024/month=01/day=05/a.parquet` and a
stray `year=2024/month=02.parquet`, `main` fails with:
```
Error during planning: Cannot merge statistics with different number of
columns: 4 vs 3 (item 1)
```
This PR fixes it, and the assertion below fails on `main` with `Some(["v1",
"v2.csv"])`:
```suggestion
assert_eq!(
None,
parse_partitions_for_path(
&ListingTableUrl::parse("file:///bucket/mytable").unwrap(),
&Path::from("bucket/mytable/mypartition=v1/otherpartition=v2.csv"),
vec!["mypartition", "otherpartition"]
)
);
assert_eq!(
None,
parse_partitions_for_path(
&ListingTableUrl::parse("file:///bucket/mytable").unwrap(),
&Path::from("bucket/mytable/mypartition=v1/otherpartition=v2.csv"),
vec!["mypartition", "otherpartition", "thirdpartition"]
)
);
```
##########
datafusion/sqllogictest/test_files/listing_table_partitions.slt:
##########
@@ -257,3 +257,81 @@ select a, p from null_part_write order by a;
statement ok
set datafusion.execution.listing_table_hive_default_partition_name =
'__HIVE_DEFAULT_PARTITION__';
+
+# A root filename cannot supply an integer partition value. Previously this
+# attempted to cast "2025.parquet" to INT and failed the query; it is now
skipped.
+query I
+copy (values(7))
+to
'test_files/scratch/listing_table_partitions/filename_partition_int/year=2024/data.parquet';
+----
+1
+
+statement ok
+create external table filename_partition_int (column1 bigint)
+partitioned by (year int)
+stored as parquet
+location 'test_files/scratch/listing_table_partitions/filename_partition_int/';
+
+# A new stray file arrives after the table's valid layout has been registered.
+query I
+copy (values(8))
+to
'test_files/scratch/listing_table_partitions/filename_partition_int/year=2025.parquet';
+----
+1
+
+statement ok
+set datafusion.runtime.list_files_cache_limit = '0K';
+
+query II
+select column1, year from filename_partition_int order by column1;
+----
+7 2024
+
+# A filename shaped like a partition must not supply a missing directory level.
+query I
+copy (values(3))
+to
'test_files/scratch/listing_table_partitions/filename_partition/year=2024/month=01/a.parquet';
+----
+1
+
+statement ok
+create external table filename_partition (column1 bigint)
+partitioned by (year varchar, month varchar)
+stored as parquet
+location 'test_files/scratch/listing_table_partitions/filename_partition/';
+
+query I
+copy (values(4))
+to
'test_files/scratch/listing_table_partitions/filename_partition/year=2024/month=02.parquet';
+----
+1
+
+query I
+copy (values(5))
+to
'test_files/scratch/listing_table_partitions/filename_partition/year=2024/data.parquet';
+----
+1
+
+# A key=value filename inside a complete partition path remains a valid file.
+query I
+copy (values(6))
+to
'test_files/scratch/listing_table_partitions/filename_partition/year=2024/month=01/month=03.parquet';
+----
+1
+
+statement ok
+set datafusion.runtime.list_files_cache_limit = '0K';
Review Comment:
The list files cache is already off from the `set` at line 283, before
`filename_partition` is created, so this second `set` has no effect. It also
suggests the cache was turned back on somewhere. Could we drop it and add
`reset datafusion.runtime.list_files_cache_limit;` next to the `RESET` at the
end, the way
[`parquet.slt`](https://github.com/apache/datafusion/blob/13efc9c27d59b7fbc6503025042f14064c5fa0f7/datafusion/sqllogictest/test_files/parquet.slt#L1066)
does? The runner's config check doesn't cover runtime settings, so anything
added to the end of this file later would run with the cache off without anyone
noticing.
```suggestion
```
##########
datafusion/sqllogictest/test_files/listing_table_partitions.slt:
##########
@@ -257,3 +257,81 @@ select a, p from null_part_write order by a;
statement ok
set datafusion.execution.listing_table_hive_default_partition_name =
'__HIVE_DEFAULT_PARTITION__';
+
+# A root filename cannot supply an integer partition value. Previously this
+# attempted to cast "2025.parquet" to INT and failed the query; it is now
skipped.
Review Comment:
This comment describes the code before the PR, which a reader of the file
won't have. Could it describe the current behavior instead?
```suggestion
# A root filename cannot supply an integer partition value, so the stray
# year=2025.parquet below is skipped instead of being cast to INT.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]