hareshkh commented on code in PR #26943: URL: https://github.com/apache/datafusion/pull/26943#discussion_r4443896640
##########
docs/source/user-guide/configs.md:
##########
@@ -139,6 +139,7 @@ The following configuration settings are available:
| datafusion.execution.max_buffered_batches_per_output_file | 2
| This is the maximum number of RecordBatches buffered
for each output file being worked. Higher values can potentially give faster
write performance at the cost of higher peak memory consumption. This budget is
split evenly between two independent points in the write pipeline (see the
demuxer diagram in #7791): how many files can be in flight from the demuxer to
a writer task, and how many RecordBatches are buffered for a single file's
writer. Must be at least 2 so each half gets at least 1 unit of buffering - 0
or 1 would leave one side with a zero-capacity channel and panic at write time.
|
| datafusion.execution.listing_table_ignore_subdirectory |
true | Should sub directories be ignored when scanning
directories for data files. Defaults to true (ignores subdirectories),
consistent with Hive. Note that this setting does not affect reading
partitioned tables (e.g. `/table/year=2021/month=01/data.parquet`).
|
| datafusion.execution.listing_table_factory_infer_partitions |
true | Should a `ListingTable` created through the
`ListingTableFactory` infer table partitions from Hive compliant directories.
Defaults to true (partition columns are inferred and will be represented in the
table schema).
|
+| datafusion.execution.listing_table_strict_partitions |
false | Should a `ListingTable` return an error, rather
than skip the file, when a listed file's directories do not match the table's
partition columns. When true, every file must be under one `key=value`
directory per partition column, in the declared order, with no further
`key=value` directories below them (this also applies to tables without
partition columns). Defaults to false (non-matching files are skipped). Only
files listed for a query are checked, so partition filters that narrow the
listing prefix narrow the check.
|
Review Comment:
Thoughts on having two different setting to control both behaviours:
- `datafusion.execution.listing_table_error_on_missing_partition_values` -
error on root files, missing levels, or wrong partition keys.
- `datafusion.execution.listing_table_error_on_extra_partition_directories`
-error on additional key=value directories beyond the declared partitions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
