xanderbailey opened a new pull request, #26310:
URL: https://github.com/apache/datafusion/pull/26310
## Which issue does this PR close?
- Closes #18083
## Rationale for this change
Hive compliant writers put the rows whose partition value is null into a
directory named `__HIVE_DEFAULT_PARTITION__` (Hive's
`hive.exec.default.partition.name`). `ListingTable` casts that directory name
straight to the partition column's type, so a table with a null partition
cannot be queried at all unless the partition column happens to be a string:
```sql
> create external table t (id int, d date) stored as parquet partitioned by
(d) location 'data/';
> select * from t;
Arrow error: Cast error: Cannot cast string '__HIVE_DEFAULT_PARTITION__' to
value of Date32 type
```
For a string partition column the query runs, but the value comes back as
the literal `__HIVE_DEFAULT_PARTITION__` rather than NULL, and `WHERE d IS
NULL` never matches it because partition columns are declared non-nullable.
Hive-written tables are a common input, and both Spark and Impala read these
values as NULL.
## What changes are included in this PR?
1. A partition value equal to the placeholder is read as a typed NULL rather
than parsed
(`helpers::try_into_partitioned_file`).
2. `ListingTable` declares partition columns nullable, in both the table
schema and the file
source's `TableSchema`, so the NULL can be represented and `IS NULL`
selects the null partition.
3. A new option,
`datafusion.execution.listing_table_hive_default_partition_name`, defaulting to
`__HIVE_DEFAULT_PARTITION__`, so a table written with a different
`hive.exec.default.partition.name` can be read. The empty name turns the
mapping off and reads the directory name as the value. The default is also
exposed as`datafusion_common::config::HIVE_DEFAULT_PARTITION_NAME`.
4. `filter_partitioned_file` no longer treats a NULL predicate result as a
match. That is reachable now that a partition value can be NULL: `part = 1`
evaluates to NULL, not false, for the null partition.
5. In the generated `configs.md`, defaults containing `_` or `*` are shown
as code. Markdown would otherwise read `__HIVE_DEFAULT_PARTITION__` as emphasis
and render it as bold text; this also backticks the two existing such defaults,
`nulls_max` and `start_offset`.
## What is the testing strategy for this PR?
Unit tests and slt tests added.
## Are there any user-facing changes?
listing_table_hive_default_partition_name added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
