rahulsmahadev opened a new pull request, #4435:
URL: https://github.com/apache/iceberg-python/pull/4435
# Rationale for this change
Bug: when a data file lacks a date, time, timestamp or timestamptz column,
`=` and `IN` miss its non-null `initial-default`, while `!=` matches it.
`_ColumnNameTranslator` compares a Python temporal value with a bound literal
in days or microseconds:
```python
# v3 table: rows 1-3 were written before `d` was added with initial-default
2020-01-01
tbl.scan(row_filter=EqualTo("d", date(2020, 1, 1))).to_arrow() # 0 rows,
expected 3
```
Fix: add a private `NestedField` accessor that returns a temporal
initial-default in the literal's form (the conversion `add_column` uses), and
use it in the translator, the Avro default reader and `Record` defaults.
Partition-projected values are already in that form and keep precedence.
Java: `SchemaParser` and `SchemaUpdate` supply typed defaults to row-group
filtering; `GenericReader` normalizes reader output through
`InternalRecordWrapper` before `Evaluator`. This PR normalizes defaults used in
internal records.
- `NestedField.initial_default` stays a Python object; internal default
consumers share the conversion.
## Are these changes tested?
Yes, filter tests in `tests/expressions/test_visitors.py` and
`tests/io/test_pyarrow.py`, plus Avro and `Record` default tests in
`tests/avro/test_resolver.py` and `tests/test_typedef.py`. All fail on main
(wrong filter result, or a Python date where an int is expected) and pass with
this change.
## Are there any user-facing changes?
Yes, this fixes incorrect temporal-default filter results and range errors.
Avro now returns defaulted temporal fields as integer days or microseconds,
matching stored fields.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]