rahulsmahadev opened a new pull request, #4390:
URL: https://github.com/apache/iceberg-python/pull/4390
# Rationale for this change
Two defaults give ascending sort fields the nulls-last order, which
disagrees with Java and with `SortField`'s own default for ascending fields.
The same ascending field can therefore be recorded with different null orders
depending on how it is built.
`Table.update_sort_order().asc(name, transform)` with no `null_order`
records nulls-last:
```python
tbl.update_sort_order().asc("user_id", IdentityTransform()).commit()
tbl.sort_order().fields[0].null_order # NullOrder.NULLS_LAST, expected
NULLS_FIRST
```
`SortField` built with a str direction also gets nulls-last, while the enum
member gets nulls-first:
```python
SortField(source_id=1, transform=IdentityTransform(),
direction="asc").null_order # NULLS_LAST
SortField(source_id=1, transform=IdentityTransform(),
direction=SortDirection.ASC).null_order # NULLS_FIRST
```
Java's `SortOrderBuilder.asc(String)` uses `NullOrder.NULLS_FIRST` (its
Javadoc says "ascending with nulls first"), and `desc(String)` uses
`NULLS_LAST`. `SortField` picks nulls-first for `SortDirection.ASC` and
nulls-last otherwise. So `desc()` agrees in all three places, and only the
ascending default in `UpdateSortOrder.asc()` differs. The spec requires
`null-order` in every sort field and does not define a default, so Java and
`SortField` are the reference.
In `SortField`, the `set_null_order` validator chooses the default by
comparing the raw `direction` with `SortDirection.ASC`. `SortDirection` is a
plain `Enum`, so the str `"asc"` never matches and gets the descending default.
Pydantic converts the value to the enum afterwards, so the field ends up
ascending with nulls-last.
This change:
- `UpdateSortOrder.asc()` defaults to `NullOrder.NULLS_FIRST`. `desc()`
keeps `NULLS_LAST`.
- `set_null_order` converts the direction to `SortDirection` before it
chooses the default.
An explicit null order is used as given. Existing sort orders are not
changed, because their null order is already stored. The `asc()` default came
in with `UpdateSortOrder` in #2552. Calls that omit `null_order` now request
nulls-first, also on existing tables, where a matching existing sort order may
be reused. Pass `NullOrder.NULLS_LAST` explicitly to keep the previous
behaviour.
## Are these changes tested?
Yes. `test_update_sort_order_default_null_order` in
`tests/catalog/test_catalog_behaviors.py` runs through the `catalog` fixture
(memory, SQL, and SQL without rowcount catalogs). It commits `asc()` and
`desc()` with no null order and checks the stored sort order, then commits
explicit null orders and checks that they are kept.
`test_deserialize_sort_field_str_direction` in `tests/table/test_sorting.py`
checks that `"asc"` and `"desc"` validate to the same `SortField` as the enum
members, with the matching default null order. Both fail on main with
`NullOrder.NULLS_LAST` where `NullOrder.NULLS_FIRST` is expected, and both pass
with this change. The integration tests in
`tests/integration/test_sort_order_update.py` pass explicit null orders, so
they do not depend on the default.
## Are there any user-facing changes?
Yes. `update_sort_order().asc()` without a null order now records
nulls-first (it recorded nulls-last), and `SortField` with a str direction now
gets the same default as the enum member. Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]