rahulsmahadev opened a new pull request, #4425:
URL: https://github.com/apache/iceberg-python/pull/4425
# Rationale for this change
Bug: `DeletionVector.to_vector()` builds the PyArrow array without a type,
so PyArrow infers it from the first bitmap. A deletion vector has one bitmap
per high 32 bits of the position. When all deleted positions are at or above
2**32 (a data file with more than 2**32 rows), the first bitmap is empty,
PyArrow types it as null and the read fails:
```python
from pyroaring import BitMap
from pyiceberg.table.deletion_vector import DeletionVector
DeletionVector("data.parquet", [BitMap(), BitMap([5])]).to_vector()
# pyarrow.lib.ArrowInvalid: Invalid null value
```
A vector with no bitmaps, like `64mapempty.bin` in `tests/table/bitmaps`,
fails with `ArrowInvalid: cannot construct ChunkedArray from empty vector and
omitted type`. Java writes an empty bitmap for key 0 in the first case
(`RoaringPositionBitmap.serialize`) and reads both fine.
Fix: pass `type=pa.int64()`. This also covers `from_positions()` in #3474,
which pads missing keys with empty bitmaps the same way.
## Are these changes tested?
Yes, added `test_to_vector_with_empty_bitmaps_for_low_keys` and extended
`test_map_empty` in `tests/table/test_deletion_vector.py`. Both fail on main
with the errors above and pass with this change.
## Are there any user-facing changes?
Yes, these deletion vectors can be read now. A vector with only empty
bitmaps now converts to an empty int64 array instead of a null-typed one.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]