rahulsmahadev commented on code in PR #4413:
URL: https://github.com/apache/iceberg-python/pull/4413#discussion_r4618910751
##########
pyiceberg/io/pyarrow.py:
##########
@@ -1871,6 +1883,20 @@ def _assign_row_ids(batch: pa.RecordBatch, row_id_index:
int, positions: pa.Arra
return batch.set_column(row_id_index, batch.schema.field(row_id_index),
row_ids)
+def _assign_last_updated_sequence_numbers(
+ batch: pa.RecordBatch, last_updated_index: int, first_row_id: int | None,
data_sequence_number: int | None
+) -> pa.RecordBatch:
+ """Replace null last updated sequence numbers with the data sequence
number of the file.
+
+ When the data file has no first row ID or data sequence number, every last
updated sequence number is null.
+ """
+ if first_row_id is None or data_sequence_number is None:
Review Comment:
When `data_sequence_number` is None, this returns nulls for every row,
including values stored in the file, while the spec says to read non-null
values "without modification" and fill nulls from the data sequence number.
`FileScanTask.from_rest_response` never sets it, so REST-planned scans now get
wrong values where main raised `NotImplementedError`. Could those keep raising
until REST planning can provide the sequence number? A case with stored values
and no sequence number would pin it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]