rahulsmahadev commented on code in PR #4413:
URL: https://github.com/apache/iceberg-python/pull/4413#discussion_r4618910751


##########
pyiceberg/io/pyarrow.py:
##########
@@ -1871,6 +1883,20 @@ def _assign_row_ids(batch: pa.RecordBatch, row_id_index: 
int, positions: pa.Arra
     return batch.set_column(row_id_index, batch.schema.field(row_id_index), 
row_ids)
 
 
+def _assign_last_updated_sequence_numbers(
+    batch: pa.RecordBatch, last_updated_index: int, first_row_id: int | None, 
data_sequence_number: int | None
+) -> pa.RecordBatch:
+    """Replace null last updated sequence numbers with the data sequence 
number of the file.
+
+    When the data file has no first row ID or data sequence number, every last 
updated sequence number is null.
+    """
+    if first_row_id is None or data_sequence_number is None:

Review Comment:
   When `data_sequence_number` is None, this returns nulls for every row, 
including values stored in the file, while the spec says to read non-null 
values "without modification" and fill nulls from the data sequence number. 
`FileScanTask.from_rest_response` never sets it, so REST-planned scans now get 
wrong values where main raised `NotImplementedError`. Could those keep raising 
until REST planning can provide the sequence number? A case with stored values 
and no sequence number would pin it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to