Fokko commented on code in PR #4273:
URL: https://github.com/apache/iceberg-python/pull/4273#discussion_r4561321536
##########
pyiceberg/table/update/__init__.py:
##########
@@ -746,11 +746,11 @@ def update_table_metadata(
for update in updates:
new_metadata = _apply_table_update(update, new_metadata, context)
- # Update last_updated_ms if it was not updated by update operations
+ # last_updated_ms must not precede the base's, which becomes the latest
metadata-log entry
if context.has_changes():
if metadata_location:
new_metadata = _update_table_metadata_log(new_metadata,
metadata_location, base_metadata.last_updated_ms)
- if base_metadata.last_updated_ms == new_metadata.last_updated_ms:
+ if new_metadata.last_updated_ms <= base_metadata.last_updated_ms:
Review Comment:
I compared this against Java's TableMetadata.Builder
(core/src/main/java/org/apache/iceberg/TableMetadata.java). Java never inherits
the snapshot timestamp into last-updated-ms at all: addSnapshot() does not
touch it, and build() stamps System.currentTimeMillis() (lines 1559-1561).
PyIceberg inherits the snapshot timestamp in AddSnapshotUpdate because
snapshots are staged eagerly (in Java the snapshot is produced at commit time),
which is the root cause of #4189. This change restores the invariant Java
enforces at lines 394-403 (last-updated-ms within one minute after the latest
metadata-log entry), so from Java's perspective the output is accepted again.
##########
tests/table/test_init.py:
##########
@@ -1683,6 +1683,23 @@ def test_update_metadata_log(table_v2: Table) -> None:
assert len(new_metadata.metadata_log) == 2
+def test_update_metadata_last_updated_ms_not_before_metadata_log(table_v2:
Table) -> None:
+ stale_snapshot = Snapshot(
+ snapshot_id=25,
+ parent_snapshot_id=19,
+ sequence_number=200,
+ timestamp_ms=table_v2.metadata.last_updated_ms - 61_000,
+ manifest_list="s3:/a/b/c.avro",
+ summary=Summary(Operation.APPEND),
+ schema_id=3,
+ )
+
+ new_metadata = update_table_metadata(
+ table_v2.metadata, (AddSnapshotUpdate(snapshot=stale_snapshot),),
False, table_v2.metadata_location
+ )
+ assert new_metadata.last_updated_ms >
new_metadata.metadata_log[-1].timestamp_ms
Review Comment:
Nit: nice regression test. The issue's repro also covers the
SetSnapshotRefUpdate path (the snapshot log gets stamped with the stale
snapshot timestamp). I probed that separately: Java tolerates it (snapshot-log
entries are validated with the same one-minute tolerance, and the stamping
actually matches Java's timeOfChange behavior for added snapshots), so this is
fine as is. Just mentioning in case you want a variant of this test with a
SetSnapshotRefUpdate to main for completeness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]