vbabenkoru commented on code in PR #9245:
URL: https://github.com/apache/paimon/pull/9245#discussion_r3801588994
##########
paimon-iceberg/src/test/java/org/apache/paimon/iceberg/IcebergRestMetadataCommitterTest.java:
##########
@@ -1348,6 +1351,44 @@ public void testRecreateWithNonZeroLineageWatermark()
throws Exception {
// Known Layer 1 limitation (documented in the spec): the server-side
next-row-id
// watermark restarts on recreation and stays behind the local
metadata; commits
// must keep succeeding regardless (validation is first-row-id >=
next-row-id).
+
+ // reader-visible lineage from the REST catalog matches the file-based
mirror:
+ // snapshot first-row-id and manifest assignments come from local
metadata, never
+ // from the server's table-level watermark. Compare by value, not just
non-nullity,
+ // against the locally-written IcebergMetadata + manifest list under
the paimon
+ // table's own metadata dir (catalogTableMetadataPath), which is the
source of truth
+ // the REST-registered table's metadata-location actually points at.
+ long latestSnapshotId = table.snapshotManager().latestSnapshotId();
+ IcebergMetadata localMetadata =
+ IcebergMetadata.fromPath(
+ table.fileIO(),
+ new Path(
+ catalogTableMetadataPath(table),
+ String.format("v%d.metadata.json",
latestSnapshotId)));
+ IcebergSnapshot localSnapshot = localMetadata.currentSnapshot();
+ assertThat(localSnapshot.firstRowId()).isNotNull();
+
+ IcebergPathFactory pathFactory = new
IcebergPathFactory(catalogTableMetadataPath(table));
+ IcebergManifestList localManifestList =
IcebergManifestList.create(table, pathFactory);
+ List<Long> localDataManifestFirstRowIds =
+ localManifestList.read(new
Path(localSnapshot.manifestList()).getName()).stream()
+ .filter(m -> m.content() ==
IcebergManifestFileMeta.Content.DATA)
+ .map(IcebergManifestFileMeta::firstRowId)
+ .collect(Collectors.toList());
+
assertThat(localDataManifestFirstRowIds).isNotEmpty().doesNotContainNull();
+
+ Table reloaded = restCatalog.loadTable(TableIdentifier.of("mydb",
"t"));
+
assertThat(reloaded.currentSnapshot().firstRowId()).isEqualTo(localSnapshot.firstRowId());
Review Comment:
Rather than patching the replay arithmetic, recreation now writes the
adjusted metadata to a file and calls `registerTable`, letting the server
import next-row-id and the rest of the metadata exactly as-is. The same
approach is used when initially creating a table with existing history and for
the exists-but-empty recovery path. Since registration has no post-create
commit step, the drop/create retry loop that path guarded against is no longer
needed.
As a side effect, the real partition spec is preserved even when a partition
column has field id 0, whereas the old create path had to fall back to an
unpartitioned spec. The test now asserts `server.nextRowId() >=
local.nextRowId()` and performs an external append through the Iceberg API to
verify that it allocates above the watermark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
