vbabenkoru commented on code in PR #9245:
URL: https://github.com/apache/paimon/pull/9245#discussion_r3801588994
##########
paimon-iceberg/src/test/java/org/apache/paimon/iceberg/IcebergRestMetadataCommitterTest.java:
##########
@@ -1348,6 +1351,44 @@ public void testRecreateWithNonZeroLineageWatermark()
throws Exception {
// Known Layer 1 limitation (documented in the spec): the server-side
next-row-id
// watermark restarts on recreation and stays behind the local
metadata; commits
// must keep succeeding regardless (validation is first-row-id >=
next-row-id).
+
+ // reader-visible lineage from the REST catalog matches the file-based
mirror:
+ // snapshot first-row-id and manifest assignments come from local
metadata, never
+ // from the server's table-level watermark. Compare by value, not just
non-nullity,
+ // against the locally-written IcebergMetadata + manifest list under
the paimon
+ // table's own metadata dir (catalogTableMetadataPath), which is the
source of truth
+ // the REST-registered table's metadata-location actually points at.
+ long latestSnapshotId = table.snapshotManager().latestSnapshotId();
+ IcebergMetadata localMetadata =
+ IcebergMetadata.fromPath(
+ table.fileIO(),
+ new Path(
+ catalogTableMetadataPath(table),
+ String.format("v%d.metadata.json",
latestSnapshotId)));
+ IcebergSnapshot localSnapshot = localMetadata.currentSnapshot();
+ assertThat(localSnapshot.firstRowId()).isNotNull();
+
+ IcebergPathFactory pathFactory = new
IcebergPathFactory(catalogTableMetadataPath(table));
+ IcebergManifestList localManifestList =
IcebergManifestList.create(table, pathFactory);
+ List<Long> localDataManifestFirstRowIds =
+ localManifestList.read(new
Path(localSnapshot.manifestList()).getName()).stream()
+ .filter(m -> m.content() ==
IcebergManifestFileMeta.Content.DATA)
+ .map(IcebergManifestFileMeta::firstRowId)
+ .collect(Collectors.toList());
+
assertThat(localDataManifestFirstRowIds).isNotEmpty().doesNotContainNull();
+
+ Table reloaded = restCatalog.loadTable(TableIdentifier.of("mydb",
"t"));
+
assertThat(reloaded.currentSnapshot().firstRowId()).isEqualTo(localSnapshot.firstRowId());
Review Comment:
Instead of rebuilding through `TableMetadata.Builder` (which recomputes
`next-row-id` from added-rows and loses any id space not starting at 0),
recreation — plus initial creation of a table with history, and the
exists-but-empty recovery — now writes the REST-adjusted metadata to a file and
**registers** it, importing every high-water mark verbatim. Two side effects
worth noting: the drop+create failure loop the empty-table recovery guarded
against can't occur (registration has no post-create commit step), and the real
partition spec now survives even with a field-id-0 partition column, where
creation used to fall back to an unpartitioned spec. The recreation test now
asserts `server.nextRowId() >= local.nextRowId()` and that an external append
through the Iceberg API allocates above the imported watermark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
