gaborkaszab commented on code in PR #17523:
URL: https://github.com/apache/iceberg/pull/17523#discussion_r4547592073
##########
api/src/main/java/org/apache/iceberg/Snapshot.java:
##########
@@ -166,10 +166,26 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO
io) {
/**
* Return the location of this snapshot's manifest list, or null if it is
not separate.
Review Comment:
The "or null if.." part is also relevant now here, right?
##########
api/src/main/java/org/apache/iceberg/Snapshot.java:
##########
@@ -166,10 +166,26 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO
io) {
/**
* Return the location of this snapshot's manifest list, or null if it is
not separate.
*
- * @return the location of the manifest list for this Snapshot
+ * <p>Only set for tables with format version 1 through 3. Snapshots in v4
tables use {@link
Review Comment:
I'm hesitant adding such a comment here. We also don't mention that for
`first-row-id`, `added-rows` etc. are only relevant from V3 onwards.
I'd like it to the spec to describe the relation between this and
rootManifestLocation.
##########
api/src/main/java/org/apache/iceberg/Snapshot.java:
##########
@@ -166,10 +166,26 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO
io) {
/**
* Return the location of this snapshot's manifest list, or null if it is
not separate.
*
- * @return the location of the manifest list for this Snapshot
+ * <p>Only set for tables with format version 1 through 3. Snapshots in v4
tables use {@link
+ * #rootManifestLocation()} instead.
+ *
+ * @return the location of the manifest list for this snapshot
*/
String manifestListLocation();
+ /**
+ * Return the location of this snapshot's root manifest, or null if this
snapshot has no root
+ * manifest.
+ *
+ * <p>v4 snapshots store this in {@code root-manifest} instead of a manifest
list. At most one of
Review Comment:
Same comment as above
##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -65,9 +66,14 @@ class BaseSnapshot implements Snapshot {
Map<String, String> summary,
Integer schemaId,
String manifestList,
+ String rootManifest,
Long firstRowId,
Long addedRows,
String keyId) {
+ Preconditions.checkArgument(
Review Comment:
This verifies they can't be set together. Should we also verify that one of
them has to be set?
##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -181,7 +189,11 @@ private void cacheManifests(FileIO fileIO) {
}
if (allManifests == null) {
- // if manifests isn't set, then the snapshotFile is set and should be
read to get the list
+ // if manifests isn't set, then the manifest list must be read
+ Preconditions.checkArgument(
Review Comment:
Don't we want to reverse this and at L181 verify that `rootManifestLocation`
is null, otherwise throw that we don't support this for V4. We could leave the
rest of the function intact.
##########
core/src/main/java/org/apache/iceberg/SnapshotParser.java:
##########
@@ -176,9 +180,11 @@ static Snapshot fromJson(JsonNode node) {
String keyId = JsonUtil.getStringOrNull(KEY_ID, node);
- if (node.has(MANIFEST_LIST)) {
- // the manifest list is stored in a manifest list file
- String manifestList = JsonUtil.getString(MANIFEST_LIST, node);
+ String manifestList = JsonUtil.getStringOrNull(MANIFEST_LIST, node);
+ String rootManifest = JsonUtil.getStringOrNull(ROOT_MANIFEST, node);
+
+ if (manifestList != null || rootManifest != null) {
+ // manifest-list is used by v1-v3 snapshots, root-manifest by v4
snapshots
Review Comment:
nit: comment is not needed
##########
core/src/test/java/org/apache/iceberg/TestMetadataUpdateParser.java:
##########
@@ -423,6 +425,37 @@ public void testAddSnapshotFromJson() throws IOException {
assertEquals(action, expected, MetadataUpdateParser.fromJson(json));
}
+ @Test
+ public void testAddSnapshotWithRootManifest() {
+ String action = MetadataUpdateParser.ADD_SNAPSHOT;
+ long snapshotId = 2;
+ int schemaId = 3;
Review Comment:
nit: this could be inline at L443
##########
core/src/main/java/org/apache/iceberg/RewriteTablePathUtil.java:
##########
@@ -241,8 +241,8 @@ private static List<Snapshot> updatePathInSnapshots(
TableMetadata metadata, String sourcePrefix, String targetPrefix) {
List<Snapshot> newSnapshots =
Lists.newArrayListWithCapacity(metadata.snapshots().size());
for (Snapshot snapshot : metadata.snapshots()) {
- String newManifestListLocation =
- newPath(snapshot.manifestListLocation(), sourcePrefix, targetPrefix);
+ String manifestList = snapshot.manifestListLocation();
Review Comment:
We're introducing the new field to `Snapshot`, I don't think that we should
implement table rewrite functionality right away. Could that be separate?
##########
core/src/test/java/org/apache/iceberg/TestSnapshotJson.java:
##########
@@ -156,13 +164,16 @@ public void testJsonConversionWithRowLineage() throws
IOException {
null,
null,
manifestList,
+ null /* rootManifest */,
firstRowId,
addedRows,
null);
String json = SnapshotParser.toJson(expected);
Snapshot snapshot = SnapshotParser.fromJson(json);
assertThat(snapshot.snapshotId()).isEqualTo(expected.snapshotId());
+
assertThat(snapshot.manifestListLocation()).isEqualTo(expected.manifestListLocation());
+
assertThat(snapshot.rootManifestLocation()).isEqualTo(expected.rootManifestLocation());
Review Comment:
Not sure what coverage we add with these asserts on rootManifestLocation,
because they are always null, except in one dedicated test. Should we assert
`isNull()` as we do with other fields below?
##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -65,9 +66,14 @@ class BaseSnapshot implements Snapshot {
Map<String, String> summary,
Integer schemaId,
String manifestList,
+ String rootManifest,
Review Comment:
We might want to introduce a dedicated contract once all the other V4
Snapshot fields are introduced, but let it be the concert of the follow-ups,
not this one.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]