stevenzwu commented on code in PR #17523:
URL: https://github.com/apache/iceberg/pull/17523#discussion_r4544819251
##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -65,9 +66,14 @@ class BaseSnapshot implements Snapshot {
Map<String, String> summary,
Integer schemaId,
String manifestList,
+ String rootManifest,
Review Comment:
I followed @gaborkaszab 's suggestion and dropped the list-only overload,
keeping a single constructor that takes both locations. Existing callers pass
null for rootManifest.
The upside is a single signature. The downside is every previous list-only
call site now passes an extra null, which leads to churning in test code.
This is how BaseSnapshot has grown before: schemaId (#2275),
firstRowId/addedRows (#11948), and keyId (#12927) were added to the same
constructor.
##########
core/src/main/java/org/apache/iceberg/BaseSnapshot.java:
##########
@@ -68,6 +69,38 @@ class BaseSnapshot implements Snapshot {
Long firstRowId,
Long addedRows,
String keyId) {
+ this(
+ sequenceNumber,
+ snapshotId,
+ parentId,
+ timestampMillis,
+ operation,
+ summary,
+ schemaId,
+ manifestList,
+ null,
+ firstRowId,
+ addedRows,
+ keyId);
+ }
+
+ BaseSnapshot(
Review Comment:
went with your recommendation and modify the existing constructor directly.
##########
core/src/main/java/org/apache/iceberg/SnapshotParser.java:
##########
@@ -87,11 +88,16 @@ static void toJson(Snapshot snapshot, JsonGenerator
generator) throws IOExceptio
// write just the location. manifests should not be embedded in JSON
along with a list
generator.writeStringField(MANIFEST_LIST, manifestList);
} else {
- // embed the manifest list in the JSON, v1 only
- JsonUtil.writeStringArray(
- MANIFESTS,
- Iterables.transform(snapshot.allManifests(DUMMY_FILE_IO),
ManifestFile::path),
- generator);
+ String rootManifest = snapshot.rootManifestLocation();
+ if (rootManifest != null) {
Review Comment:
Done. toJson now reads both locations up front and uses a flat if / else if
/ else.
##########
core/src/main/java/org/apache/iceberg/SnapshotParser.java:
##########
@@ -87,11 +88,16 @@ static void toJson(Snapshot snapshot, JsonGenerator
generator) throws IOExceptio
// write just the location. manifests should not be embedded in JSON
along with a list
generator.writeStringField(MANIFEST_LIST, manifestList);
} else {
- // embed the manifest list in the JSON, v1 only
- JsonUtil.writeStringArray(
- MANIFESTS,
- Iterables.transform(snapshot.allManifests(DUMMY_FILE_IO),
ManifestFile::path),
- generator);
+ String rootManifest = snapshot.rootManifestLocation();
+ if (rootManifest != null) {
+ generator.writeStringField(ROOT_MANIFEST, rootManifest);
+ } else {
Review Comment:
Agreed. The order is now root-manifest first, then manifest-list, then the
embedded v1 manifests.
##########
core/src/main/java/org/apache/iceberg/SnapshotsTable.java:
##########
@@ -37,7 +37,8 @@ public class SnapshotsTable extends BaseMetadataTable {
Types.NestedField.optional(
6,
"summary",
- Types.MapType.ofRequired(7, 8, Types.StringType.get(),
Types.StringType.get())));
+ Types.MapType.ofRequired(7, 8, Types.StringType.get(),
Types.StringType.get())),
Review Comment:
Agreed. I removed the snapshots metadata table changes from this PR. We can
decide how v4 should show up in metadata tables in a separate discussion.
##########
core/src/main/java/org/apache/iceberg/FileCleanupStrategy.java:
##########
@@ -92,9 +92,12 @@ public abstract void cleanFiles(
ManifestFile.DELETED_FILES_COUNT.fieldId()));
protected CloseableIterable<ManifestFile> readManifests(Snapshot snapshot) {
- if (snapshot.manifestListLocation() != null) {
+ String location = snapshot.manifestListLocation();
+ if (location != null) {
return InternalData.read(
- FileFormat.AVRO,
fileIO.newInputFile(snapshot.manifestListLocation()))
+ FileFormat.AVRO,
+ ManifestLists.newInputFile(
+ fileIO, new BaseFileWithEncryptedKey(location, null,
snapshot.keyId(), null)))
Review Comment:
Agreed on the scope. I removed the FileCleanupStrategy change and the
ManifestLists routing, so this PR no longer depends on #17545. It now only adds
the root-manifest field to Snapshot, BaseSnapshot, and SnapshotParser, plus the
path rewrite in RewriteTablePathUtil.
##########
core/src/main/java/org/apache/iceberg/SnapshotParser.java:
##########
@@ -176,9 +182,11 @@ static Snapshot fromJson(JsonNode node) {
String keyId = JsonUtil.getStringOrNull(KEY_ID, node);
- if (node.has(MANIFEST_LIST)) {
- // the manifest list is stored in a manifest list file
- String manifestList = JsonUtil.getString(MANIFEST_LIST, node);
+ String manifestList = JsonUtil.getStringOrNull(MANIFEST_LIST, node);
+ String rootManifest = JsonUtil.getStringOrNull(ROOT_MANIFEST, node);
+
+ if (manifestList != null || rootManifest != null) {
+ // manifest-list is used by v1-v3 snapshots, root-manifest by v4
snapshots
return new BaseSnapshot(
Review Comment:
Yes, with the list-only constructor gone, fromJson now has a single call
that passes both manifestList and rootManifest. The v1 embedded-manifests
constructor is still separate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]