stevenzwu commented on code in PR #19030:
URL: https://github.com/apache/iceberg/pull/19030#discussion_r4648724792
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -59,6 +63,31 @@ class V4ManifestReader extends CloseableGroup implements
CloseableIterable<Track
TrackedFile.CONTENT_TYPE.fieldId(), // needed for content filtering
TrackedFile.RECORD_COUNT.fieldId()); // needed for first_row_id
assignment and filtering
+ /**
+ * Reads leaf manifest locations from a root manifest file.
+ *
+ * @param root a root {@link ManifestFile}
+ * @param io a {@link FileIO}
+ * @return a list of manifest locations tracked by the root manifest
+ */
+ static List<String> readManifestLocations(ManifestFile root, FileIO io) {
Review Comment:
Should this take a `RootManifestFile` instead of a `ManifestFile`? Passing a
leaf manifest would not fail here; the filter would return an empty list, and
cleanup would then treat none of the committed manifests as referenced.
##########
core/src/main/java/org/apache/iceberg/SnapshotProducer.java:
##########
@@ -614,32 +662,30 @@ private void notifyListeners() {
}
protected void cleanAll() {
- for (String manifestList : manifestsByManifestList.keySet()) {
+ for (String manifestList : rootToManifestLocations.keySet()) {
Review Comment:
nit: `manifestList` -> `rootLocation`/`rootFile`
##########
core/src/main/java/org/apache/iceberg/RootManifestProducer.java:
##########
@@ -0,0 +1,188 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableSet;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Types;
+
+/** Responsible for producing v4 root manifest files for a commit. */
+class RootManifestProducer {
+ private final TableOperations ops;
+ private final FileIO io;
+ private final long snapshotId;
+ private final Map<String, String> writerProperties = Map.of();
+ private final List<TrackedFile> appendedDataFiles = Lists.newArrayList();
+
+ // internal state
+ private final String commitUUID = UUID.randomUUID().toString();
+ private final AtomicInteger attempt = new AtomicInteger(0);
+
+ // commit metadata
+ private final Set<String> manifestLocations = Sets.newHashSet();
+ private long assignedRowIds = 0;
+
+ RootManifestProducer(TableOperations ops, long snapshotId) {
+ this.ops = ops;
+ this.io = ops.io();
+ this.snapshotId = snapshotId;
+ }
+
+ void appendFile(DataFile dataFile) {
+ appendedDataFiles.add(TrackedFileAdapters.asTrackedFile(dataFile,
ops.current().schema()));
+ }
+
+ TrackedFile apply(TableMetadata base, Snapshot parent, long
attemptSequenceNumber) {
+ RootMetadata root = RootMetadata.fromSnapshot(base, parent, io);
+
+ manifestLocations.clear();
+ V4ManifestWriter rootWriter = newRootWriter(base, attemptSequenceNumber,
base.nextRowId());
+ try (rootWriter) {
+ for (TrackedFile added : appendedDataFiles) {
+ rootWriter.append(added);
+ }
+
+ if (root != null) {
+ for (TrackedFile file : root.dataFiles()) {
+ if (file.tracking().isLive()) {
+ rootWriter.existing(file);
+ }
+ }
+
+ for (TrackedFile dataManifest : root.dataManifests()) {
+ if (dataManifest.tracking().isLive()) {
+ rootWriter.existing(dataManifest);
+ manifestLocations.add(dataManifest.location());
+ }
+ }
+
+ for (TrackedFile deleteManifest : root.deleteManifests()) {
+ if (deleteManifest.tracking().isLive()) {
+ rootWriter.existing(deleteManifest);
+ manifestLocations.add(deleteManifest.location());
+ }
+ }
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to write root manifest", e);
+ }
+
+ this.assignedRowIds = rootWriter.assignedRowIds();
+
+ return rootWriter.toTrackedManifest();
+ }
+
+ long assignedRowIds() {
+ return assignedRowIds;
+ }
+
+ Set<String> manifestLocations() {
+ return ImmutableSet.copyOf(manifestLocations);
+ }
+
+ private EncryptedOutputFile newRootOutputFile() {
+ String filename =
+ String.format(
+ Locale.ROOT, "root-%d-%d-%s", snapshotId,
attempt.incrementAndGet(), commitUUID);
+ String rootManifestLocation =
+ ops.metadataFileLocation(FileFormat.PARQUET.addExtension(filename));
+ OutputFile rawOutputFile = ops.io().newOutputFile(rootManifestLocation);
+ return ops.encryption().encrypt(rawOutputFile);
Review Comment:
I assume encryption is out of scope for this first version, which makes
sense. Should `writeRootManifest` fail fast when table encryption is enabled,
though? Otherwise the commit succeeds with an encrypted root that can't be read
back.
##########
core/src/main/java/org/apache/iceberg/RootManifestProducer.java:
##########
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import java.io.IOException;
+import java.io.UncheckedIOException;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iceberg.encryption.EncryptedOutputFile;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.io.OutputFile;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+import org.apache.iceberg.types.Types;
+
+/** Responsible for producing v4 root manifest files for a commit. */
+class RootManifestProducer {
+ private final TableOperations ops;
+ private final FileIO io;
+ private final long snapshotId;
+ private final Types.StructType contentStatsType;
+ private final int startingSchemaId;
+ private final Map<String, String> writerProperties = Map.of();
+ private final List<TrackedFile> appendedDataFiles = Lists.newArrayList();
+
+ // internal state
+ private final String commitUUID = UUID.randomUUID().toString();
+ private final AtomicInteger attempt = new AtomicInteger(0);
+ private final AtomicInteger manifestCount = new AtomicInteger(0);
+
+ // commit metadata
+ private final Set<String> manifestLocations = Sets.newHashSet();
+ private long assignedRowIds = 0;
+
+ RootManifestProducer(TableOperations ops, long snapshotId) {
+ this.ops = ops;
+ this.io = ops.io();
+ this.snapshotId = snapshotId;
+ this.contentStatsType =
+ StatsUtil.statsWriteSchema(ops.current().schema(),
MetricsConfig.forTable(ops.current()));
+ this.startingSchemaId = ops.current().currentSchemaId();
+ }
+
+ void appendFile(DataFile dataFile) {
+ appendedDataFiles.add(TrackedFileAdapters.asTrackedFile(dataFile,
ops.current().schema()));
+ }
+
+ TrackedFile apply(TableMetadata base, Snapshot parent, long
attemptSequenceNumber) {
+ RootMetadata root = RootMetadata.fromSnapshot(base, parent, io);
+
+ manifestLocations.clear();
+ V4ManifestWriter rootWriter = newRootWriter(base, attemptSequenceNumber,
base.nextRowId());
+ try (rootWriter) {
+ for (TrackedFile added : appendedDataFiles) {
+ rootWriter.append(added);
+ }
+
+ if (root != null) {
+ for (TrackedFile file : root.dataFiles()) {
+ if (file.tracking().isLive()) {
+ rootWriter.existing(file);
+ }
+ }
+
+ for (TrackedFile dataManifest : root.dataManifests()) {
+ if (dataManifest.tracking().isLive()) {
+ rootWriter.existing(dataManifest);
+ manifestLocations.add(dataManifest.location());
+ }
+ }
+
+ for (TrackedFile deleteManifest : root.deleteManifests()) {
+ if (deleteManifest.tracking().isLive()) {
+ rootWriter.existing(deleteManifest);
+ manifestLocations.add(deleteManifest.location());
+ }
+ }
+ }
+ } catch (IOException e) {
+ throw new UncheckedIOException("Failed to write root manifest", e);
+ }
+
+ this.assignedRowIds = rootWriter.assignedRowIds();
+
+ return rootWriter.toTrackedManifest();
+ }
+
+ long assignedRowIds() {
+ return assignedRowIds;
+ }
+
+ Set<String> manifestLocations() {
+ return manifestLocations;
+ }
+
+ private EncryptedOutputFile newRootOutputFile() {
+ String filename =
+ String.format(
+ Locale.ROOT, "root-%d-%d-%s", snapshotId,
attempt.incrementAndGet(), commitUUID);
+ String rootManifestLocation =
+ ops.metadataFileLocation(FileFormat.PARQUET.addExtension(filename));
+ OutputFile rawOutputFile = ops.io().newOutputFile(rootManifestLocation);
+ return ops.encryption().encrypt(rawOutputFile);
+ }
+
+ private V4ManifestWriter newRootWriter(TableMetadata base, long
sequenceNumber, long firstRowId) {
+ // always use the latest stats type to prevent dropping stats from new
files in the root
+ Types.StructType statsType =
+ StatsUtil.statsWriteSchema(base.schema(),
MetricsConfig.forTable(base));
+ return V4ManifestWriter.createRoot(
+ newRootOutputFile(),
+ base.specsById(),
+ statsType,
+ writerProperties,
+ base.currentSchemaId(),
+ snapshotId,
+ sequenceNumber,
+ firstRowId);
+ }
+
+ private record RootMetadata(
+ List<TrackedFile> dataFiles,
+ List<TrackedFile> dataManifests,
+ List<TrackedFile> deleteManifests) {
+ private static RootMetadata fromRootManifest(TableMetadata meta, Snapshot
snapshot, FileIO io) {
+ ManifestFile root = RootManifestFile.from(snapshot, meta.location(), io);
+
+ ImmutableList.Builder<TrackedFile> rootDataFiles =
ImmutableList.builder();
+ ImmutableList.Builder<TrackedFile> dataManifests =
ImmutableList.builder();
+ ImmutableList.Builder<TrackedFile> deleteManifests =
ImmutableList.builder();
+
+ try (V4ManifestReader reader =
+ V4ManifestReader.builder(root, io, meta.schema(), meta.specsById())
Review Comment:
The second half of this still looks open:
`V4ManifestReader.readManifestLocations` builds its reader without
`.tableLocation(...)`.
##########
core/src/main/java/org/apache/iceberg/inmemory/InMemoryInputFile.java:
##########
@@ -29,36 +30,59 @@ public class InMemoryInputFile implements InputFile {
private final String location;
private final byte[] contents;
+ private final InMemoryFileIO parentFileIO;
public InMemoryInputFile(byte[] contents) {
this("memory:" + UUID.randomUUID(), contents);
}
public InMemoryInputFile(String location, byte[] contents) {
Preconditions.checkNotNull(location, "location is null");
- Preconditions.checkNotNull(contents, "contents is null");
Review Comment:
curious why did we drop the null check and defensive clone? For this
constructor, shouldn't `contents` be required?
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -59,6 +63,31 @@ class V4ManifestReader extends CloseableGroup implements
CloseableIterable<Track
TrackedFile.CONTENT_TYPE.fieldId(), // needed for content filtering
TrackedFile.RECORD_COUNT.fieldId()); // needed for first_row_id
assignment and filtering
+ /**
+ * Reads leaf manifest locations from a root manifest file.
+ *
+ * @param root a root {@link ManifestFile}
+ * @param io a {@link FileIO}
+ * @return a list of manifest locations tracked by the root manifest
+ */
+ static List<String> readManifestLocations(ManifestFile root, FileIO io) {
+ try (V4ManifestReader reader =
+ builder(root, io, new Schema(), ImmutableMap.of())
+ .metricsConfig(MetricsConfig.getDefault())
+ .select(TrackedFile.CONTENT_TYPE.name(),
TrackedFile.LOCATION.name())
Review Comment:
nit: `content_type` is always projected along with `status`, which the
reader uses to skip deleted entries. Selecting only
`TrackedFile.LOCATION.name()` would be enough here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]