laskoviymishka commented on code in PR #16859:
URL: https://github.com/apache/iceberg/pull/16859#discussion_r3843662425
##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
// latest snapshot in this file corresponding to the log entry
latestSnapshotId,
latestSnapshot != null ? latestSnapshot.schemaId() : null,
- latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+ latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+ properties);
+ }
+
+ private static Map<String, String> tablePropertiesResolver(
Review Comment:
Small thing: `tablePropertiesResolver` reads as a noun, but it's a static
helper doing an action, and every other method in this file is a verb phrase
(`metadataLogEntryToRow`, `task`, `planFiles`). I'd rename to
`resolveTableProperties` (or `loadTableProperties`) — the call site then reads
as `loadTableProperties(metadataLogEntry, io, current, skipPropertiesLoad)`.
##########
core/src/main/java/org/apache/iceberg/MetadataLogEntriesTable.java:
##########
@@ -119,6 +140,32 @@ private static StaticDataTask.Row metadataLogEntryToRow(
// latest snapshot in this file corresponding to the log entry
latestSnapshotId,
latestSnapshot != null ? latestSnapshot.schemaId() : null,
- latestSnapshot != null ? latestSnapshot.sequenceNumber() : null);
+ latestSnapshot != null ? latestSnapshot.sequenceNumber() : null,
+ properties);
+ }
+
+ private static Map<String, String> tablePropertiesResolver(
+ TableMetadata.MetadataLogEntry metadataLogEntry,
+ FileIO io,
+ TableMetadata current,
+ boolean skipPropertiesLoad) {
+
+ // Avoid loading metadata file when properties are not projected.
+ if (skipPropertiesLoad) {
+ return null;
+ }
+
+ // Reuse the already loaded current metadata.
+ if (metadataLogEntry.file().equals(current.metadataFileLocation())) {
+ return current.properties();
+ }
+
+ try {
+ return TableMetadataParser.read(io,
metadataLogEntry.file()).properties();
+ } catch (NotFoundException e) {
Review Comment:
The null-on-missing-file behavior looks intentional and matches what you
noted in the design thread — no issue there. What I'm less sure about is the
other failure modes: `TableMetadataParser.read` can also throw
`RuntimeIOException` (corrupt or partially-written metadata, permission
denied), and those propagate and fail the whole scan. Since the use case is
post-incident RCA, that's exactly when a historical file is most likely
inconsistent. Do you want those to surface, or fold into the same null+warn
path? Either's fine — just worth being deliberate about which.
##########
spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMetadataTables.java:
##########
@@ -644,19 +644,22 @@ public void testMetadataLogEntries() throws Exception {
metadataLogEntries.get(0).file(),
null,
null,
- null),
+ null,
+ tableMetadata.properties()),
Review Comment:
These assertions don't quite exercise the feature. The expected value is
`tableMetadata.properties()` — the current map — for every row, including the
historical ones, and nothing here changes a property between metadata versions,
so every metadata file carries the same set. An impl that just returned
`current.properties()` for every row regardless of history would pass all of
these.
The core unit test does cover the old→new case, so this isn't a correctness
gap — but the engine-level tests give no regression protection for the one
thing the column is for. In at least one of them I'd `ALTER TABLE ... SET
TBLPROPERTIES` between two appends and assert the earlier row shows the old
value and the later row the new one.
(Same applies to the Spark v4.0/v4.1 copies and the Flink
`TestFlinkMetaDataTable` versions — they share this pattern.)
##########
core/src/test/java/org/apache/iceberg/TestMetadataLogEntriesTableProperties.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.never;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.apache.iceberg.inmemory.InMemoryCatalog;
+import org.apache.iceberg.io.CloseableIterable;
+import org.apache.iceberg.io.FileIO;
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.relocated.com.google.common.collect.Iterables;
+import org.apache.iceberg.relocated.com.google.common.collect.Lists;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+class TestMetadataLogEntriesTableProperties {
+ private static final String PROPERTY = "key";
+ private static final String INITIAL_VALUE = "old";
+ private static final String UPDATED_VALUE = "new";
+ private static final TableIdentifier TABLE_IDENTIFIER =
TableIdentifier.of("ns", "table");
+ private static final Schema SCHEMA =
+ new Schema(Types.NestedField.required(1, "id", Types.IntegerType.get()));
+
+ private InMemoryCatalog catalog;
+ private BaseTable table;
+ private Map<String, String> initialProperties;
+ private Map<String, String> updatedProperties;
+
+ @BeforeEach
+ public void setupTableWithProperties() {
+ this.catalog = new InMemoryCatalog();
+ catalog.initialize("test", ImmutableMap.of());
+ catalog.createNamespace(TABLE_IDENTIFIER.namespace());
+ this.table =
+ (BaseTable)
+ catalog.createTable(
+ TABLE_IDENTIFIER,
+ SCHEMA,
+ PartitionSpec.unpartitioned(),
+ ImmutableMap.of(PROPERTY, INITIAL_VALUE));
+ this.initialProperties = ImmutableMap.copyOf(table.properties());
+ table.updateProperties().set(PROPERTY, UPDATED_VALUE).commit();
+ this.updatedProperties = ImmutableMap.copyOf(table.properties());
+ }
+
+ @AfterEach
+ public void after() throws IOException {
+ catalog.dropTable(TABLE_IDENTIFIER);
+ catalog.close();
+ }
+
+ @Test
+ public void loadsHistoricalPropertiesAndReusesCurrentMetadata() throws
IOException {
+ TableMetadata current = table.operations().current();
+ TableMetadata.MetadataLogEntry previous =
Iterables.getOnlyElement(current.previousFiles());
+ FileIO io = spy(table.io());
+
+ DataTask task = planTask(metadataLogEntriesTable(current,
io).newScan().select("properties"));
+
+ assertThat(firstColumnValues(task)).containsExactly(initialProperties,
updatedProperties);
+ verify(io).newInputFile(previous.file());
+ verify(io).newInputFile(current.metadataFileLocation());
Review Comment:
`verify(io).newInputFile(current.metadataFileLocation())` passes because
`StaticDataTask.of(...)` calls `newInputFile` once at construction — not
because the reuse fast path did anything. So this reads like it's checking the
current-metadata reuse, but it's really asserting a side effect of an unrelated
code path; if `StaticDataTask` ever stopped calling `newInputFile` on
construction, it'd still pass with the optimization broken.
`doesNotLoadPropertiesWhenNotProjected` uses `verify(io, never())` cleanly —
could we assert the reuse the same way (no second read of the current file), or
at least add a comment on the invariant? The missing-metadata test is well
structured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
