danielcweeks commented on code in PR #18935:
URL: https://github.com/apache/iceberg/pull/18935#discussion_r4569732608


##########
core/src/main/java/org/apache/iceberg/MetricsConfig.java:
##########
@@ -190,6 +190,11 @@ public Set<Integer> struct(Types.StructType struct, 
Iterable<Set<Integer>> field
             return null;
           }
 
+          @Override
+          public Set<Integer> file(Types.FileType file, Iterable<Set<Integer>> 
fieldResults) {
+            return struct(file.asStruct(), fieldResults);
+          }

Review Comment:
   nit: is treating a file exactly like a struct the right default for inferred 
metrics? Every file column now uses six slots of the inferred-column limit, and 
with several file columns that could push real data columns out of default 
metrics. It also collects truncated bounds on `inline` and `checksum`, which I 
doubt help pruning. If that's intended, let's say so. Otherwise, `file()` could 
add only `uri`, `size`, and `content_type` by default, possibly in a later part 
of the series.
   
   <!-- lenses: weeks-review, russell-review -->



##########
core/src/main/java/org/apache/iceberg/SchemaOrder.java:
##########
@@ -110,6 +110,11 @@ public List<Integer> map(
       return null;
     }
 
+    @Override
+    public List<Integer> file(Types.FileType file, Iterable<List<Integer>> 
fieldResults) {
+      return struct(file.asStruct(), fieldResults);

Review Comment:
   Neither this nor the `MetricsConfig.limitFieldIds` override has a direct 
test, so nothing pins the order of the nested IDs or which file subfields get 
default metrics. `TestSchemaOrder` already has one test per type. Can we add a 
`fileOrder()` case there and a `limitFieldIds` case in `TestMetricsConfig`? 
Derive the expected IDs from `Types.FileType.of(n).fields()` rather than 
hardcoding them.
   
   <!-- lenses: nastra-test-review, blue-review -->



##########
core/src/test/java/org/apache/iceberg/TestFileTypeTableMetadata.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iceberg;
+
+import static org.apache.iceberg.types.Types.NestedField.optional;
+import static org.apache.iceberg.types.Types.NestedField.required;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.types.Types;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+class TestFileTypeTableMetadata {
+  private static final Schema SCHEMA =
+      new Schema(
+          required(1, "id", Types.LongType.get()), optional(2, "photo", 
Types.FileType.of(2)));
+
+  @ParameterizedTest
+  @ValueSource(ints = {1, 2, 3})
+  void rejectsAFileColumnBeforeFormatVersion4(int formatVersion) {
+    assertThatThrownBy(() -> newTableMetadata(formatVersion))
+        .isInstanceOf(IllegalStateException.class)
+        .hasMessageContaining("Invalid type for photo: file is not supported 
until v4");
+  }
+
+  @Test
+  void acceptsAMetricsModeForAFileNestedField() {
+    TableMetadata metadata =
+        TableMetadata.newTableMetadata(
+            SCHEMA,
+            PartitionSpec.unpartitioned(),
+            "file:/tmp/table",
+            ImmutableMap.of(
+                TableProperties.FORMAT_VERSION,
+                "4",
+                TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + "photo.size",
+                "full"));
+
+    assertThat(metadata.properties())
+        .containsEntry(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + 
"photo.size", "full");
+  }

Review Comment:
   I don't think this tests what it says. `metadata.properties()` just returns 
the map we passed in, so the only real check is that `newTableMetadata` doesn't 
throw. Can we assert the resolved mode instead? This is `MetricsConfig` 
behavior, so it probably belongs in `TestMetricsConfig`:
   
   ```java
   MetricsConfig config =
       MetricsConfig.from(
           ImmutableMap.of(TableProperties.METRICS_MODE_COLUMN_CONF_PREFIX + 
"photo.size", "full"),
           schema,
           SortOrder.unsorted());
   
   
assertThat(config.columnMode("photo.size")).isEqualTo(MetricsModes.Full.get());
   ```
   
   That would replace the `containsEntry` assertion here. It's also worth 
checking that a nested field with no override gets the default mode.
   
   <!-- lenses: weeks-review, russell-review, nastra-test-review, blue-review 
-->



##########
core/src/test/java/org/apache/iceberg/TestSchemaUpdate.java:
##########
@@ -2590,4 +2590,119 @@ public void 
testCaseInsensitiveMoveAfterNewlyAddedField() {
 
     assertThat(actual.asStruct()).isEqualTo(expected.asStruct());
   }
+
+  private static final Schema FILE_SCHEMA =
+      new Schema(
+          required(1, "id", Types.LongType.get()),
+          optional(2, "photo", Types.FileType.of(2)),
+          optional(9, "data", Types.StringType.get()));
+
+  private static SchemaUpdate fileUpdate() {
+    return new SchemaUpdate(FILE_SCHEMA, FILE_SCHEMA.highestFieldId());
+  }
+
+  @Test
+  void cannotAddColumnToFileColumn() {
+    assertThatThrownBy(() -> fileUpdate().addColumn("photo", "extra", 
Types.StringType.get()))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot add to non-struct column: photo: file");
+  }
+
+  @Test
+  void cannotDeleteFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().deleteColumn("photo.checksum"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.checksum");
+  }
+
+  @Test
+  void cannotRenameFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().renameColumn("photo.uri", 
"location"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotPromoteFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().updateColumn("photo.size", 
Types.LongType.get()))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.size");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldDoc() {
+    assertThatThrownBy(() -> fileUpdate().updateColumnDoc("photo.uri", "the 
location"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldDefault() {
+    assertThatThrownBy(
+            () -> fileUpdate().updateColumnDefault("photo.uri", 
Literal.of("s3://bucket/key")))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldRequirement() {
+    assertThatThrownBy(() -> fileUpdate().requireColumn("photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+    assertThatThrownBy(() -> fileUpdate().makeColumnOptional("photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotMoveFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().moveFirst("photo.checksum"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+    assertThatThrownBy(() -> fileUpdate().moveBefore("photo.checksum", 
"photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+    assertThatThrownBy(() -> fileUpdate().moveAfter("photo.uri", 
"photo.inline"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+  }
+
+  @Test
+  void renameFileColumn() {
+    Schema renamed = fileUpdate().renameColumn("photo", "image").apply();
+
+    
assertThat(renamed.findField("image").type()).isEqualTo(Types.FileType.of(2));
+    assertThat(renamed.findField("image.uri").fieldId()).isEqualTo(3);
+  }

Review Comment:
   nit: we know the full expected schema here, so can we assert the whole 
struct like `deleteFileColumn` does? Feels like we're taking a shortcut with a 
half-assert.
   
   <!-- lenses: nastra-test-review -->



##########
core/src/test/java/org/apache/iceberg/TestSchemaUpdate.java:
##########
@@ -2590,4 +2590,119 @@ public void 
testCaseInsensitiveMoveAfterNewlyAddedField() {
 
     assertThat(actual.asStruct()).isEqualTo(expected.asStruct());
   }
+
+  private static final Schema FILE_SCHEMA =

Review Comment:
   Shouldn't `FILE_SCHEMA` and `fileUpdate()` move to the top of the class with 
the other fields?



##########
api/src/test/java/org/apache/iceberg/types/TestReadabilityChecks.java:
##########
@@ -286,6 +286,27 @@ public void testIncompatibleStructAndPrimitive() {
         .contains("struct cannot be read as a string");
   }
 
+  @Test
+  void incompatibleFileAndStruct() {
+    Schema fileSchema = new Schema(optional(1, "photo", Types.FileType.of(1)));
+    Schema structSchema =
+        new Schema(optional(1, "photo", 
Types.StructType.of(Types.FileType.of(1).fields())));
+
+    List<String> asFile = 
CheckCompatibility.readCompatibilityErrors(fileSchema, structSchema);
+    assertThat(asFile).hasSize(1);
+    assertThat(asFile.get(0))
+        .as("Should complain that a struct cannot be read as a file")
+        .contains("cannot be read as a file");
+
+    List<String> asStruct = 
CheckCompatibility.readCompatibilityErrors(structSchema, fileSchema);
+    assertThat(asStruct).hasSize(1);
+    assertThat(asStruct.get(0))
+        .as("Should complain that a file cannot be read as a struct")
+        .contains("file cannot be read as a struct");
+
+    assertThat(CheckCompatibility.readCompatibilityErrors(fileSchema, 
fileSchema)).isEmpty();
+  }

Review Comment:
   nit: this covers three behaviors (struct as file, file as struct, file as 
file). Can we split it into three tests? `hasSize(1)` followed by `get(0)` also 
hides the actual error on failure. Prefer 
`.singleElement().asString().endsWith("cannot be read as a file")`, and drop 
the `.as()` descriptions since they just repeat the assertion.
   
   <!-- lenses: nastra-test-review, blue-review -->



##########
core/src/test/java/org/apache/iceberg/TestSchemaUpdate.java:
##########
@@ -2590,4 +2590,119 @@ public void 
testCaseInsensitiveMoveAfterNewlyAddedField() {
 
     assertThat(actual.asStruct()).isEqualTo(expected.asStruct());
   }
+
+  private static final Schema FILE_SCHEMA =
+      new Schema(
+          required(1, "id", Types.LongType.get()),
+          optional(2, "photo", Types.FileType.of(2)),
+          optional(9, "data", Types.StringType.get()));
+
+  private static SchemaUpdate fileUpdate() {
+    return new SchemaUpdate(FILE_SCHEMA, FILE_SCHEMA.highestFieldId());
+  }
+
+  @Test
+  void cannotAddColumnToFileColumn() {
+    assertThatThrownBy(() -> fileUpdate().addColumn("photo", "extra", 
Types.StringType.get()))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot add to non-struct column: photo: file");
+  }
+
+  @Test
+  void cannotDeleteFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().deleteColumn("photo.checksum"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.checksum");
+  }
+
+  @Test
+  void cannotRenameFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().renameColumn("photo.uri", 
"location"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotPromoteFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().updateColumn("photo.size", 
Types.LongType.get()))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.size");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldDoc() {
+    assertThatThrownBy(() -> fileUpdate().updateColumnDoc("photo.uri", "the 
location"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldDefault() {
+    assertThatThrownBy(
+            () -> fileUpdate().updateColumnDefault("photo.uri", 
Literal.of("s3://bucket/key")))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldRequirement() {
+    assertThatThrownBy(() -> fileUpdate().requireColumn("photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+    assertThatThrownBy(() -> fileUpdate().makeColumnOptional("photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotMoveFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().moveFirst("photo.checksum"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+    assertThatThrownBy(() -> fileUpdate().moveBefore("photo.checksum", 
"photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+    assertThatThrownBy(() -> fileUpdate().moveAfter("photo.uri", 
"photo.inline"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+  }
+
+  @Test
+  void renameFileColumn() {
+    Schema renamed = fileUpdate().renameColumn("photo", "image").apply();
+
+    
assertThat(renamed.findField("image").type()).isEqualTo(Types.FileType.of(2));
+    assertThat(renamed.findField("image.uri").fieldId()).isEqualTo(3);
+  }
+
+  @Test
+  void deleteFileColumn() {
+    Schema deleted = fileUpdate().deleteColumn("photo").apply();
+
+    assertThat(deleted.findField("photo")).isNull();
+    assertThat(deleted.asStruct())
+        .isEqualTo(
+            new Schema(
+                    required(1, "id", Types.LongType.get()),
+                    optional(9, "data", Types.StringType.get()))
+                .asStruct());
+  }
+
+  @Test
+  void addFileColumnReservesNestedIds() {
+    Schema schema = new Schema(required(1, "id", Types.LongType.get()));
+    Schema expected =
+        new Schema(
+            required(1, "id", Types.LongType.get()),
+            optional(2, "photo", Types.FileType.of(2)),
+            optional(9, "data", Types.StringType.get()));
+
+    Schema updated =
+        new SchemaUpdate(schema, schema.highestFieldId())
+            .addColumn("photo", Types.FileType.of(0))
+            .addColumn("data", Types.StringType.get())
+            .apply();
+
+    assertThat(updated.asStruct()).isEqualTo(expected.asStruct());
+  }

Review Comment:
   We're only testing top-level.  Should we add one at a lower depth (in map or 
list), since it goes through a slightly different path?
   
   <!-- lenses: nastra-test-review, russell-review -->



##########
api/src/main/java/org/apache/iceberg/types/CheckCompatibility.java:
##########
@@ -260,6 +260,16 @@ public List<String> variant(Types.VariantType readVariant) 
{
     return ImmutableList.of(String.format(": %s cannot be read as a %s", 
currentType, readVariant));
   }
 
+  @Override
+  public List<String> file(Types.FileType readFile, Iterable<List<String>> 
fieldErrorLists) {
+    if (currentType.isFileType()) {
+      // the nested fields are derived from the base id, so matching ids means 
matching fields

Review Comment:
   This seems like an unnecessary (and unrelated) comment.



##########
api/src/main/java/org/apache/iceberg/Schema.java:
##########
@@ -161,6 +163,20 @@ public Schema(
     this.highestFieldId = lazyIdToName().keySet().stream().mapToInt(i -> 
i).max().orElse(0);
   }
 
+  private void validateFileBaseIds() {
+    for (NestedField field : lazyIdToField().values()) {

Review Comment:
   This seems like the convenient way to do this, but I agree, we're 
sacrificing the lazy behavior.  We could have a schema visitor that does the 
same behavior and avoid eager creation of the mapping.



##########
core/src/test/java/org/apache/iceberg/TestSchemaUpdate.java:
##########
@@ -2590,4 +2590,119 @@ public void 
testCaseInsensitiveMoveAfterNewlyAddedField() {
 
     assertThat(actual.asStruct()).isEqualTo(expected.asStruct());
   }
+
+  private static final Schema FILE_SCHEMA =
+      new Schema(
+          required(1, "id", Types.LongType.get()),
+          optional(2, "photo", Types.FileType.of(2)),
+          optional(9, "data", Types.StringType.get()));
+
+  private static SchemaUpdate fileUpdate() {
+    return new SchemaUpdate(FILE_SCHEMA, FILE_SCHEMA.highestFieldId());
+  }
+
+  @Test
+  void cannotAddColumnToFileColumn() {
+    assertThatThrownBy(() -> fileUpdate().addColumn("photo", "extra", 
Types.StringType.get()))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot add to non-struct column: photo: file");
+  }
+
+  @Test
+  void cannotDeleteFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().deleteColumn("photo.checksum"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.checksum");
+  }
+
+  @Test
+  void cannotRenameFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().renameColumn("photo.uri", 
"location"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotPromoteFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().updateColumn("photo.size", 
Types.LongType.get()))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.size");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldDoc() {
+    assertThatThrownBy(() -> fileUpdate().updateColumnDoc("photo.uri", "the 
location"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldDefault() {
+    assertThatThrownBy(
+            () -> fileUpdate().updateColumnDefault("photo.uri", 
Literal.of("s3://bucket/key")))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotUpdateFileNestedFieldRequirement() {
+    assertThatThrownBy(() -> fileUpdate().requireColumn("photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+    assertThatThrownBy(() -> fileUpdate().makeColumnOptional("photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot change a nested field of a file column: 
photo.uri");
+  }
+
+  @Test
+  void cannotMoveFileNestedField() {
+    assertThatThrownBy(() -> fileUpdate().moveFirst("photo.checksum"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+    assertThatThrownBy(() -> fileUpdate().moveBefore("photo.checksum", 
"photo.uri"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+    assertThatThrownBy(() -> fileUpdate().moveAfter("photo.uri", 
"photo.inline"))
+        .isInstanceOf(IllegalArgumentException.class)
+        .hasMessage("Cannot move fields in non-struct type: file");
+  }

Review Comment:
   nit: these two tests each check several operations, so a failure doesn't 
tell you which one broke. Can we split them into one test per operation like 
other tests above?
   
   <!-- lenses: nastra-test-review -->



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to