difin commented on code in PR #6927:
URL: https://github.com/apache/hive/pull/6927#discussion_r4622726937


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/BaseHiveIcebergMetaHook.java:
##########
@@ -453,6 +456,45 @@ private Schema getSchemaWithIdentifierFields(Schema 
schema, Set<String> identifi
     return new Schema(cols, identifierFieldIds);
   }
 
+  /**
+   * UNKNOWN values are omitted from data files, so Iceberg can only write 
them as top-level columns or as struct
+   * fields that have at least one sibling of another type. Lists, maps and 
all-UNKNOWN structs would have no
+   * physical representation left.
+   */
+  static void validateUnknownType(String columnPath, Type type) {
+    switch (type.typeId()) {
+      case STRUCT -> {
+        List<Types.NestedField> fields = type.asStructType().fields();
+        Preconditions.checkArgument(fields.stream().anyMatch(field -> 
!isUnknown(field.type())),
+            "Column '%s' of type %s is not supported: a struct must have at 
least one field that is not of type " +

Review Comment:
   Creating it works, but the table isn't usable on every format. UNKNOWN 
columns aren't written to data files, so a table whose columns are all UNKNOWN 
has an empty file schema. Parquet refuses to write it ("Cannot write a schema 
with an empty group"), and Hive's vectorized ORC reader can't open it. Avro and 
non-vectorized ORC work. This is the same reason structs whose fields are all 
UNKNOWN are already rejected.
   
   I'll reject it at DDL time instead: CREATE TABLE, REPLACE COLUMNS and DROP 
COLUMN will fail with "a table must have at least one column that is not of 
type unknown". I'll add a negative q-test for it. A lone UNKNOWN column next to 
other columns stays supported, and `positive/unknown_type.q` covers that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to