jerryshao commented on code in PR #14659:
URL: https://github.com/apache/gravitino/pull/14659#discussion_r4510040321


##########
core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java:
##########
@@ -714,6 +717,22 @@ private static void applyColumnDefaultValue(
         capabilities.columnDefaultValue().unsupportedMessage() + " Illegal 
column: " + columnName);
   }
 
+  /**
+   * Rejects setting {@code initialDefaultValue} on create/add-column. 
Catalogs that support write
+   * defaults (e.g. MySQL) still do not apply Iceberg-style initial defaults, 
so reusing {@link
+   * Capability#columnDefaultValue()} would accept a value the catalog 
silently drops. Until a
+   * catalog can write initial defaults, require the unset sentinel.
+   */
+  private static void applyColumnInitialDefaultValue(
+      String columnName, Expression initialDefaultValue) {
+    Preconditions.checkArgument(
+        INITIAL_DEFAULT_VALUE_NOT_SET.equals(initialDefaultValue),

Review Comment:
   The check fails for null, since the sentinel's equals(null) is false. Any 
third-party Column that returns null here, or a caller that copies a loaded 
Iceberg v3 column into create/add-column, will now be rejected. Treat null as 
unset and add a test for that path.



##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ConvertUtil.java:
##########
@@ -76,9 +89,112 @@ public static IcebergColumn 
fromNestedField(Types.NestedField nestedField) {
         .withNullable(nestedField.isOptional())
         .withComment(nestedField.doc())
         
.withType(IcebergDataTypeConverter.CONVERTER.toGravitino(nestedField.type()))
+        .withDefaultValue(
+            fromIcebergLiteral(
+                nestedField.writeDefaultLiteral(),
+                nestedField.type(),
+                Column.DEFAULT_VALUE_NOT_SET))
+        .withInitialDefaultValue(
+            fromIcebergLiteral(
+                nestedField.initialDefaultLiteral(),
+                nestedField.type(),
+                Column.INITIAL_DEFAULT_VALUE_NOT_SET))
         .build();
   }
 
+  private static Expression fromIcebergLiteral(
+      Literal<?> icebergDefault, Type type, Expression unset) {
+    if (icebergDefault == null) {
+      return unset;
+    }
+    return toGravitinoLiteral(icebergDefault, type, unset);
+  }
+
+  private static Expression toGravitinoLiteral(
+      Literal<?> icebergDefault, Type type, Expression unset) {
+    Object value = icebergDefault.value();
+    if (value == null) {
+      return Literals.NULL;
+    }
+
+    switch (type.typeId()) {
+      case BOOLEAN:
+        return Literals.booleanLiteral((Boolean) value);
+      case INTEGER:
+        return Literals.integerLiteral((Integer) value);
+      case LONG:
+        return Literals.longLiteral((Long) value);
+      case FLOAT:
+        return Literals.floatLiteral((Float) value);
+      case DOUBLE:
+        return Literals.doubleLiteral((Double) value);
+      case STRING:
+        return Literals.stringLiteral(value.toString());
+      case DATE:
+        return Literals.dateLiteral(DateTimeUtil.dateFromDays((Integer) 
value));
+      case TIME:
+        return Literals.timeLiteral(DateTimeUtil.timeFromMicros((Long) value));
+      case TIMESTAMP:
+        {
+          Types.TimestampType timestampType = (Types.TimestampType) type;
+          if (timestampType.shouldAdjustToUTC()) {
+            // Iceberg timestamptz is a UTC instant (micros). Expose it as UTC 
LocalDateTime while
+            // keeping TimestampType.withTimeZone so the type still says "with 
time zone".
+            return Literals.of(
+                DateTimeUtil.timestamptzFromMicros((Long) 
value).toLocalDateTime(),
+                
org.apache.gravitino.rel.types.Types.TimestampType.withTimeZone(
+                    FromIcebergType.PRECISION_MICROSECOND));
+          }
+          return Literals.of(
+              DateTimeUtil.timestampFromMicros((Long) value),
+              
org.apache.gravitino.rel.types.Types.TimestampType.withoutTimeZone(
+                  FromIcebergType.PRECISION_MICROSECOND));
+        }
+      case TIMESTAMP_NANO:
+        {
+          Types.TimestampNanoType timestampNanoType = 
(Types.TimestampNanoType) type;
+          if (timestampNanoType.shouldAdjustToUTC()) {
+            // Same UTC-instant → LocalDateTime convention as TIMESTAMP with 
zone above.
+            return Literals.of(
+                DateTimeUtil.timestamptzFromNanos((Long) 
value).toLocalDateTime(),
+                
org.apache.gravitino.rel.types.Types.TimestampType.withTimeZone(
+                    FromIcebergType.PRECISION_NANOSECOND));
+          }
+          return Literals.of(
+              DateTimeUtil.timestampFromNanos((Long) value),
+              
org.apache.gravitino.rel.types.Types.TimestampType.withoutTimeZone(
+                  FromIcebergType.PRECISION_NANOSECOND));
+        }
+      case DECIMAL:
+        {
+          Types.DecimalType decimalType = (Types.DecimalType) type;
+          return Literals.decimalLiteral(
+              Decimal.of((BigDecimal) value, decimalType.precision(), 
decimalType.scale()));
+        }
+      case UUID:
+        return Literals.of(
+            ((UUID) value).toString(), 
org.apache.gravitino.rel.types.Types.UUIDType.get());
+      case FIXED:
+        return Literals.of(
+            toByteArray(value),
+            
org.apache.gravitino.rel.types.Types.FixedType.of(((Types.FixedType) 
type).length()));
+      case BINARY:
+        return Literals.of(
+            toByteArray(value), 
org.apache.gravitino.rel.types.Types.BinaryType.get());
+      default:
+        // Nested types cannot carry non-null Iceberg defaults; other V3 types 
are left unset
+        // rather than failing table load.
+        return unset;

Review Comment:
   A default of an unsupported type silently becomes unset, so users cannot 
tell it was dropped. Consider logging a warning here.



##########
maintenance/optimizer/build.gradle.kts:
##########
@@ -26,9 +26,11 @@ plugins {
 
 val scalaVersion: String =
   project.properties["scalaVersion"] as? String ?: 
extra["defaultScalaVersion"].toString()
-val sparkVersion: String = libs.versions.spark33.get()
+// Match maintenance/jobs and catalog Iceberg 1.11 so IsolatedClassLoader's 
shared
+// org.apache.iceberg.* types include NestedField default APIs used by 
lakehouse-iceberg.
+val sparkVersion: String = libs.versions.spark35.get()

Review Comment:
   Why does this fix switch the optimizer module from Spark 3.3 to Spark 3.5 
and the matching Iceberg version? Could this change the supported Spark 
versions of that module, and should it be a separate PR?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to