lasdf1234 commented on code in PR #14659:
URL: https://github.com/apache/gravitino/pull/14659#discussion_r4492614764


##########
maintenance/optimizer/build.gradle.kts:
##########
@@ -26,9 +26,11 @@ plugins {
 
 val scalaVersion: String =
   project.properties["scalaVersion"] as? String ?: 
extra["defaultScalaVersion"].toString()
-val sparkVersion: String = libs.versions.spark33.get()
+// Match maintenance/jobs and catalog Iceberg 1.11 so IsolatedClassLoader's 
shared
+// org.apache.iceberg.* types include NestedField default APIs used by 
lakehouse-iceberg.
+val sparkVersion: String = libs.versions.spark35.get()

Review Comment:
   Follow-up: this explanation still stands on the latest head (`493c7edc3`, 
after merging `main`). The optimizer Spark 3.5 / Iceberg 1.11 bump remains in 
the PR for the `writeDefaultLiteral` / IsolatedClassLoader reason above.



##########
core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java:
##########
@@ -638,6 +641,19 @@ private static void applyColumnDefaultValue(
         capabilities.columnDefaultValue().unsupportedMessage() + " Illegal 
column: " + columnName);
   }
 
+  /**
+   * Rejects setting {@code initialDefaultValue} on create/add-column. 
Catalogs that support write
+   * defaults (e.g. MySQL) still do not apply Iceberg-style initial defaults, 
so reusing {@link
+   * Capability#columnDefaultValue()} would accept a value the catalog 
silently drops. Until a
+   * catalog can write initial defaults, require the unset sentinel.
+   */
+  private static void applyColumnInitialDefaultValue(
+      String columnName, Expression initialDefaultValue) {
+    Preconditions.checkArgument(

Review Comment:
   Follow-up: `testLoadThenCreateRejectsColumnsCarryingInitialDefault` is on 
the latest head (`493c7edc3`). Create/add-column still rejects a set 
`initialDefaultValue`; the test documents the load-then-create failure.



##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ConvertUtil.java:
##########
@@ -76,9 +89,109 @@ public static IcebergColumn 
fromNestedField(Types.NestedField nestedField) {
         .withNullable(nestedField.isOptional())
         .withComment(nestedField.doc())
         
.withType(IcebergDataTypeConverter.CONVERTER.toGravitino(nestedField.type()))
+        .withDefaultValue(
+            fromIcebergLiteral(
+                nestedField.writeDefaultLiteral(),
+                nestedField.type(),
+                Column.DEFAULT_VALUE_NOT_SET))
+        .withInitialDefaultValue(
+            fromIcebergLiteral(
+                nestedField.initialDefaultLiteral(),
+                nestedField.type(),
+                Column.INITIAL_DEFAULT_VALUE_NOT_SET))
         .build();
   }
 
+  private static Expression fromIcebergLiteral(
+      Literal<?> icebergDefault, Type type, Expression unset) {
+    if (icebergDefault == null) {
+      return unset;
+    }
+    return toGravitinoLiteral(icebergDefault, type, unset);
+  }
+
+  private static Expression toGravitinoLiteral(
+      Literal<?> icebergDefault, Type type, Expression unset) {
+    Object value = icebergDefault.value();
+    if (value == null) {
+      return Literals.NULL;
+    }
+
+    switch (type.typeId()) {
+      case BOOLEAN:
+        return Literals.booleanLiteral((Boolean) value);
+      case INTEGER:
+        return Literals.integerLiteral((Integer) value);
+      case LONG:
+        return Literals.longLiteral((Long) value);
+      case FLOAT:
+        return Literals.floatLiteral((Float) value);
+      case DOUBLE:
+        return Literals.doubleLiteral((Double) value);
+      case STRING:
+        return Literals.stringLiteral(value.toString());
+      case DATE:
+        return Literals.dateLiteral(DateTimeUtil.dateFromDays((Integer) 
value));
+      case TIME:
+        return Literals.timeLiteral(DateTimeUtil.timeFromMicros((Long) value));
+      case TIMESTAMP:
+        {
+          Types.TimestampType timestampType = (Types.TimestampType) type;
+          if (timestampType.shouldAdjustToUTC()) {
+            return Literals.of(
+                DateTimeUtil.timestamptzFromMicros((Long) 
value).toLocalDateTime(),

Review Comment:
   Follow-up: the UTC `LocalDateTime` + `withTimeZone` mapping, comment, and 
`testFromNestedFieldTimestamptzDefaultUsesUtcLocalDateTime` are on the latest 
head (`493c7edc3`).



##########
common/src/main/java/org/apache/gravitino/dto/util/DTOConverters.java:
##########
@@ -1651,4 +1673,46 @@ public static PolicyContent fromDTO(PolicyContentDTO 
policyContentDTO) {
     throw new IllegalArgumentException(
         "Unsupported policy content type: " + policyContentDTO.getClass());
   }
+
+  private static boolean isBinaryOrFixed(Type dataType) {
+    return dataType instanceof Types.BinaryType || dataType instanceof 
Types.FixedType;
+  }
+
+  private static String literalValueToDtoString(Object value, Type dataType) {
+    // byte[].toString() is an object identity and cannot preserve the 
literal's bytes.
+    if (isBinaryOrFixed(dataType) && value instanceof byte[]) {
+      return BINARY_HEX_PREFIX + BINARY_HEX.encode((byte[]) value);
+    }
+    return value.toString();
+  }
+
+  /**
+   * Decodes {@code 0x}-prefixed hex for binary/fixed. Unprefixed values keep 
their previous string
+   * interpretation (#14609).
+   */
+  private static Object literalValueFromDtoString(String value, Type dataType) 
{
+    if (!isBinaryOrFixed(dataType) || value == null) {
+      return value;

Review Comment:
   Follow-up: unprefixed binary/fixed values are rejected (Java + Python) on 
the latest head (`493c7edc3`), with unit coverage for the rejection path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to