lasdf1234 commented on code in PR #14659:
URL: https://github.com/apache/gravitino/pull/14659#discussion_r4512521573
##########
core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java:
##########
@@ -714,6 +717,22 @@ private static void applyColumnDefaultValue(
capabilities.columnDefaultValue().unsupportedMessage() + " Illegal
column: " + columnName);
}
+ /**
+ * Rejects setting {@code initialDefaultValue} on create/add-column.
Catalogs that support write
+ * defaults (e.g. MySQL) still do not apply Iceberg-style initial defaults,
so reusing {@link
+ * Capability#columnDefaultValue()} would accept a value the catalog
silently drops. Until a
+ * catalog can write initial defaults, require the unset sentinel.
+ */
+ private static void applyColumnInitialDefaultValue(
+ String columnName, Expression initialDefaultValue) {
+ Preconditions.checkArgument(
+ INITIAL_DEFAULT_VALUE_NOT_SET.equals(initialDefaultValue),
Review Comment:
Fixed in `7d1ef0a46`: `null` is now treated as unset alongside
`INITIAL_DEFAULT_VALUE_NOT_SET`. Added
`testApplyCapabilitiesTreatsNullInitialDefaultAsUnset` for a third-party
`Column` that returns `null`.
##########
maintenance/optimizer/build.gradle.kts:
##########
@@ -26,9 +26,11 @@ plugins {
val scalaVersion: String =
project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
-val sparkVersion: String = libs.versions.spark33.get()
+// Match maintenance/jobs and catalog Iceberg 1.11 so IsolatedClassLoader's
shared
+// org.apache.iceberg.* types include NestedField default APIs used by
lakehouse-iceberg.
+val sparkVersion: String = libs.versions.spark35.get()
Review Comment:
Agreed — the optimizer Spark 3.5 / Iceberg 1.11 bump is unrelated to column
defaults, so it went in a separate PR:
https://github.com/apache/gravitino/pull/14782 (already merged as #14783). This
PR no longer needs to carry that change on its own.
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ConvertUtil.java:
##########
@@ -76,9 +89,112 @@ public static IcebergColumn
fromNestedField(Types.NestedField nestedField) {
.withNullable(nestedField.isOptional())
.withComment(nestedField.doc())
.withType(IcebergDataTypeConverter.CONVERTER.toGravitino(nestedField.type()))
+ .withDefaultValue(
+ fromIcebergLiteral(
+ nestedField.writeDefaultLiteral(),
+ nestedField.type(),
+ Column.DEFAULT_VALUE_NOT_SET))
+ .withInitialDefaultValue(
+ fromIcebergLiteral(
+ nestedField.initialDefaultLiteral(),
+ nestedField.type(),
+ Column.INITIAL_DEFAULT_VALUE_NOT_SET))
.build();
}
+ private static Expression fromIcebergLiteral(
+ Literal<?> icebergDefault, Type type, Expression unset) {
+ if (icebergDefault == null) {
+ return unset;
+ }
+ return toGravitinoLiteral(icebergDefault, type, unset);
+ }
+
+ private static Expression toGravitinoLiteral(
+ Literal<?> icebergDefault, Type type, Expression unset) {
+ Object value = icebergDefault.value();
+ if (value == null) {
+ return Literals.NULL;
+ }
+
+ switch (type.typeId()) {
+ case BOOLEAN:
+ return Literals.booleanLiteral((Boolean) value);
+ case INTEGER:
+ return Literals.integerLiteral((Integer) value);
+ case LONG:
+ return Literals.longLiteral((Long) value);
+ case FLOAT:
+ return Literals.floatLiteral((Float) value);
+ case DOUBLE:
+ return Literals.doubleLiteral((Double) value);
+ case STRING:
+ return Literals.stringLiteral(value.toString());
+ case DATE:
+ return Literals.dateLiteral(DateTimeUtil.dateFromDays((Integer)
value));
+ case TIME:
+ return Literals.timeLiteral(DateTimeUtil.timeFromMicros((Long) value));
+ case TIMESTAMP:
+ {
+ Types.TimestampType timestampType = (Types.TimestampType) type;
+ if (timestampType.shouldAdjustToUTC()) {
+ // Iceberg timestamptz is a UTC instant (micros). Expose it as UTC
LocalDateTime while
+ // keeping TimestampType.withTimeZone so the type still says "with
time zone".
+ return Literals.of(
+ DateTimeUtil.timestamptzFromMicros((Long)
value).toLocalDateTime(),
+
org.apache.gravitino.rel.types.Types.TimestampType.withTimeZone(
+ FromIcebergType.PRECISION_MICROSECOND));
+ }
+ return Literals.of(
+ DateTimeUtil.timestampFromMicros((Long) value),
+
org.apache.gravitino.rel.types.Types.TimestampType.withoutTimeZone(
+ FromIcebergType.PRECISION_MICROSECOND));
+ }
+ case TIMESTAMP_NANO:
+ {
+ Types.TimestampNanoType timestampNanoType =
(Types.TimestampNanoType) type;
+ if (timestampNanoType.shouldAdjustToUTC()) {
+ // Same UTC-instant → LocalDateTime convention as TIMESTAMP with
zone above.
+ return Literals.of(
+ DateTimeUtil.timestamptzFromNanos((Long)
value).toLocalDateTime(),
+
org.apache.gravitino.rel.types.Types.TimestampType.withTimeZone(
+ FromIcebergType.PRECISION_NANOSECOND));
+ }
+ return Literals.of(
+ DateTimeUtil.timestampFromNanos((Long) value),
+
org.apache.gravitino.rel.types.Types.TimestampType.withoutTimeZone(
+ FromIcebergType.PRECISION_NANOSECOND));
+ }
+ case DECIMAL:
+ {
+ Types.DecimalType decimalType = (Types.DecimalType) type;
+ return Literals.decimalLiteral(
+ Decimal.of((BigDecimal) value, decimalType.precision(),
decimalType.scale()));
+ }
+ case UUID:
+ return Literals.of(
+ ((UUID) value).toString(),
org.apache.gravitino.rel.types.Types.UUIDType.get());
+ case FIXED:
+ return Literals.of(
+ toByteArray(value),
+
org.apache.gravitino.rel.types.Types.FixedType.of(((Types.FixedType)
type).length()));
+ case BINARY:
+ return Literals.of(
+ toByteArray(value),
org.apache.gravitino.rel.types.Types.BinaryType.get());
+ default:
+ // Nested types cannot carry non-null Iceberg defaults; other V3 types
are left unset
+ // rather than failing table load.
+ return unset;
Review Comment:
Fixed in `7d1ef0a46`: unsupported Iceberg default types now `LOG.warn` with
the type id before falling back to unset, so table load still succeeds but the
drop is visible in the server log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]