jerryshao commented on code in PR #14659:
URL: https://github.com/apache/gravitino/pull/14659#discussion_r4509190121
##########
core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java:
##########
@@ -714,6 +717,19 @@ private static void applyColumnDefaultValue(
capabilities.columnDefaultValue().unsupportedMessage() + " Illegal
column: " + columnName);
}
+ /**
+ * Rejects setting {@code initialDefaultValue} on create/add-column.
Catalogs that support write
+ * defaults (e.g. MySQL) still do not apply Iceberg-style initial defaults,
so reusing {@link
+ * Capability#columnDefaultValue()} would accept a value the catalog
silently drops. Until a
+ * catalog can write initial defaults, require the unset sentinel.
+ */
+ private static void applyColumnInitialDefaultValue(
+ String columnName, Expression initialDefaultValue) {
+ Preconditions.checkArgument(
+ INITIAL_DEFAULT_VALUE_NOT_SET.equals(initialDefaultValue),
+ "Setting column initial default value is not supported. Illegal
column: " + columnName);
Review Comment:
The error message is only visible to the caller and does not say which
catalogs would support it. Consider naming the catalog capability so users know
the value is read-only for now.
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ConvertUtil.java:
##########
@@ -76,9 +89,112 @@ public static IcebergColumn
fromNestedField(Types.NestedField nestedField) {
.withNullable(nestedField.isOptional())
.withComment(nestedField.doc())
.withType(IcebergDataTypeConverter.CONVERTER.toGravitino(nestedField.type()))
+ .withDefaultValue(
+ fromIcebergLiteral(
+ nestedField.writeDefaultLiteral(),
+ nestedField.type(),
+ Column.DEFAULT_VALUE_NOT_SET))
+ .withInitialDefaultValue(
+ fromIcebergLiteral(
+ nestedField.initialDefaultLiteral(),
+ nestedField.type(),
+ Column.INITIAL_DEFAULT_VALUE_NOT_SET))
.build();
}
+ private static Expression fromIcebergLiteral(
+ Literal<?> icebergDefault, Type type, Expression unset) {
+ if (icebergDefault == null) {
+ return unset;
+ }
+ return toGravitinoLiteral(icebergDefault, type, unset);
+ }
+
+ private static Expression toGravitinoLiteral(
+ Literal<?> icebergDefault, Type type, Expression unset) {
+ Object value = icebergDefault.value();
+ if (value == null) {
+ return Literals.NULL;
+ }
+
+ switch (type.typeId()) {
+ case BOOLEAN:
+ return Literals.booleanLiteral((Boolean) value);
+ case INTEGER:
+ return Literals.integerLiteral((Integer) value);
+ case LONG:
+ return Literals.longLiteral((Long) value);
+ case FLOAT:
+ return Literals.floatLiteral((Float) value);
+ case DOUBLE:
+ return Literals.doubleLiteral((Double) value);
+ case STRING:
+ return Literals.stringLiteral(value.toString());
+ case DATE:
+ return Literals.dateLiteral(DateTimeUtil.dateFromDays((Integer)
value));
+ case TIME:
+ return Literals.timeLiteral(DateTimeUtil.timeFromMicros((Long) value));
+ case TIMESTAMP:
+ {
+ Types.TimestampType timestampType = (Types.TimestampType) type;
+ if (timestampType.shouldAdjustToUTC()) {
+ // Iceberg timestamptz is a UTC instant (micros). Expose it as UTC
LocalDateTime while
+ // keeping TimestampType.withTimeZone so the type still says "with
time zone".
+ return Literals.of(
+ DateTimeUtil.timestamptzFromMicros((Long)
value).toLocalDateTime(),
Review Comment:
timestamptz is converted to a UTC LocalDateTime but typed as with-time-zone,
so the zone is lost and clients may read it in local time. Is this the intended
wire convention, and is it tested end to end through the DTO?
##########
core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java:
##########
@@ -705,6 +707,7 @@ private static void applyColumnNotNull(
private static void applyColumnDefaultValue(Column column, Capability
capabilities) {
applyColumnDefaultValue(column.name(), column.defaultValue(),
capabilities);
+ applyColumnInitialDefaultValue(column.name(),
column.initialDefaultValue());
Review Comment:
Any column loaded from an Iceberg v3 table with an initial default now fails
here when reused in createTable or addColumn, since this check rejects every
non-unset value. Clients that copy loaded columns will break; consider ignoring
the field for catalogs that do not support it, or add a test for this round
trip.
##########
maintenance/optimizer/build.gradle.kts:
##########
@@ -26,9 +26,11 @@ plugins {
val scalaVersion: String =
project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
-val sparkVersion: String = libs.versions.spark33.get()
+// Match maintenance/jobs and catalog Iceberg 1.11 so IsolatedClassLoader's
shared
+// org.apache.iceberg.* types include NestedField default APIs used by
lakehouse-iceberg.
+val sparkVersion: String = libs.versions.spark35.get()
Review Comment:
Switching the optimizer module from Spark 3.3 to Spark 3.5 looks unrelated
to column defaults and can affect users of that module. Can this go in a
separate PR, or can you explain why it is required here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]