lasdf1234 commented on code in PR #14659:
URL: https://github.com/apache/gravitino/pull/14659#discussion_r4492522738
##########
maintenance/optimizer/build.gradle.kts:
##########
@@ -26,9 +26,11 @@ plugins {
val scalaVersion: String =
project.properties["scalaVersion"] as? String ?:
extra["defaultScalaVersion"].toString()
-val sparkVersion: String = libs.versions.spark33.get()
+// Match maintenance/jobs and catalog Iceberg 1.11 so IsolatedClassLoader's
shared
+// org.apache.iceberg.* types include NestedField default APIs used by
lakehouse-iceberg.
+val sparkVersion: String = libs.versions.spark35.get()
Review Comment:
Thanks for the review.
The Spark 3.3 → 3.5 / Iceberg 1.11 bump in `maintenance/optimizer` is
required for this read path, not an unrelated cleanup:
`ConvertUtil.fromNestedField` now calls Iceberg
`NestedField.writeDefaultLiteral()` / `initialDefaultLiteral()`. Optimizer
integration tests load lakehouse-iceberg through `IsolatedClassLoader` with
shared `org.apache.iceberg.*` types, so an older `iceberg-spark-runtime-3.3`
(1.8.1) on the optimizer classpath causes `NoSuchMethodError` even for tables
without defaults (caught by mchades earlier; Maintenance CI showed the same).
Keeping the bump in this PR keeps the runtime API aligned with
`maintenance/jobs` and `catalog-lakehouse-iceberg`. Happy to split a follow-up
if we want the version change reviewed separately once this lands.
##########
core/src/main/java/org/apache/gravitino/catalog/CapabilityHelpers.java:
##########
@@ -638,6 +641,19 @@ private static void applyColumnDefaultValue(
capabilities.columnDefaultValue().unsupportedMessage() + " Illegal
column: " + columnName);
}
+ /**
+ * Rejects setting {@code initialDefaultValue} on create/add-column.
Catalogs that support write
+ * defaults (e.g. MySQL) still do not apply Iceberg-style initial defaults,
so reusing {@link
+ * Capability#columnDefaultValue()} would accept a value the catalog
silently drops. Until a
+ * catalog can write initial defaults, require the unset sentinel.
+ */
+ private static void applyColumnInitialDefaultValue(
+ String columnName, Expression initialDefaultValue) {
+ Preconditions.checkArgument(
Review Comment:
Agreed this is a real load-then-create / copy-table risk with the read-only
scope.
I kept the reject behavior (option A) and added
`testLoadThenCreateRejectsColumnsCarryingInitialDefault` in
`TestCapabilityHelpers` to document that columns loaded with Iceberg v3
`initialDefaultValue` set fail `applyCapabilities` on create/add-column, rather
than silently dropping the value.
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ConvertUtil.java:
##########
@@ -76,9 +89,109 @@ public static IcebergColumn
fromNestedField(Types.NestedField nestedField) {
.withNullable(nestedField.isOptional())
.withComment(nestedField.doc())
.withType(IcebergDataTypeConverter.CONVERTER.toGravitino(nestedField.type()))
+ .withDefaultValue(
+ fromIcebergLiteral(
+ nestedField.writeDefaultLiteral(),
+ nestedField.type(),
+ Column.DEFAULT_VALUE_NOT_SET))
+ .withInitialDefaultValue(
+ fromIcebergLiteral(
+ nestedField.initialDefaultLiteral(),
+ nestedField.type(),
+ Column.INITIAL_DEFAULT_VALUE_NOT_SET))
.build();
}
+ private static Expression fromIcebergLiteral(
+ Literal<?> icebergDefault, Type type, Expression unset) {
+ if (icebergDefault == null) {
+ return unset;
+ }
+ return toGravitinoLiteral(icebergDefault, type, unset);
+ }
+
+ private static Expression toGravitinoLiteral(
+ Literal<?> icebergDefault, Type type, Expression unset) {
+ Object value = icebergDefault.value();
+ if (value == null) {
+ return Literals.NULL;
+ }
+
+ switch (type.typeId()) {
+ case BOOLEAN:
+ return Literals.booleanLiteral((Boolean) value);
+ case INTEGER:
+ return Literals.integerLiteral((Integer) value);
+ case LONG:
+ return Literals.longLiteral((Long) value);
+ case FLOAT:
+ return Literals.floatLiteral((Float) value);
+ case DOUBLE:
+ return Literals.doubleLiteral((Double) value);
+ case STRING:
+ return Literals.stringLiteral(value.toString());
+ case DATE:
+ return Literals.dateLiteral(DateTimeUtil.dateFromDays((Integer)
value));
+ case TIME:
+ return Literals.timeLiteral(DateTimeUtil.timeFromMicros((Long) value));
+ case TIMESTAMP:
+ {
+ Types.TimestampType timestampType = (Types.TimestampType) type;
+ if (timestampType.shouldAdjustToUTC()) {
+ return Literals.of(
+ DateTimeUtil.timestamptzFromMicros((Long)
value).toLocalDateTime(),
Review Comment:
Yes, intentional for the Iceberg path.
Iceberg `timestamptz` is a UTC instant (micros/nanos). We keep
`TimestampType.withTimeZone` on the type and materialize the value as that
instant’s UTC `LocalDateTime` (offset is not retained on the literal). Added a
short comment in `ConvertUtil` and
`testFromNestedFieldTimestamptzDefaultUsesUtcLocalDateTime` to lock that
mapping. Aligning with JDBC’s `OffsetDateTime` convention can be a follow-up if
we want one literal representation everywhere.
##########
common/src/main/java/org/apache/gravitino/dto/util/DTOConverters.java:
##########
@@ -1651,4 +1673,46 @@ public static PolicyContent fromDTO(PolicyContentDTO
policyContentDTO) {
throw new IllegalArgumentException(
"Unsupported policy content type: " + policyContentDTO.getClass());
}
+
+ private static boolean isBinaryOrFixed(Type dataType) {
+ return dataType instanceof Types.BinaryType || dataType instanceof
Types.FixedType;
+ }
+
+ private static String literalValueToDtoString(Object value, Type dataType) {
+ // byte[].toString() is an object identity and cannot preserve the
literal's bytes.
+ if (isBinaryOrFixed(dataType) && value instanceof byte[]) {
+ return BINARY_HEX_PREFIX + BINARY_HEX.encode((byte[]) value);
+ }
+ return value.toString();
+ }
+
+ /**
+ * Decodes {@code 0x}-prefixed hex for binary/fixed. Unprefixed values keep
their previous string
+ * interpretation (#14609).
+ */
+ private static Object literalValueFromDtoString(String value, Type dataType)
{
+ if (!isBinaryOrFixed(dataType) || value == null) {
+ return value;
Review Comment:
Agreed — silent String pass-through for Binary/Fixed is unsafe now that
Iceberg defaults also ride this codec.
`literalValueFromDtoString` (and the Python counterpart) now reject
unprefixed values with a clear `0x`+hex error. Added Java/Python unit coverage
for the rejection path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]