Copilot commented on code in PR #14659:
URL: https://github.com/apache/gravitino/pull/14659#discussion_r4480870733
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalogCapability.java:
##########
@@ -37,9 +37,9 @@ public IcebergCatalogCapability(String schemaSeparator) {
@Override
public CapabilityResult columnDefaultValue() {
- // Iceberg column default value is WIP, see
- // https://github.com/apache/iceberg/pull/4525
- return CapabilityResult.unsupported("Iceberg does not support column
default value.");
+ // Format-version 3 column defaults: write-default maps to
Column.defaultValue() and
+ // initial-default maps to Column.initialDefaultValue().
+ return CapabilityResult.SUPPORTED;
Review Comment:
The PR description states that creating/altering column defaults via the
Gravitino API remains unsupported
(`IcebergCatalogCapability.columnDefaultValue()` unchanged), but this change
flips Iceberg to `SUPPORTED` and the PR also adds/updates tests for add/update
default behavior. Please update the PR description (or revert the
capability/behavior change) so it accurately reflects the new supported
behavior.
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ConvertUtil.java:
##########
@@ -97,5 +106,28 @@ private static
org.apache.gravitino.rel.types.Types.StructType toGravitinoStruct
return org.apache.gravitino.rel.types.Types.StructType.of(fields);
}
+ private static Schema applyColumnDefaults(
+ List<Types.NestedField> icebergFields, Column[] columns) {
+ List<Types.NestedField> withDefaults = new
ArrayList<>(icebergFields.size());
+ for (int i = 0; i < icebergFields.size(); i++) {
+ Types.NestedField field = icebergFields.get(i);
+ Literal<?> write =
+ ToIcebergColumnDefault.fromGravitino(columns[i].defaultValue(),
field.type());
+ Literal<?> initial =
+
ToIcebergColumnDefault.fromGravitino(columns[i].initialDefaultValue(),
field.type());
+ if (write == null && initial == null) {
+ withDefaults.add(field);
+ } else {
+ // Seed both when only one side is set (Iceberg addColumn behavior for
new fields).
+ withDefaults.add(
+ Types.NestedField.from(field)
+ .withWriteDefault(write != null ? write : initial)
+ .withInitialDefault(initial != null ? initial : write)
+ .build());
Review Comment:
This forces write-default and initial-default to be identical whenever only
one of `defaultValue`/`initialDefaultValue` is provided. That makes it
impossible to represent cases where users intentionally want only one side set
(which is the main reason to expose two separate fields). Consider only setting
the defaults that are explicitly provided, and reserve 'seed both from one
value' behavior for the Iceberg `addColumn` path where the underlying API
requires it.
##########
clients/client-python/gravitino/api/rel/column.py:
##########
@@ -224,5 +261,10 @@ def __hash__(self):
if self._default_value is Column.DEFAULT_VALUE_NOT_SET
else self._default_value
),
+ (
+ tuple(self._initial_default_value)
+ if self._initial_default_value is
Column.INITIAL_DEFAULT_VALUE_NOT_SET
+ else self._initial_default_value
+ ),
Review Comment:
This `__hash__` implementation will raise at runtime when defaults are
unset: it attempts `tuple(self._initial_default_value)` (and similarly for
`_default_value`) exactly when the value is the NOT_SET sentinel, which is not
an iterable. The hash normalization should treat NOT_SET as a stable scalar
(e.g., `None` or the sentinel itself) and only convert to `tuple(...)` when the
value is actually a list.
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/ops/IcebergCatalogWrapperHelper.java:
##########
@@ -170,22 +176,42 @@ private void doAddColumn(UpdateSchema
icebergUpdateSchema, AddColumn addColumn)
getParentName(addColumn.fieldName()),
getLeafName(addColumn.fieldName()),
IcebergDataTypeConverter.CONVERTER.fromGravitino(addColumn.getDataType()),
- addColumn.getComment());
+ addColumn.getComment(),
+ ToIcebergColumnDefault.fromGravitino(
+ addColumn.getDefaultValue(),
+
IcebergDataTypeConverter.CONVERTER.fromGravitino(addColumn.getDataType())));
} else {
// TODO: figure out how to enable users to add required columns
// icebergUpdateSchema.allowIncompatibleChanges();
icebergUpdateSchema.addRequiredColumn(
getParentName(addColumn.fieldName()),
getLeafName(addColumn.fieldName()),
IcebergDataTypeConverter.CONVERTER.fromGravitino(addColumn.getDataType()),
- addColumn.getComment());
+ addColumn.getComment(),
+ ToIcebergColumnDefault.fromGravitino(
+ addColumn.getDefaultValue(),
+
IcebergDataTypeConverter.CONVERTER.fromGravitino(addColumn.getDataType())));
}
if (!ColumnPosition.defaultPos().equals(addColumn.getPosition())) {
doMoveColumn(icebergUpdateSchema, addColumn.fieldName(),
addColumn.getPosition());
}
}
+ private void doUpdateColumnDefaultValue(
+ UpdateSchema icebergUpdateSchema,
+ UpdateColumnDefaultValue updateColumnDefaultValue,
+ Schema icebergTableSchema) {
+ String fieldName = DOT.join(updateColumnDefaultValue.fieldName());
+ NestedField field = icebergTableSchema.findField(fieldName);
+ Preconditions.checkArgument(field != null, "Cannot update missing column:
%s", fieldName);
+ // Iceberg updateColumnDefault updates write-default only; initial-default
is fixed at add time.
+ Literal<?> newDefault =
+ ToIcebergColumnDefault.fromGravitino(
+ updateColumnDefaultValue.getNewDefaultValue(), field.type());
Review Comment:
`ToIcebergColumnDefault.fromGravitino(...)` can return `null` (unset /
NULL), but this code passes the result directly into `updateColumnDefault`. If
Iceberg does not accept a null literal here, this will fail at runtime. If the
intent is to support removing the default, handle that explicitly (e.g., call
the appropriate Iceberg API to delete the default) or reject null/unset values
for `UpdateColumnDefaultValue` with a clear error.
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/converter/ConvertUtil.java:
##########
@@ -97,5 +106,28 @@ private static
org.apache.gravitino.rel.types.Types.StructType toGravitinoStruct
return org.apache.gravitino.rel.types.Types.StructType.of(fields);
}
+ private static Schema applyColumnDefaults(
+ List<Types.NestedField> icebergFields, Column[] columns) {
+ List<Types.NestedField> withDefaults = new
ArrayList<>(icebergFields.size());
+ for (int i = 0; i < icebergFields.size(); i++) {
+ Types.NestedField field = icebergFields.get(i);
+ Literal<?> write =
+ ToIcebergColumnDefault.fromGravitino(columns[i].defaultValue(),
field.type());
+ Literal<?> initial =
+
ToIcebergColumnDefault.fromGravitino(columns[i].initialDefaultValue(),
field.type());
Review Comment:
This loop indexes `columns[i]` based on `icebergFields.size()` without
validating that `columns.length` matches `icebergFields.size()`. If they ever
diverge (e.g., due to projection/reordering changes in type conversion), this
will throw `ArrayIndexOutOfBoundsException` or mis-apply defaults to the wrong
field. Add an explicit precondition that sizes match (and ideally fail with a
descriptive message).
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/ops/IcebergCatalogWrapperHelper.java:
##########
@@ -271,6 +300,27 @@ public IcebergTableChange buildIcebergTableChanges(
IcebergTableChange icebergTableChange =
new IcebergTableChange(icebergTableIdentifier, transaction);
if (!gravitinoColumnChanges.isEmpty()) {
+ boolean setsColumnDefault =
+ gravitinoColumnChanges.stream()
+ .anyMatch(
+ change -> {
+ if (change instanceof UpdateColumnDefaultValue) {
+ return true;
+ }
+ if (change instanceof AddColumn) {
+ Expression defaultValue = ((AddColumn)
change).getDefaultValue();
+ return defaultValue != null
+ &&
!defaultValue.equals(Column.DEFAULT_VALUE_NOT_SET);
Review Comment:
The format-version guard is triggered for any `AddColumn` whose
`defaultValue` is not `DEFAULT_VALUE_NOT_SET`, but the actual converter
(`ToIcebergColumnDefault.fromGravitino`) treats multiple values as effectively
'no default' (e.g., `Literals.NULL`, `INITIAL_DEFAULT_VALUE_NOT_SET`). This can
cause false positives where the code requires format-version 3 even though no
Iceberg default will be applied. Consider reusing the same predicate as the
converter (or calling the converter here and checking whether it returns a
non-null Iceberg `Literal`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]