mchades commented on code in PR #14609:
URL: https://github.com/apache/gravitino/pull/14609#discussion_r4467431435
##########
spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTableChangeConverter.java:
##########
@@ -93,13 +106,49 @@ public org.apache.gravitino.rel.TableChange
toGravitinoTableChange(TableChange c
(TableChange.UpdateColumnDefaultValue) change;
return org.apache.gravitino.rel.TableChange.updateColumnDefaultValue(
updateColumnDefaultValue.fieldNames(),
- Literals.stringLiteral(updateColumnDefaultValue.newDefaultValue()));
+ convertDefaultValue(updateColumnDefaultValue.newDefaultValue()));
} else {
throw new UnsupportedOperationException(
String.format("Unsupported table change %s",
change.getClass().getName()));
}
}
+ private Expression convertDefaultValue(String sql) {
+ // Spark uses an empty SQL string for DROP DEFAULT; a string literal
containing no characters
+ // is represented by two quote characters instead.
+ if (sql.isEmpty()) {
+ throw new UnsupportedOperationException("DROP DEFAULT is not supported
by Gravitino");
+ }
+ try {
+ org.apache.spark.sql.catalyst.expressions.Expression parsed =
+ CatalystSqlParser$.MODULE$.parseExpression(sql);
+ boolean literal =
+ parsed instanceof Literal
+ || (parsed instanceof UnaryMinus && ((UnaryMinus)
parsed).child() instanceof Literal)
+ || (parsed instanceof UnaryPositive
+ && ((UnaryPositive) parsed).child() instanceof Literal);
+ Preconditions.checkArgument(
+ literal && parsed.resolved() && parsed.foldable(),
+ "Only literal column defaults are supported: %s",
+ sql);
+ Object value = parsed.eval(null);
+ if (parsed.dataType() instanceof TimestampType) {
Review Comment:
[P2] Please account for the target column type when converting timestamp
defaults. With Spark's session zone set to `Asia/Shanghai`, a UTC MySQL
connection, and a `DATETIME(6)` target column, `TIMESTAMP '2026-11-04
12:34:56.123456'` becomes `'2026-11-04 04:34:56.123456+00:00'` and is stored as
`04:34:56.123456`. Spark's coercion to the target NTZ type yields
`12:34:56.123456`. The reverse NTZ-to-LTZ assignment also depends on the
database connection zone. Convert using the target column type and Spark
session zone, or reject unsupported cross-type defaults instead of silently
changing their values. Reproduced with Spark 3.5/4.0 and MySQL 8.0.45.
##########
catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/converter/JdbcColumnDefaultValueConverter.java:
##########
@@ -85,14 +86,29 @@ public String fromGravitino(Expression defaultValue) {
value = validateNumericLiteral(value);
}
return value;
+ } else if (type instanceof Types.BinaryType && literal.value()
instanceof byte[]) {
+ return binaryLiteral((byte[]) literal.value());
+ } else if (type instanceof Types.BinaryType
+ && literal.value() instanceof String
+ && ((String) literal.value()).startsWith("0x")) {
+ byte[] value;
+ try {
+ value =
+ BaseEncoding.base16()
+ .lowerCase()
+ .decode(((String)
literal.value()).substring(2).toLowerCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(
+ "Invalid binary column default value: use 0x followed by
hexadecimal byte pairs", e);
+ }
+ return binaryLiteral(value);
} else if (type instanceof Types.TimestampType) {
- /**
- * @see LocalDateTime#toString() would return like
'yyyy-MM-ddTHH:mm:ss'
- */
- if (literal.value() instanceof String || literal.value() instanceof
LocalDateTime) {
- return quoteStringLiteral(literal.value().toString().replace("T", "
"));
+ String value = literal.value().toString().replace("T", " ");
Review Comment:
[P2] `OffsetDateTime.toString()` omits the seconds field when seconds and
nanoseconds are both zero. For example, `TIMESTAMP '2026-11-04 12:34:00'`
passes through the Spark/REST conversion and is rendered here as `'2026-11-04
12:34+00:00'`. MySQL 8.0.45 rejects that default with `ERROR 1067: Invalid
default value`; midnight fails too. Please format timestamps with an explicit
seconds field, preserving fractions and offsets, and add whole-minute/midnight
regression cases. The existing `12:34:56.123456` cases do not exercise this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]