yuqi1129 commented on code in PR #14609:
URL: https://github.com/apache/gravitino/pull/14609#discussion_r4468081561
##########
spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTableChangeConverter.java:
##########
@@ -93,13 +106,49 @@ public org.apache.gravitino.rel.TableChange
toGravitinoTableChange(TableChange c
(TableChange.UpdateColumnDefaultValue) change;
return org.apache.gravitino.rel.TableChange.updateColumnDefaultValue(
updateColumnDefaultValue.fieldNames(),
- Literals.stringLiteral(updateColumnDefaultValue.newDefaultValue()));
+ convertDefaultValue(updateColumnDefaultValue.newDefaultValue()));
} else {
throw new UnsupportedOperationException(
String.format("Unsupported table change %s",
change.getClass().getName()));
}
}
+ private Expression convertDefaultValue(String sql) {
+ // Spark uses an empty SQL string for DROP DEFAULT; a string literal
containing no characters
+ // is represented by two quote characters instead.
+ if (sql.isEmpty()) {
+ throw new UnsupportedOperationException("DROP DEFAULT is not supported
by Gravitino");
+ }
+ try {
+ org.apache.spark.sql.catalyst.expressions.Expression parsed =
+ CatalystSqlParser$.MODULE$.parseExpression(sql);
+ boolean literal =
+ parsed instanceof Literal
+ || (parsed instanceof UnaryMinus && ((UnaryMinus)
parsed).child() instanceof Literal)
+ || (parsed instanceof UnaryPositive
+ && ((UnaryPositive) parsed).child() instanceof Literal);
+ Preconditions.checkArgument(
+ literal && parsed.resolved() && parsed.foldable(),
+ "Only literal column defaults are supported: %s",
+ sql);
+ Object value = parsed.eval(null);
+ if (parsed.dataType() instanceof TimestampType) {
Review Comment:
Fixed in b7bdd208f7afde2f11a25b039c6bb7808ba02c2c.
Timestamp defaults now resolve the target column type and coerce LTZ/NTZ
values using Spark's session time zone before REST/JDBC conversion. An LTZ
literal assigned to DATETIME/NTZ retains the local fields produced by Spark's
cast; the reverse assignment produces an explicit UTC instant. The schema is
loaded lazily once through Gravitino. Batches combining timestamp defaults with
schema changes are rejected before any write; the schema-load permission and
batch restriction are documented.
Validated on Spark 3.5.3 and 4.0.3 across UTC, Asia/Shanghai and
America/Los_Angeles, both timestamp configurations, both assignment directions
and explicit offsets, including a Spark CAST comparison. The filtered catalog
tests also cover schema lookup and failure isolation. Actual MySQL CREATE/ALTER
default inserts verify TIMESTAMP instants and DATETIME local fields under
+00:00/+08:00/-07:00 database sessions. Removing target-aware conversion makes
all 6 selected Spark regression cases fail; restoring it passes them.
##########
catalogs/catalog-jdbc-common/src/main/java/org/apache/gravitino/catalog/jdbc/converter/JdbcColumnDefaultValueConverter.java:
##########
@@ -85,14 +86,29 @@ public String fromGravitino(Expression defaultValue) {
value = validateNumericLiteral(value);
}
return value;
+ } else if (type instanceof Types.BinaryType && literal.value()
instanceof byte[]) {
+ return binaryLiteral((byte[]) literal.value());
+ } else if (type instanceof Types.BinaryType
+ && literal.value() instanceof String
+ && ((String) literal.value()).startsWith("0x")) {
+ byte[] value;
+ try {
+ value =
+ BaseEncoding.base16()
+ .lowerCase()
+ .decode(((String)
literal.value()).substring(2).toLowerCase(Locale.ROOT));
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException(
+ "Invalid binary column default value: use 0x followed by
hexadecimal byte pairs", e);
+ }
+ return binaryLiteral(value);
} else if (type instanceof Types.TimestampType) {
- /**
- * @see LocalDateTime#toString() would return like
'yyyy-MM-ddTHH:mm:ss'
- */
- if (literal.value() instanceof String || literal.value() instanceof
LocalDateTime) {
- return quoteStringLiteral(literal.value().toString().replace("T", "
"));
+ String value = literal.value().toString().replace("T", " ");
Review Comment:
Fixed in b7bdd208f7afde2f11a25b039c6bb7808ba02c2c.
JDBC timestamp rendering now restores an explicit seconds field when
java.time or a REST literal string omits it, while preserving fractional
precision and numeric offsets. This covers both zoned and local timestamps,
including 12:34:00 and midnight.
The 27-case JDBC converter class passes. Actual MySQL CREATE/ALTER default
inserts pass for TIMESTAMP(6) and DATETIME(6), fractional/whole-minute/midnight
values, and database session zones +00:00/+08:00/-07:00. Removing the seconds
fix makes both new JDBC regression cases fail; restoring it passes them. The
complete relevant validation totals 246 passing cases with zero
failures/errors/skips.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]