yuqi1129 commented on code in PR #14609:
URL: https://github.com/apache/gravitino/pull/14609#discussion_r4479875890
##########
spark-connector/spark-common/src/main/java/org/apache/gravitino/spark/connector/SparkTableChangeConverter.java:
##########
@@ -93,13 +139,95 @@ public org.apache.gravitino.rel.TableChange
toGravitinoTableChange(TableChange c
(TableChange.UpdateColumnDefaultValue) change;
return org.apache.gravitino.rel.TableChange.updateColumnDefaultValue(
updateColumnDefaultValue.fieldNames(),
- Literals.stringLiteral(updateColumnDefaultValue.newDefaultValue()));
+ convertDefaultValue(
+ updateColumnDefaultValue.newDefaultValue(),
+ updateColumnDefaultValue.fieldNames(),
+ columns));
} else {
throw new UnsupportedOperationException(
String.format("Unsupported table change %s",
change.getClass().getName()));
}
}
+ private Expression convertDefaultValue(
+ String sql, String[] fieldNames, Supplier<Column[]> columns) {
+ // Spark uses an empty SQL string for DROP DEFAULT; a string literal
containing no characters
+ // is represented by two quote characters instead.
+ if (sql.isEmpty()) {
+ throw new UnsupportedOperationException("DROP DEFAULT is not supported
by Gravitino");
+ }
+ org.apache.spark.sql.catalyst.expressions.Expression parsed;
+ Object value;
+ try {
+ parsed = CatalystSqlParser$.MODULE$.parseExpression(sql);
+ boolean literal =
+ parsed instanceof Literal
+ || (parsed instanceof UnaryMinus && ((UnaryMinus)
parsed).child() instanceof Literal)
+ || (parsed instanceof UnaryPositive
+ && ((UnaryPositive) parsed).child() instanceof Literal);
+ Preconditions.checkArgument(
+ literal && parsed.resolved() && parsed.foldable(),
+ "Only literal column defaults are supported: %s",
+ sql);
+ value = parsed.eval(null);
+ } catch (Exception e) {
+ throw new IllegalArgumentException("Unsupported column default SQL: " +
sql, e);
+ }
+ if (parsed.dataType() instanceof TimestampType
Review Comment:
Fixed in 6fb03b9d70c44fc9cbea32745e1e3d4cad76c0cf.
DATE defaults now resolve the target column type. LTZ targets use Spark's
session-zone start of day; NTZ targets retain local midnight. In the reported
Asia/Shanghai case, DATE '2026-11-04' now becomes 2026-11-03T16:00:00Z before
REST/JDBC conversion.
For STRING-to-timestamp assignments, this change takes the
explicit-rejection option: both Spark 3.5 and 4.0 fail before writing and
suggest a TIMESTAMP_LTZ or TIMESTAMP_NTZ literal. Date/string defaults
targeting non-timestamp columns keep their values and types. The additional
schema-read permission and schema-changing batch restriction now cover
timestamp, date and string defaults and are documented in the code/PR.
Validation: 108 related tests passed on Spark 3.5.3 and 4.0.3 (54 each).
DATE conversions match Spark CAST in UTC, Asia/Shanghai, America/Los_Angeles
and Pacific/Apia, including pre-epoch dates and a skipped day, and survive REST
JSON and JDBC rendering. Catalog tests cover the reported instant, one schema
read shared across these literal types, read-failure isolation, and rejection
before any write. The 14 selected regression cases all fail on the preceding
implementation and pass with this fix. No deployed Spark-to-MySQL end-to-end
run was performed for this follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]