Copilot commented on code in PR #14658:
URL: https://github.com/apache/gravitino/pull/14658#discussion_r4478238733
##########
flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/catalog/BaseCatalog.java:
##########
@@ -888,6 +889,66 @@ private static Optional<List<String>>
getFlinkPrimaryKey(Table table) {
return Optional.of(primaryKeyFieldList);
}
+ /**
+ * Returns whether this catalog can store a table as a Flink generic table,
which serializes the
+ * full Flink schema, including computed columns, metadata columns and
watermarks, into {@code
+ * flink.*} properties. Only the Hive catalog can.
+ *
+ * @return true if generic tables are supported, false by default
+ */
+ protected boolean supportsGenericTables() {
+ return false;
+ }
+
+ /**
+ * Rejects computed columns, metadata columns and watermarks, which the
Gravitino column model
+ * cannot represent. Storing them anyway would turn each such column into a
plain physical column
+ * and drop the watermark, without any error. This path serves Paimon,
Iceberg and JDBC tables,
+ * and Hive tables with {@code 'connector'='hive'}.
+ *
+ * @param resolvedTable the resolved create-table request
+ */
+ private void rejectUnsupportedSchemaElements(ResolvedCatalogBaseTable<?>
resolvedTable) {
+ for (org.apache.flink.table.catalog.Column column :
+ resolvedTable.getResolvedSchema().getColumns()) {
+ rejectUnsupportedColumn(column);
+ }
+ if (!resolvedTable.getResolvedSchema().getWatermarkSpecs().isEmpty()) {
+ throw new UnsupportedOperationException(
+ String.format(
+ "The watermark cannot be stored in the Gravitino catalog
'%s'.%s",
+ getName(), genericTableAdvice()));
Review Comment:
The watermark error message doesn’t identify which rowtime attribute(s) had
watermarks and uses the singular form even though multiple watermark specs are
possible. Including the affected attribute name(s) (and/or count) would make
this significantly easier to debug for users.
##########
flink-connector/flink-common/src/main/java/org/apache/gravitino/flink/connector/catalog/BaseCatalog.java:
##########
@@ -425,6 +425,7 @@ public void createTable(ObjectPath tablePath,
CatalogBaseTable table, boolean ig
NameIdentifier.of(tablePath.getDatabaseName(),
tablePath.getObjectName());
ResolvedCatalogBaseTable<?> resolvedTable = (ResolvedCatalogBaseTable<?>)
table;
+ rejectUnsupportedSchemaElements(resolvedTable);
Column[] columns =
Review Comment:
With validation happening unconditionally at the start of `createTable`,
`CREATE TABLE IF NOT EXISTS` can now throw `UnsupportedOperationException` even
when the table already exists (because the schema is validated before any
potential no-op short-circuit). If the intent is to preserve typical IF NOT
EXISTS semantics, move/guard this validation so it only runs when a create is
actually attempted; if the new behavior is intentional, consider adding an
explicit test covering the 'already exists + ignoreIfExists' case to lock in
that contract.
##########
flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/catalog/TestBaseCatalog.java:
##########
@@ -546,6 +550,220 @@ public void
testAlterTableWithCommentChangeForwardsAlterCall() throws Exception
Mockito.verify(tableCatalog, Mockito.times(1)).alterTable(Mockito.any(),
Mockito.any());
}
+ /**
+ * Computed columns, metadata columns and watermarks cannot be represented
by Gravitino columns.
+ * createTable must reject them instead of storing plain physical columns
without the watermark.
+ */
+ @Test
+ void testCreateTableRejectsComputedColumnsAndWatermarks() {
Review Comment:
This test also asserts rejection of metadata columns, but the method name
only mentions computed columns and watermarks. Consider renaming it to include
metadata columns (or splitting into separate tests) so failures are easier to
diagnose and the name matches covered behavior.
##########
flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/catalog/TestBaseCatalog.java:
##########
@@ -546,6 +550,220 @@ public void
testAlterTableWithCommentChangeForwardsAlterCall() throws Exception
Mockito.verify(tableCatalog, Mockito.times(1)).alterTable(Mockito.any(),
Mockito.any());
}
+ /**
+ * Computed columns, metadata columns and watermarks cannot be represented
by Gravitino columns.
+ * createTable must reject them instead of storing plain physical columns
without the watermark.
+ */
+ @Test
+ void testCreateTableRejectsComputedColumnsAndWatermarks() {
+ TestableBaseCatalog catalog =
+ new TestableBaseCatalog(Mockito.mock(AbstractCatalog.class),
mockedCatalogHandle());
+
+ // computed column: b AS a + 1
+ List<Column> computed =
+ Arrays.asList(
+ Column.physical("a", DataTypes.INT()),
+ Column.computed("b", resolvedExpression(DataTypes.INT())));
+ ResolvedCatalogTable computedTable =
+ resolvedCatalogTable(resolvedSchema(computed,
Collections.emptyList()));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t1"), computedTable,
false));
+
+ // metadata column: m INT METADATA
+ List<Column> metadata =
+ Arrays.asList(
+ Column.physical("a", DataTypes.INT()),
+ Column.metadata("m", DataTypes.INT(), null, false));
+ ResolvedCatalogTable metadataTable =
+ resolvedCatalogTable(resolvedSchema(metadata,
Collections.emptyList()));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t4"), metadataTable,
false));
+
+ // watermark spec on a timestamp column
+ List<WatermarkSpec> watermarkSpecs =
+ Collections.singletonList(
+ WatermarkSpec.of("a",
resolvedExpression(DataTypes.TIMESTAMP_LTZ(3))));
+ ResolvedCatalogTable watermarkedTable =
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(Column.physical("a",
DataTypes.TIMESTAMP_LTZ(3))),
+ watermarkSpecs));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t2"),
watermarkedTable, false));
+
+ // plain physical column still creates
+ ResolvedCatalogTable plainTable =
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(Column.physical("a",
DataTypes.INT())),
+ Collections.emptyList()));
+ Assertions.assertDoesNotThrow(
+ () -> catalog.createTable(new ObjectPath("db", "t3"), plainTable,
false));
+ }
+
+ /**
+ * ALTER TABLE ... ADD of a computed column must be rejected like
createTable does, while adding a
+ * physical column still goes through.
+ */
+ @Test
+ void testAlterTableRejectsComputedColumns() throws Exception {
Review Comment:
The add-column rejection path is only tested for computed columns, but
`BaseCatalog.addColumn(...)` also rejects metadata columns via the same helper.
Add a parallel `assertThrows` for `Column.metadata(...)` to ensure metadata ADD
is rejected consistently.
##########
flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/catalog/TestBaseCatalog.java:
##########
@@ -546,6 +550,220 @@ public void
testAlterTableWithCommentChangeForwardsAlterCall() throws Exception
Mockito.verify(tableCatalog, Mockito.times(1)).alterTable(Mockito.any(),
Mockito.any());
}
+ /**
+ * Computed columns, metadata columns and watermarks cannot be represented
by Gravitino columns.
+ * createTable must reject them instead of storing plain physical columns
without the watermark.
+ */
+ @Test
+ void testCreateTableRejectsComputedColumnsAndWatermarks() {
+ TestableBaseCatalog catalog =
+ new TestableBaseCatalog(Mockito.mock(AbstractCatalog.class),
mockedCatalogHandle());
+
+ // computed column: b AS a + 1
+ List<Column> computed =
+ Arrays.asList(
+ Column.physical("a", DataTypes.INT()),
+ Column.computed("b", resolvedExpression(DataTypes.INT())));
+ ResolvedCatalogTable computedTable =
+ resolvedCatalogTable(resolvedSchema(computed,
Collections.emptyList()));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t1"), computedTable,
false));
+
+ // metadata column: m INT METADATA
+ List<Column> metadata =
+ Arrays.asList(
+ Column.physical("a", DataTypes.INT()),
+ Column.metadata("m", DataTypes.INT(), null, false));
+ ResolvedCatalogTable metadataTable =
+ resolvedCatalogTable(resolvedSchema(metadata,
Collections.emptyList()));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t4"), metadataTable,
false));
+
+ // watermark spec on a timestamp column
+ List<WatermarkSpec> watermarkSpecs =
+ Collections.singletonList(
+ WatermarkSpec.of("a",
resolvedExpression(DataTypes.TIMESTAMP_LTZ(3))));
+ ResolvedCatalogTable watermarkedTable =
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(Column.physical("a",
DataTypes.TIMESTAMP_LTZ(3))),
+ watermarkSpecs));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t2"),
watermarkedTable, false));
+
+ // plain physical column still creates
+ ResolvedCatalogTable plainTable =
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(Column.physical("a",
DataTypes.INT())),
+ Collections.emptyList()));
+ Assertions.assertDoesNotThrow(
+ () -> catalog.createTable(new ObjectPath("db", "t3"), plainTable,
false));
+ }
+
+ /**
+ * ALTER TABLE ... ADD of a computed column must be rejected like
createTable does, while adding a
+ * physical column still goes through.
+ */
+ @Test
+ void testAlterTableRejectsComputedColumns() throws Exception {
+ Catalog gravitinoCatalogHandle = Mockito.mock(Catalog.class);
+ TableCatalog tableCatalog = Mockito.mock(TableCatalog.class);
+
Mockito.when(gravitinoCatalogHandle.asTableCatalog()).thenReturn(tableCatalog);
+
+ Schema schema = Schema.newBuilder().column("a", DataTypes.INT()).build();
+ CatalogTable existing =
+ catalogCompat().createCatalogTable(schema, null, ImmutableList.of(),
ImmutableMap.of());
+ TestableBaseCatalog catalog =
+ new TestableBaseCatalog(
+ Mockito.mock(AbstractCatalog.class), gravitinoCatalogHandle,
existing);
+
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () ->
+ catalog.alterTable(
+ new ObjectPath("db", "t"),
+ existing,
+ Collections.singletonList(
+ org.apache.flink.table.catalog.TableChange.add(
+ Column.computed("b",
resolvedExpression(DataTypes.INT())))),
+ false));
+
Review Comment:
The add-column rejection path is only tested for computed columns, but
`BaseCatalog.addColumn(...)` also rejects metadata columns via the same helper.
Add a parallel `assertThrows` for `Column.metadata(...)` to ensure metadata ADD
is rejected consistently.
##########
docs/flink-connector/flink-catalog-hive.md:
##########
@@ -39,6 +39,9 @@ When loading or altering a table, Gravitino Flink connector
detects generic tabl
reconstructed from the serialized `flink.*` properties. Hive tables continue
to use the native
Hive schema.
+Only generic tables keep computed columns, metadata columns and watermarks.
Creating a Hive table
+(`connector=hive`) with them, or adding such a column to it, fails.
Review Comment:
The docs use `connector=hive` here, while other places (and error messages)
use `'connector'='hive'`. Aligning the formatting/quoting avoids confusion and
makes it easier to copy/paste the property correctly.
##########
flink-connector/flink-common/src/test/java/org/apache/gravitino/flink/connector/catalog/TestBaseCatalog.java:
##########
@@ -546,6 +550,220 @@ public void
testAlterTableWithCommentChangeForwardsAlterCall() throws Exception
Mockito.verify(tableCatalog, Mockito.times(1)).alterTable(Mockito.any(),
Mockito.any());
}
+ /**
+ * Computed columns, metadata columns and watermarks cannot be represented
by Gravitino columns.
+ * createTable must reject them instead of storing plain physical columns
without the watermark.
+ */
+ @Test
+ void testCreateTableRejectsComputedColumnsAndWatermarks() {
+ TestableBaseCatalog catalog =
+ new TestableBaseCatalog(Mockito.mock(AbstractCatalog.class),
mockedCatalogHandle());
+
+ // computed column: b AS a + 1
+ List<Column> computed =
+ Arrays.asList(
+ Column.physical("a", DataTypes.INT()),
+ Column.computed("b", resolvedExpression(DataTypes.INT())));
+ ResolvedCatalogTable computedTable =
+ resolvedCatalogTable(resolvedSchema(computed,
Collections.emptyList()));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t1"), computedTable,
false));
+
+ // metadata column: m INT METADATA
+ List<Column> metadata =
+ Arrays.asList(
+ Column.physical("a", DataTypes.INT()),
+ Column.metadata("m", DataTypes.INT(), null, false));
+ ResolvedCatalogTable metadataTable =
+ resolvedCatalogTable(resolvedSchema(metadata,
Collections.emptyList()));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t4"), metadataTable,
false));
+
+ // watermark spec on a timestamp column
+ List<WatermarkSpec> watermarkSpecs =
+ Collections.singletonList(
+ WatermarkSpec.of("a",
resolvedExpression(DataTypes.TIMESTAMP_LTZ(3))));
+ ResolvedCatalogTable watermarkedTable =
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(Column.physical("a",
DataTypes.TIMESTAMP_LTZ(3))),
+ watermarkSpecs));
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> catalog.createTable(new ObjectPath("db", "t2"),
watermarkedTable, false));
+
+ // plain physical column still creates
+ ResolvedCatalogTable plainTable =
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(Column.physical("a",
DataTypes.INT())),
+ Collections.emptyList()));
+ Assertions.assertDoesNotThrow(
+ () -> catalog.createTable(new ObjectPath("db", "t3"), plainTable,
false));
+ }
+
+ /**
+ * ALTER TABLE ... ADD of a computed column must be rejected like
createTable does, while adding a
+ * physical column still goes through.
+ */
+ @Test
+ void testAlterTableRejectsComputedColumns() throws Exception {
+ Catalog gravitinoCatalogHandle = Mockito.mock(Catalog.class);
+ TableCatalog tableCatalog = Mockito.mock(TableCatalog.class);
+
Mockito.when(gravitinoCatalogHandle.asTableCatalog()).thenReturn(tableCatalog);
+
+ Schema schema = Schema.newBuilder().column("a", DataTypes.INT()).build();
+ CatalogTable existing =
+ catalogCompat().createCatalogTable(schema, null, ImmutableList.of(),
ImmutableMap.of());
+ TestableBaseCatalog catalog =
+ new TestableBaseCatalog(
+ Mockito.mock(AbstractCatalog.class), gravitinoCatalogHandle,
existing);
+
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () ->
+ catalog.alterTable(
+ new ObjectPath("db", "t"),
+ existing,
+ Collections.singletonList(
+ org.apache.flink.table.catalog.TableChange.add(
+ Column.computed("b",
resolvedExpression(DataTypes.INT())))),
+ false));
+
+ Assertions.assertDoesNotThrow(
+ () ->
+ catalog.alterTable(
+ new ObjectPath("db", "t"),
+ existing,
+ Collections.singletonList(
+ org.apache.flink.table.catalog.TableChange.add(
+ Column.physical("c", DataTypes.INT()))),
+ false));
+ }
+
+ /**
+ * The generic table advice is only given by catalogs that support generic
tables, which is only
+ * the Hive catalog.
+ */
+ @Test
+ void testUnsupportedSchemaAdviceMentionsGenericTablesOnlyWhenSupported() {
+ TestableBaseCatalog plain =
+ new TestableBaseCatalog(Mockito.mock(AbstractCatalog.class),
mockedCatalogHandle());
+ TestableBaseCatalog genericLike =
+ new TestableBaseCatalog(Mockito.mock(AbstractCatalog.class),
mockedCatalogHandle()) {
+ @Override
+ protected boolean supportsGenericTables() {
+ return true;
+ }
+ };
+
+ UnsupportedOperationException plainError =
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () ->
+ plain.createTable(
+ new ObjectPath("db", "t"),
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(
+ Column.computed("b",
resolvedExpression(DataTypes.INT()))),
+ Collections.emptyList())),
+ false));
+ Assertions.assertFalse(
+ plainError.getMessage().contains("generic table"),
plainError.getMessage());
+
+ UnsupportedOperationException genericLikeError =
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () ->
+ genericLike.createTable(
+ new ObjectPath("db", "t"),
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(
+ Column.computed("b",
resolvedExpression(DataTypes.INT()))),
+ Collections.emptyList())),
+ false));
+ Assertions.assertTrue(
+ genericLikeError.getMessage().contains("generic table"),
genericLikeError.getMessage());
+
+ // the watermark message follows the same conditional advice
+ ResolvedCatalogTable watermarked =
+ resolvedCatalogTable(
+ resolvedSchema(
+ Collections.singletonList(Column.physical("ts",
DataTypes.TIMESTAMP_LTZ(3))),
+ Collections.singletonList(
+ WatermarkSpec.of("ts",
resolvedExpression(DataTypes.TIMESTAMP_LTZ(3))))));
+ UnsupportedOperationException plainWatermarkError =
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> plain.createTable(new ObjectPath("db", "t"), watermarked,
false));
+ Assertions.assertFalse(
+ plainWatermarkError.getMessage().contains("generic table"),
+ plainWatermarkError.getMessage());
+ UnsupportedOperationException genericLikeWatermarkError =
+ Assertions.assertThrows(
+ UnsupportedOperationException.class,
+ () -> genericLike.createTable(new ObjectPath("db", "t"),
watermarked, false));
+ Assertions.assertTrue(
+ genericLikeWatermarkError.getMessage().contains("generic table"),
+ genericLikeWatermarkError.getMessage());
+ }
+
+ private static Catalog mockedCatalogHandle() {
+ Catalog handle = Mockito.mock(Catalog.class);
+
Mockito.doReturn(Mockito.mock(TableCatalog.class)).when(handle).asTableCatalog();
+ return handle;
+ }
+
+ private static ResolvedExpression resolvedExpression(DataType dataType) {
+ ResolvedExpression expression = Mockito.mock(ResolvedExpression.class);
+ Mockito.when(expression.getOutputDataType()).thenReturn(dataType);
+ Mockito.when(expression.asSerializableString()).thenReturn("`a`");
Review Comment:
The helper always serializes expressions as \"`a`\", but some tests build
watermarks for other rowtime attributes (e.g., \"ts\"). If Flink starts
validating watermark expressions more strictly (or changes builder behavior
across versions), this can become a brittle / misleading setup. Consider
parameterizing the serialized string (e.g., `resolvedExpression(dataType,
\"`ts`\")`) or matching the rowtime attribute in each test to avoid accidental
invalid schemas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]