tkaymak commented on code in PR #41331:
URL: https://github.com/apache/beam/pull/41331#discussion_r4601008835
##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -152,10 +167,17 @@ public SourceRecordJson(@Nullable SourceRecord
sourceRecord, boolean fullMetadat
if (this.value == null) {
this.event = new Event(null, null, null, null);
this.transactionEvent = null;
+ this.changeSchemaEvent = null;
this.metadataExtras = Collections.emptyMap();
} else if (isTransactionMetadataEvent(sourceRecord.keySchema())) {
this.event = null;
this.transactionEvent = this.loadTransactionEvent();
+ this.changeSchemaEvent = null;
+ this.metadataExtras = Collections.emptyMap();
+ } else if (isSchemaChangeEventKeyName(sourceRecord.keySchema())) {
Review Comment:
On master these records fall through to the data branch and are emitted with
`metadata` and null `before` and `after`. They are not skipped. The new payload
drops `source`, so `withFullMetadata(true)` has no effect here and `snapshot`,
`file` and `pos` are lost. Please consider keeping `metadata` in this payload,
with the extras under full metadata.
##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -592,6 +766,14 @@ private static boolean
isTransactionMetadataEvent(@Nullable Schema keySchema) {
return keySchema != null &&
SCHEMA_TRANSACTION_METADATA_EVENT_KEY_NAME.equals(keySchema.name());
}
+ /**
+ * Whether the record is a schema change event, emitted for each DDL
statement when {@code
+ * include.schema.changes} is enabled. These are identified by their key
schema name.
+ */
+ private static boolean isSchemaChangeEventKeyName(@Nullable Schema
keySchema) {
Review Comment:
Nit. The sibling is `isTransactionMetadataEvent` and this method takes a
schema, so `isSchemaChangeEvent` would read better. Debezium and the Javadoc
here say schema change, so `SchemaChangeEvent` would be more consistent than
`ChangeSchemaEvent`.
##########
CHANGES.md:
##########
@@ -89,6 +89,7 @@
* Support for X source added (Java/Python)
([#X](https://github.com/apache/beam/issues/X)).
* Upgraded Iceberg dependency to 1.12.0
([#40971](https://github.com/apache/beam/issues/40971)).
* (Java/Python/Go) DebeziumIO: `readAsJson()` now emits Debezium transaction
metadata with `provide.transaction.metadata=true` (Python/Go via
`connection_properties`)
([#40784](https://github.com/apache/beam/issues/40784)).
+* (Java/Python/Go) DebeziumIO: `readAsJson()` now emits MySQL, Oracle and SQL
Server schema change events with `include.schema.changes=true` (Python/Go via
`connection_properties`)
([#41330](https://github.com/apache/beam/issues/41330)).
Review Comment:
The wording with `include.schema.changes=true` reads as opt in, but true is
the Debezium default. Please say that the JSON shape of these records changes
and that `include.schema.changes=false` drops them. A line under Breaking
Changes for 2.79.0 would help users who read `metadata` on every element. The
same wording is in `README.md:100`, `debezium.py:155` and `debezium.go:151`.
The expansion service passes only the user properties, so Python and Go get the
default of true as well.
##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -119,11 +126,19 @@ public class SourceRecordJson {
private static final String SCHEMA_TRANSACTION_METADATA_EVENT_KEY_NAME =
"io.debezium.connector.common.TransactionMetadataKey";
+ /** Key schema names of the schema change events emitted by the supported
connectors. */
+ private static final Set<String> SCHEMA_CHANGE_EVENT_KEY_NAMES =
+ ImmutableSet.of(
+ "io.debezium.connector.mysql.SchemaChangeKey",
+ "io.debezium.connector.oracle.SchemaChangeKey",
+ "io.debezium.connector.sqlserver.SchemaChangeKey");
Review Comment:
Db2 is in `Connectors` and uses the same schemas with the key name
`io.debezium.connector.db2.SchemaChangeKey`. With this list a Db2 schema change
still comes out in the old data shape, I ran that. Debezium has
`SchemaFactory.get().isSchemaChangeSchema(schema)`, which covers every
connector and returns true for the Db2 key. If the list of three is intended,
please say in the docs what Db2 emits.
##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -584,6 +666,98 @@ public DataCollection(@Nullable String dataCollection,
@Nullable Long eventCount
}
}
+ /** Depicts a schema change SourceRecord as a ChangeSchemaEvent to be mapped
as JSON. */
+ @SuppressWarnings("unused") // Unused params are validated as JSON
+ static class ChangeSchemaEvent implements Serializable {
+
+ private final @Nullable String databaseName;
+ private final @Nullable String schemaName;
+ private final @Nullable String ddl;
+
+ @SerializedName("ts_ms")
+ private final @Nullable Long tsMs;
+
+ private final @Nullable List<TableChange> tableChanges;
+
+ /**
+ * ChangeSchemaEvent Initializer.
+ *
+ * @param databaseName Database the DDL statement was applied to
+ * @param schemaName Schema the DDL statement was applied to, null for
MySQL
+ * @param ddl DDL statement that changed the schema
+ * @param tsMs Time the connector processed the event, in milliseconds
+ * @param tableChanges Tables created, altered or dropped by the DDL
statement
+ */
+ public ChangeSchemaEvent(
+ @Nullable String databaseName,
+ @Nullable String schemaName,
+ @Nullable String ddl,
+ @Nullable Long tsMs,
+ @Nullable List<TableChange> tableChanges) {
+ this.databaseName = databaseName;
+ this.schemaName = schemaName;
+ this.ddl = ddl;
+ this.tsMs = tsMs;
+ this.tableChanges = tableChanges;
+ }
+
+ /**
+ * Transforms the ChangeSchemaEvent to a JSON string.
+ *
+ * @return JSON String
+ */
+ public String toJson() {
+ return GSON.toJson(this);
+ }
+
+ /** Depicts an entry of the tableChanges field within SourceRecord. */
+ static class TableChange implements Serializable {
+ private final @Nullable String type;
+ private final @Nullable String id;
+ private final @Nullable Table table;
+
+ /**
+ * TableChange Initializer.
+ *
+ * @param type Kind of change: {@code CREATE}, {@code ALTER} or {@code
DROP}
+ * @param id Fully qualified table id, for example {@code
"inventory"."customers"}
+ * @param table Table definition after the change
Review Comment:
`table` is null for a DROP, Debezium only sets the id there. Please mention
it.
##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumIOMySqlConnectorIT.java:
##########
@@ -367,4 +371,91 @@ public void testDebeziumIOMySqlFullMetadata() {
p.run().waitUntilFinish();
}
+
+ /**
+ * Debezium - MySQL connector Test with {@code include.schema.changes}
enabled.
+ *
+ * <p>Tests that the snapshot's CREATE schema change event for {@code
inventory.customers} carries
+ * the table's columns, each with every field of Debezium's column schema.
+ */
+ @Test
+ public void testDebeziumIOMySqlSchemaChangeEventColumns() {
+
+ String kafkaBootstrapServers = KAFKA_CONTAINER.getBootstrapServers();
+ String schemaHistoryTopic = "mysql-schema-history-schema-change-" +
System.nanoTime();
+
+ String host = MY_SQL_CONTAINER.getHost();
+ String port = MY_SQL_CONTAINER.getMappedPort(3306).toString();
+
+ PipelineOptions options = PipelineOptionsFactory.create();
+ Pipeline p = Pipeline.create(options);
+ PCollection<String> results =
+ p.apply(
+ DebeziumIO.readAsJson()
+ .withConnectorConfiguration(
+ DebeziumIO.ConnectorConfiguration.create()
+ .withUsername("debezium")
+ .withPassword("dbz")
+ .withConnectorClass(MySqlConnector.class)
+ .withHostName(host)
+ .withPort(port)
+ .withConnectionProperty("database.server.id", "184056")
+ .withConnectionProperty("database.include.list",
"inventory")
+ .withConnectionProperty("table.include.list",
"inventory.customers")
+ .withConnectionProperty("include.schema.changes",
"true")
+ .withConnectionProperty(
+ "schema.history.internal.kafka.bootstrap.servers",
+ kafkaBootstrapServers)
+ .withConnectionProperty(
+ "schema.history.internal.kafka.topic",
schemaHistoryTopic)
+ .withConnectionProperty(
+ "schema.history.internal",
+
"io.debezium.storage.kafka.history.KafkaSchemaHistory")
+ .withConnectionProperty(
+
"schema.history.internal.store.only.captured.tables.ddl", "true"))
+ // The snapshot emits its schema change events before the
customers rows.
+ .withMaxNumberOfRecords(10)
Review Comment:
This test reads the snapshot CREATE event, it never runs DDL. The use case
in the issue is a live `ALTER TABLE`. A second table in `table.include.list`
that the test alters would cover the binlog path without touching `customers`.
The limit of 10 depends on how many schema events the snapshot emits before the
rows, so a short comment with the count would help. By my reading of the
Debezium snapshot source it is up to six. That count is inference, I did not
run the IT.
##########
.github/trigger_files/beam_PostCommit_XVR_Direct.json:
##########
@@ -1,4 +1,4 @@
{
- "modification": 4,
+ "modification": 5,
Review Comment:
I could not find a Debezium test in the XVR Direct suite, so I think this
trigger can go. That is inference from the Gradle files. The Debezium precommit
already runs the new IT on any change under `sdks/java/io/debezium`.
##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/SourceRecordJsonTest.java:
##########
@@ -251,6 +252,125 @@ public void testFullMetadataWithoutOpField() {
assertTrue(json.contains("\"lsn\":37094192"));
}
+ @Test
+ public void testChangeSchemaEvent() {
+ Schema keySchema =
+ SchemaBuilder.struct()
+ .name("io.debezium.connector.mysql.SchemaChangeKey")
Review Comment:
Could you add cases for a DROP with a null `table`, two entries in
`tableChanges`, an empty list, and one Oracle or SQL Server key name. I ran
these against the head code and they pass, so this is only about pinning the
behaviour.
`SchemaFactory.get().schemaHistoryChangeSchema(SchemaNameAdjuster.NO_OP)` gives
you the real change, table and column schemas and saves about 45 lines.
##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -91,6 +91,13 @@
* ({@link #loadTransactionEvent}): {@code status}, {@code id}, {@code ts_ms},
{@code event_count}
* and {@code data_collections}.
*
+ * <h3>Schema change events</h3>
+ *
+ * <p>With {@code include.schema.changes=true}, the MySQL, Oracle and SQL
Server connectors also
Review Comment:
The snapshot emits these events as well. For MySQL that is one CREATE per
captured table plus database level statements with an empty `tableChanges`. It
is worth a sentence, since a pipeline that alerts on schema changes will see
them at every initial snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]