tkaymak commented on code in PR #41331:
URL: https://github.com/apache/beam/pull/41331#discussion_r4601008835


##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -152,10 +167,17 @@ public SourceRecordJson(@Nullable SourceRecord 
sourceRecord, boolean fullMetadat
     if (this.value == null) {
       this.event = new Event(null, null, null, null);
       this.transactionEvent = null;
+      this.changeSchemaEvent = null;
       this.metadataExtras = Collections.emptyMap();
     } else if (isTransactionMetadataEvent(sourceRecord.keySchema())) {
       this.event = null;
       this.transactionEvent = this.loadTransactionEvent();
+      this.changeSchemaEvent = null;
+      this.metadataExtras = Collections.emptyMap();
+    } else if (isSchemaChangeEventKeyName(sourceRecord.keySchema())) {

Review Comment:
   On master these records fall through to the data branch and are emitted with 
`metadata` and null `before` and `after`. They are not skipped. The new payload 
drops `source`, so `withFullMetadata(true)` has no effect here and `snapshot`, 
`file` and `pos` are lost. Please consider keeping `metadata` in this payload, 
with the extras under full metadata.



##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -592,6 +766,14 @@ private static boolean 
isTransactionMetadataEvent(@Nullable Schema keySchema) {
     return keySchema != null && 
SCHEMA_TRANSACTION_METADATA_EVENT_KEY_NAME.equals(keySchema.name());
   }
 
+  /**
+   * Whether the record is a schema change event, emitted for each DDL 
statement when {@code
+   * include.schema.changes} is enabled. These are identified by their key 
schema name.
+   */
+  private static boolean isSchemaChangeEventKeyName(@Nullable Schema 
keySchema) {

Review Comment:
   Nit. The sibling is `isTransactionMetadataEvent` and this method takes a 
schema, so `isSchemaChangeEvent` would read better. Debezium and the Javadoc 
here say schema change, so `SchemaChangeEvent` would be more consistent than 
`ChangeSchemaEvent`.



##########
CHANGES.md:
##########
@@ -89,6 +89,7 @@
 * Support for X source added (Java/Python) 
([#X](https://github.com/apache/beam/issues/X)).
 * Upgraded Iceberg dependency to 1.12.0 
([#40971](https://github.com/apache/beam/issues/40971)).
 * (Java/Python/Go) DebeziumIO: `readAsJson()` now emits Debezium transaction 
metadata with `provide.transaction.metadata=true` (Python/Go via 
`connection_properties`) 
([#40784](https://github.com/apache/beam/issues/40784)).
+* (Java/Python/Go) DebeziumIO: `readAsJson()` now emits MySQL, Oracle and SQL 
Server schema change events with `include.schema.changes=true` (Python/Go via 
`connection_properties`) 
([#41330](https://github.com/apache/beam/issues/41330)).

Review Comment:
   The wording with `include.schema.changes=true` reads as opt in, but true is 
the Debezium default. Please say that the JSON shape of these records changes 
and that `include.schema.changes=false` drops them. A line under Breaking 
Changes for 2.79.0 would help users who read `metadata` on every element. The 
same wording is in `README.md:100`, `debezium.py:155` and `debezium.go:151`. 
The expansion service passes only the user properties, so Python and Go get the 
default of true as well.



##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -119,11 +126,19 @@ public class SourceRecordJson {
   private static final String SCHEMA_TRANSACTION_METADATA_EVENT_KEY_NAME =
       "io.debezium.connector.common.TransactionMetadataKey";
 
+  /** Key schema names of the schema change events emitted by the supported 
connectors. */
+  private static final Set<String> SCHEMA_CHANGE_EVENT_KEY_NAMES =
+      ImmutableSet.of(
+          "io.debezium.connector.mysql.SchemaChangeKey",
+          "io.debezium.connector.oracle.SchemaChangeKey",
+          "io.debezium.connector.sqlserver.SchemaChangeKey");

Review Comment:
   Db2 is in `Connectors` and uses the same schemas with the key name 
`io.debezium.connector.db2.SchemaChangeKey`. With this list a Db2 schema change 
still comes out in the old data shape, I ran that. Debezium has 
`SchemaFactory.get().isSchemaChangeSchema(schema)`, which covers every 
connector and returns true for the Db2 key. If the list of three is intended, 
please say in the docs what Db2 emits.



##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -584,6 +666,98 @@ public DataCollection(@Nullable String dataCollection, 
@Nullable Long eventCount
     }
   }
 
+  /** Depicts a schema change SourceRecord as a ChangeSchemaEvent to be mapped 
as JSON. */
+  @SuppressWarnings("unused") // Unused params are validated as JSON
+  static class ChangeSchemaEvent implements Serializable {
+
+    private final @Nullable String databaseName;
+    private final @Nullable String schemaName;
+    private final @Nullable String ddl;
+
+    @SerializedName("ts_ms")
+    private final @Nullable Long tsMs;
+
+    private final @Nullable List<TableChange> tableChanges;
+
+    /**
+     * ChangeSchemaEvent Initializer.
+     *
+     * @param databaseName Database the DDL statement was applied to
+     * @param schemaName Schema the DDL statement was applied to, null for 
MySQL
+     * @param ddl DDL statement that changed the schema
+     * @param tsMs Time the connector processed the event, in milliseconds
+     * @param tableChanges Tables created, altered or dropped by the DDL 
statement
+     */
+    public ChangeSchemaEvent(
+        @Nullable String databaseName,
+        @Nullable String schemaName,
+        @Nullable String ddl,
+        @Nullable Long tsMs,
+        @Nullable List<TableChange> tableChanges) {
+      this.databaseName = databaseName;
+      this.schemaName = schemaName;
+      this.ddl = ddl;
+      this.tsMs = tsMs;
+      this.tableChanges = tableChanges;
+    }
+
+    /**
+     * Transforms the ChangeSchemaEvent to a JSON string.
+     *
+     * @return JSON String
+     */
+    public String toJson() {
+      return GSON.toJson(this);
+    }
+
+    /** Depicts an entry of the tableChanges field within SourceRecord. */
+    static class TableChange implements Serializable {
+      private final @Nullable String type;
+      private final @Nullable String id;
+      private final @Nullable Table table;
+
+      /**
+       * TableChange Initializer.
+       *
+       * @param type Kind of change: {@code CREATE}, {@code ALTER} or {@code 
DROP}
+       * @param id Fully qualified table id, for example {@code 
"inventory"."customers"}
+       * @param table Table definition after the change

Review Comment:
   `table` is null for a DROP, Debezium only sets the id there. Please mention 
it.



##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/DebeziumIOMySqlConnectorIT.java:
##########
@@ -367,4 +371,91 @@ public void testDebeziumIOMySqlFullMetadata() {
 
     p.run().waitUntilFinish();
   }
+
+  /**
+   * Debezium - MySQL connector Test with {@code include.schema.changes} 
enabled.
+   *
+   * <p>Tests that the snapshot's CREATE schema change event for {@code 
inventory.customers} carries
+   * the table's columns, each with every field of Debezium's column schema.
+   */
+  @Test
+  public void testDebeziumIOMySqlSchemaChangeEventColumns() {
+
+    String kafkaBootstrapServers = KAFKA_CONTAINER.getBootstrapServers();
+    String schemaHistoryTopic = "mysql-schema-history-schema-change-" + 
System.nanoTime();
+
+    String host = MY_SQL_CONTAINER.getHost();
+    String port = MY_SQL_CONTAINER.getMappedPort(3306).toString();
+
+    PipelineOptions options = PipelineOptionsFactory.create();
+    Pipeline p = Pipeline.create(options);
+    PCollection<String> results =
+        p.apply(
+            DebeziumIO.readAsJson()
+                .withConnectorConfiguration(
+                    DebeziumIO.ConnectorConfiguration.create()
+                        .withUsername("debezium")
+                        .withPassword("dbz")
+                        .withConnectorClass(MySqlConnector.class)
+                        .withHostName(host)
+                        .withPort(port)
+                        .withConnectionProperty("database.server.id", "184056")
+                        .withConnectionProperty("database.include.list", 
"inventory")
+                        .withConnectionProperty("table.include.list", 
"inventory.customers")
+                        .withConnectionProperty("include.schema.changes", 
"true")
+                        .withConnectionProperty(
+                            "schema.history.internal.kafka.bootstrap.servers",
+                            kafkaBootstrapServers)
+                        .withConnectionProperty(
+                            "schema.history.internal.kafka.topic", 
schemaHistoryTopic)
+                        .withConnectionProperty(
+                            "schema.history.internal",
+                            
"io.debezium.storage.kafka.history.KafkaSchemaHistory")
+                        .withConnectionProperty(
+                            
"schema.history.internal.store.only.captured.tables.ddl", "true"))
+                // The snapshot emits its schema change events before the 
customers rows.
+                .withMaxNumberOfRecords(10)

Review Comment:
   This test reads the snapshot CREATE event, it never runs DDL. The use case 
in the issue is a live `ALTER TABLE`. A second table in `table.include.list` 
that the test alters would cover the binlog path without touching `customers`. 
The limit of 10 depends on how many schema events the snapshot emits before the 
rows, so a short comment with the count would help. By my reading of the 
Debezium snapshot source it is up to six. That count is inference, I did not 
run the IT.



##########
.github/trigger_files/beam_PostCommit_XVR_Direct.json:
##########
@@ -1,4 +1,4 @@
 {
-  "modification": 4,
+  "modification": 5,

Review Comment:
   I could not find a Debezium test in the XVR Direct suite, so I think this 
trigger can go. That is inference from the Gradle files. The Debezium precommit 
already runs the new IT on any change under `sdks/java/io/debezium`.



##########
sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/SourceRecordJsonTest.java:
##########
@@ -251,6 +252,125 @@ public void testFullMetadataWithoutOpField() {
     assertTrue(json.contains("\"lsn\":37094192"));
   }
 
+  @Test
+  public void testChangeSchemaEvent() {
+    Schema keySchema =
+        SchemaBuilder.struct()
+            .name("io.debezium.connector.mysql.SchemaChangeKey")

Review Comment:
   Could you add cases for a DROP with a null `table`, two entries in 
`tableChanges`, an empty list, and one Oracle or SQL Server key name. I ran 
these against the head code and they pass, so this is only about pinning the 
behaviour. 
`SchemaFactory.get().schemaHistoryChangeSchema(SchemaNameAdjuster.NO_OP)` gives 
you the real change, table and column schemas and saves about 45 lines.



##########
sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/SourceRecordJson.java:
##########
@@ -91,6 +91,13 @@
  * ({@link #loadTransactionEvent}): {@code status}, {@code id}, {@code ts_ms}, 
{@code event_count}
  * and {@code data_collections}.
  *
+ * <h3>Schema change events</h3>
+ *
+ * <p>With {@code include.schema.changes=true}, the MySQL, Oracle and SQL 
Server connectors also

Review Comment:
   The snapshot emits these events as well. For MySQL that is one CREATE per 
captured table plus database level statements with an empty `tableChanges`. It 
is worth a sentence, since a pipeline that alerts on schema changes will see 
them at every initial snapshot.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to