Lee SeungMin created FLINK-34949:
------------------------------------
Summary: Suggestion notify / skip DDL at data inegration famework
v3.X
Key: FLINK-34949
URL: https://issues.apache.org/jira/browse/FLINK-34949
Project: Flink
Issue Type: New Feature
Components: Flink CDC
Reporter: Lee SeungMin
Discussion under feature for data inegration famework v3.X ([~renqs] )
---
We *implemented a notification DDL with GTIDs and skip DDL and after all
queries after DDL*
Our team use Flink CDC to perform MySql CDC.
But since our team sources other team's database and table, we had a need to be
notified about DDL.
Because without appling DDL to our target table, kafak connect that we used
occurs erros (conflict table schema between chagne event log and kafka connect).
----
Here's the our team use case with more detail.
# We source tables from other teams.
# And we send change event log from flink to kafka to use kafka connect (w.
Debezium's JDBC Sink Connector).
# When DDL performed and schema in change event log changed, error occur in
kafka connect. Because Kafka connect loads the table schema on startup, but the
schema is different from the schema in the change event log and kafka connect
has.
# We don't want to occur above error, so we get notified that DDL has been
performed in source table and skip the DDL and any subsequent queries.
# When notified, we apply the same DDL to our target table, restart Kafka
connect (to load the changed target table's schema), and then rerun the flink
cdc app using the returned GTIDs.
git pr
-
[https://github.com/apache/flink-cdc/pull/3055?notification_referrer_id=NT_kwDOAM9bI7M5MzAxOTA4OTcxOjEzNTg5Mjgz#issuecomment-2022183412]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
