supreetha created FLINK-38788:
---------------------------------
Summary: flink-connector-mysql-cdc silently fails when unable to
reconnect due to credentials rotation
Key: FLINK-38788
URL: https://issues.apache.org/jira/browse/FLINK-38788
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.5.0, cdc-3.4.0
Reporter: supreetha
When MySQL credentials are rotated, the {{debezium-mysql}} source connector (
flink-connector-mysql-cdc) fails to reconnect. It also does not throw any
errors. Instead, it appears to enter an infinite retry loop, creating a new
thread on each retry attempt.
The connector ignores the configured connect.timeout and connect.max-retries
parameters, resulting in a silent failure where the job stops consuming data
but never restarts or fails.
Steps to reproduce:
# Create a MySQL CDC source using the Debezium MySQL connector by setting the
below config:
** {{connect.timeout}}
** connect.max-retries
** Use a non-root MySQL user.
# Start a Flink job.
# Confirm that the job connects successfully and begins reading MySQL binlog
events.
# Rotate the MySQL user's password.
# Run {{KILL CONNECTION <connection_id>}} on the MySQL server to force a
reconnect.
# Generate new data in MySQL using the updated credentials to confim the new
credentials are working.
# Observe that the Flink job stops consuming data from MySQL but does *not*
throw an error.
# Notice that the MySqlStreamingChangeEventSource tries to create a new thread
infinitely.
Stack trace:
{code:java}
2025-12-05 17:05:57,113 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 8: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138589, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:06:56,069 INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Stopped
reading binlog after 0 events, last recorded offset: {transaction_id=null,
ts_sec=1764954381, file=mysql-bin.000003, pos=139087,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-17,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
server_id=1, event=1} 2025-12-05 17:06:57,109 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 9: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:07:27,733 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:07:57,113 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 10: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:08:27,791 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:08:57,112 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 11: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:09:27,845 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:09:57,112 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 12: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:10:27,901 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:10:57,114 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 13: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:11:27,957 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:11:57,114 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 14: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
