[
https://issues.apache.org/jira/browse/FLINK-38788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
supreetha updated FLINK-38788:
------------------------------
Description:
When MySQL credentials are rotated, the {{debezium-mysql}} source connector (
flink-connector-mysql-cdc) fails to reconnect. It also does not throw any
errors. Instead, it appears to enter an infinite retry loop, creating a new
thread on each retry attempt.
The connector ignores the configured connect.timeout and connect.max-retries
parameters, resulting in a silent failure where the job stops consuming data
but never restarts or fails.
Steps to reproduce:
# Create a MySQL CDC source using the Debezium MySQL connector by setting the
below config:
- {{connect.timeout}}
- connect.max-retries
- Use a non-root MySQL user.
# Start a Flink job.
# Confirm that the job connects successfully and begins reading MySQL binlog
events.
# Rotate the MySQL user's password.
# Run {{KILL CONNECTION <connection_id>}} on the MySQL server to force a
reconnect.
# Generate new data in MySQL using the updated credentials to confim the new
credentials are working.
# Observe that the Flink job stops consuming data from MySQL but does *not*
throw an error.
# Notice that the MySqlStreamingChangeEventSource tries to create a new thread
infinitely.
Stack trace:
{code:java}
2025-12-05 17:05:57,113 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 8: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138589, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:06:56,069 INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Stopped
reading binlog after 0 events, last recorded offset: {transaction_id=null,
ts_sec=1764954381, file=mysql-bin.000003, pos=139087,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-17,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
server_id=1, event=1} 2025-12-05 17:06:57,109 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 9: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:07:27,733 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:07:57,113 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 10: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:08:27,791 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:08:57,112 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 11: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:09:27,845 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:09:57,112 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 12: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:10:27,901 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:10:57,114 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 13: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:11:27,957 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:11:57,114 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 14: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1}{code}
was:
When MySQL credentials are rotated, the {{debezium-mysql}} source connector (
flink-connector-mysql-cdc) fails to reconnect. It also does not throw any
errors. Instead, it appears to enter an infinite retry loop, creating a new
thread on each retry attempt.
The connector ignores the configured connect.timeout and connect.max-retries
parameters, resulting in a silent failure where the job stops consuming data
but never restarts or fails.
Steps to reproduce:
# Create a MySQL CDC source using the Debezium MySQL connector by setting the
below config:
{{- connect.timeout}}
- connect.max-retries
- Use a non-root MySQL user.
# Start a Flink job.
# Confirm that the job connects successfully and begins reading MySQL binlog
events.
# Rotate the MySQL user's password.
# Run {{KILL CONNECTION <connection_id>}} on the MySQL server to force a
reconnect.
# Generate new data in MySQL using the updated credentials to confim the new
credentials are working.
# Observe that the Flink job stops consuming data from MySQL but does *not*
throw an error.
# Notice that the MySqlStreamingChangeEventSource tries to create a new thread
infinitely.
Stack trace:
{code:java}
2025-12-05 17:05:57,113 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 8: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138589, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:06:56,069 INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Stopped
reading binlog after 0 events, last recorded offset: {transaction_id=null,
ts_sec=1764954381, file=mysql-bin.000003, pos=139087,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-17,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
server_id=1, event=1} 2025-12-05 17:06:57,109 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 9: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:07:27,733 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:07:57,113 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 10: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:08:27,791 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:08:57,112 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 11: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:09:27,845 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:09:57,112 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 12: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:10:27,901 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:10:57,114 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 13: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1} 2025-12-05 17:11:27,957 INFO
io.debezium.util.Threads [] - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
17:11:57,114 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [local_agr.application_setting, local_agr.asset,
local_agr.asset_attribute] on checkpoint 14: {transaction_id=null,
ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
row=0, event=0, server_id=1}{code}
> flink-connector-mysql-cdc silently fails when unable to reconnect due to
> credentials rotation
> ---------------------------------------------------------------------------------------------
>
> Key: FLINK-38788
> URL: https://issues.apache.org/jira/browse/FLINK-38788
> Project: Flink
> Issue Type: Bug
> Components: Flink CDC
> Affects Versions: cdc-3.4.0, cdc-3.5.0
> Reporter: supreetha
> Priority: Major
>
> When MySQL credentials are rotated, the {{debezium-mysql}} source connector (
> flink-connector-mysql-cdc) fails to reconnect. It also does not throw any
> errors. Instead, it appears to enter an infinite retry loop, creating a new
> thread on each retry attempt.
> The connector ignores the configured connect.timeout and connect.max-retries
> parameters, resulting in a silent failure where the job stops consuming data
> but never restarts or fails.
> Steps to reproduce:
> # Create a MySQL CDC source using the Debezium MySQL connector by setting
> the below config:
> - {{connect.timeout}}
> - connect.max-retries
> - Use a non-root MySQL user.
> # Start a Flink job.
> # Confirm that the job connects successfully and begins reading MySQL binlog
> events.
> # Rotate the MySQL user's password.
> # Run {{KILL CONNECTION <connection_id>}} on the MySQL server to force a
> reconnect.
> # Generate new data in MySQL using the updated credentials to confim the new
> credentials are working.
> # Observe that the Flink job stops consuming data from MySQL but does *not*
> throw an error.
> # Notice that the MySqlStreamingChangeEventSource tries to create a new
> thread infinitely.
> Stack trace:
>
> {code:java}
> 2025-12-05 17:05:57,113 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset for tables [local_agr.application_setting, local_agr.asset,
> local_agr.asset_attribute] on checkpoint 8: {transaction_id=null,
> ts_sec=1764954296, file=mysql-bin.000003, pos=138589, kind=SPECIFIC,
> gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
> row=0, event=0, server_id=1} 2025-12-05 17:06:56,069 INFO
> io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Stopped
> reading binlog after 0 events, last recorded offset: {transaction_id=null,
> ts_sec=1764954381, file=mysql-bin.000003, pos=139087,
> gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-17,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
> server_id=1, event=1} 2025-12-05 17:06:57,109 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset for tables [local_agr.application_setting, local_agr.asset,
> local_agr.asset_attribute] on checkpoint 9: {transaction_id=null,
> ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
> gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
> row=0, event=0, server_id=1} 2025-12-05 17:07:27,733 INFO
> io.debezium.util.Threads [] - Creating thread
> debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
> 17:07:57,113 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset for tables [local_agr.application_setting, local_agr.asset,
> local_agr.asset_attribute] on checkpoint 10: {transaction_id=null,
> ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
> gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
> row=0, event=0, server_id=1} 2025-12-05 17:08:27,791 INFO
> io.debezium.util.Threads [] - Creating thread
> debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
> 17:08:57,112 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset for tables [local_agr.application_setting, local_agr.asset,
> local_agr.asset_attribute] on checkpoint 11: {transaction_id=null,
> ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
> gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
> row=0, event=0, server_id=1} 2025-12-05 17:09:27,845 INFO
> io.debezium.util.Threads [] - Creating thread
> debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
> 17:09:57,112 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset for tables [local_agr.application_setting, local_agr.asset,
> local_agr.asset_attribute] on checkpoint 12: {transaction_id=null,
> ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
> gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
> row=0, event=0, server_id=1} 2025-12-05 17:10:27,901 INFO
> io.debezium.util.Threads [] - Creating thread
> debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
> 17:10:57,114 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset for tables [local_agr.application_setting, local_agr.asset,
> local_agr.asset_attribute] on checkpoint 13: {transaction_id=null,
> ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
> gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
> row=0, event=0, server_id=1} 2025-12-05 17:11:27,957 INFO
> io.debezium.util.Threads [] - Creating thread
> debezium-mysqlconnector-mysql_binlog_source-binlog-client 2025-12-05
> 17:11:57,114 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset for tables [local_agr.application_setting, local_agr.asset,
> local_agr.asset_attribute] on checkpoint 14: {transaction_id=null,
> ts_sec=1764954296, file=mysql-bin.000003, pos=138668, kind=SPECIFIC,
> gtids=6d2eb014-d1fb-11f0-9e96-0242ac130006:1-16,af6ffbc8-7459-11f0-bd64-0242ac130005:1-729,
> row=0, event=0, server_id=1}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
