MOBIN created FLINK-35058:
-----------------------------
Summary: Encountered change event for table db.table whose schema
isn't known to this connector
Key: FLINK-35058
URL: https://issues.apache.org/jira/browse/FLINK-35058
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: 1.17.1
Reporter: MOBIN
Flink1.17.1
flink-cdc：flink-sql-connector-mysql-cdc-2.4.1.jar
{code:java}
CREATE TABLE `test_cdc_timestamp` (
`id` BIGINT COMMENT '主键id',
....
proctime AS PROCTIME(),
PRIMARY KEY(id) NOT ENFORCED
) WITH (
'connector' = 'mysql-cdc',
'hostname' = 'xxxxx',
'scan.startup.mode' = 'timestamp',
'scan.startup.timestamp-millis' = '1712419200000' ,
'port' = '3306',
'username' = 'xxx',
'password' = 'xxx',
'database-name' = 'xxtablename',
'table-name' = 'xxdatabase',
'scan.incremental.snapshot.enabled' = 'false',
'debezium.snapshot.locking.mode' = 'none',
'server-id' = '5701',
'server-time-zone' = 'Asia/Shanghai',
'debezium.skipped.operations' = 'd'
); {code}
When I use 'scan.startup.mode' = 'latent-offset 'or'initial' to synchronize
data normally, when I use 'scan.startup.mode' = 'timestamp', the following
error is reported
{code:java}
2024-04-09 11:11:15.619 [debezium-engine] INFO io.debezium.util.Threads -
Requested thread factory for connector MySqlConnector, id = mysql_binlog_source
named = change-event-source-coordinator
2024-04-09 11:11:15.621 [debezium-engine] INFO io.debezium.util.Threads -
Creating thread
debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator
2024-04-09 11:11:15.629
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Metrics registered
2024-04-09 11:11:15.630
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Context created
2024-04-09 11:11:15.642
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.connector.mysql.MySqlSnapshotChangeEventSource - No previous
offset has been found
2024-04-09 11:11:15.642
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.connector.mysql.MySqlSnapshotChangeEventSource - According
to the connector configuration only schema will be snapshotted
2024-04-09 11:11:15.644
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Snapshot ended with
SnapshotResult [status=SKIPPED, offset=null]
2024-04-09 11:11:15.652
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.util.Threads - Requested thread factory for connector
MySqlConnector, id = mysql_binlog_source named = binlog-client
2024-04-09 11:11:15.656
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Starting streaming
2024-04-09 11:11:15.682
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - GTID set
purged on server:
0969640a-1d48-11ed-b6cf-28dee561557c:1-27603868993,70958f24-2253-11eb-891d-f875a48ad7b1:1-50323,ec1e6593-2251-11eb-9c18-f875a48ad539:1-25345454762
2024-04-09 11:11:15.682
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0
events on streaming start
2024-04-09 11:11:15.682
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0
rows on streaming start
2024-04-09 11:11:15.683
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.util.Threads - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client
2024-04-09 11:11:15.686 [blcxxxx.mysql.com:3306] INFO io.debezium.util.Threads
- Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2024-04-09 11:11:15.700 [blcxxxx.mysql.com:3306] INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Connected to
MySQL binlog at xxx.mysql.com:3306, starting at MySqlOffsetContext
[sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT},
sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=,
currentBinlogPosition=0, currentRowNumber=0, serverId=0, sourceTime=null,
threadId=-1, currentQuery=null, tableIds=[], databaseName=null],
snapshotCompleted=false, transactionContext=TransactionContext
[currentTransactionId=null, perTableEventCount={}, totalEventCount=0],
restartGtidSet=null, currentGtidSet=null, restartBinlogFilename=,
restartBinlogPosition=0, restartRowsToSkip=0, restartEventsToSkip=0,
currentEventLengthInBytes=0, inTransaction=false, transactionId=null,
incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false,
chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null,
maximumKey=null]]
2024-04-09 11:11:15.700
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Waiting
for keepalive thread to start
2024-04-09 11:11:15.701 [blcxxxx.mysql.com:3306] INFO io.debezium.util.Threads
- Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2024-04-09 11:11:15.701
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Keepalive
thread is running
2024-04-09 11:11:15.818 [blcxxxx.mysql.com:3306] ERROR
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Encountered
change event 'Event{header=EventHeaderV4{timestamp=1711939504000,
eventType=TABLE_MAP, serverId=431777, headerLength=19, dataLength=81,
nextPosition=981190, flags=0}, data=TableMapEventData{tableId=933966,
database='xxxdatabase', table='xxxtablename', columnTypes=8, 15, 8, 15, 8, 3,
2, 1, -11, 18, 18, 1, columnMetadata=0, 400, 0, 1024, 0, 0, 0, 0, 4, 0, 0, 0,
columnNullability={4, 6, 8, 11}, eventMetadata=null}}' at offset
{transaction_id=null, file=mysql-bin.016482, pos=981027,
gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019,
server_id=431777, event=1} for table xxxdatabase.xxxtablename whose schema
isn't known to this connector. One possible cause is an incomplete database
history topic. Take a new snapshot in this case.
Use the mysqlbinlog tool to view the problematic event: mysqlbinlog
--start-position=981090 --stop-position=981190 --verbose mysql-bin.016482
2024-04-09 11:11:15.819 [blcxxxx.mysql.com:3306] ERROR
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error during
binlog processing. Last offset stored = {transaction_id=null,
file=mysql-bin.016482, pos=981027,
gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019,
server_id=431777, event=1}, binlog reader near position =
mysql-bin.016482/981090
2024-04-09 11:11:15.821 [blcxxxx.mysql.com:3306] ERROR
io.debezium.pipeline.ErrorHandler - Producer failure
io.debezium.DebeziumException: Error processing binlog event
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:429)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:1095)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
at
com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
at java.lang.Thread.run(Thread.java:745)
Caused by: io.debezium.DebeziumException: Encountered change event for table
xxdatabase.xxtablename whose schema isn't known to this connector
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:753)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:819)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:720)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:1046)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:408)
... 6 common frames omitted
2024-04-09 11:11:15.821 [blcxxxx.mysql.com:3306] INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Error processing
binlog event, and propagating to Kafka Connect so it stops this connector.
Future binlog events read before connector is shutdown will be ignored.
2024-04-09 11:11:16.124 [debezium-engine] INFO
io.debezium.embedded.EmbeddedEngine - Stopping the task and engine
2024-04-09 11:11:16.124 [debezium-engine] INFO
io.debezium.connector.common.BaseSourceTask - Stopping down connector
2024-04-09 11:11:16.216
[debezium-mysqlconnector-mysql_binlog_source-change-event-source-coordinator]
INFO io.debezium.pipeline.ChangeEventSourceCoordinator - Finished streaming
2024-04-09 11:11:16.216 [blcxxxx.mysql.com:3306] INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Stopped reading
binlog after 0 events, last recorded offset: {transaction_id=null,
file=mysql-bin.016482, pos=981027,
gtids=0969640a-1d48-11ed-b6cf-28dee561557c:27603868994-27603870019,
server_id=431777, event=71904}
2024-04-09 11:11:16.218 [pool-10-thread-1] INFO
io.debezium.jdbc.JdbcConnection - Connection gracefully closed
2024-04-09 11:11:16.221 [debezium-engine] ERROR
com.ververica.cdc.debezium.internal.Handover - Reporting error:
com.ververica.cdc.connectors.shaded.org.apache.kafka.connect.errors.ConnectException:
An exception occurred in the change event producer. This connector will be
stopped.
at
io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:50)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:429)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$25(MySqlStreamingChangeEventSource.java:1095)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1246)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:1072)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:631)
at
com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:932)
at java.lang.Thread.run(Thread.java:745)
Caused by: io.debezium.DebeziumException: Error processing binlog event
... 7 common frames omitted
Caused by: io.debezium.DebeziumException: Encountered change event for table
xxdatabase.xxtablename whose schema isn't known to this connector
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:753)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.informAboutUnknownTableIfRequired(MySqlStreamingChangeEventSource.java:819)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleUpdateTableMetadata(MySqlStreamingChangeEventSource.java:720)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.lambda$execute$13(MySqlStreamingChangeEventSource.java:1046)
at
io.debezium.connector.mysql.MySqlStreamingChangeEventSource.handleEvent(MySqlStreamingChangeEventSource.java:408)
... 6 common frames omitted
2024-04-09 11:11:16.223 [Source: source_ascribe_all_result_data[1] -> Calc[2]
-> Sink: print[3] (1/1)#0] INFO io.debezium.embedded.EmbeddedEngine -
Stopping the embedded engine
2024-04-09 11:11:16.227 [Source: source_ascribe_all_result_data[1] -> Calc[2]
-> Sink: print[3] (1/1)#0] INFO io.debezium.embedded.EmbeddedEngine -
Stopping the embedded engine
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
