Fengt-en commented on issue #8213:
URL: https://github.com/apache/seatunnel/issues/8213#issuecomment-2550581922
> > > #7901
> >
> >
> > Please provide the detailed exception info, thx
>
>
任务是正常的，日志也没有出现异常信息，数据也可以正常同步，看起来整条任务链都是正常的，可是状态长期为busy.提供此issues后，我又从checkpoint方面入手调研，当我checkpoint时间间隔调整为5分钟，source算子也是busy状态，所以问题点也不在checkpoint方面
我可以提供一些执行日志，希望能从中获取一些信息
2024-12-18 15:25:54,567 INFO
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter -
subtaskIndex=1 rowIndex=1: SeaTunnelRow#tableId=test.test
SeaTunnelRow#kind=INSERT : 1, haha
2024-12-18 15:25:54,567 INFO
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter -
subtaskIndex=0 rowIndex=2: SeaTunnelRow#tableId=test.test
SeaTunnelRow#kind=INSERT : 4, dddddddd
2024-12-18 15:25:54,567 INFO
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter -
subtaskIndex=1 rowIndex=2: SeaTunnelRow#tableId=test.test
SeaTunnelRow#kind=INSERT : 3, bbbbbbbbbbb
2024-12-18 15:25:54,567 INFO
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter -
subtaskIndex=0 rowIndex=3: SeaTunnelRow#tableId=test.test
SeaTunnelRow#kind=INSERT : 6, aaa
2024-12-18 15:25:54,567 INFO
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter -
subtaskIndex=1 rowIndex=3: SeaTunnelRow#tableId=test.test
SeaTunnelRow#kind=INSERT : 5, eeeeeeeeeee
2024-12-18 15:25:54,567 INFO
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter -
subtaskIndex=0 rowIndex=4: SeaTunnelRow#tableId=test.test
SeaTunnelRow#kind=INSERT : 8, xxxxx
2024-12-18 15:25:54,567 INFO
org.apache.seatunnel.connectors.seatunnel.console.sink.ConsoleSinkWriter -
subtaskIndex=1 rowIndex=4: SeaTunnelRow#tableId=test.test
SeaTunnelRow#kind=INSERT : 7, xxxxx
2024-12-18 15:25:54,573 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator - Source Source:
MySQL-CDC-Source received split request from parallel task 0
2024-12-18 15:25:54,647 INFO
org.apache.seatunnel.connectors.seatunnel.common.source.reader.SourceReaderBase
- Received unhandled source event:
CompletedSnapshotSplitsAckEvent(completedSplits=[test.test:0])
2024-12-18 15:25:54,649 INFO
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader
- subtask 0 add splits: incremental-split-0
2024-12-18 15:25:54,650 INFO
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader
- The incremental split[incremental-split-0] startup position {ts_sec=0,
file=LAPTOP-5CC3T63L-bin.000335, pos=157, gtids=, row=0, event=0} is equal the
maxSnapshotSplitsHighWatermark {ts_sec=0, file=LAPTOP-5CC3T63L-bin.000335,
pos=157, gtids=, row=0, event=0}, auto enter pure increment phase.
2024-12-18 15:25:54,650 INFO
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceReader
- Clean the IncrementalSplit#completedSnapshotSplitInfos to empty.
2024-12-18 15:25:54,664 INFO
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader
- It's turn to read incremental split, close current snapshot fetcher.
2024-12-18 15:25:54,666 INFO
org.apache.seatunnel.connectors.cdc.base.source.enumerator.IncrementalSourceEnumerator
- Clean the SnapshotSplitAssigner#assignedSplits/splitCompletedOffsets to
empty.
2024-12-18 15:25:54,667 INFO io.debezium.jdbc.JdbcConnection - Connection
gracefully closed
2024-12-18 15:25:54,671 INFO
org.apache.seatunnel.connectors.cdc.base.source.reader.IncrementalSourceSplitReader
- Stream fetcher is created.
2024-12-18 15:25:54,669 WARN io.debezium.connector.mysql.MySqlConnection -
Database configuration option 'serverTimezone' is set but is obsolete, please
use 'connectionTimeZone' instead
2024-12-18 15:25:54,704 INFO io.debezium.connector.mysql.MySqlConnection -
Get all known binlogs from MySQL
2024-12-18 15:25:54,760 INFO
org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.reader.fetch.MySqlSourceFetchTaskContext
- MySQL has the binlog file 'LAPTOP-5CC3T63L-bin.000335' required by the
connector
2024-12-18 15:25:54,765 INFO
org.apache.seatunnel.connectors.cdc.base.source.reader.external.IncrementalSourceStreamFetcher
- Start incremental read task for incremental split:
org.apache.seatunnel.connectors.cdc.base.source.split.IncrementalSplit@dcd4a369
exactly-once: false
2024-12-18 15:25:54,777 INFO io.debezium.util.Threads - Requested thread
factory for connector MySqlConnector, id = mysql_binlog_source named =
binlog-client
2024-12-18 15:25:54,831 INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0 events on
streaming start
2024-12-18 15:25:54,831 INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Skip 0 rows on
streaming start
2024-12-18 15:25:54,833 INFO io.debezium.util.Threads - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client
2024-12-18 15:25:54,863 INFO io.debezium.util.Threads - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client
十二月 18, 2024 3:25:54 下午 com.github.shyiko.mysql.binlog.BinaryLogClient
connect
信息: Connected to localhost:3306 at LAPTOP-5CC3T63L-bin.000335/157
(sid:1745957373, cid:14)
2024-12-18 15:25:54,898 INFO io.debezium.jdbc.JdbcConnection - Connection
gracefully closed
2024-12-18 15:25:54,898 INFO
org.apache.seatunnel.connectors.seatunnel.cdc.mysql.source.reader.fetch.binlog.MySqlBinlogFetchTask
- Binlog client connected, closed idle jdbc connection.
2024-12-18 15:25:54,902 INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Connected to
MySQL binlog at localhost:3306, starting at MySqlOffsetContext
[sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT},
sourceInfo=SourceInfo [currentGtid=null,
currentBinlogFilename=LAPTOP-5CC3T63L-bin.000335, currentBinlogPosition=157,
currentRowNumber=0, serverId=0, sourceTime=null, threadId=-1,
currentQuery=null, tableIds=[], databaseName=null], snapshotCompleted=false,
transactionContext=TransactionContext [currentTransactionId=null,
perTableEventCount={}, totalEventCount=0], restartGtidSet=null,
currentGtidSet=null, restartBinlogFilename=LAPTOP-5CC3T63L-bin.000335,
restartBinlogPosition=157, restartRowsToSkip=0, restartEventsToSkip=0,
currentEventLengthInBytes=0, inTransaction=false, transactionId=null,
incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false,
chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null,
maximum
Key=null]]
2024-12-18 15:25:54,904 INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Waiting for
keepalive thread to start
2024-12-18 15:25:54,906 INFO io.debezium.util.Threads - Creating thread
debezium-mysqlconnector-mysql_binlog_source-binlog-client
2024-12-18 15:25:54,913 INFO
io.debezium.connector.mysql.MySqlStreamingChangeEventSource - Keepalive thread
is running
2024-12-18 15:25:59,918 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering
checkpoint 1 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1734506759896 for job
9d90f70b0b1f2918352c1a0072ffe088.
2024-12-18 15:26:00,079 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed
checkpoint 1 for job 9d90f70b0b1f2918352c1a0072ffe088 (4805 bytes,
checkpointDuration=173 ms, finalizationTime=10 ms).
2024-12-18 15:26:00,079 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator - Marking
checkpoint 1 as completed for source Source: MySQL-CDC-Source.
2024-12-18 15:26:09,891 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering
checkpoint 2 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1734506769890 for job
9d90f70b0b1f2918352c1a0072ffe088.
2024-12-18 15:26:09,948 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed
checkpoint 2 for job 9d90f70b0b1f2918352c1a0072ffe088 (4805 bytes,
checkpointDuration=55 ms, finalizationTime=3 ms).
2024-12-18 15:26:09,948 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator - Marking
checkpoint 2 as completed for source Source: MySQL-CDC-Source.
2024-12-18 15:26:19,891 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering
checkpoint 3 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1734506779891 for job
9d90f70b0b1f2918352c1a0072ffe088.
2024-12-18 15:26:19,937 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed
checkpoint 3 for job 9d90f70b0b1f2918352c1a0072ffe088 (4805 bytes,
checkpointDuration=46 ms, finalizationTime=0 ms).
2024-12-18 15:26:19,937 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator - Marking
checkpoint 3 as completed for source Source: MySQL-CDC-Source.
2024-12-18 15:26:29,892 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering
checkpoint 4 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1734506789891 for job
9d90f70b0b1f2918352c1a0072ffe088.
2024-12-18 15:26:29,898 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed
checkpoint 4 for job 9d90f70b0b1f2918352c1a0072ffe088 (4805 bytes,
checkpointDuration=7 ms, finalizationTime=0 ms).
2024-12-18 15:26:29,899 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator - Marking
checkpoint 4 as completed for source Source: MySQL-CDC-Source.
2024-12-18 15:26:39,892 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering
checkpoint 5 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1734506799891 for job
9d90f70b0b1f2918352c1a0072ffe088.
2024-12-18 15:26:39,995 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed
checkpoint 5 for job 9d90f70b0b1f2918352c1a0072ffe088 (4805 bytes,
checkpointDuration=104 ms, finalizationTime=0 ms).
2024-12-18 15:26:39,996 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator - Marking
checkpoint 5 as completed for source Source: MySQL-CDC-Source.
从这开始，日志会重复打印,自增checkpoint序列
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
