[
https://issues.apache.org/jira/browse/FLINK-36573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Runkang He updated FLINK-36573:
-------------------------------
Description:
For mysql binlog split, currently when checkpointing, we only print the binlog
offset in logs, and difficult to know which table the offset belongs to,
especially when there are multiple mysql cdc sources in one job.
{code:java}
2024-09-27 18:16:10,488 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset on checkpoint 4: {transaction_id=null, ts_sec=0,
file=mysql-binlog.000666, pos=147223270, kind=SPECIFIC,
gtids=e6e6e8e3-6351-11ef-8340-52540bd1ce99:81433397-81559559, row=0, event=20,
server_id=1}
{code}
So it's very useful to add table information of binlog offsets, to help user
clearly see the binlog reading progress of each table. Another benefit is that
when the job topology modification causes job state incompatibility, users can
use the binlog file and position information here to restart from the
specific-offsets mode to recover.
{code:java}
2024-09-27 18:16:10,488 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [test_db.test] on checkpoint 4: {transaction_id=null,
ts_sec=0, file=mysql-binlog.000666, pos=147223270, kind=SPECIFIC,
gtids=e6e6e8e3-6351-11ef-8340-52540bd1ce99:81433397-81559559, row=0, event=20,
server_id=1}
{code}
was:
For mysql binlog split, currently when checkpointing, we only print the binlog
offset in logs, and difficult to know which table the offset belongs to,
especially when there are multiple mysql cdc sources in one job.
{code:java}
2024-09-27 18:16:10,488 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset on checkpoint 4: {transaction_id=null, ts_sec=0,
file=mysql-binlog.000666, pos=147223270, kind=SPECIFIC,
gtids=e6e6e8e3-6351-11ef-8340-52540bd1ce99:81433397-81559559, row=0, event=20,
server_id=1}
{code}
So it's very useful to add table information of binlog offsets, to help user
clearly see the binlog reading progress of each table. Another benefit is that
when the job topology modification causes job status incompatibility, users can
use the binlog file and position information here to restart from the
specific-offsets mode to recover.
{code:java}
2024-09-27 18:16:10,488 INFO
org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
Binlog offset for tables [test_db.test] on checkpoint 4: {transaction_id=null,
ts_sec=0, file=mysql-binlog.000666, pos=147223270, kind=SPECIFIC,
gtids=e6e6e8e3-6351-11ef-8340-52540bd1ce99:81433397-81559559, row=0, event=20,
server_id=1}
{code}
> Add table information of binlog offsets when checkpointing
> ----------------------------------------------------------
>
> Key: FLINK-36573
> URL: https://issues.apache.org/jira/browse/FLINK-36573
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.2.1
> Reporter: Runkang He
> Priority: Minor
> Labels: pull-request-available
> Fix For: cdc-3.2.1
>
>
> For mysql binlog split, currently when checkpointing, we only print the
> binlog offset in logs, and difficult to know which table the offset belongs
> to, especially when there are multiple mysql cdc sources in one job.
> {code:java}
> 2024-09-27 18:16:10,488 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset on checkpoint 4: {transaction_id=null, ts_sec=0,
> file=mysql-binlog.000666, pos=147223270, kind=SPECIFIC,
> gtids=e6e6e8e3-6351-11ef-8340-52540bd1ce99:81433397-81559559, row=0,
> event=20, server_id=1}
> {code}
> So it's very useful to add table information of binlog offsets, to help user
> clearly see the binlog reading progress of each table. Another benefit is
> that when the job topology modification causes job state incompatibility,
> users can use the binlog file and position information here to restart from
> the specific-offsets mode to recover.
> {code:java}
> 2024-09-27 18:16:10,488 INFO
> org.apache.flink.cdc.connectors.mysql.source.reader.MySqlSourceReader [] -
> Binlog offset for tables [test_db.test] on checkpoint 4:
> {transaction_id=null, ts_sec=0, file=mysql-binlog.000666, pos=147223270,
> kind=SPECIFIC, gtids=e6e6e8e3-6351-11ef-8340-52540bd1ce99:81433397-81559559,
> row=0, event=20, server_id=1}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
