lzshlzsh commented on code in PR #3845:
URL: https://github.com/apache/flink-cdc/pull/3845#discussion_r2298111455
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/io/debezium/connector/mysql/GtidUtils.java:
##########
@@ -36,36 +38,64 @@ public class GtidUtils {
public static GtidSet fixRestoredGtidSet(GtidSet serverGtidSet, GtidSet
restoredGtidSet) {
Map<String, GtidSet.UUIDSet> newSet = new HashMap<>();
serverGtidSet.getUUIDSets().forEach(uuidSet ->
newSet.put(uuidSet.getUUID(), uuidSet));
- for (GtidSet.UUIDSet uuidSet : restoredGtidSet.getUUIDSets()) {
- GtidSet.UUIDSet serverUuidSet = newSet.get(uuidSet.getUUID());
+ for (GtidSet.UUIDSet restoredUuidSet : restoredGtidSet.getUUIDSets()) {
+ GtidSet.UUIDSet serverUuidSet =
newSet.get(restoredUuidSet.getUUID());
if (serverUuidSet != null) {
- long restoredIntervalEnd = getIntervalEnd(uuidSet);
- List<com.github.shyiko.mysql.binlog.GtidSet.Interval>
newIntervals =
- new ArrayList<>();
- for (GtidSet.Interval serverInterval :
serverUuidSet.getIntervals()) {
- if (serverInterval.getEnd() <= restoredIntervalEnd) {
- newIntervals.add(
- new
com.github.shyiko.mysql.binlog.GtidSet.Interval(
- serverInterval.getStart(),
serverInterval.getEnd()));
- } else if (serverInterval.getStart() <= restoredIntervalEnd
- && serverInterval.getEnd() > restoredIntervalEnd) {
- newIntervals.add(
+ List<GtidSet.Interval> serverIntervals =
serverUuidSet.getIntervals();
+ List<GtidSet.Interval> restoredIntervals =
restoredUuidSet.getIntervals();
+
+ long earliestRestoredTx =
getMinIntervalStart(restoredIntervals);
+
+ List<com.github.shyiko.mysql.binlog.GtidSet.Interval> merged =
new ArrayList<>();
+
+ // Process each server interval
+ for (GtidSet.Interval serverInterval : serverIntervals) {
+ // First, check if any part comes before earliest restored
+ if (serverInterval.getStart() < earliestRestoredTx) {
+ long end = Math.min(serverInterval.getEnd(),
earliestRestoredTx - 1);
+ merged.add(
new
com.github.shyiko.mysql.binlog.GtidSet.Interval(
- serverInterval.getStart(),
restoredIntervalEnd));
+ serverInterval.getStart(), end));
Review Comment:
> There is also a risk of data loss here as we cannot confirm whether this
gtid has been executed(although the probability seems very small), right?
Yes, you are right, we can construct the following data loss case:
```
mysql> show master status;
+------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
Executed_Gtid_Set
|
+------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
| mysql-bin.000197 | 237 | | |
ad268c5d-2f18-11ef-8eac-0242ac120003:9-12,
fc992a75-c2a9-11ee-82e7-0242ac120004:1-182571 |
+------------------+----------+--------------+------------------+------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> set gtid_next='fc992a75-c2a9-11ee-82e7-0242ac120004:182580';
Query OK, 0 rows affected (0.01 sec)
mysql> update full_types_bitsail_2x set tiny_c = 1 where id = 10;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> set gtid_next='fc992a75-c2a9-11ee-82e7-0242ac120004:182579';
Query OK, 0 rows affected (0.00 sec)
mysql> update full_types_bitsail_2x set tiny_c = 2 where id = 10;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
-- do a savepoint:
{"transaction_id":null,"ts_sec":"1756127179","file":"mysql-bin.000197","pos":"634","kind":"SPECIFIC","gtids":"fc992a75-c2a9-11ee-82e7-0242ac120004:182580-182580","row":"1","event":"2","server_id":"123"}
mysql> set gtid_next='fc992a75-c2a9-11ee-82e7-0242ac120004:182578';
Query OK, 0 rows affected (0.00 sec)
mysql> update full_types_bitsail_2x set tiny_c = 3 where id = 10;
Query OK, 1 row affected (0.02 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> show master status;
+------------------+----------+--------------+------------------+--------------------------------------------------------------------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
Executed_Gtid_Set
|
+------------------+----------+--------------+------------------+--------------------------------------------------------------------------------------------------------+
| mysql-bin.000197 | 1191 | | |
ad268c5d-2f18-11ef-8eac-0242ac120003:9-12,
fc992a75-c2a9-11ee-82e7-0242ac120004:1-182571:182578-182580 |
+------------------+----------+--------------+------------------+--------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
-- restore from savepoint, update set tiny_c = 3 lost, and because of
"event":"2", 2 following events are dropped also.
GTID set available on server:
ad268c5d-2f18-11ef-8eac-0242ac120003:9-12,fc992a75-c2a9-11ee-82e7-0242ac120004:1-182571:182578-182580
Final merged GTID set to use when connecting to MySQL:
ad268c5d-2f18-11ef-8eac-0242ac120003:9-12,fc992a75-c2a9-11ee-82e7-0242ac120004:1-182571:182578-182580
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
