ricky2129 opened a new pull request, #12555:
URL: https://github.com/apache/seatunnel/pull/12555
### Purpose of this pull request
Closes #12554.
On a MySQL source that carries a GTID lineage inherited from a previous
master, a restored
MySQL-CDC job re-reads everything the server still retains for that lineage.
In our case four jobs
walked back about 100 binlog files and took roughly two and a half days to
catch up.
`MySqlStreamingChangeEventSource#filterGtidSet` drops any lineage the
restored GTID set does not
mention, and only `purgedServerGtid` brings it back, which reaches no
further than `gtid_purged`.
Everything the server retains past the purge point then looks unconsumed and
is sent again. A job
started with `startup.mode = specific` never records a lineage that went
quiet before its start
position, so it is exposed from its first checkpoint.
The `Previous_gtids` event of the binlog file being resumed from is the set
executed before that
file, which is the one thing that separates a dormant lineage from one that
has just become active
after a failover. This reads it and merges in only the lineages the
checkpoint does not track, so a
position already recorded is never overridden.
One detail worth flagging for review: `GtidUtils.mergeGtidSetInto` keeps the
entry already held for
a UUID rather than replacing it, so `Previous_gtids` has to be merged
**before** the purged set.
The other order silently discards it, since the purged set only reaches the
purge point.
If `SHOW BINLOG EVENTS` fails, for example when the account has no
`REPLICATION SLAVE` privilege,
the warning is logged and the merge behaves exactly as it does today rather
than blocking startup.
`GtidUtilsTest` covers the three cases that have to be told apart, and
asserts the old result as
well as the new one so the difference is visible:
| case | claimed before | claimed with this change |
|---|---|---|
| lineage missing from the checkpoint, executed earlier | `B:1-900` (purge
point) | `B:1-1000` |
| checkpoint tracks every lineage | unchanged | unchanged |
| lineage active after the resume position | not claimed | not claimed |
The same merge exists in upstream Debezium and is reported there as
debezium/dbz#2731, with a fix
in debezium/debezium#8135. SeaTunnel needs its own change because of the
merge-order difference
described above.
### Does this PR introduce _any_ user-facing change?
No new configuration. A job restored on a server carrying a lineage its
checkpoint does not track
no longer re-reads that lineage from the oldest retained binlog. Behaviour
is unchanged when the
checkpoint already tracks every lineage on the server.
### How was this patch tested?
`GtidUtilsTest`, four new cases alongside the two existing ones, all passing:
```
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0
```
`Previous_gtids` parsing was checked over JDBC against MySQL 8.4.5. The
value carries a newline
after the comma when more than one lineage is present, and is empty for the
oldest retained file.
### Check list
* [X] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [X] If necessary, please update the documentation to describe the new
feature.
https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md
* [X] If you are contributing the connector code, please check that the
following files are updated:
1. Update change log that in connector document. For more details you can
refer to
[connector-v2](https://github.com/apache/seatunnel/tree/dev/docs/en/connector-v2)
2. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
3. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
* [X] Update the
[`release-note`](https://github.com/apache/seatunnel/blob/dev/release-note.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
