xinnyuli commented on issue #12382:
URL: https://github.com/apache/seatunnel/issues/12382#issuecomment-5908833957
A correction and some upstream context for my previous comment.
The dbz#76 / PG 17+ reference was wrong. The related upstream change is
dbz#1554 ("Fix data loss on restart when multiple events share the same LSN",
PR #7531, commit a51931d92). It adds a new offset field
(lastProcessedMessageType) so WalPositionLocator can tell what the stored LSN
refers to. That issue is triggered by several messages sharing one LSN (COPY),
not by a commit end LSN equal to the next record's start, so I can't say yet
that it covers this case exactly. The change first appears in 3.7.0.Alpha1
(3.6.3 does not have it), and 3.7 connectors target Java 17, while SeaTunnel
pins 1.9.8.Final and still runs on Java 8/11.
Possible directions, for discussion only (no PR from me until you decide):
1. Upgrade Debezium: would bring the dbz#1554 change, but 3.7 needs Java 17,
so it doesn't fit while SeaTunnel supports Java 8/11.
2. Port a similar guard: store something extra in the offset so a stored
commit-end LSN isn't treated as an already processed change. This changes the
offset/savepoint format, so old savepoints need a compatibility path.
3. A narrower SeaTunnel-side workaround on restore, so a message at exactly
the stored commit-end LSN isn't skipped. Smaller change, but it needs care not
to replay events that really were processed.
Whichever you prefer, I think the first step is a deterministic test that
puts the id=15 INSERT right at the restored LSN, so it fails every time instead
of ~3% under load. Happy to write that test if you agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
