xinnyuli commented on issue #12382: URL: https://github.com/apache/seatunnel/issues/12382#issuecomment-5978110950
Here is the real-run comparison you asked for. Setup: the same PostgresCDCIT file (taken from 146a1b5c, the PR's base) run against two production revisions, so only the connector code differs: the PR base 146a1b5c and #12454's head 3080371c. Both on GitHub-hosted ubuntu-latest, Java 11, stress-ng on, no tracing agent, the three Debezium loggers at INFO in the test log4j2.properties, plus one test-only change that logs pg_current_wal_insert_lsn()/pg_current_wal_lsn() right before and after the id=15 insert. No production code changed. Before (146a1b5c): 30 runs reached the id=15 check, 3 missing. After (#12454, 3080371c): 28 runs reached the check, 0 missing (2 setup timeouts at the committed-LSN wait, not counted). The three values side by side. In every missing-row run the equality holds: | run | stored offset (lsn_proc = lsn_commit) | first LSN on restart | LSN filtered as "already processed" | id=15 WAL range (insert before -> after) | resume - stored | |---|---|---|---|---|---| | before n7 (missing) | 0/222A058 | 0/222A058 | 0/222A058 | 0/222A058 -> 0/222A190 | 312 | | before n18 (missing) | 0/2228138 | 0/2228138 | 0/2228138 | 0/2228138 -> 0/2228270 | 312 | | before n25 (missing) | 0/2227B38 | 0/2227B38 | 0/2227B38 | 0/2227B38 -> 0/2227C70 | 312 | | after n4 (delivered) | 0/222A318 | 0/222A318 | none | 0/222A318 -> 0/222A450 | 0 | | after n30 (delivered) | 0/222D3A8 | 0/222D3A8 | none | 0/222D3A8 -> 0/222D4E0 | 0 | So in the failing runs the id=15 transaction starts exactly at the stored commit-end LSN, and that LSN is the one filtered as already processed. None of the 27 passing "before" runs hit this boundary. With #12454, the two runs that hit the same boundary resumed at the stored LSN, filtered nothing there, and delivered id=15. One "after" run (n10) also filtered a message at its stored LSN 0/221F028, but it was a single message with nothing else on the stream until 0/2227898, and id=15 started later at 0/22278D8 and was delivered. So that was not id=15 and no row was dropped. Caveats: 3/30 vs 0/28 alone is not statistically significant (one-sided Fisher p ~0.13), so I'd lean on the boundary-hit rows rather than the rate. Java 11 only. I did not run the PR's own extended IT here. Runs: before https://github.com/xinnyuli/seatunnel-12382-repro/actions/runs/37185551736 , after https://github.com/xinnyuli/seatunnel-12382-repro/actions/runs/37187621867 As @DanielLeens noted on #12454, the existing IT doesn't force this boundary deterministically. If useful, I'd like to follow up with a deterministic regression test for it as a separate PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
