github-actions[bot] commented on code in PR #65645:
URL: https://github.com/apache/doris/pull/65645#discussion_r3594640110
##########
regression-test/suites/job_p0/streaming_job/cdc/test_streaming_mysql_job_snapshot_with_concurrent_dml.groovy:
##########
@@ -79,27 +98,54 @@
suite("test_streaming_mysql_job_snapshot_with_concurrent_dml", "p0,external,mysq
"user" = "root",
"password" = "123456",
"database" = "${mysqlDb}",
- "include_tables" = "${table1}",
+ "include_tables" = "${table1},${table2}",
"offset" = "initial",
"snapshot_split_size" = "10",
- "snapshot_parallelism" = "1"
+ "snapshot_parallelism" = "1",
+ "skip_snapshot_backfill" = "false"
)
TO DATABASE ${currentDb} (
"table.create.properties.replication_num" = "1"
)
"""
- // Concurrent DML on source while cdc-client is still snapshotting.
+ def succeedTaskCount = {
+ def rows = sql """select SucceedTaskCount from
jobs("type"="insert") where Name='${jobName}' and ExecuteType='STREAMING'"""
+ rows.size() == 1 ? (rows.get(0).get(0).toString() as long) : 0L
+ }
+ Awaitility.await().atMost(120, SECONDS).pollInterval(100,
MILLISECONDS).until({
+ succeedTaskCount() >= 1
+ })
+
+ def writeProbeDml = {
+ connect("root", "123456",
"jdbc:mysql://${externalEnvIp}:${mysql_port}") {
+ sql """UPDATE ${mysqlDb}.${table1} SET version=version+1 WHERE
id=500"""
+ sql """UPDATE ${mysqlDb}.${table2} SET version=version+1 WHERE
other_id=500"""
+ }
+ }
+
+ writeProbeDml()
+ long probeStartTaskCount = succeedTaskCount()
+ long probeTargetTaskCount = probeStartTaskCount + 10
+ Awaitility.await().atMost(120, SECONDS).pollInterval(200,
MILLISECONDS).until({
+ writeProbeDml()
Review Comment:
The probe loop can still pass without putting a foreign-table record between
a split's low and high watermarks. `SucceedTaskCount` advances only after a
task commits, and the next snapshot task is merely created afterward; each
`writeProbeDml()` runs before reading the counter, so it can land in the
inter-task gap while several short 10-row splits finish before the next 200 ms
poll. Those updates precede the next low watermark and never enter snapshot
backfill, allowing the old all-schema behavior to pass. Please keep a writer
continuously active until the task-count window closes, or add a hook/barrier
proving a split captured its low watermark before issuing the foreign-table
update. The PostgreSQL loop needs the same guarantee.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
