nielifeng commented on issue #11472: URL: https://github.com/apache/seatunnel/issues/11472#issuecomment-5000059543
Additional validation for control-plane + worker failover: I tested a deterministic Kafka -> MySQL batch job with both the submitting master and one worker stopped during the job. Setup: - Kafka topic contains exactly `200000` visible records, `id = 1..200000` - Sink table has primary key on `id` - Sink query uses `insert ignore` for idempotent retry behavior - Submit job through `master-01` - After submit, stop `master-01` and one worker - Observe job through another master - Restart both stopped nodes after the job finishes Result: ```text jobStatus = FINISHED errorMsg = null MySQL sink: count = 200000 distinct_ids = 200000 min_id = 1 max_id = 200000 missing_in_range = 0 Finished job metrics: SourceReceivedCount = 292908 SinkWriteCount = 292024 SinkCommittedCount = 200000 IntermediateQueueSize = 884 ``` For comparison, master-only failover did not reproduce the metrics problem: - Kafka -> MySQL, submitting master stopped: metrics stayed `200000 / 200000 / 200000`, queue `0` - MySQL -> MySQL, submitting master stopped: metrics stayed `200000 / 200000 / 200000`, queue `0` So the inaccurate finished metrics appear when a worker task attempt is canceled/retried, including master+worker combined failover. Master-only failover looks healthy in these deterministic tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
