nielifeng commented on issue #11472: URL: https://github.com/apache/seatunnel/issues/11472#issuecomment-4999996342
Additional validation: this is not specific to Kafka source. I also reproduced the same finished-job metrics inconsistency with a deterministic MySQL -> MySQL batch job. Setup: - Source table: `id = 1..200000`, primary key on `id` - Sink table: primary key on `id` - Job: JDBC source -> JDBC sink - `parallelism = 2` - `checkpoint.interval = 10000` - JDBC source uses `table_path` with `split.size = 5000` - During the job, one Zeta worker was stopped with `systemctl stop seatunnel-worker` No-failover baseline: ```text jobStatus = FINISHED SourceReceivedCount = 200000 SinkWriteCount = 200000 SinkCommittedCount = 200000 IntermediateQueueSize = 0 MySQL sink: count = 200000 distinct_ids = 200000 min_id = 1 max_id = 200000 missing_in_range = 0 ``` Worker failover run: ```text jobStatus = FINISHED errorMsg = null SourceReceivedCount = 250074 SinkWriteCount = 248024 SinkCommittedCount = 200000 IntermediateQueueSize = 2049 MySQL sink: count = 200000 distinct_ids = 200000 min_id = 1 max_id = 200000 missing_in_range = 0 ``` So the target data is still correct, but the finished job metrics over-count source/sink records and leave a non-zero `IntermediateQueueSize`. This suggests the issue is likely in Zeta metrics aggregation / task attempt cleanup after worker failover, rather than in Kafka source itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
