SEPURI-SAI-KRISHNA commented on issue #13137:
URL: https://github.com/apache/seatunnel/issues/13137#issuecomment-6682629726
I reproduced this on current `dev` (`df59bb4cd1`) without MySQL. In this
reproduction the extra instances are not retained task group contexts. I do not
plan to open a PR.
Job: FakeSource (`parallelism = 5`, `split.num = 5`, `row.num = 100000000`,
`split.read-interval = 100`) -> Sql -> Console, streaming, realtime
observability on, default `queue-type`, JDK 11. `subtaskCount` after each
same-ID savepoint stop and restore, no failover:
```
start 1st 2nd 3rd 4th
1 node 5 6 7 9 9
1 node 5 8 8 9 9
2 nodes 5 7 9 9 9
2 nodes 5 5 6 6 -
```
Row 3 shows the same `7 / 14336` as the report. Row 2 had logging added. At
every probe the task mapping had 5 queue groups and a class histogram showed 6
live `TaskGroupContext` objects in the cluster: those 5 plus the enumerator.
**Cause.** `getQueueCache` registers the queue counters on the metrics
context of the task that reaches it first. A group's two tasks initialize on
separate worker threads, so the winner can change per attempt.
`SeaTunnelTask.init()` reloads each task's context from the metrics snapshot
store, which `removeMetricsContext` keeps on a savepoint end. If the other task
wins after a restore, the earlier owner still reports its old counters, so the
group has two rows for one queue. No worker move is needed.
I logged the registrations and the rows seen by `RawAggConsumer.consumeLong`
on 1 node through 5 restores. The number of tasks that had ever registered
matched `subtaskCount` in all 6 attempts (5, 6, 7, 7, 9, 9). Size rows of tasks
that did not register in that attempt stayed at 0, so `queueFillRatio` and
`bpRatio` read low: 5/7 of their value when the count is 7.
**Possible fix.** A throwaway prototype removes `IntermediateQueueSize*`,
`IntermediateQueuePutBlockedNs#*` and `IntermediateQueueCapacity#*` in
`getOrCreateMetricsContext`, since that queue no longer exists. One run on 2
nodes stayed at `5 / 10240` for all 7 restores. Not a tested patch.
`--get_running_job_metrics` merges job history and still lists the old rows, so
that view needs its own decision.
**Limits.** I did not run MySQL CDC or its idle reader close, so I cannot
say this explains both extra instances in the report. The disruptor queue
registers the same way, but I did not run it.
Client arguments:
```
-c job.conf --async --set-job-id 13137
-s 13137
-c job.conf -r 13137 --async
```
I can post the config and raw responses if useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]