HuangZL-github commented on issue #10993: URL: https://github.com/apache/seatunnel/issues/10993#issuecomment-4676338692
> Thanks, this makes the thread much clearer. > > The new Kafka `EXACTLY_ONCE` observation is a different path from the original `PENDING` question. > > For the original scheduling question: yes, your understanding is basically right. With `dynamic-slot: false` and `job-schedule-strategy: WAIT`, the worker-side slot capacity becomes the first bottleneck as the number of concurrent jobs grows. In that setup, more jobs will queue in `PENDING`, and even jobs that do finish can take much longer end-to-end because they spend more time waiting for available execution slots. > > So if the symptom is "the same task used to finish in 10 seconds, but now takes minutes after the cluster becomes busy", that is still consistent with scheduling pressure on the worker side rather than a master-only bottleneck. > > Separately, the Kafka sink/source behavior with `EXACTLY_ONCE` should be tracked as a different issue, because that is no longer about pending-job scheduling. If you can reproduce it stably, please open a separate issue with: > > 1. a minimal config for the Kafka sink job > 2. a minimal config for the Kafka source job > 3. the exact semantics settings on both sides > 4. whether the consumer sees no committed records at all, or stalls only after a checkpoint / transaction boundary > > For this current thread, testing `dynamic-slot` and checking slot usage while the cluster is busy is the right next step. 你好，昨天将worker上的slot配置改成了动态slot，跑了一夜效果感觉没有静态slot好，因为任务多调度频率频繁，动态slot情况下，集群中会出现僵死任务，即curl -s http://$MASTER_IP:$HTTP_PORT/running-jobs | grep -c "UNKNOWABLE"，出现僵死任务需要重启集群才能继续跑任务，而静态slot暂时没出现这种情况。是不是动态slot其实没上限限制导致的。现在任务也从开始的10s左右，现在都基本在200~400s左右跑完。 这边seatunnel版本是2.3.11，查询系统监控情况：http://172.23.57.231:18080/system-monitoring-information，返回结果如下： `[{"isMaster":"true","host":"node01","port":"5801","processors":"10","physical.memory.total":"56.8G","physical.memory.free":"915.3M","swap.space.total":"0","swap.space.free":"0","heap.memory.used":"4.1G","heap.memory.free":"25.9G","heap.memory.total":"30.0G","heap.memory.max":"30.0G","heap.memory.used/total":"13.69%","heap.memory.used/max":"13.69%","minor.gc.count":"114","minor.gc.time":"26971ms","major.gc.count":"0","major.gc.time":"0ms","load.process":"9.13%","load.system":"43.98%","load.systemAverage":"4.56","thread.count":"221","thread.peakCount":"356","cluster.timeDiff":"0","event.q.size":"0","executor.q.async.size":"0","executor.q.client.size":"0","executor.q.client.query.size":"0","executor.q.client.blocking.size":"0","executor.q.query.size":"0","executor.q.scheduled.size":"0","executor.q.io.size":"0","executor.q.system.size":"0","executor.q.operations.size":"0","executor.q.priorityOperation.size":"0","operations.completed.count":"2216313","executor.q.mapLoad.size":"0","exe cutor.q.mapLoadAllKeys.size":"0","executor.q.cluster.size":"0","executor.q.response.size":"0","operations.running.count":"6","operations.pending.invocations.percentage":"0.00%","operations.pending.invocations.count":"0","proxy.count":"11","clientEndpoint.count":"0","connection.active.count":"4","client.connection.count":"0","connection.count":"2"},{"isMaster":"false","host":"node02","port":"5802","processors":"10","physical.memory.total":"46.9G","physical.memory.free":"3.8G","swap.space.total":"0","swap.space.free":"0","heap.memory.used":"7.6G","heap.memory.free":"12.4G","heap.memory.total":"20.0G","heap.memory.max":"20.0G","heap.memory.used/total":"37.91%","heap.memory.used/max":"37.91%","minor.gc.count":"279","minor.gc.time":"10603ms","major.gc.count":"0","major.gc.time":"0ms","load.process":"10.32%","load.system":"28.95%","load.systemAverage":"3.78","thread.count":"227","thread.peakCount":"434","cluster.timeDiff":"-356","event.q.size":"0","executor.q.async.size":"0","executor.q.c lient.size":"0","executor.q.client.query.size":"0","executor.q.client.blocking.size":"0","executor.q.query.size":"0","executor.q.scheduled.size":"0","executor.q.io.size":"0","executor.q.system.size":"0","executor.q.operations.size":"27","executor.q.priorityOperation.size":"0","operations.completed.count":"124918","executor.q.mapLoad.size":"0","executor.q.mapLoadAllKeys.size":"0","executor.q.cluster.size":"0","executor.q.response.size":"11","operations.running.count":"2","operations.pending.invocations.percentage":"0.00%","operations.pending.invocations.count":"0","proxy.count":"11","clientEndpoint.count":"0","connection.active.count":"4","client.connection.count":"0","connection.count":"2"},{"isMaster":"false","host":"node03","port":"5802","processors":"10","physical.memory.total":"41.0G","physical.memory.free":"4.7G","swap.space.total":"0","swap.space.free":"0","heap.memory.used":"7.6G","heap.memory.free":"12.4G","heap.memory.total":"20.0G","heap.memory.max":"20.0G","heap.memory.us ed/total":"38.21%","heap.memory.used/max":"38.21%","minor.gc.count":"282","minor.gc.time":"10609ms","major.gc.count":"0","major.gc.time":"0ms","load.process":"20.55%","load.system":"35.26%","load.systemAverage":"3.95","thread.count":"192","thread.peakCount":"467","cluster.timeDiff":"16348","event.q.size":"0","executor.q.async.size":"0","executor.q.client.size":"0","executor.q.client.query.size":"0","executor.q.client.blocking.size":"0","executor.q.query.size":"0","executor.q.scheduled.size":"0","executor.q.io.size":"0","executor.q.system.size":"0","executor.q.operations.size":"0","executor.q.priorityOperation.size":"0","operations.completed.count":"121027","executor.q.mapLoad.size":"0","executor.q.mapLoadAllKeys.size":"0","executor.q.cluster.size":"0","executor.q.response.size":"0","operations.running.count":"1","operations.pending.invocations.percentage":"0.00%","operations.pending.invocations.count":"0","proxy.count":"11","clientEndpoint.count":"0","connection.active.count":"4","c lient.connection.count":"0","connection.count":"2"}]` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
