DanielLeens commented on issue #10993: URL: https://github.com/apache/seatunnel/issues/10993#issuecomment-4671711222
Thanks, this makes the thread much clearer. The new Kafka `EXACTLY_ONCE` observation is a different path from the original `PENDING` question. For the original scheduling question: yes, your understanding is basically right. With `dynamic-slot: false` and `job-schedule-strategy: WAIT`, the worker-side slot capacity becomes the first bottleneck as the number of concurrent jobs grows. In that setup, more jobs will queue in `PENDING`, and even jobs that do finish can take much longer end-to-end because they spend more time waiting for available execution slots. So if the symptom is "the same task used to finish in 10 seconds, but now takes minutes after the cluster becomes busy", that is still consistent with scheduling pressure on the worker side rather than a master-only bottleneck. Separately, the Kafka sink/source behavior with `EXACTLY_ONCE` should be tracked as a different issue, because that is no longer about pending-job scheduling. If you can reproduce it stably, please open a separate issue with: 1. a minimal config for the Kafka sink job 2. a minimal config for the Kafka source job 3. the exact semantics settings on both sides 4. whether the consumer sees no committed records at all, or stalls only after a checkpoint / transaction boundary For this current thread, testing `dynamic-slot` and checking slot usage while the cluster is busy is the right next step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
