Hi Kamal, Autoscaling consists of two parts: 1. policy: when should we scale up/down the job and for how much? 2. mechanism: how to rescale the job. The adaptive scheduler just gives you the mechanism part, which allows rescaling the job without kill-and-start the whole job. You shall use it together with the operator or K8s HPA which provides the policy part (there's a subtlety that K8s HPA needs to be used with the Reactive Mode [1], a special mode for Adaptive Scheduler).
It is totally fine to apply Autoscaling without the K8s Operator at the cost of extra complexity, e.g. the just mentioned K8s HPA + the Reactive Mode pair. However, the K8s HPA policy is more targeted towards stateless workloads without complex internal topology, and it does not perform quite well on Flink. The K8s Operator's approach [2] is more suited. You can mimic its algorithm to get the suggested operator parallelisms, and use [3] to apply the new parallelisms. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#reactive-mode [2] https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ [3] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#externalized-declarative-resource-management Best, Zhanghao Chen ________________________________ From: Kamal Mittal via user <user@flink.apache.org> Sent: Monday, May 12, 2025 13:35 To: user@flink.apache.org <user@flink.apache.org> Subject: Flink task manager PODs autoscaling - K8s installation Hello, For Flink Task Manager K8s PODs autoscaling, it is mandatory to use Flink Kubernetes Operator/HPA or only flink provided scheduler is sufficient as below? jobmanager.scheduler: adaptive Also what is the additional work done by below configuration? job.autoscaler.enabled: "true" Please give documentation link if available without Flink Kubernetes Operator? Rgds, Kamal
