gyfora opened a new pull request, #227: URL: https://github.com/apache/flink-kubernetes-operator/pull/227
This is a proof of concept of introducing Kubernetes scaling capabilities and support for using the Horizontal Pod Autoscaler for application deployments. **Background:** https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#scale-subresource https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/ **Scaling for Flink applications:** Our initial idea was to use the parallelism as the scaling unit/parameter. This is however not a really good match for the Kubernetes scaling behaviour, where you have a concept of replicas and watch pods/containers to decide whether to scale up or down. By using parallelism you will end up with confusing terminology and half used task managers (where not all the slots are taken) We decided it might be better to use the number of taskmanager pods as the replica count for scaling purposes as it more closely matches what is actually happening under the hood **In this PR:** We introduce `taskManager.replicas` in the spec, which is by default `null`. When defined it takes precedence over the job parallelism setting and the parallelism is computed using `replicas * numTaskSlots`. We also expose the taskmanager replicas together with the pod label selector in the status (this is necessary for the scale subresource). After setting the correct annotations the generate subresource looks like this: ``` subresources: scale: labelSelectorPath: .status.taskManager.labelSelector specReplicasPath: .spec.taskManager.replicas statusReplicasPath: .status.taskManager.replicas ``` With this we can now crate a job and hpa: <img width="1035" alt="image" src="https://user-images.githubusercontent.com/5880972/169222154-eac7c296-1b05-4a59-b897-36688d080fe8.png";> The HorizontalPodAutoscaler policy will continuously monitor the TM pods and scale the number up/down. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://user-images.githubusercontent.com/5880972/169222154-eac7c296-1b05-4a59-b897-36688d080fe8.png"](https://user-images.githubusercontent.com/5880972/169222154-eac7c296-1b05-4a59-b897-36688d080fe8.png"){kind=link}