GitHub user karjaljo edited a discussion: Custom worker images to lower the number of kubernetespodoperator tasks
I've been running Airflow on distributed virtual machine environment and now I'm moving to OpenShift. Big chunk of my tasks are run using DockerOperator in the VM environment and I have switched those to be KubernetesPodOperator tasks. My processing is time sensitive so I can't use too strict pool sizes or other limiting conditions. Also, many of the pod runs are short lived, only few seconds. This leads to problems within the OpenShift cluster, since it can't handle too many pods too fast. One solution I was thinking, would be to build custom worker images that contain one or more applications. Since the workers would be always running and I could convert the KubernetesPodOperator tasks to regular tasks, the pod count would stay low. Is this possible and is there a better solution to this problem? So far it has been quite nice to have separate images for each application and building workers with multiple installations would also be a messy way to tackle this issue. GitHub link: https://github.com/apache/airflow/discussions/76535 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
