c-thiel commented on issue #19135: URL: https://github.com/apache/airflow/issues/19135#issuecomment-1029155560
@hterik I also think the Scheduler bit can be good inspiration. I just want to address your first point briefly: I really do think that there are many use-cases for this: Having an operator allows much more freedoms than using the KubernetesExecutor only - for example in specifying the exact dependencies you need for individual tasks. Additionaly, spawning pods for every little thing really might stress the cluster. In our deployment we are runnig ~20 thousand tasks each morning in an hour and then a few more over the day. We tried the KubernetesExecutor but having to schedule a new pod for every sensor and mundane task was a bit too much for our control plane. Thus we are using Celery with lots of KubernetesOperator with self implemented taskflow API. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
