hterik commented on issue #19135: URL: https://github.com/apache/airflow/issues/19135#issuecomment-1029776459
Did you see the [CeleryKubernetes executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/celery_kubernetes.html) ? That allows mixing K8 with celery Can also recommend `["executor_config"]["pod_override"]` on taskflow api if you have need to specify exact dependencies per task. We have custom wrappers on top of this so you just have to write: ``` @node_type=HighPerformance @task def foo(): .... ``` Actually it is a lot more granular than this but you get the idea. For small lightweight sensors there is possibility to batch them into https://airflow.apache.org/docs/apache-airflow/stable/concepts/smart-sensors.html or https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html Not against your idea of a custom operator, just handing out tips :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
