mik-laj edited a comment on issue #9755: URL: https://github.com/apache/airflow/issues/9755#issuecomment-695778993
> But is not that abuse of pools? The pool is a feature taken from celery Pools are used to reduce the number of tasks that access a given resource ie. limit the execution parallelism on arbitrary sets of tasks.. You can configure the value from 0 to block tasks from running, any number or -1 to disable quotas. In my opinion, it fits perfectly here, but we do not have an automatic mechanism that will assign task with specific connections to a specific pool. See: https://airflow.readthedocs.io/en/latest/concepts.html#pools > The pool is a feature taken from celery + now it is also used for CeleryKubernetesExcecutor to distinguish "big" tasks that you can send to Kubernetes from "small" tasks that you want to send to Celery. No. CeleryExecutor and CeleryKubernetesExecutor use a queue. Queues can also be used here, but it would be an abuse and it would be very difficult to configure because the queues are configured on the target server and Airflow cannot automatically change its configuration. If a worker is assigned to queue A, then it cannot be assigned to queue B without restarting the worker. It is different with pools that are fully managed by Airflow and we have a much greater influence on their configuration and behavior. See: https://airflow.readthedocs.io/en/latest/executor/celery.html#queues > The fact that we can do something does not always mean that we should do it. I agree, but if there is one concept that has a similar application, we can think about reusing or extending it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
