mik-laj commented on issue #9755: URL: https://github.com/apache/airflow/issues/9755#issuecomment-695778993
> But is not that abuse of pools? The pool is a feature taken from celery Pools are used to reduce the number of tasks that access a given resource ie. limit the execution parallelism on arbitrary sets of tasks.. You can configure the value from 0 to block tasks from running, any number or -1 to disable quotas. In my opinion, it fits perfectly here, but we do not have an automatic mechanism that will assign task with specific connections to a specific pool. See: https://airflow.readthedocs.io/en/latest/concepts.html#pools > The pool is a feature taken from celery + now it is also used for CeleryKubernetesExcecutor to distinguish "big" tasks that you can send to Kubernetes from "small" tasks that you want to send to Celery. No. CeleryExecutor and CeleryKubernetesExecutor uses a queue. See: https://airflow.readthedocs.io/en/latest/executor/celery.html#queues > The fact that we can do something does not always mean that we should do it. I agree, but if there is one concept that has a similar application, we can think about reusing or extending it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
