ginevragaudioso commented on issue #13975:
URL: https://github.com/apache/airflow/issues/13975#issuecomment-771785185
Thanks @anushkrishnav ! I can't provide much guidance on the implementation
side of things, as I stated before I did not take the time to look into airflow
source code deep enough to be able to give any meaningful advice. All I know is
that it seems that the one-pool assumption goes all the way into SQL, so this
may require a table model change, but I could be wrong here.
In terms of feature request, I can help clarify the desired outcome. The
goal is to be able to call something like (`BashOperator` is only an example,
but this change would need to go into `BaseOperator` so that all operators
inherit it):
`
my_task = BashOperator(
# some kwargs here, unrelated to pool(s)
pools=[RESOURCE_A_POOL, RESOURCE_B_POOL],
pools_slots=[1, 1], # this could be default, but just mentioning here
since this kwarg also needs to change
)
`
Currently, the `pools` argument that takes a list of pools does not exist,
we only have a `pool` argument that takes one pool.
With the proposed feature in this issue, and the proposed new call in this
comment, `my_task` can only be scheduled when there is one slot available in
all declared pools, and `my_task` will take up one slot in all pools. I think
it's ok to leave resource ordering (which is needed to avoid deadlocks
probably) to the user, and airflow could just "book" the pools in the order
they are given.
Please let me know if you need any more guidance from me, and thanks for
replying to this issue!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
