Yes, after looking at the code, it looks like implementing a many to many between pools and tasks requires significant redesign.
My workaround is to implement it externally by borrowing required resources from an external pool using a sensor, and returning them back once the task is done using another sensor. They are basically entry/exit sensors for a task Thx On Sat, May 12, 2018, 2:28 PM Dennis O'Brien <den...@dennisobrien.net> wrote: > A task is assigned to a pool by the task specifying the name of the pool. > The docs suggest that the pool argument is a string, not a list of strings. > https://airflow.apache.org/code.html#baseoperator > > And looking at the code it does seem like this relationship of one task > assigned to zero or one pools is strict. > > It's an interesting idea of trying to associated a task with multiple > pools. I think things would get complicated fast to avoid deadlock > situations. Do you know of any other work flow systems that support > one-to-many relationship? > > Is it possible to bundle the resources into a single resource? I'm > guessing this is not an option, and that tasks use different combinations > of these resources. > > cheers, > Dennis > > > On Fri, May 11, 2018 at 1:04 PM Neelesh <neele...@gmail.com> wrote: > > > Hello, > > > > Is it possible to assign multiple pools to a task? > > > > I'm wrapping a bash operator around some heavy-weight custom data > > processing script that cannot be modified. This script cannot be > modified, > > and it uses multiple resources. Im trying to use pools as proxies for > these > > resources. > > > > Thanks > > >
