Thanks for the details; yeah was expecting this behavior On Aug 17, 2017 12:36 PM, "Maxime Beauchemin" <[email protected]> wrote:
> The point of pools is to limit parallelism on a logical set of tasks > instances to a certain number. Overflowing into another pool would break > the only guarantee it provides. > > `priority_weigth` works along with pool to define which task should be > scheduled first once slots open up. It won't kill any other tasks if higher > priority tasks show up, it just re-orders the queue. > > Max > > On Wed, Aug 16, 2017 at 10:47 PM, David Capwell <[email protected]> > wrote: > > > I'm looking into pools and had a few questions > > > > Let's say I have two pools, each of 50% of the cluster. If one pool is > at > > capacity and has a backlog, but the other pool is idle, will airflow > allow > > the first pool's work to start consuming the slots from the idle pool, > and > > if so is their preemption to rectify if needed? Since the docs don't > cover > > this I expect that this is not supported? > > > > Also, let's say I have a DAG that I need to always run regardless of if > > there are free slot or not; how could I accomplish this? Best I can see > is > > pools and make one just slightly smaller than the full cluster. > > > > Thanks for your time reading this email > > >
