Hi Brian,

Really appreciate your quick reply. Just to be clear, I did not intend to
run them in particular order. as a matter of fact, these are expensive db
queries that I cant afford to run in parallel.
I think I have setup the tasks correctly to use pool but may be missing the
priority_weight setting correctly. Appreciate if you could run by your
configs just to see if I am not missing any simple point.

thanks much,
Manish

On Fri, Apr 6, 2018 at 6:18 PM, Brian Greene <
br...@heisenbergwoodworking.com> wrote:

> To be clear, you’re hoping that setting the slots to 1 will cause the
> tasks across district dags to run in order based on the assumption that
> they’ll queue up and then execute off the pool?
>
> I don’t think it will quite work that way - there’s no guarantee the
> scheduler will execute your tasks across dags in any particular sequence,
> and if 1 is “faster” than the other for sure they don’t “line up”.  Thus,
> no way to ensure they’ll queue in the right order.
>
> I successfully use pools across many dags to limit access to an expensive
> resource and it works really well, but my design doesn’t require they
> execute in any particular order, each idempotent.
>
> I’m curious as to your design/constraints - could you elaborate?
>
> Brian
>
> Sent from a device with less than stellar autocorrect
>
> > On Apr 6, 2018, at 3:46 PM, Manish Trivedi <trivman...@gmail.com> wrote:
> >
> > Hi Airflow devs,
> >
> > I have a use case to limit the # of calls to a certain database. I am
> using
> > the pool along with priority weight to schedule the tasks to the slot
> pool.
> > I have around 5 operators that I need to execute in serial order across
> > different dags.
> >
> > Slot pool is created with "1" slot to ensure sequential exection. I am
> not
> > able to achieve the desired function with current setup.
>

Reply via email to