Re: Slot pools correct usage

2018-04-07 Thread Brian Greene
So what’s it doing (your config)? Does it work if you don’t use pools? What about if the pool is if size 2? What if just one dag runs? Have you ever seen this query work, or is it just since you started messing with pools that it stopped working? I use 1 pool, no priority (I don’t care abo

Re: Slot pools correct usage

2018-04-06 Thread Manish Trivedi
Hi Brian, Really appreciate your quick reply. Just to be clear, I did not intend to run them in particular order. as a matter of fact, these are expensive db queries that I cant afford to run in parallel. I think I have setup the tasks correctly to use pool but may be missing the priority_weight s

Re: Slot pools correct usage

2018-04-06 Thread Brian Greene
To be clear, you’re hoping that setting the slots to 1 will cause the tasks across district dags to run in order based on the assumption that they’ll queue up and then execute off the pool? I don’t think it will quite work that way - there’s no guarantee the scheduler will execute your tasks ac

Slot pools correct usage

2018-04-06 Thread Manish Trivedi
Hi Airflow devs, I have a use case to limit the # of calls to a certain database. I am using the pool along with priority weight to schedule the tasks to the slot pool. I have around 5 operators that I need to execute in serial order across different dags. Slot pool is created with "1" slot to en