David, When you say "massive" oversubscribing, are you running a lot of dags in parallel that use your configured pools? Access to pools is not atomic at the moment.
Can you also quantify "massive" ? Not that it matters, but to get a better idea. Rgds, Gerard On Tue, Nov 1, 2016 at 4:09 PM, siddharth anand <[email protected]> wrote: > Yes.. we have seen the over-subscription of pools. We do need a fix for it > --- I don't believe there is one. We need someone to own and fix it.. happy > to review a PR. > > We use pools at Agari for all of our needs. We are okay with mild > oversubscription, so we do see numbers slightly higher, but our pipelines > work fine with that level. In your example, the oversubscription is much > higher. > -s > > On Tue, Nov 1, 2016 at 6:38 AM, David Kegley <[email protected]> wrote: > > > I've been seeing some weird behavior when using Airflow's execution pool > > feature. Pools have been massively over-filled leading to failed tasks. > I > > created a bug for this issue, but in the mean time, has anyone else > > experienced this behavior? > > > > https://issues.apache.org/jira/browse/AIRFLOW-584 > > > > Best, > > David > > >
