Have a look at this: https://github.com/apache/incubator-airflow/pull/1509
Sent from my iPhone > On 30 mei 2016, at 14:03, Ryabchuk, Pavlo <[email protected]> wrote: > > Hi all, > > Maybe I am misusing airflow a bit, because I am using it as on demand > (triggered) complex data processing system, but still, the question is, what > are the actual parameters I should play around with in order to speedup > execution? > I have around 250 Dummy tasks (which do nothing) in my DAG and running it > locally with celery executor takes around 1000 sec, which is pretty strange. > I've noticed that a single Dummy task takes some milliseconds. I've tried > playing around with celery concurrency, airflow executor parallelism and > heartbeat, but with almost no result... it's really strange, what am I doing > wrong :) > > Best, > Pavlo > >
