Thanks @ashb for pointing out, I've changed the commit title and PR title to `Parallelize Celery Executor task state fetching` to better reflect the content of the change. Also open for discussion on the default value of `sync_parallelism`( given the small difference different value can cause let's keep it short). I vote for `16`, it has proven to provide great performance, ~1s running time for querying 30k running task states( of course this also heavily depends on the network, celery broker, backend etc, we use redis for broker and backend), and is not a crazy number that will create much burden on the box.
[ Full content available at: https://github.com/apache/incubator-airflow/pull/3830 ] This message was relayed via gitbox.apache.org for [email protected]
