o-nikolas commented on issue #41055: URL: https://github.com/apache/airflow/issues/41055#issuecomment-2256264249
> Hi @o-nikolas , in the base executor, it's described that setting the parallelism to '0' means infinite parallelism. See : > > https://github.com/apache/airflow/blob/c0cc614b2d494d610431fc2796388b86b64af731/airflow/executors/base_executor.py#L113 > > The LocalExecutor specifically implemented this unlimited parallelism see: > > https://github.com/apache/airflow/blob/9468a72b75c2880f032f654165012a9f66b83fbf/airflow/executors/local_executor.py#L388 > > Another evidence that it can be set to `0` is > > https://github.com/apache/airflow/blob/9468a72b75c2880f032f654165012a9f66b83fbf/airflow/executors/local_executor.py#L233 > > So I believe this to be a breaking change @ephraimbuddy Thanks for the reply! Yupp, there are snippets of proof on both sides. I'm aware of the code in the base exec and local executor. But also this feature is undocumented, untested, some metrics break when you use it, etc. So as I said originally, it's a half-baked feature. > my 2c: I think there is a valid case for the infinite/unknown parallelism for KE. For example, if we have a cluster with some capacity X Gi of memory, now how do we decide the optimal parallelism in such scenarios? @utkarsharma2 thanks for the reply! There is nothing stopping you from setting parallelism to 100000 if you're not sure. But I think forcing folks to take a minute to think through what their parallelism _should be_ is a useful exercise. Airflow performance can degrade quite quickly if you let any of the many adjustment knobs get too far out of a good performance band (e.g. if you have noisy DAGs and set parallelism to infinity, you may choke your scheduler or workers with too many tasks concurrently). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
