jbkc85 commented on issue #13542: URL: https://github.com/apache/airflow/issues/13542#issuecomment-877542235
> This is also why I increased the `parallelism` option to 1000 and increased the `pool` size. ooh, so it wasn't just the pool size. That makes more sense on why it worked. That being said, @caleboverman I think stumbled onto a great solution that not only eliminates the potential of (when this is fixed) parallelism, when airflow fixes the issue, ends up scheduling so many processes it takes down the Airflow box itself. The gist of it, is just create more pools and use the pools as a way to help queue processes. This of course works when you have a smaller installation, but really helped our stuff go through after we implemented the solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
