shenoykarthikd commented on issue #11266: URL: https://github.com/apache/airflow/issues/11266#issuecomment-708014805
Good point, @teastburn! I have typically changed the max_threads value for my Airflow clusters so that CPU utilization doesn't increase beyond 80% over a reasonable time period. Rarely ever checked the RAM as memory usage is typically pretty low on the master node. It may make sense to change the FAQ as below unless someone feels memory should also be a consideration. max_threads: Scheduler will spawn multiple threads in parallel to schedule dags. This is controlled by max_threads with default value of 2. User should increase it to a higher value in production. Note that an increase in this value causes a corresponding increase in CPU utilization. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
