dlesco commented on issue #35806: URL: https://github.com/apache/airflow/issues/35806#issuecomment-1828138095
Google Composer 2 does allow to set cpu requests/limit to 1: the three options allowed by Composer are (0.5, 0.75, 1). You can see I put logs with the limit set to 1. But with a limit set to 1, it still takes more than 200ms sometimes. That's because you have a workload that is more than 1 CPU; besides the async-task thread, you have: 1. The main thread, running the _run_trigger_loop function. 2. The kubernetes configured livenessProbe for the container. 3. gunicorn master and worker processes. For Composer 2, the livenessProbe is running a script that calls the `airflow jobs check` command. So, it's starting up a whole new Python process, loading modules, etc, which takes a lot of CPU. It would've been better if the gunicorn server had a /liveness URL to do an http liveness check; then it's not spending a lot of cpu starting up a new process. You can see from my logs that the magnitude of the delays are less when I removed the livenessProbe; with the livenessProbe, the max delay was **0.78s**; without the livenessProbe, **0.37s**. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
