alex-magno opened a new issue, #34013:
URL: https://github.com/apache/airflow/issues/34013

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### What happened
   
   We are facing a quite interesting problem with our celery workers. **We run 
on AWS EKS (m-series nodes running Bottlerocket OS), Airflow v2.6.2**.
   
   When the amount of running tasks is high (close to our max cap), we notice 
an increase in the celery workers memory usage followed by a stale period. At 
the same time, CPU utilization goes to zero.
   
![image](https://github.com/apache/airflow/assets/87935605/55b8faba-0b46-48f5-a22b-a4a0b98713d6)
   
   During this time, the worker container seems to freeze (no logs are 
produced). Of the tasks that were running, almost all of them failed with the 
only log being:
   ```
   [2023-08-31, 13:20:20 UTC] {taskinstance.py:1093} INFO - Dependencies not 
met for <TaskInstance: ***** scheduled__2023-08-31T10:00:00+00:00 [failed]>, 
dependency 'Task Instance State' FAILED: Task is in the 'failed' state.
   [2023-08-31, 13:20:20 UTC] {local_task_job_runner.py:154} INFO - Task is not 
able to be run
   ```
   
   It is important to note that this problem happens sometimes, but not every 
time. Which makes it even more confusing. Also, workers pods are not killed in 
the process, nor do they throw any exceptions.
   
   Some of our configurations are:
   ```
   AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG = 32
   AIRFLOW__CORE__PARALLELISM = 64
   AIRFLOW__CELERY__WORKER_CONCURRENCY = 12
   ```
   
   We are running at the moment with 5 celery workers, with no CPU or memory 
limits. The scheduler seems fine during those periods, producing regular logs. 
The only thing I can think of at the moment is that its probably due to a 
shortage of resources in the celery workers? However the worker concurrency 
limit should prevent a high usage of CPU/memory. Any ideas will be appreciated.
   
   Some executor metrics on the same time period:
   
![image](https://github.com/apache/airflow/assets/87935605/64157a01-dfad-4c0e-ad26-e0a005270f7f)
   
![image](https://github.com/apache/airflow/assets/87935605/dec42938-1285-4fa1-88a7-e1d64293ac86)
   
   ### What you think should happen instead
   
   CPU and Memory resources should not go stale. Also, workers should keep 
logging as usual, instead of "freezing".
   
   ### How to reproduce
   
   Deploy through Helm chart, using the `CeleryExecutor`. Run the maximum 
number of allowed tasks at the same time. However, the problem is not 
guaranteed to happen.
   
   ### Operating System
   
   Bottlerocket OS
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   AWS EKS with k8s `v1.26.1` - Nodes (m-series) using Bottlerocket OS.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to