[GitHub] [airflow] bharatk-meesho commented on issue #27589: Scheduler pods memory leak on airflow 2.3.2

Fri, 11 Nov 2022 06:16:59 -0800 


bharatk-meesho commented on issue #27589:
URL: https://github.com/apache/airflow/issues/27589#issuecomment-1311749973

   Thanks @Taragolis I will take a look at these links.
   However I did tried one thing, I cleared the logs in /opt/airflow/scheduler 
and that didn't decreased the memory consumption. Also I checked the parameter 
in grafana dashboard which denotes memory utilized and the parameter is 
"container_memory_working_set_bytes".
   
   Below is graph of scheduler pod level memory increasing from grafana 
although all dags are paused. And there has been no changes in the system 
whatsoever.
   
   <img width="1352" alt="Screenshot 2022-11-11 at 7 28 09 PM" 
src="https://user-images.githubusercontent.com/95409716/201355221-a2183d7a-966e-4dca-a16b-690f75b6379c.png";>
   
   
   One another weird thing I noticed is that the memory utilization from top 
combined of all process doesn't match what I get from running "kubectl 
command". Attaching both screen-shots below.
   
   HPA command output shows about 55% memory utilization and eventually it 
keeps on increasing so far i have obsreved.
   <img width="860" alt="Screenshot 2022-11-11 at 7 40 27 PM" 
src="https://user-images.githubusercontent.com/95409716/201357385-6cc4421e-ef85-4d5c-9df2-8886de8cd815.png";>
   
   
   While via top it doesn't seem to be more than 2%
   <img width="674" alt="Screenshot 2022-11-11 at 7 43 51 PM" 
src="https://user-images.githubusercontent.com/95409716/201358161-0377ca23-e35d-4049-949a-88b0ad006920.png";>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to