potiuk commented on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-911882319
> my screenshot shows the metric container_memory_working_set_bytes, it decreased after 10 seconds (scraping time of metrics) after rm execution, scheduler did not restart Ah. so investigation begins again then. @suhanov , Need some more answers :). Just to clarify: So the 4GB memory drop (out of 6GB) was all from `container_memory_working_set_bytes` ? Do you know which processes they were ? Do you have any other processes running in the container besides airflow ? Was there a drop in a number of processes when you deleted the files? Maybe you can even look now and show the memory usage of the processes you have now in the container after some time of running the scheduler after deletion. I'd love to get to the bottom of it, because I find it really surprising to find that removal of files causes memory drop. I think - besides the kernel cache - which is low level, you'd really have some kind of service that is subscribed to those files via FS "notify" kind of system to be able to free any memory as result of deleting a file. Normally even if you have a file opened in linux and the file gets deleted, nothing special happens, the file is not actually removed until the last process that keeps the file opened is closed. So I find it really surprising to see such behaviour (that's why it is so interesting - because it is counter-intuitive). Additional question: What KIND of filesystem you have for the logs in scheduler ? Is it a usual "local filesystem" or is it some kind of distributed, user-space kind of filesystem. Because if it is the latter, then it could be the (for example user-space run) filesystem that keeps the log files data in memory and frees them after they are deleted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
