potiuk commented on issue #14924: URL: https://github.com/apache/airflow/issues/14924#issuecomment-912479114
I looked a bit more. I have a hypothesis what could be the problem - I looked at the possibility that we have growing "dirty" memory (i.e. not flushed/synchronised to disk). My hypothesis is that (at least some of) the logs are not flushed and they remain in "dirty" (or rather "unsynchronized") state. That would be fairly strange as usually "dirty" memory is synchronized after at most few seconds. And I believe that the standard "RotatingFileHandler" we use to write processor manager lgos should properly close and flush the streams anyway. But maybe you have some optimisatios/settings on your OS to prolong/disable auto-flushing (would be strange though) and maybe there is some special configuration/handler of logs that you write? I found this nice PDF describing how PageCache actually works (fascinating read) http://sylab-srv.cs.fiu.edu/lib/exe/fetch.php?media=paperclub:lkd3ch16.pdf - Linux uses "write-back" cache strategy where it first writes data to cache and then it is flushed to disk (and remains in cache as non-dirty). The os will mark the files as dirty (and thus you can see them in `Container_memory_working_set_bytes'). If for whatever reason those files would remain as "dirty" they will still be counted as "working_set" memory. And that would also explain why the memory is freed after deleting the files - when the file gets deleted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
