Hi all, KSM is a really cool way to lower memory usage, but due to the delay between worker spawn and when ksmd kicks in and merges the pages, you end up requiring enough free memory for the initial amount rather than the (sometimes significantly lower) merged amount.
To make this clearer, consider an app with 10 workers that use 1GB each. However, when KSM kicks in it will merge pages to bring down the per-worker memory usage to 500MB. If we spawn all the workers immediately, we need 10GB of free memory immediately (assume we do some initialization that allocates all of this memory). However, in "steady-state" after ksm has done its work, we may need just 5GB plus 500MB for each unmerged worker memory (for example when a worker reloads). This is significantly lower on average, though there is some probability that many workers reload at the same time, causing a memory usage spike. Techniques like pre-fork warmup to take advantage of copy-on-write are good in theory but, in my experience, can cause problems with non-fork-safe code as well as problems with garbage collection in python specifically. My thought is to have some sort of rate limiter to worker spawning so that we can give ksmd enough time to merge the pages, though this seems pretty hacky and fragile (e.g. how do you tune the rate?). Has anyone dealt with this issue? Anyone have any alternate ideas? Thanks -Ben
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
