On 06/05/2015 12:09 PM, Markus "Shorty" Uckelmann wrote:
Am 05.06.2015 um 18:33 schrieb Gordon Messmer:
On 06/05/2015 03:29 AM, Markus "Shorty" Uckelmann wrote:
some (probably unused) parts are swapped out. But, some of
those parts are the salt-minion, php-fpm or mysqld. All services which
are important for us and which suffer badly from being swapped out.
Those two things can't really both be true. If the pages swapped out
are unused, then the application won't suffer as a result.
Why not? If you have an application which sees action only every 12 to
24 hours,I think this can happen.
Well, that's not "unused," then.
To measure the swap use of your processes, install "smem". It will show
you the amount of swap that each process is using.
For more specific information, make a copy of /proc/<pid>/smaps.
To quantify your problem, let bacula run then save the output of smem,
or /proc/<pid>/smaps for each of your critical services, or both, and
then access each of the services and quantify the latency relative to
the normal latency. Finally, after collecting latency information, get
the output of smem and/or /proc/<pid>/smaps again. You can compare swap
use before and after accessing the service to see how much was swapped
out beforehand (presumably because of the backup), and how much had to
be recovered for your test query.
I'd suggest collecting that information at the normal swappiness setting
and at 0.
If the kernel is swapping out processes in favor of filesystem cache
when swappiness is 0, I believe that would be a bug, and should be
reported to the kernel developers.
Our salt-minion would be a candidate
for this. Allthough we constantly check if it's alive, we only do once
or twice a day something "heavy" like a deployment. And very often we
have to run thos deployments twice, because the first time we get a lot
of timeouts. Sure, it might be the software itself. But I think it could
be possible that it is because of swapped out pages.
"Timeouts" is pretty vague. Very generally, it's possible that you have
a timeout configured somewhere that is failing on the first run because
the filesystem cache now contains content from your backup, and your
process only completes in time when the files needed for the deployment
are in the filesystem cache. That's a stretch as far as explanations
go, but if that is the case, then swappiness isn't going to fix the
problem. You need to fix your timeout so that it allows enough time for
the deployment to finish when the server is cold booted (using no
cache), or prime your caches before doing deployments.
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
