dan...@zoltak.com:
> When load spikes, which only happens when running AUFS the performance
> suffers i.e. the number of requests per second drops off on the AUFS
> node and the request per second graphs becomes jittery. The latency
> also goes up quite a bit on the node running AUFS.
Then let's identify the major part from the http client side. For
example, how about this approach?
The sequence on the http client is
- connect to server
- send an HTTP request
- recevie the HTTP response
- close the connection?
For each step, how long does it take? and why?
You may say the longest is absolutely between send and receive. then
I'd suggest you to capture the tcp packets and confirm it.
After confirming the problem exists on the server side, let's think
about the steps on the server similarly.
- accept the connection
- write to a log file? including dns query?
- parse the http request
- access to the file (on NFS? not aufs?)
- the content may be dynamic which may run php, mysql or something.
- open and read from the file and send it back to the client, or sendfile.
- close the connection?
Similarly, for each step, we need to consider how long does it take and
why?
strace is one option which we already tried. You may want to try other
tools such like profiler, tracer, perf or some statistics in kernel.
As I wrote previously, I'd also suggest you to see cpu usage, network
i/o, local disk i/o, memory usage, lock contention or something in order
to identify where in the server takes along time.
Since you wrote only apache server processes are busy and the web
content exists out of aufs, currently I am afraid the problem seems to
be out of aufs. Repating the load average 2 becomes 4 on 8 cpu is less
informative.
J. R. Okajima
--
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
