Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-30 Thread sfjro
dan...@zoltak.com: > When load spikes, which only happens when running AUFS the performance > suffers i.e. the number of requests per second drops off on the AUFS > node and the request per second graphs becomes jittery. The latency > also goes up quite a bit on the node running AUFS. Then

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-29 Thread sfjro
dan...@zoltak.com: > I down graded the kernel to 2.6.31-gentoo-r6, applied the AUFS patch > and compiled the kernel and module. > > I then booted a node without AUFS loaded and ran it in an 8 node > cluster where the other nodes were running on the pre upgraded image. > > The rootfs was a tmpf

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-28 Thread daniel
Quoting "Michael S. Zick" : > Could it be something as simple as network load? > I don't recall your posting what NICs are on that box. > > Apache + NFS on an 8-core box can have a need to send a few packets. ;-) Thanks for the suggestion. We run Gigabit Ethernet with one adapter for dedicated S

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-27 Thread Michael S. Zick
On Wed January 26 2011, dan...@zoltak.com wrote: > Quoting sf...@users.sourceforge.net: > > > > If you don't mind, would you clarify your story? > > Other aufs users may meet the similar situation and may think "there > > must exist something bad in aufs" similarly. For such users, what you > > h

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread daniel
Quoting sf...@users.sourceforge.net: > If you don't mind, would you clarify your story? > Other aufs users may meet the similar situation and may think "there > must exist something bad in aufs" similarly. For such users, what you > have experienced and found may be helpful. > Some numbers about

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread sfjro
dan...@zoltak.com: > Yes my mistake. The latency has gone up a bit but not by much. ::: > I've performed more tests and have found that there is something going > on with NFS unrelated to AUFS as I have now managed to verify that > AUFS is not causing the performance issue. > > Thanks

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread daniel
Quoting sf...@users.sourceforge.net: > dan...@zoltak.com: >> > How much RAM does your system have? >> >> 16GB and 8 cores. > > Memory large enough. > CPU many enough. > Let's keep in mind about lock contension. > > Just to make sure, you can see them all via /proc. Right? > Everything is showi

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread sfjro
dan...@zoltak.com: > Is there anyway to validate this? Do you mean that you want to make sure that aufs code is unrelated to the activity out of aufs? How about adding printk in d_revalidate or code review? > > How much RAM does your system have? > > 16GB and 8 cores. Memory large enough.

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread daniel
Quoting sf...@users.sourceforge.net: > dan...@zoltak.com: >> Due to the AUFS being the RootFS and the raw NFS mounts are mounted on >> the RootFS could this in fact be causing AUFS to do unnecessary lookups? > > I don't think so. > Because the root dir is always cached and its cache will never b

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread sfjro
dan...@zoltak.com: > Due to the AUFS being the RootFS and the raw NFS mounts are mounted on > the RootFS could this in fact be causing AUFS to do unnecessary lookups? I don't think so. Because the root dir is always cached and its cache will never be discarded. Every access (to anywhere) may ca

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread daniel
Quoting sf...@users.sourceforge.net: > So your apache is running out of aufs mostly. Then the cause of your > load average may exist somewhere else. The only problem is that with the same setup i.e. same kernel, mounts e.t.c without AUFS running there is no performance issue! When AUFS is loa

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread sfjro
dan...@zoltak.com: > The thing I don't understand is even though the RootFS is AUFS that is > a Union with tmpfs and NFS, Apache is serving it's content from a > non-aufs volume. This being the case why is the performance being > impacted i.e. the /home folder where Apache is serving from is

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread daniel
Thanks for the script and the info. I have a few answers and questions below... Quoting sf...@users.sourceforge.net: > dan...@zoltak.com: >> It patches the sources with your aufs: dynop supports grsec/pax patch. > > Ok. > So the grsec/pax patch is applied to your kernel, and it is applied for >

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-26 Thread sfjro
dan...@zoltak.com: > It patches the sources with your aufs: dynop supports grsec/pax patch. Ok. So the grsec/pax patch is applied to your kernel, and it is applied for both cases (aufs and non-aufs). So we can ignore about the patch. Right? > I have provided straces of the same system with and w

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-25 Thread sfjro
dan...@zoltak.com: > I will try to be concise with my answers. ::: > The following translates to: > compiled without: debug support, hardening support, inotify support, > nfs export support ::: > The Apache 2.2 processes appear to be running hotter when AUFS is the root FS. Thak

Re: NFS root + aufs tmpfs + Apache performance problem

2011-01-24 Thread sfjro
Hello Daniel, dan...@zoltak.com: > I have found that Dracut and aufs2 seems to work fine with my FTP > servers and Mail servers however with the Web servers previously I was > seeing a load average around ~2. Since switching to aufs2 the load > average has double to ~4. First, - I am not a