Thanks for the script and the info.
I have a few answers and questions below...
Quoting sf...@users.sourceforge.net:
> dan...@zoltak.com:
>> It patches the sources with your aufs: dynop supports grsec/pax patch.
>
> Ok.
> So the grsec/pax patch is applied to your kernel, and it is applied for
> both cases (aufs and non-aufs). So we can ignore about the patch. Right?
The grsec/pax patch isn't applied to either.
>> I have provided straces of the same system with and without AUFS.
>>
>> The lstats seem to be slower. It's as if the NFS cache is not working?
>> The NFS mounts Apache serve's the content from are under /home which
>> are normal NFS mounts so AUFS shouldn't impact performance?
> :::
>> Could it be possible AUFS is not allowing the NFS cache to work on the
>> raw mounted NFS volumes?
>
> Not only lstat.
> Here is a result by my simple analysing script (attached).
>
> aufs_apache.strace nonaufs.strace
> --------+-----------------------------+--------------------------
> 0.000069 0.000397 open 418 .165775 0.000328 open 198 .064918
> 0.000065 0.000077 read 763 .058690 0.000012 read 488 .005694
> 0.000052 0.000062 lstat 11201 .697812 0.000010 lstat 2189 .021072
> 0.000043 0.000047 lseek 24 .001128 0.000004 lseek 4 .000017
> 0.000035 0.000048 writev 2 .000096 0.000013 writev 4 .000050
> 0.000016 0.000023 mmap 178 .004154 0.000007 mmap 78 .000558
> 0.000013 0.000028 access 179 .005014 0.000015 access 24 .000354
> 0.000013 0.000023 chdir 4 .000090 0.000010 chdir 2 .000020
> 0.000010 0.000020 munmap 178 .003486 0.000010 munmap 78 .000772
> 0.000009 0.000020 readlink 15 .000301 0.000011 readlink 3 .000033
> 0.000009 0.000016 fstat 399 .006467 0.000007 fstat 159 .001070
> :::
>
> These columns shows (see the first line)
> - in your aufs_apache.strace, there exist 418 open systemcalls
> - and its total amount of time is 0.165775
> - its average time is 0.000397
> - also in your nonaufs.strace, 198 open calls, 0.064918 total time,
> 0.000328 ave time
> - the first column shows the difference of these two average times,
> 0.000069.
> And the result is sorted by the first column.
>
> Due to the amount of your sampling is not enough, such analysis may not
> be meaningful, but I tried.
>
> Both of open and lstat receive a pathname and internally lookup it (the
> path resolution). It is one of the heavy work for aufs since aufs has to
> search every branch. In your case, aufs first search the given filename
> on tmpfs, and if it is not found, then search the same name on nfs.
> If the result is already cached on memory, it is used. But the amount of
> system cache is limited (it depends upon your system RAM size) and it
> will be discarded by the order of least usage.
> The important issue here is that, when you use aufs, your system caches
> not only the filename on nfs but alst the name on aufs. So roughly
> speaking, the amount of cached name will be double. Finally if your
> apache server accesses the previously accessed file AFTER accessing
> various other files, the file may not be cached (may already be
> discarded from cache).
> This nature behaviour of aufs _may_ be one reason of your problem.
> To confirm the amount of system caching,
> /proc/sys/fs/{dentry-state,inode-state} may be helpful.
Using AUFS:
cat /proc/sys/fs/dentry-state
791734 724106 45 0 0 0
cat /proc/sys/fs/inode-state
835729 249181 0 0 0 0 0
No AUFS:
cat /proc/sys/fs/dentry-state
714262 633454 45 0 0 0
cat /proc/sys/fs/inode-state
860710 272418 0 0 0 0 0
> I think NFS caches expectedly. But it may be discarded due to the
> limited size of system cache.
> Honestly speaking, I am unsure whether the caching of dentry and inode
> is the cause of your problem, but it may be one reason I guess.
The thing I don't understand is even though the RootFS is AUFS that is
a Union with tmpfs and NFS, Apache is serving it's content from a
non-aufs volume. This being the case why is the performance being
impacted i.e. the /home folder where Apache is serving from is
directly mounted to the NFS server and not in Union with the tmpfs.
In this case shouldn't any AUFS lookups be bypassed?
------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
