dan...@zoltak.com:
> It patches the sources with your aufs: dynop supports grsec/pax patch.
Ok.
So the grsec/pax patch is applied to your kernel, and it is applied for
both cases (aufs and non-aufs). So we can ignore about the patch. Right?
> I have provided straces of the same system with and without AUFS.
>
> The lstats seem to be slower. It's as if the NFS cache is not working?
> The NFS mounts Apache serve's the content from are under /home which
> are normal NFS mounts so AUFS shouldn't impact performance?
:::
> Could it be possible AUFS is not allowing the NFS cache to work on the
> raw mounted NFS volumes?
Not only lstat.
Here is a result by my simple analysing script (attached).
aufs_apache.strace nonaufs.strace
--------+-----------------------------+--------------------------
0.000069 0.000397 open 418 .165775 0.000328 open 198 .064918
0.000065 0.000077 read 763 .058690 0.000012 read 488 .005694
0.000052 0.000062 lstat 11201 .697812 0.000010 lstat 2189 .021072
0.000043 0.000047 lseek 24 .001128 0.000004 lseek 4 .000017
0.000035 0.000048 writev 2 .000096 0.000013 writev 4 .000050
0.000016 0.000023 mmap 178 .004154 0.000007 mmap 78 .000558
0.000013 0.000028 access 179 .005014 0.000015 access 24 .000354
0.000013 0.000023 chdir 4 .000090 0.000010 chdir 2 .000020
0.000010 0.000020 munmap 178 .003486 0.000010 munmap 78 .000772
0.000009 0.000020 readlink 15 .000301 0.000011 readlink 3 .000033
0.000009 0.000016 fstat 399 .006467 0.000007 fstat 159 .001070
:::
These columns shows (see the first line)
- in your aufs_apache.strace, there exist 418 open systemcalls
- and its total amount of time is 0.165775
- its average time is 0.000397
- also in your nonaufs.strace, 198 open calls, 0.064918 total time,
0.000328 ave time
- the first column shows the difference of these two average times,
0.000069.
And the result is sorted by the first column.
Due to the amount of your sampling is not enough, such analysis may not
be meaningful, but I tried.
Both of open and lstat receive a pathname and internally lookup it (the
path resolution). It is one of the heavy work for aufs since aufs has to
search every branch. In your case, aufs first search the given filename
on tmpfs, and if it is not found, then search the same name on nfs.
If the result is already cached on memory, it is used. But the amount of
system cache is limited (it depends upon your system RAM size) and it
will be discarded by the order of least usage.
The important issue here is that, when you use aufs, your system caches
not only the filename on nfs but alst the name on aufs. So roughly
speaking, the amount of cached name will be double. Finally if your
apache server accesses the previously accessed file AFTER accessing
various other files, the file may not be cached (may already be
discarded from cache).
This nature behaviour of aufs _may_ be one reason of your problem.
To confirm the amount of system caching,
/proc/sys/fs/{dentry-state,inode-state} may be helpful.
I think NFS caches expectedly. But it may be discarded due to the
limited size of system cache.
Honestly speaking, I am unsure whether the caching of dentry and inode
is the cause of your problem, but it may be one reason I guess.
J. R. Okajima
#!/bin/sh
tmp=/tmp/$$
files="aufs_apache.strace nonaufs.strace"
set -eu
cut -c7- $files | cut -f1 -d'(' | sort | uniq > $tmp.syscall
for f in $files
do
#echo
#echo $f
while read sc
do
case $sc in
semop|poll|restart_syscall|epoll_wait) continue;;
esac
fgrep -w $sc $f > $tmp || :
echo -n $sc $(wc -l < $tmp) ''
if [ -s $tmp ]
then
{
rev $tmp | cut -f1 -d'<' | cut -f2 -d '>' | rev |
sed -e 's/^/a+=/'
echo a
} | bc
else
echo 0
fi
done < $tmp.syscall |
awk '{
ave = 0;
if ($2)
ave = $3 / $2;
printf("%0.6f %s\n", ave, $0);
}' > $tmp.$f
echo $tmp.$f
done |
xargs paste |
tee $tmp.pasted |
awk '{
diff = 0;
if ($3 && $7)
diff = $1 - $5;
printf("%0.6f %s\n", diff, $0)
}' |
sort -nr
rm -f $tmp $tmp.*
------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
