So I wrote some more sophisticated code to benchmark the writes. This
program monitors the write performance second by second, reporting the
apparent number of bytes written, number of bytes actually written (reported
by the block layer), and total CPU use for all jfsCommit kthreads.
I let run this program run for a long time until I saw the same sort of
severe performance degradation I saw before. Then I dumped out the xtree and
compared the allocated xad's to the periods of higher CPU use. This yielded
some very interesting observations:
1. The larger xad's do correspond to periods of higher CPU use, lower
apparent write throughput, and higher raw block write rates
2. There is a modest impact to performance if any of the first 16 xads
become large (15% cpu, 25% slowdown)
3. Once the xtree grows to 17 entries (and spills out of the inode)
performance suffers greatly. CPU rises rapidly and write throughput drops
dramatically (75-90% slowdown)
4. The performance drops very rapidly once the xtree grows to 17 entries,
and the impact is quite large even if the xad is relatively small. Once the
xad had 100k blocks the write performance was 50% of the first 16 xads.
5. If the xad's beyond the 17th fill completely write performance becomes
very, very poor (~1% of the original throughput for the first 16) and CPU
use hits 100%.
I don't know why this is happening but it seems the performance suffers
quite a bit once the 17th xad is reached. Interestingly, it doesn't even
matter if the file is actually very large. This can be reproduced quickly
by creating a file that contains 16 sparse extents and then appending to the
end of it:
> for a in 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31; do dd if=/dev/zero
of=test.dat seek=$a count=1 bs=4k; done
> dd if=/dev/zero of=test.dat bs=4k seek=33
This should immediately cause poor write performance and high CPU
consumption.
Shaggy, do you have any ideas why the xtree lazy commit performance might
suffer so greatly once it spills out of the inode?
-Bob
------------------------------------------------------------------------------
Achieve unprecedented app performance and reliability
What every C/C++ and Fortran developer should know.
Learn how Intel has extended the reach of its next-generation tools
to help boost performance applications - inlcuding clusters.
http://p.sf.net/sfu/intel-dev2devmay
_______________________________________________
Jfs-discussion mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/jfs-discussion
