I am using 2.6.35.11-83.fc14.x86_64 (stock from Fedora Core 14). Did anything else change? How exactly do you do the file writes? What block size do you use? Is there a lot of create / delete churn?
I ask because I believe both write block size and the degree of free space fragmentation matter. Larger block sizes and systems with more free space fragmentation may mitigate the issue. Can you try a simple dd test? Copy from /dev/zero with (default) 512 byte blocksize and watch the jfsCommit CPU usage as the file gets larger. The hard part is that it matters how exactly JFS allocates blocks so you might have to repeat this test a few times to see it happening. -Bob On May 10, 2011, at 0:06, Sandon Van Ness <[email protected]> wrote: > What kernel version do you use? I only ask as I *used* to have this problem > but it seemed to have gone away in one of the following: > > Kernel upgrade > Hardware upgrade (much more ram, 24GB+ now) > 128MB logsize used when creating the volume. > > Since these three things I have written files that are serveral hundred > gigabytes without problems. > > In the past I definitely had the same issue you did. Here is a post I made on > a forum back in 2008 about it: > > http://www.dslreports.com/forum/r20737283-weird-JFS-behavior > > On 05/09/2011 12:15 PM, Robert Evans wrote: >> I've observed some performance degradation when writing large files. >> jfsCommit reaches nearly 100% cpu use and the write rate slows to a crawl. >> I've done a lot of digging, and I think I know the cause. In summary, I >> believe txUpdateMap is spending a *lot* of extra time re-writing the >> persistent block maps when extending the last xad in the file's xtree. xtLog >> stores the *entire last xad* for update, and when appending a large file on >> a sparsely allocated volume this xad might cover a lot of blocks. And this >> causes each commit to update millions of already allocated blocks. As the >> file grows the total number of updates grows exponentially causing huge >> performance loss. >> >> The reason I investigated this was a 320GB file write slowed to about >> 7-10MB/s when the underlying RAID volume can usually write 400MB/s. >> jfsCommit cpu was 99.4% at the same time. For the record I'm running Linux >> 2.6.35.11-83.fc14.x86_64. >> >> I confirmed the file was not fragmented by dumping out the xtree for the >> inode in question. (I used some hacky python code that I wrote for exploring >> JFS volumes, and I used sync and drop_caches to ensure metadata was flushed >> out to the disk). >> >> Here's the xtree (4kb allocation size), and each tuple is (offset, address, >> length): >> (0, 336166839, 1) >> (1, 992337163, 2) >> (3, 992398076, 1) >> (4, 992471283, 22) >> (26, 992771329, 13861631) >> (13861657, 1026074340, 14113052) >> (27974709, 1059503188, 14238636) >> (42213345, 1093863162, 2) >> (42213347, 1094345396, 175) >> (42213522, 1094345577, 91) >> (42213613, 1094588258, 12707998) >> (54921611, 1115498095, 1) >> (54921612, 1115498097, 2) >> (54921614, 1115787154, 1) >> (54921615, 1116292353, 102) >> (54921717, 1116650670, 39713) >> (54961430, 1116695739, 16777215) >> (71738645, 1133472954, 6404161) >> >> The performance issue occurred while writing the last 50 GB or so which is >> consistent with the last two XADs. When the file was around 266Gib long the >> last xad (at offset 54961430) would have had around 10M blocks. Every >> commit would have updated 10M blocks, until the xad overflowed and the new >> xad at offset 71738645 was created. And sure enough, when the file got to >> be around 293Gib the performance went up again but slowly fell again. >> >> This should be reproducible by writing a large file on a large empty volume. >> As each xad fills the number of blocks being updated will increase and the >> total number of updates will grow exponentially causing the write to slow >> down. Once the xad overflows performance should go up again. >> >> -Bob >> >> >> ------------------------------------------------------------------------------ >> WhatsUp Gold - Download Free Network Management Software >> The most intuitive, comprehensive, and cost-effective network >> management toolset available today. Delivers lowest initial >> acquisition cost and overall TCO of any competing solution. >> http://p.sf.net/sfu/whatsupgold-sd >> >> >> _______________________________________________ >> Jfs-discussion mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/jfs-discussion > ------------------------------------------------------------------------------ Achieve unprecedented app performance and reliability What every C/C++ and Fortran developer should know. Learn how Intel has extended the reach of its next-generation tools to help boost performance applications - inlcuding clusters. http://p.sf.net/sfu/intel-dev2devmay _______________________________________________ Jfs-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/jfs-discussion
