On 05/09/2011 02:15 PM, Robert Evans wrote: > I've observed some performance degradation when writing large files. > jfsCommit reaches nearly 100% cpu use and the write rate slows to a > crawl. I've done a lot of digging, and I think I know the cause. In > summary, I believe txUpdateMap is spending a *lot* of extra time > re-writing the persistent block maps when extending the last xad in the > file's xtree. xtLog stores the *entire last xad* for update, and when > appending a large file on a sparsely allocated volume this xad might > cover a lot of blocks. And this causes each commit to update millions > of already allocated blocks. As the file grows the total number of > updates grows exponentially causing huge performance loss.
I see. Thanks for the in-depth investigation. Everything you describe makes sense. I'll see what kind of fix I can come up with. It probably wouldn't be to hard to hang some hint off the in-memory inode to keep track of the previous extent location and size that has been processed by txUpdateMap. It wouldn't handle every corner case involving random access, but it should handle sequential writes. Anything non-sequential would likely end up using smaller extents anyway. Thanks again, Shaggy > The reason I investigated this was a 320GB file write slowed to about > 7-10MB/s when the underlying RAID volume can usually write 400MB/s. > jfsCommit cpu was 99.4% at the same time. For the record I'm running > Linux 2.6.35.11-83.fc14.x86_64. > > I confirmed the file was not fragmented by dumping out the xtree for the > inode in question. (I used some hacky python code that I wrote for > exploring JFS volumes, and I used sync and drop_caches to ensure > metadata was flushed out to the disk). > > Here's the xtree (4kb allocation size), and each tuple is (offset, > address, length): > (0, 336166839, 1) > (1, 992337163, 2) > (3, 992398076, 1) > (4, 992471283, 22) > (26, 992771329, 13861631) > (13861657, 1026074340, 14113052) > (27974709, 1059503188, 14238636) > (42213345, 1093863162, 2) > (42213347, 1094345396, 175) > (42213522, 1094345577, 91) > (42213613, 1094588258, 12707998) > (54921611, 1115498095, 1) > (54921612, 1115498097, 2) > (54921614, 1115787154, 1) > (54921615, 1116292353, 102) > (54921717, 1116650670, 39713) > (54961430, 1116695739, 16777215) > (71738645, 1133472954, 6404161) > > The performance issue occurred while writing the last 50 GB or so which > is consistent with the last two XADs. When the file was around 266Gib > long the last xad (at offset 54961430) would have had around 10M > blocks. Every commit would have updated 10M blocks, until the xad > overflowed and the new xad at offset 71738645 was created. And sure > enough, when the file got to be around 293Gib the performance went up > again but slowly fell again. > > This should be reproducible by writing a large file on a large empty > volume. As each xad fills the number of blocks being updated will > increase and the total number of updates will grow exponentially causing > the write to slow down. Once the xad overflows performance should go up > again. > > -Bob ------------------------------------------------------------------------------ WhatsUp Gold - Download Free Network Management Software The most intuitive, comprehensive, and cost-effective network management toolset available today. Delivers lowest initial acquisition cost and overall TCO of any competing solution. http://p.sf.net/sfu/whatsupgold-sd _______________________________________________ Jfs-discussion mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/jfs-discussion
