Hi,
On 2014/12/22 6:32, Robert White wrote:
On 12/21/2014 08:32 AM, Charles Cazabon wrote:
Hi, Robert,
Thanks for the response. Many of the things you mentioned I have tried, but
for completeness:
Have you taken SMART (smartmotools etc) to these disks
There are no errors or warnings from SMART for the disks.
Do make sure you are regularly running the long "offline" test. [offline is a
bad name, what it really should be called is the long idle-interval test. sigh] about
once a week. Otherwise SMART is just going to tell you the disk just died when it dies.
I'm not saying this is relevant to the current circumstance. But since you
didn't mention a testing schedule I figured it bared a mention
Have you tried segregating some of your system memory for to make
sure that you aren't actually having application performance issues?
The system isn't running out of memory; as I say, about the only userspace
processes running are ssh, my shell, and rsync.
The thing with "movablecore=" will not lead to an "out of memory" condition or
not, its a question of cache and buffer evictions.
I figured that you'd have said something about actual out of memory errors.
But here's the thing.
Once storage pressure gets "high enough" the system will start forgetting things intermittently to
make room for other things. One of the things it will "forget" is pages of code from running
programs. The other thing it can "forget" is dirent (directory entries) relevant to ongoing
activity.
The real killer can involve "swappiness" (e.g. /proc/sys/vm/swapiness :: the
tendency of the system to drop pages of program code, do not adjust this till you
understand it fully) and overall page fault rates on the system. You'll start geting
evictions long before you start using _any_ swap file space.
So if your effective throughput is low, the first thing to really look at is if
your page fault rates are rising. Variations of sar, ps, and top may be able
to tell you about the current system and/or per-process page fault rates.
You'll have to compare your distro's tool set to the procedures you can find
online.
It's a little pernicious because it's a silent performance drain. There are no system
messages to tell you "uh, hey dude, I'm doing a lot of reclaims lately and even
going back to disk for pages of this program you really like". You just have to know
how to look in that area.
However, your first suggestion caused me to slap myself:
Have you tried increasing the number of stripe buffers for the
filesystem?
This I had totally forgotten. When I bump up the stripe cache size, it
*seems* (so far, at least) to eliminate the slowest performance I'm seeing -
specifically, the periods I've been seeing where no I/O at all seems to
happen, plus the long runs of 1-3MB/s. The copy is now staying pretty much in
the 22-27MB/s range.
That's not as fast as the hardware is capable of - as I say, with other
filesystems on the same hardware, I can easily see 100+MB/s - but it's much
better than it was.
Is this remaining difference (25 vs 100+ MB/s) simply due to btrfs not being
tuned for performance yet, or is there something else I'm probably
overlooking?
I find BTRFS can be a little slow on my laptop, but I blame memory pressure
evicting important structures somewhat system wide. Which is part of why I did
the moveablecore= parametric tuning. I don't think there is anything that will
pack the locality of the various trees, so you can end up needing bits of
things from all over your disk in order to sequentially resolve a large
directory and compute the running checksums for rsync (etc.).
Simple rule of thumb, if "wait for I/O time" has started to rise you've got
some odd memory pressure that's sending you to idle land. It's not hard-and-fast as a
rule, but since you've said that your CPU load (wich I'm taking to be the user+system
time) is staying low you are likely waiting for something.
Capturing "echo t >/proc/magic-sysrq" on waiting for I/O may help you.
It shows us where kernel actually be waiting for.
In addition, to confirm whether this problem is caused only by
Btrfs or not, the following way can be used.
1. preparing the extra storage,
2. copy Btrfs's data into int by dd if=<LVM volume> of=<extra storage>
3. Use it and confirm whether this problem still happen or not
However, since the size of your Btrfs is quite large, I guess you
can't do it. If you have such extra storage, you've already
embed it to Btrfs.
Thanks,
Satoru
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
