Re: ReiserFS v3 choking when free space falls below 10% - FIXED
Hello, I'm hope I'm doing this right here, The question is: Is or since when is the the patch which helped Mike Benoit integrated to the kernel source? Greetings bernd_b
Re: ReiserFS v3 choking when free space falls below 10% - FIXED
Interesting that you sent this today. Just last night I really started to notice slow down on my MythTV box. It had been going on for a while, but I finally got fed up last night enough to look in to it, and what I found was this: 0 1324 4904 10184 6732800 1408 1060 1033 1393 17 5 5 73 0 1324 5204 10248 6692400 1284 3372 872 1324 16 5 23 57 0 1324 5288 10264 6674800 1280 1696 856 1361 13 2 23 62 1 1324 5076 10228 6701600 1032 2264 872 1399 16 2 46 37 0 3324 5160 10212 6713600 1428 2064 1007 1398 14 4 21 61 0 1324 4896 10272 6716800 1124 1712 867 1358 15 5 21 59 0 1324 5024 10256 6716000 1412 2328 1013 1379 16 4 21 59 0 1324 5100 10252 6701200 1024 1704 848 1378 14 4 40 43 0 1324 5584 10196 6655200 1284 1672 856 1360 13 4 28 55 0 1324 5608 10200 6681600 1168 2344 940 1489 17 4 27 52 0 0324 5880 10280 6640000 1288 3192 1092 1447 20 4 31 45 1 0324 5824 10160 6656400 1152 1996 997 1366 14 3 27 56 1 0324 5716 10144 6668000 1280 1616 855 1364 16 3 12 70 0 0324 6220 10084 6614000 1152 1960 991 1351 14 4 19 63 0 0324 6120 10076 6635600 1416 2184 1122 1556 19 4 48 29 Why is IO wait so high when its only reading/writing combined less then 4mb/sec? Before the patch there was a cut off point where the box just died, but now it seems like the box is just always slow. Could the patch be causing even more fragmentation then before, so while the corner case of virtually bringing the box to its knees is fixed, it is just always slow now? BTW: The vmstat output was with 25gb free, I've seen it happen with as much as 40gb free too. :( I really hope Reiser4 gets in to the kernel soon, hopefully allocate on flush greatly reduces the fragmentation caused by Myth. On Fri, 2006-08-18 at 16:41 +0200, Bernd Butscheidt wrote: Hello, I'm hope I'm doing this right here, The question is: Is or since when is the the patch which helped Mike Benoit integrated to the kernel source? Greetings bernd_b -- Mike Benoit [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: ReiserFS v3 choking when free space falls below 10% - FIXED
Sorry, here is the vmstat output with column headers. procs ---memory-- ---swap-- -io --system-- cpu r b swpd free buff cache si sobibo incs us sy id wa 1 0324 5976 10196 6635600 1722 1831 759 1496 89 4 1 7 0 1324 4904 10184 6732800 1408 1060 1033 1393 17 5 5 73 0 1324 5204 10248 6692400 1284 3372 872 1324 16 5 23 57 0 1324 5288 10264 6674800 1280 1696 856 1361 13 2 23 62 1 1324 5076 10228 6701600 1032 2264 872 1399 16 2 46 37 0 3324 5160 10212 6713600 1428 2064 1007 1398 14 4 21 61 0 1324 4896 10272 6716800 1124 1712 867 1358 15 5 21 59 0 1324 5024 10256 6716000 1412 2328 1013 1379 16 4 21 59 0 1324 5100 10252 6701200 1024 1704 848 1378 14 4 40 43 0 1324 5584 10196 6655200 1284 1672 856 1360 13 4 28 55 0 1324 5608 10200 6681600 1168 2344 940 1489 17 4 27 52 0 0324 5880 10280 6640000 1288 3192 1092 1447 20 4 31 45 1 0324 5824 10160 6656400 1152 1996 997 1366 14 3 27 56 1 0324 5716 10144 6668000 1280 1616 855 1364 16 3 12 70 0 0324 6220 10084 6614000 1152 1960 991 1351 14 4 19 63 0 0324 6120 10076 6635600 1416 2184 1122 1556 19 4 48 29 0 1324 5388 10172 6685600 1404 2260 1153 1418 14 6 21 59 0 1324 6288 10180 6600800 1412 2040 1012 1377 14 3 0 83 On Fri, 2006-08-18 at 16:41 +0200, Bernd Butscheidt wrote: Hello, I'm hope I'm doing this right here, The question is: Is or since when is the the patch which helped Mike Benoit integrated to the kernel source? Greetings bernd_b -- Mike Benoit [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: ReiserFS v3 choking when free space falls below 10% - FIXED
David Masover wrote: As a future MythTV user a bit late to this discussion, I'm curious -- was this Reiser3 or 4? Are there any known MythTV issues with v4? I say this because the box with my capture card is running on a Reiser4 root right now... I think you get to be the one to tell us
Re: ReiserFS v3 choking when free space falls below 10% - FIXED
Mike Benoit wrote: Thanks for all your hard work, I'm sure many other MythTV users will be appreciate it. As a future MythTV user a bit late to this discussion, I'm curious -- was this Reiser3 or 4? Are there any known MythTV issues with v4? I say this because the box with my capture card is running on a Reiser4 root right now... signature.asc Description: OpenPGP digital signature
Re: ReiserFS v3 choking when free space falls below 10% - FIXED
On Tue, 2006-07-25 at 19:10 -0500, David Masover wrote: Mike Benoit wrote: Thanks for all your hard work, I'm sure many other MythTV users will be appreciate it. As a future MythTV user a bit late to this discussion, I'm curious -- was this Reiser3 or 4? Are there any known MythTV issues with v4? I say this because the box with my capture card is running on a Reiser4 root right now... It was Reiser3. I personally don't know of any issues with Reiser4 and MythTV, but if Reiser4 has pauses or hangs during flush that I have heard so much about, I could see that posing a problem to MythTV. -- Mike Benoit [EMAIL PROTECTED] signature.asc Description: This is a digitally signed message part
Re: ReiserFS v3 choking when free space falls below 10% - FIXED
I applied the attached patch that Jeff supplied me and so far it is
working flawlessly. I currently have less than 4% free space on my drive
and the CPU usage is less then 3% with two recordings going. I'll let it
run until about 2% free space just to test further.
It also _appears_ that overall CPU usage is down slightly based on the
vmstat output from when we were trying to diagnose the problem before
compared to now. The SYS CPU time was hovering between 3-10% before, and
now it seems to be between 0-2%. I haven't done any actual performance
tests though.
Jeff, what drawbacks does this patch have?
Thanks for all your hard work, I'm sure many other MythTV users will be
appreciate it.
On Thu, 2006-06-29 at 10:41 -0700, Mike Benoit wrote:
My MythTV box recently started showing odd behavior during recordings,
at certain times the load of the box would spike to 10+ and recordings
would start losing frames and become unwatchable. TOP would show
mythbackend as using 90+% SYS CPU usage, which under normal
circumstances it normally uses about 5% USR.
So I finally got around to profiling mythbackend when the load starts to
spike. To my surprise it appears that once I have less then 10% (30GB)
free on the drive reiserfs can't up, even just writing at 1mb/sec is too
much for it.
Is there something that can be done to fix this, 30gb seems like a lot
of wasted space.
#opreport
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
TIMER:0|
samples| %|
--
77863 78.7856 reiserfs
18183 18.3984 vmlinux
695 0.7032 mysqld
452 0.4574 libc-2.4.so
360 0.3643 libmythtv-0.19.so.0.19.0
324 0.3278 ivtv
323 0.3268 nvidia
242 0.2449 libqt-mt.so.3.3.6
110 0.1113 libpthread-2.4.so
53 0.0536 libstdc++.so.6.0.8
35 0.0354 ld-2.4.so
23 0.0233 libperl.so
22 0.0223 libz.so.1.2.3
snip
#opreport -l /usr/src/linux/vmlinux
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples %symbol name
9607 52.8351 default_idle
7694 42.3142 find_next_zero_bit
183 1.0064 __copy_from_user_ll
570.3135 handle_IRQ_event
370.2035 __copy_to_user_ll
340.1870 ide_outb
300.1650 ide_end_request
220.1210 ioread8
220.1210 schedule
210.1155 get_page_from_freelist
170.0935 mmx_clear_page
snip
System Details:
---
Kernel v2.6.16.21 (custom compiled)
- This issue also happened with 2.6.14 too though.
FilesystemSize Used Avail Use% Mounted on
/dev/hda1 280G 269G 12G 97% /
[EMAIL PROTECTED] cat /proc/mounts
rootfs / rootfs rw 0 0
/dev /dev tmpfs rw 0 0
/dev/root / reiserfs rw,noatime,nodiratime 0 0
[EMAIL PROTECTED] cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 6
model name : AMD Athlon(tm) XP 2100+
stepping: 2
cpu MHz : 1759.680
cache size : 256 KB
[EMAIL PROTECTED] free
total used free sharedbuffers
cached
Mem:515992 496256 19736 0 36256
271728
-/+ buffers/cache: 188272 327720
Swap: 262136408 261728
[EMAIL PROTECTED] ~]# hdparm -i /dev/hda
/dev/hda:
Model=ST3300622A, FwRev=3.AND, SerialNo=3NF1GAGW
Config={ HardSect NotMFM HdSw15uSec Fixed DTR10Mbs RotSpdTol.5% }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
BuffType=unknown, BuffSize=16384kB, MaxMultSect=16, MultSect=16
CurCHS=4047/16/255, CurSects=16511760, LBA=yes, LBAsects=268435455
IORDY=on/off, tPIO={min:240,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5
AdvancedPM=no WriteCache=enabled
Drive conforms to: Unspecified: ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3
ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6 ATA/ATAPI-7
* signifies the current active mode
[EMAIL PROTECTED] ~]# hdparm -tT /dev/hda
/dev/hda:
Timing cached reads: 1296 MB in 2.00 seconds = 646.99 MB/sec
Timing buffered disk reads: 166 MB in 3.02 seconds = 55.05 MB/sec
vmstat 1 output:
--
procs ---memory-- ---swap-- -io --system--
cpu
r b swpd free buff cache si sobibo incs us sy
id wa
8 0408 5800 29308 24860400 0 1036 406 132 2 98
0 0
4 0408 5644 29396 24860800 0 1128 437 184 2 92
0 6
7 0408 6316 29428 24802000 0 1316 539 287 0 86
0 14
5 0408 6104 29480 24818000 0 588 415 187 0 99
0 1
4 0408 5764 29536 24836400 0 1092 421
Re: ReiserFS v3 choking when free space falls below 10% - FIXED
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mike Benoit wrote: I applied the attached patch that Jeff supplied me and so far it is working flawlessly. I currently have less than 4% free space on my drive and the CPU usage is less then 3% with two recordings going. I'll let it run until about 2% free space just to test further. It also _appears_ that overall CPU usage is down slightly based on the vmstat output from when we were trying to diagnose the problem before compared to now. The SYS CPU time was hovering between 3-10% before, and now it seems to be between 0-2%. I haven't done any actual performance tests though. Jeff, what drawbacks does this patch have? Thanks for all your hard work, I'm sure many other MythTV users will be appreciate it. Hi Mike - There really shouldn't be any. I suspect that the window searching was actually causing more problems than it was solving. The original goal would have been to try to keep chunks of blocks contiguous for better access patterns, but if those chunks end up getting spread out all over the disk, that's hardly the outcome we were looking for. So, what will now happen is that the allocator will allocate the next n blocks it can find, regardless of the window size. If there happens to be a window of the size we needed, it will automatically find it through the normal process of allocating one block at a time. - -Jeff - -- Jeff Mahoney SUSE Labs -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFExUqOLPWxlyuTD7IRAuaFAJ47W+zr2ZwIs//vMgm3RNHuw4dpwACdECdv ueI91PGuCLQdeKipY5G9kqk= =vk6Z -END PGP SIGNATURE-
