Hi Jeff,
Like clock work the problem showed up pretty much exactly when I
expected. This time however I discovered a few other interesting tidbits
along the way.
I tried to re-create the problem much faster by writing a little script
that would append data to a file at about 5mb/s, I ran two instances of
this script simultaneously, so each script was writing to two separate
files at the same time. After the files reached 2gb (close to the same
size of a recording) both scripts would start writing to new files. Two
recordings were also going on at the same time as all of this. I filled
the drive up so about 10gb (5%) was free, and although the write speed
dropped off significantly as the drive filled up, the SYS CPU time never
increased. So this method obviously failed to re-create the problem, so
I deleted all these files the script created (~60gb worth) and let
MythTV do its thing until the drive filled up that way. The pattern
MythTV writes data out must have something to do with this?
The other interesting thing that happened was at about 11pm tonight I
noticed the problem started occurring, (SYS CPU was high) but MythTV was
transcoding MPEG2 recordings to MPEG4 which was using USR CPU time, so I
tried to stop this process so the oprofile data wouldn't be so
cluttered. Well when I stopped the transcoding, MythTV deleted the temp
file it had been creating and this caused the problem stop almost
immediately!
I didn't realize MythTV deleted this temp file at first, so I was
disappointed and therefore started the transcoding again in hopes of
re-creating the problem. Within about 45mins, it started happening
again, but this time I renamed the temp file (so it wouldn't get
deleted) MythTV was trancoding to before I killed that process, and was
able to keep the problem happening long enough to get a clean oprofile
and better vmstat data.
The really interesting thing is how much free space was available each
time the problem hit:
Time #1:
Thu Jul 6 22:53:26 PDT 2006
/dev/hda1 293024652 269089456 23935196 92% /
Time #2 (45mins later):
Thu Jul 6 23:35:38 PDT 2006
/dev/hda1 293024652 269227580 23797072 92% /
It seems like once the free space hits a very specific point, the
problem is triggered. As you will notice in the vmstat logs, within
about 30seconds the SYS CPU time goes from 4% to 75+% and hovers there.
Attached are the vmstat logs and oprofile report and I'm sending you the
output of debugreiserfs -p /dev/hda1 when the drive is 92% full
privately (when it finishes, could be morning).
Just so you know, I had vmstat set to output 6 times every 10 seconds,
then I ran date/df, rinse and repeat.
This is the script I used to collect the data:
{
while [ "1" != "0" ] ; do
date
df
vmstat 10 6
done
} 2>>/tmp/monitor.log 1>>/tmp/monitor.log
vmstat_1.txt is the first time the problem occurred.
vmstat_2.txt is the second time the problem occurred.
Hopefully this helps you track down the issue. If not, let me know if
you want me to collect more data. I'll try to keep the drive as full as
possible so I can re-create the problem much faster. Also if you need
access to the box, that can be arranged.
Thanks.
PS. I'm running kernel v2.6.16.21-rfsfix, "rfsfix" is the following
patch you sent me. I experienced the problem on kernels as old as
2.6.14.
diff -ruNpX ../dontdiff linux-2.6.15.orig.staging1/fs/reiserfs/bitmap.c
linux-2.6.15.orig.staging2/fs/reiserfs/bitmap.c
--- linux-2.6.15.orig.staging1/fs/reiserfs/bitmap.c 2006-01-16
16:53:35.663319136 -0500
+++ linux-2.6.15.orig.staging2/fs/reiserfs/bitmap.c 2006-01-16
16:53:35.673317616 -0500
@@ -187,7 +187,10 @@ static int scan_bitmap_block(struct reis
return 0; // No free blocks in this bitmap
}
- /* search for a first zero bit -- beggining of a window
*/
+ if (*beg < bi->first_zero_hint)
+ *beg = bi->first_zero_hint;
+
+ /* search for a first zero bit -- beginning of a window
*/
*beg = reiserfs_find_next_zero_le_bit
((unsigned long *)(bh->b_data), boundary, *beg);
On Thu, 2006-07-06 at 14:39 -0400, Jeff Mahoney wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Mike Benoit wrote:
> > On Thu, 2006-07-06 at 14:02 -0400, Jeff Mahoney wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> Mike Benoit wrote:
> >>> My desktop machine (v2.6.16, same as my MythTV box) is running with 9%
> >>> free space right now and it is not experiencing any slow down. I think
> >>> the problem is caused by the usage pattern of MythTV and how it
> >>> simultaneously streams one or more large files to the HD in relatively
> >>> small chunks over a long period of time.
> >> Ok, if you run into the problem again, can you dump the metadata before
> >> freeing the space? The code itself looks sound, and I'm wondering if
> >> you've managed to create pathological fragmentation that's mucking
> >> things up. Being able to see the fs metadata would confirm or disprove
> >> that theory, and help in fixing it.
> >
> > Will do, I've started a bunch of recordings so I should start seeing the
> > problem again by tonight or tomorrow morning. Is there any other data
> > you would like me to collect? Additional oprofile reports, vmstat
> > information before the problem occurs and/or after?
>
> Great. Any information you can provide would help quite a bit. oprofile
> would be useful, as would vmstat information.
>
> - -Jeff
>
> - --
> Jeff Mahoney
> SUSE Labs
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
> Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org
>
> iD8DBQFErVjRLPWxlyuTD7IRAlgoAKCRqtHLk6Uq9Bp3yZq/18tHt8l2mwCfT206
> UMSE1Om/pvg+svHImWkwLT8=
> =I5Oj
> -----END PGP SIGNATURE-----
--
Mike Benoit <[EMAIL PROTECTED]>
[EMAIL PROTECTED] tmp]# opreport
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
TIMER:0|
samples| %|
------------------
49983 87.9595 reiserfs
6146 10.8157 vmlinux
162 0.2851 libc-2.4.so
111 0.1953 nvidia
99 0.1742 libmythtv-0.19.so.0.19.0
90 0.1584 ivtv
42 0.0739 mysqld
35 0.0616 libstdc++.so.6.0.8
33 0.0581 opreport
26 0.0458 libqt-mt.so.3.3.6
14 0.0246 ld-2.4.so
14 0.0246 libperl.so
13 0.0229 libbfd-2.16.91.0.6.so
<snip>
[EMAIL PROTECTED] tmp]# opreport -l /usr/src/linux-2.6.16.21-rfsfix/vmlinux
CPU: CPU with timer interrupt, speed 0 MHz (estimated)
Profiling through timer interrupt
samples % symbol name
3393 55.2066 find_next_zero_bit
2256 36.7068 default_idle
114 1.8549 __do_softirq
56 0.9112 __copy_from_user_ll
16 0.2603 __copy_to_user_ll
13 0.2115 __find_get_block
13 0.2115 mmx_clear_page
12 0.1952 ide_outb
10 0.1627 number
8 0.1302 ioread8
7 0.1139 find_get_page
<snip>
Thu Jul 6 22:51:45 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 268876336 24148316 92% /
ipso:/home/ipso/backup
116130272 103483184 12647088 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 0 80144 6100 12596 108080 6 7 15 62 8 82 46 2 42 10
1 1 80144 6236 12592 107936 0 0 1577 2174 571 1567 97 3 0 0
1 1 80144 6076 12608 107896 0 0 1607 2215 558 1556 97 3 0 0
4 0 80144 6192 12500 108128 0 0 1591 2456 562 1558 96 4 0 0
1 1 80144 6096 12524 108032 0 0 1578 2224 570 1523 93 7 0 0
2 1 80144 5436 12816 108400 0 0 1437 2256 558 1428 89 10 0 1
Thu Jul 6 22:52:35 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 268983824 24040828 92% /
ipso:/home/ipso/backup
116130272 103483184 12647088 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 80144 5980 13012 107732 6 7 16 63 8 83 46 2 42 10
2 0 80144 6016 13120 107632 0 0 1526 2063 575 1496 91 9 0 1
2 1 80144 6116 13004 107392 0 0 1439 2348 571 1453 89 10 0 1
4 0 80144 6252 13092 107388 0 0 1526 2354 563 1542 94 5 0 1
3 0 80144 6268 13156 107280 0 0 1553 2080 563 1499 92 8 0 0
3 1 80144 6288 13032 107264 0 0 1540 2383 576 1504 92 6 0 2
Thu Jul 6 22:53:26 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269089456 23935196 92% /
ipso:/home/ipso/backup
116130272 103483184 12647088 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 80144 5900 13112 107640 6 7 16 63 8 83 46 2 41 10
1 0 80144 6356 13196 107128 0 0 1512 2381 566 1492 90 8 0 2
2 1 80144 6484 13224 106820 0 0 1501 2089 571 1518 86 9 0 5
1 0 80144 6600 13456 106460 0 0 1514 2458 580 1564 92 7 0 1
2 1 80144 5704 13820 107024 0 0 1577 2208 562 1459 83 15 0 2
1 0 80144 6308 13612 106788 0 0 1476 2124 561 1397 82 17 0 1
Thu Jul 6 22:54:17 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269194040 23830612 92% /
ipso:/home/ipso/backup
116130272 103483064 12647208 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 80144 6440 13640 106588 6 7 16 64 8 83 46 2 41 10
6 1 80144 6656 13160 106636 0 0 1362 2432 567 1265 71 26 0 2
11 0 80144 5756 13112 107684 0 0 669 368 438 298 17 83 0 0
5 1 80144 6716 13192 106556 0 0 835 372 464 469 30 70 0 0
9 0 80144 5656 13268 107676 0 0 860 308 467 492 31 69 0 0
13 0 80144 6028 13196 107448 0 0 769 302 476 417 24 76 0 0
Thu Jul 6 22:55:09 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269227876 23796776 92% /
ipso:/home/ipso/backup
116130272 103483064 12647208 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 80144 6628 13224 106688 6 7 17 64 8 84 46 2 41 10
2 0 80144 16668 13124 97352 0 0 629 248 441 377 22 78 0 0
9 0 80144 15988 13520 100960 0 0 94 282 378 158 11 89 0 0
9 0 80144 12272 13768 104540 0 0 51 415 359 102 6 94 0 0
6 0 80144 7672 14112 108780 0 0 78 400 379 141 8 92 0 0
9 0 80144 5920 14424 110016 0 0 92 409 378 150 10 90 0 0
Thu Jul 6 22:56:00 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269243268 23781384 92% /
ipso:/home/ipso/backup
116130272 103483120 12647152 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
5 0 80144 6544 14456 109372 6 7 17 64 9 84 46 2 41 10
4 0 80144 5856 14644 109900 0 0 64 342 356 116 8 92 0 0
11 1 80140 6224 14900 109088 1 0 108 424 369 140 8 92 0 0
7 2 80140 5744 15052 109084 0 0 261 312 387 233 22 78 0 0
7 1 80132 6068 15404 107160 84 0 424 396 394 204 19 81 0 0
7 1 80128 6424 15476 107440 1 0 169 411 378 148 10 90 0 0
Thu Jul 6 22:56:52 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269259644 23765008 92% /
ipso:/home/ipso/backup
116130272 103483128 12647144 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 80128 6336 15536 107708 6 7 17 64 9 84 46 2 41 10
6 1 80124 6164 15728 107732 0 0 100 296 366 138 9 91 0 0
7 0 80124 5880 15836 107864 6 0 280 388 417 285 26 72 0 2
4 3 80120 5884 16080 106664 0 0 120 449 378 156 10 90 0 0
5 0 80116 5776 16192 107476 1 0 55 346 361 93 5 95 0 0
8 0 80112 6236 16488 106688 0 0 120 389 399 182 12 88 0 0
Thu Jul 6 22:57:43 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269276516 23748136 92% /
ipso:/home/ipso/backup
116130272 103483128 12647144 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
6 0 80112 6236 16488 106692 6 7 17 64 9 84 46 2 41 10
3 0 80112 6428 16648 106336 0 0 77 353 366 128 9 91 0 0
8 1 80108 6224 16872 105456 0 0 116 417 371 154 9 91 0 0
4 0 80108 6396 17084 105804 0 0 121 366 373 172 12 87 0 1
13 0 80108 6532 17320 105392 0 0 128 412 384 172 13 87 0 0
5 0 80104 5720 17684 105000 0 0 132 424 381 179 13 87 0 0
Thu Jul 6 22:58:34 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269293472 23731180 92% /
ipso:/home/ipso/backup
116130272 103483128 12647144 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
9 0 80104 5484 17684 105136 6 7 17 64 9 84 46 2 41 10
7 0 80104 6572 17896 104072 3 0 100 381 360 124 8 92 0 0
15 1 80104 6468 18332 103100 0 0 265 393 422 291 23 76 0 1
7 1 80104 6360 18816 103324 0 0 164 369 377 174 11 89 0 0
8 0 80076 6656 19040 102456 6 0 210 451 399 201 14 86 0 0
5 0 80076 5724 19220 103624 0 0 206 360 402 254 21 79 0 0
Thu Jul 6 22:59:26 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269309984 23714668 92% /
ipso:/home/ipso/backup
116130272 103483128 12647144 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 0 80076 5788 19220 103624 6 7 17 64 9 84 46 2 41 10
0 2 80052 7976 21904 121856 1 0 328 2766 466 391 3 59 0 37
2 0 80052 6352 21508 104400 0 0 728 2993 546 1090 60 6 1 33
1 0 80056 5732 21476 105328 0 0 770 2494 493 923 86 2 0 12
1 2 80052 6116 18996 95456 0 0 1622 1520 480 1023 93 4 0 3
4 0 80052 6392 19060 107020 0 0 1028 2472 512 1004 91 7 0 2
Thu Jul 6 23:00:16 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 268256840 24767812 92% /
ipso:/home/ipso/backup
116130272 103483128 12647144 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 80052 6824 19192 106672 6 7 17 65 9 84 46 3 41 10
2 0 80052 8016 17464 109484 0 0 799 2246 486 977 95 3 0 2
2 0 80052 6468 15016 113108 0 0 1027 2664 518 1066 96 3 0 1
1 0 80052 6000 15268 113620 0 0 808 2368 475 959 97 2 0 0
1 0 80052 6452 14864 113308 0 0 804 2577 497 984 96 2 0 2
0 1 80052 25848 16052 111012 18 0 478 1777 517 775 19 2 23 56
Thu Jul 6 23:01:07 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 268326500 24698152 92% /
ipso:/home/ipso/backup
116130272 103483128 12647144 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 1 80052 25788 16088 111052 6 7 17 66 9 84 46 3 41 10
0 0 80048 5644 17124 131912 1 0 178 2403 492 698 3 2 63 32
0 0 80048 5732 18144 130416 0 0 100 2128 512 727 1 2 66 31
0 0 80048 5784 18884 129856 0 0 8 1929 497 717 1 2 85 11
0 0 80048 6604 18592 129432 0 0 0 1841 453 647 0 1 95 5
0 0 80048 5960 18648 129904 0 0 2 1912 477 696 1 1 95 4
Thu Jul 6 23:01:57 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 268421444 24603208 92% /
ipso:/home/ipso/backup
116130272 103483136 12647136 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 80048 5704 18656 130184 6 7 17 66 9 84 46 3 41 10
0 0 80048 5596 18796 130104 0 0 2 2048 480 698 1 1 94 5
0 0 80048 6244 19416 128764 0 0 5 1982 477 678 1 2 85 12
0 0 80048 6052 19232 129272 0 0 6 1874 481 697 1 1 90 9
0 0 80048 5928 19604 128920 0 0 15 2039 488 716 1 2 93 5
0 0 80048 5560 19668 129188 0 0 22 1775 466 667 0 1 96 4
Thu Jul 6 23:31:25 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 268695440 24329212 92% /
ipso:/home/ipso/backup
116130272 103483600 12646672 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 1 53388 5600 11960 98968 6 7 28 87 15 96 46 3 41 10
1 1 53388 5496 11980 99376 0 0 1552 2160 555 1546 95 4 0 1
1 1 53388 6608 11612 98364 0 0 1552 2310 576 1575 96 3 0 0
2 1 53388 5288 11720 99584 0 0 1553 2129 577 1600 96 4 0 0
3 0 53388 5944 11920 98960 0 0 1541 2321 557 1566 96 4 0 1
2 1 53388 6444 12476 97612 0 0 1592 2315 587 1580 94 4 0 2
Thu Jul 6 23:32:15 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 268800636 24224016 92% /
ipso:/home/ipso/backup
116130272 103483600 12646672 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 53388 6388 12564 97604 6 7 29 87 15 96 46 3 41 10
1 0 53388 6576 12364 97692 0 0 1526 2268 581 1579 95 4 0 2
1 1 53388 5288 12312 98976 0 0 1552 2187 559 1561 96 4 0 0
2 1 53388 5308 12044 99084 0 0 1540 2144 574 1573 94 5 0 1
1 0 53388 6144 12052 98480 0 0 1529 2486 582 1566 96 3 0 0
1 1 53388 5604 11644 99240 0 0 1538 2150 550 1531 96 3 0 0
Thu Jul 6 23:33:06 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 268908104 24116548 92% /
ipso:/home/ipso/backup
116130272 103483600 12646672 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 1 53388 6448 11800 98196 6 7 29 88 15 97 46 3 41 10
1 1 53388 4964 11952 99604 0 0 1488 2306 575 1540 93 4 0 3
2 1 53388 5624 12276 98516 0 0 1491 2370 586 1570 95 4 0 1
2 0 53388 6012 12480 97808 0 0 1514 2235 577 1552 96 4 0 0
1 1 53388 5308 12712 98296 0 0 1564 2327 590 1624 95 4 0 1
1 0 53388 5836 12536 98120 0 0 1565 2404 599 1596 96 4 0 0
Thu Jul 6 23:33:57 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269015876 24008776 92% /
ipso:/home/ipso/backup
116130272 103483648 12646624 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 1 53388 4960 12572 98752 6 7 30 89 15 97 46 3 41 10
1 0 53388 5856 12652 98060 0 0 1542 2270 586 1576 96 3 0 1
1 1 53388 5452 12372 98472 0 0 1564 2261 572 1615 97 3 0 0
1 1 53388 5188 12136 98964 0 0 1554 2206 586 1558 95 4 0 1
4 0 53388 6292 12120 97976 0 0 1567 2288 582 1562 94 4 0 3
1 0 53388 6380 12184 97688 0 0 1553 2417 572 1562 95 3 0 2
Thu Jul 6 23:34:47 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269121368 23903284 92% /
ipso:/home/ipso/backup
116130272 103483648 12646624 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 53388 5328 12316 98556 6 7 30 89 15 98 46 3 41 10
1 1 53388 5116 12568 98580 0 0 1514 2254 578 1571 96 4 0 0
1 2 53388 6280 12264 97592 0 0 1554 2137 574 1562 95 4 0 1
1 1 53388 4980 12220 99000 0 0 1564 2381 560 1552 96 4 0 0
2 0 53388 6484 12156 97656 0 0 1528 2440 576 1553 94 6 0 0
2 1 53388 6080 12104 97992 0 0 1475 2201 572 1397 82 17 0 1
Thu Jul 6 23:35:38 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269227580 23797072 92% /
ipso:/home/ipso/backup
116130272 103483648 12646624 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 1 53388 5392 12140 98460 6 7 30 90 15 98 46 3 41 10
2 2 53388 5040 12340 98800 0 0 1398 2354 556 1345 82 18 0 1
10 0 53388 5820 12464 97976 0 0 1349 2136 574 1318 77 22 0 1
1 0 53388 6348 12268 97352 0 0 1207 1725 543 1029 61 39 0 1
10 1 53388 18020 12472 90884 0 0 366 1730 445 490 27 73 0 0
8 0 53388 13616 12744 95320 0 0 39 478 348 93 3 97 0 0
Thu Jul 6 23:36:30 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269306828 23717824 92% /
ipso:/home/ipso/backup
116130272 103483648 12646624 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
4 0 53388 12564 12796 96276 6 7 31 90 16 98 46 3 41 10
6 1 53388 6844 13148 100976 1 0 115 392 374 135 7 93 0 0
9 1 53388 6376 13368 101052 0 0 316 365 427 304 29 70 0 1
8 1 53384 6464 13960 100628 3 0 262 446 435 263 18 80 0 2
8 1 53384 6564 14556 99936 0 0 274 407 398 289 14 85 0 1
11 0 53384 9348 14768 96812 0 0 246 399 395 229 11 89 0 0
Thu Jul 6 23:37:24 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269323968 23700684 92% /
ipso:/home/ipso/backup
116130272 103483656 12646616 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 0 53384 9332 14812 97072 6 7 31 90 16 98 46 3 41 10
6 0 53384 5636 15028 100700 0 0 90 407 377 146 10 90 0 0
10 0 53384 6600 15944 98740 0 0 106 495 390 396 10 90 0 0
11 1 53380 6656 16168 98304 0 0 86 436 392 199 9 91 0 0
6 1 53376 26724 16540 100164 1 0 72 465 385 171 8 86 0 6
0 2 52636 20920 17076 104348 1 0 49 497 383 149 1 88 0 11
Thu Jul 6 23:38:16 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269342424 23682228 92% /
ipso:/home/ipso/backup
116130272 103483656 12646616 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 2 52636 20920 17076 104348 6 7 31 91 16 98 46 3 41 10
8 1 52636 16932 17996 107840 0 0 75 476 439 258 1 68 0 30
5 2 52636 8120 18392 114748 0 0 206 570 379 124 1 91 0 8
7 1 52636 6448 18636 115504 0 0 11 470 350 66 1 98 0 1
0 2 52636 5784 18972 115464 0 0 47 518 344 76 1 97 0 2
7 0 52636 5960 18528 98312 0 0 213 372 370 187 13 86 0 1
Thu Jul 6 23:39:07 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269362396 23662256 92% /
ipso:/home/ipso/backup
116130272 103483656 12646616 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
10 0 52636 5812 18528 98452 6 7 31 91 16 98 46 3 41 10
3 1 52636 8996 18800 94600 0 0 5 583 394 180 11 89 0 0
4 0 52636 6140 18876 97804 0 0 110 404 376 149 11 88 0 1
8 0 52636 6020 18932 97824 0 0 103 407 369 150 10 90 0 0
8 0 52636 6324 19176 97244 0 0 52 470 364 128 7 93 0 0
5 0 52636 6552 19272 97024 0 0 97 379 405 214 11 89 0 0
Thu Jul 6 23:39:58 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269381820 23642832 92% /
ipso:/home/ipso/backup
116130272 103483656 12646616 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
7 0 52636 6528 19272 97024 6 7 31 91 16 98 46 3 41 10
9 0 52636 6692 19464 98536 0 0 62 339 386 181 13 87 0 0
7 0 52636 6320 19512 99264 0 0 90 440 386 169 12 88 0 0
8 0 52636 6280 19812 98972 0 0 90 459 373 140 9 91 0 0
4 0 52636 6100 20084 98796 0 0 102 347 392 214 14 86 0 0
7 0 52636 17288 20376 99256 0 0 46 514 368 124 6 92 0 1
Thu Jul 6 23:40:50 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269399972 23624680 92% /
ipso:/home/ipso/backup
116130272 103483664 12646608 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
10 0 52636 17044 20376 99516 6 7 31 91 16 98 46 3 41 10
7 0 52636 25408 20900 98952 0 0 68 486 392 159 1 87 6 6
2 0 52636 20736 21580 102904 0 0 21 547 404 164 1 88 0 10
3 0 52636 16760 21812 106748 0 0 0 482 358 93 1 93 0 6
3 1 52636 11900 22012 110892 0 0 8 392 346 73 1 98 0 1
2 0 52636 7336 22988 114776 0 0 6 501 344 87 1 98 0 1
Thu Jul 6 23:41:41 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269414460 23610192 92% /
ipso:/home/ipso/backup
116130272 103483664 12646608 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 0 52636 7336 22988 114776 6 7 31 91 16 98 46 3 41 10
3 1 52636 5840 23260 115876 0 0 0 521 366 107 1 90 0 9
4 0 52636 6508 23600 114952 0 0 0 464 368 104 0 93 0 7
0 1 52636 9292 24336 111212 0 0 2 525 386 171 2 87 3 8
3 0 52636 6136 24540 114204 0 0 0 400 345 76 1 97 0 3
6 0 52636 6428 24408 114164 0 0 0 436 358 84 1 93 0 6
Thu Jul 6 23:42:32 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269433368 23591284 92% /
ipso:/home/ipso/backup
116130272 103483664 12646608 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
7 0 52636 6420 24408 114176 6 7 31 91 16 98 46 3 41 10
4 1 52636 9080 23816 112132 0 0 3 486 361 90 2 98 0 0
4 0 52636 5832 24008 115140 0 0 0 417 330 53 0 99 0 0
6 0 52636 6100 23812 115112 0 0 0 416 346 74 0 97 0 3
5 0 52636 6336 23504 115224 0 0 1 455 364 87 0 95 0 4
3 0 52636 6316 23400 115240 0 0 1 525 379 141 1 90 3 7
Thu Jul 6 23:43:23 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269453124 23571528 92% /
ipso:/home/ipso/backup
116130272 103483664 12646608 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
6 0 52636 8492 23212 112768 6 7 31 91 16 98 46 3 41 10
9 0 52636 5968 23452 114660 0 0 0 538 346 74 1 97 0 2
6 0 52636 8756 23268 112824 0 0 7 535 375 102 1 97 0 2
5 1 52636 5988 22920 116024 0 0 0 484 345 58 1 99 0 0
3 0 52636 6396 22988 115488 0 0 0 507 339 58 1 99 0 0
0 0 52636 6436 23324 115120 0 0 0 533 362 111 1 92 2 5
Thu Jul 6 23:44:16 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269476608 23548044 92% /
ipso:/home/ipso/backup
116130272 103483704 12646568 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 52636 6420 23324 115120 6 7 31 91 17 98 46 3 41 10
2 1 52636 6160 23528 115164 0 0 2 511 370 139 2 93 5 0
5 0 52636 5956 23740 115184 0 0 0 502 339 56 1 99 0 1
6 0 52636 6000 24144 114028 0 0 4 450 347 85 1 98 0 1
6 0 52636 5956 24328 114596 0 0 14 518 373 94 1 98 0 0
4 0 52636 6248 23716 114932 0 0 0 549 347 75 1 99 0 0
Thu Jul 6 23:45:07 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269499692 23524960 92% /
ipso:/home/ipso/backup
116130272 103483704 12646568 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 1 52636 5876 23800 115156 6 7 31 91 17 98 46 3 41 10
4 0 52636 5724 23088 116072 0 0 3 490 382 105 1 94 0 5
9 1 52632 5640 23468 115632 0 0 5 564 387 156 1 90 5 4
5 0 52632 5844 23608 115392 0 0 0 460 368 89 1 95 0 4
5 0 52632 9172 23528 112108 0 0 3 442 385 120 1 98 0 0
4 0 52632 5628 23824 115468 0 0 0 447 337 56 1 99 0 0
Thu Jul 6 23:45:58 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269520312 23504340 92% /
ipso:/home/ipso/backup
116130272 103483704 12646568 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 0 52632 5620 23824 115480 6 7 31 92 17 98 46 3 41 10
4 0 52632 5856 24092 114976 0 0 0 452 339 56 1 99 0 0
10 2 52632 6916 24400 111148 0 0 9 469 368 109 2 93 0 5
1 0 52632 6232 24704 110184 0 0 6 434 386 167 5 88 3 5
10 1 52632 5744 24736 110920 0 0 36 457 376 153 1 93 0 6
6 1 52632 6288 25004 110036 0 0 121 561 367 112 5 94 0 1
Thu Jul 6 23:46:50 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269540344 23484308 92% /
ipso:/home/ipso/backup
116130272 103483712 12646560 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
6 1 52632 6148 25004 110184 6 7 31 92 17 98 46 3 41 10
5 1 52632 16564 24844 100752 0 0 375 486 405 224 10 81 0 9
5 1 52632 12296 25336 105868 0 0 82 514 370 125 8 89 0 3
10 0 52632 8568 25696 109268 0 0 74 453 383 198 12 81 2 5
5 0 52632 6044 25880 111568 0 0 0 529 362 221 25 74 0 0
0 0 52616 17924 25336 102048 0 0 272 352 356 333 11 36 48 5
Thu Jul 6 23:47:41 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269555780 23468872 92% /
ipso:/home/ipso/backup
116130272 103483712 12646560 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 52616 17924 25336 102048 6 7 31 92 17 98 46 3 41 10
0 0 52616 17992 25348 102048 0 0 0 2 312 322 0 0 100 0
0 0 52616 17996 25360 102048 0 0 0 2 312 324 0 0 100 0
0 0 52616 17996 25600 102112 0 0 0 52 320 325 0 0 100 0
0 0 52616 17996 25612 102112 0 0 0 2 312 324 0 0 99 0
0 0 52616 17996 25624 102112 0 0 0 2 312 323 0 0 100 0
Thu Jul 6 23:48:31 PDT 2006
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda1 293024652 269555860 23468792 92% /
ipso:/home/ipso/backup
116130272 103483712 12646560 90% /mnt/backup
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 52616 17972 25624 102112 6 7 31 92 17 99 46 3 41 10
0 0 52616 17988 25664 102116 0 0 0 11 314 324 0 1 100 0
0 0 52616 17872 25688 102116 0 0 0 4 312 322 0 0 100 0
0 0 52616 17872 25716 102120 0 0 0 6 312 324 0 0 99 0
0 0 52616 17872 25732 102120 0 0 0 8 314 323 0 0 100 0
0 0 52616 17748 25896 102120 0 0 0 33 315 322 0 1 100 0
signature.asc
Description: This is a digitally signed message part
