Re: [BackupPC-users] BackupPC_nightly takes too much time
Hello Ryan, 1) Raid5X is going to kill your performance. As mentioned in a previous message, convert to RAID10 if you can afford the space decrease. The write speed of RAID 50 is higher than RAID 5 and i think this is not the problem on my system. The backuppc server input - reading from disk - takes to much time. The backups of new hosts with a lot of data are fast. It is only the BackupPC_nightly process. Yesterday i changed BackupPCNightlyPeriod to 4 and it seems ok. http://www.astro.ufl.edu/~ken/perf_tests.html 2) Make sure you are setting the 'noatime' option on your backuppc pool storage partition Ok, i change this now. 3) Edit your updatedb.conf and add the path to your backuppc pool to PRUNEPATHS. Ok, i change this now. Also, are you using a BBU on your Adaptec 3805 card? I think it is enabled. Regards Norbert -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
What filesystem are you using? The standard filesystem of ubuntu 9.10 = EXT4 Regards Norbert -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Using PAE, you can have 3.5 G of usable ram on a system. HOWEVER, each r...@server:~# uname -a Linux server 2.6.31-20-generic-pae #58-Ubuntu SMP Fri Mar 12 06:25:51 UTC 2010 i686 GNU/Linux Currently still running admin4/8 01:00 BackupPC_nightly -m 0 255 11896 top - 09:47:18 up 19 days, 18:23, 1 user, load average: 2.08, 2.10, 2.07 Tasks: 151 total, 1 running, 150 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 85.3%id, 14.7%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.3%sy, 0.0%ni, 94.3%id, 5.4%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.3%us, 0.0%sy, 0.0%ni, 91.1%id, 8.5%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.2%sy, 0.0%ni, 25.0%id, 74.8%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8261580k total, 991692k used, 7269888k free, 687680k buffers Swap: 3906240k total,18016k used, 3888224k free,89412k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 11896 backuppc 20 0 10364 7448 2136 D1 0.1 5:20.53 BackupPC_nightl r...@server:~# vmstat 10 10 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 3 18016 7271832 687296 8771200 3 231 2 1 78 18 0 2 18016 7270656 687868 8891200 5794 2 570 244 0 0 61 39 0 2 18016 7271176 687488 8891200 651551 497 271 0 0 71 29 0 2 18016 7271068 687496 8892000 549650 494 188 0 0 68 32 0 2 18016 7271208 687360 8904000 5476 2 478 194 0 0 69 31 0 3 18016 7271456 687224 8862400 5507 5 499 207 0 0 68 32 0 2 18016 7270992 688024 8873200 5826 0 506 216 0 0 69 31 0 2 18016 7270868 687884 8875200 5502 3 495 201 0 0 69 31 0 2 18016 7272132 686736 8841600 5440 2 505 203 0 0 68 31 0 2 18016 7271768 687400 8834000 5660 0 482 202 0 0 69 30 IMHO the server is reading from the filesystem and this is to slow? Regards Norbert -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On Wed, Apr 07, 2010 at 05:02:33PM -0400, Josh Malone wrote: OS is Ubuntu 9.04 32Bit IMHO it is better to migrate to a 64Bit-System!? I don't see an urgent reason to migrate to 64 bit... I would have installed this machine 64 bit at the beginning, just because it's a 64 bit machine. You'll lose some performance, but it might be barely noticeable. I'm not so sure that's the case. My understanding is that a 32-bit OS can only address a little over 3GB of physical memory, since the system has 8GB, I would think you would want to upgrade to a 64-bit OK. Richard Using PAE, you can have 3.5 G of usable ram on a system. HOWEVER, each individual process only has a 4GB virtual address space, so only 4G of ram per process. If you have 1 memory-intensive process you can make use of 8G of ram on a 32-bit system. Right. But we're talking about BackupPC_nightly here which doesn't require loads of memory. As we can see in vmstat output, about 6 GB of RAM are used as disk cache. So we're on the safe side here regarding 32bit vs. 64bit. HTH, Tino. -- What we nourish flourishes. - Was wir nähren erblüht. www.lichtkreis-chemnitz.de www.tisc.de -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On Wed, Apr 07, 2010 at 01:03:18PM +0200, Norbert Schulze wrote: Or just post the output of vmstat 10 10 r...@server:~# vmstat 10 10 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 5 14764 6196504 687296 20274400 6 115 2 1 78 18 0 5 14764 6184340 688156 21416800 807252 1729 1947 2 1 27 69 0 5 14764 6175044 687184 22478400 8106 438 2160 2618 3 2 26 69 0 7 14764 6154740 686712 23560000 829728 1638 1814 3 1 16 80 0 7 14764 6145812 688652 24305200 8054 109 1602 1744 2 1 14 84 2 8 14764 6136036 687120 25358000 8205 170 1782 1997 3 1 16 80 0 5 14764 6132008 687580 26611200 778359 1546 1659 3 1 17 79 0 6 14764 6123984 687284 27517200 794030 1756 2008 2 1 29 67 0 6 14764 6117016 688716 28106800 7420 340 1590 1753 2 1 21 77 0 6 14764 6108932 688092 28939600 8068 105 1896 2182 3 1 24 73 Your I/O system seems saturated (70-80% of time is spent waiting for I/O). Try running only one BackupPC_nightly in parallel. Tino. -- What we nourish flourishes. - Was wir nähren erblüht. www.lichtkreis-chemnitz.de www.tisc.de -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Hello Tino,
Your I/O system seems saturated (70-80% of time is spent waiting for
I/O). Try running only one BackupPC_nightly in parallel.
Yesterday i set $Conf{MaxBackupPCNightlyJobs} = 2; to
$Conf{MaxBackupPCNightlyJobs} = 1;
Currently still running :-/
admin4/8 01:00 BackupPC_nightly -m 0 255 11896
Current vmstat 10 10 output:
procs ---memory-- ---swap-- -io -system-- cpu
r b swpd free buff cache si sobibo in cs us sy id wa
0 2 18012 7261024 688028 9597600 1 231 2 1 78 18
0 2 18012 7263008 687052 9484800 5634 1 483 201 0 0 68 31
0 2 18012 7267968 687212 9024800 5431 3 505 208 0 0 68 31
0 2 18012 7267604 687876 8999600 5697 0 496 196 0 0 69 31
0 2 18012 7268348 687092 8988400 6065 4 521 202 0 0 68 32
0 2 18012 7268100 687616 8968400 2 487 200 0 0 68 31
0 2 18012 7268224 687548 8964800 5571 0 492 208 0 0 69 30
0 2 18012 7267852 688032 8971200 5564 1 483 198 0 0 70 30
0 2 18012 7267976 687484 8948000 5768 3 490 197 0 0 69 31
0 2 18012 7268472 686848 8971600 5362 2 510 211 0 0 67 32
Regards
Norbert
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Hi,
- Norbert Schulze n...@nurfuerspam.de wrote:
Hello Tino,
Your I/O system seems saturated (70-80% of time is spent waiting for
I/O). Try running only one BackupPC_nightly in parallel.
Yesterday i set $Conf{MaxBackupPCNightlyJobs} = 2; to
$Conf{MaxBackupPCNightlyJobs} = 1;
Currently still running :-/
admin 4/8 01:00 BackupPC_nightly -m 0 255 11896
Current vmstat 10 10 output:
procs ---memory-- ---swap-- -io -system-- cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
0 2 18012 7261024 688028 95976 0 0 1 2 3 1 2 1 78 18
0 2 18012 7263008 687052 94848 0 0 5634 1 483 201 0 0 68 31
0 2 18012 7267968 687212 90248 0 0 5431 3 505 208 0 0 68 31
0 2 18012 7267604 687876 89996 0 0 5697 0 496 196 0 0 69 31
0 2 18012 7268348 687092 89884 0 0 6065 4 521 202 0 0 68 32
0 2 18012 7268100 687616 89684 0 0 2 487 200 0 0 68 31
0 2 18012 7268224 687548 89648 0 0 5571 0 492 208 0 0 69 30
0 2 18012 7267852 688032 89712 0 0 5564 1 483 198 0 0 70 30
0 2 18012 7267976 687484 89480 0 0 5768 3 490 197 0 0 69 31
0 2 18012 7268472 686848 89716 0 0 5362 2 510 211 0 0 67 32
Is this running in a VM? I often see high CPU wait times when doing disk I/O in
a VM.
Gerald
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Hello Gerald, Is this running in a VM? I often see high CPU wait times when doing disk I/O in a VM. No, it is a normal system with a hardware raid-controller. Regards Norbert -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Le jeudi 08 avril 2010 à 15:36 +0200, Norbert Schulze a écrit : Hello Gerald, Is this running in a VM? I often see high CPU wait times when doing disk I/O in a VM. No, it is a normal system with a hardware raid-controller. Do you have a BBU on the raid controler ? Hardware RAID without BBU (and of course, without write-back cache enabled) is usually *very* slow, much slower than simple disk system, or software RAID. Regards, Daniel Regards Norbert -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Daniel Berteaud FIREWALL-SERVICES SARL. Société de Services en Logiciels Libres Technopôle Montesquieu 33650 MARTILLAC Tel : 05 56 64 15 32 Fax : 05 56 64 15 32 Mail: dan...@firewall-services.com Web : http://www.firewall-services.com -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On 4/8/2010 8:36 AM, Norbert Schulze wrote:
Hello Gerald,
Is this running in a VM? I often see high CPU wait times when doing disk
I/O in a VM.
No, it is a normal system with a hardware raid-controller.
Raid5? That has horrible write performance, especially with small
writes like the directory/inode/freespace updates for file deletions.
If you aren't pressed for disk space you might improve things by
increasing $Conf{BackupPCNightlyPeriod} so it only traverses part of the
pool each night.
--
Les Mikesell
lesmikes...@gmail.com
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Hello Daniel, Do you have a BBU on the raid controler ? ADAPTEC 3805 RAID SAS/SATA 8−Kanal Hardware RAID without BBU (and of course, without write-back cache enabled) is usually *very* slow, much slower than simple disk system, or software RAID. When you look at my other postings, the output of vmstat said that there many blocks received from a block device and minimal blocks sent to a block device. IMHO: BackupPC_nightly mainly reads from disk!? Regards Norbert procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 5 18012 6913580 687380 43347200 5 252 2 1 78 19 0 2 18012 6920324 686976 43579200 6011 4 553 248 1 0 62 37 1 2 18012 6920696 687196 43568400 532118 484 212 0 0 69 30 0 2 18012 6920076 687804 43570000 5780 6 499 203 0 0 67 33 0 2 18012 6921572 687500 43443200 5383 6 466 201 0 0 70 30 0 2 18012 6922324 686708 43410000 5476 4 478 204 0 0 70 30 0 2 18012 6921728 687484 43406800 5747 2 495 194 0 0 71 29 0 2 18012 6921256 687628 43418800 5632 1 497 209 0 0 70 29 0 2 18012 6921404 687488 43409200 558622 499 210 0 0 69 31 0 2 18012 6921428 687596 43388800 5784 2 496 206 0 0 69 31 -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Raid5? That has horrible write performance,
It is a RAID50
IMHO: BackupPC_nightly mainly reads from disk!?
If you aren't pressed for disk space you might improve things by
increasing $Conf{BackupPCNightlyPeriod} so it only traverses part
I change this now to 4
Regards
Norbert
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On 4/8/2010 9:48 AM, Norbert Schulze wrote: Raid5? That has horrible write performance, It is a RAID50 IMHO: BackupPC_nightly mainly reads from disk!? It traverses the pool directory, stat()'ing each inode (which isn't like to be near the directory) looking for entries that only have one link, meaning that all of the pc backups that previously linked to it have expired, and deletes these files. The deletes involve small writes back to the directory, inode, and free space list which will be slow as the raid5 must do a read before the write and then update the parity, essentially tying the speed to the slowest disk. How many of these you do will relate to the turnover of files on the targets. If you do a large number of maildir type folders with a lot of activity there could be a lot. -- Les Mikesell lesmikes...@gmail.com -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On 4/8/2010 12:23 PM, Les Mikesell wrote: On 4/8/2010 9:48 AM, Norbert Schulze wrote: Raid5? That has horrible write performance, It is a RAID50 IMHO: BackupPC_nightly mainly reads from disk!? It traverses the pool directory, stat()'ing each inode (which isn't like to be near the directory) looking for entries that only have one link, meaning that all of the pc backups that previously linked to it have expired, and deletes these files. The deletes involve small writes back to the directory, inode, and free space list which will be slow as the raid5 must do a read before the write and then update the parity, essentially tying the speed to the slowest disk. How many of these you do will relate to the turnover of files on the targets. If you do a large number of maildir type folders with a lot of activity there could be a lot. A few comments. 1) Raid5X is going to kill your performance. As mentioned in a previous message, convert to RAID10 if you can afford the space decrease. 2) Make sure you are setting the 'noatime' option on your backuppc pool storage partition 3) Edit your updatedb.conf and add the path to your backuppc pool to PRUNEPATHS. updatedb usually runs nightly on most systems around 4am as a cron job and allowing it to index your backuppc pool will take quite a while if many files are changing nightly. 4) Look at this page for tips on speeding up your backup. http://sourceforge.net/apps/mediawiki/backuppc/index.php?title=Speedupbackups Also, are you using a BBU on your Adaptec 3805 card? Adaptec's page indicates the controller does support a BBU. If you are, as a previous person mentioned, make sure you have write-caching enabled on the card. -- Ryan Manikowski ]] Devision Media Services LLC [[ www.devision.us r...@devision.us | 716.771.2282 -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Hi Norbert, On Wed, Apr 07, 2010 at 11:34:57AM +0200, Norbert Schulze wrote: the BackupPC_nightly takes too much time. Is this too much data for this server? Intel(R) Core(TM)2 Quad CPUQ9650 @ 3.00GHz Memory: 8GB General Server Information The servers PID is 15477, on host BACKUPPC-Server, version 3.1.0, started at 3/29 14:35. This status was generated at 4/7 11:29. The configuration was last loaded at 4/7 11:29. PCs will be next queued at 4/7 12:00. Other info: 9 pending backup requests from last scheduled wakeup, 0 pending user backup requests, 10 pending command requests, Uncompressed pool: Pool is 357.26GB comprising 469442 files and 4369 directories (as of 4/6 01:05), Pool hashing gives 87 repeated files with longest chain 12, Nightly cleanup removed 314 files of size 0.24GB (around 4/6 01:05), Compressed pool: Pool is 472.31GB comprising 5608440 files and 4369 directories (as of 4/6 14:03), Pool hashing gives 30090 repeated files with longest chain 8, Nightly cleanup removed 7149 files of size 6.13GB (around 4/6 14:03), Pool file system was recently at 29% (4/7 11:28), today's max is 29% (4/7 01:00) and yesterday's max was 29 Currently Running Jobs admin4/7 01:00 BackupPC_nightly -m 0 127 5259 admin1 4/7 01:00 BackupPC_nightly 128 255 5260 take a look at what vmstat 10 prints (ignore the first line). It will show you the I/O load of the system. What kind of storage are you using? Maybe it is saturated, then you might want to try not running BackupPC_nightly in parallel since it is I/O bound. HTH, Tino. -- What we nourish flourishes. - Was wir nähren erblüht. www.lichtkreis-chemnitz.de www.tisc.de -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
take a look at what vmstat 10 r...@server:/var/www# vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 4 14764 6235884 688976 13520400 2 104 2 1 78 18 r...@server:/var/www# vmstat 10 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 4 14764 6217932 687500 15443200 2 104 2 1 78 18 What kind of storage are you using? Storage: 6 x WD RE4−GP 2002FYPS 2TB 7k S−ATA−II 24/7 (RAID50) Raid Controller: ADAPTEC 3805 RAID SAS/SATA 8−Kanal OS is Ubuntu 9.04 32Bit IMHO it is better to migrate to a 64Bit-System!? Regards Norbert -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On Wed, Apr 07, 2010 at 12:11:31PM +0200, Norbert Schulze wrote: take a look at what vmstat 10 r...@server:/var/www# vmstat procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 4 14764 6235884 688976 13520400 2 104 2 1 78 18 r...@server:/var/www# vmstat 10 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 1 4 14764 6217932 687500 15443200 2 104 2 1 78 18 I suppose, this is the first line of vmstat output? It's useless - this is overall statistics since system boot. Drop the first line, then wait another 10 seconds for another line to appear. Or just post the output of vmstat 10 10 What kind of storage are you using? Storage: 6 x WD RE4−GP 2002FYPS 2TB 7k S−ATA−II 24/7 (RAID50) Raid Controller: ADAPTEC 3805 RAID SAS/SATA 8−Kanal OS is Ubuntu 9.04 32Bit IMHO it is better to migrate to a 64Bit-System!? I don't see an urgent reason to migrate to 64 bit... I would have installed this machine 64 bit at the beginning, just because it's a 64 bit machine. You'll lose some performance, but it might be barely noticeable. Tino. -- What we nourish flourishes. - Was wir nähren erblüht. www.lichtkreis-chemnitz.de www.tisc.de -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
I suppose, this is the first line of vmstat output? Sorry :-) Or just post the output of vmstat 10 10 r...@server:~# vmstat 10 10 procs ---memory-- ---swap-- -io -system-- cpu r b swpd free buff cache si sobibo in cs us sy id wa 0 5 14764 6196504 687296 20274400 6 115 2 1 78 18 0 5 14764 6184340 688156 21416800 807252 1729 1947 2 1 27 69 0 5 14764 6175044 687184 22478400 8106 438 2160 2618 3 2 26 69 0 7 14764 6154740 686712 23560000 829728 1638 1814 3 1 16 80 0 7 14764 6145812 688652 24305200 8054 109 1602 1744 2 1 14 84 2 8 14764 6136036 687120 25358000 8205 170 1782 1997 3 1 16 80 0 5 14764 6132008 687580 26611200 778359 1546 1659 3 1 17 79 0 6 14764 6123984 687284 27517200 794030 1756 2008 2 1 29 67 0 6 14764 6117016 688716 28106800 7420 340 1590 1753 2 1 21 77 0 6 14764 6108932 688092 28939600 8068 105 1896 2182 3 1 24 73 Regards Norbert -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
Josh, Interesting that. Is there an easy way to convert from ext3 to ext4? Or do you need to reformat? Also, did you change all your hard drives to ext4 or just the drive backuppc backs up to? ___ Dave Williams Check out our WebOS mobile phone app for the Palm Pre and Pixi: Golf Caddie | Golf Caddie Forum | Golf Caddie FAQ by DTW-Consulting, Inc. -Original Message- From: Josh Malone [mailto:jmal...@nrao.edu] Sent: Wednesday, April 07, 2010 4:44 PM To: General list for user discussion, questions and support Subject: Re: [BackupPC-users] BackupPC_nightly takes too much time On Wed, 07 Apr 2010 11:34:57 +0200, Norbert Schulze n...@nurfuerspam.de wrote: Hello, the BackupPC_nightly takes too much time. Is this too much data for this server? Intel(R) Core(TM)2 Quad CPUQ9650 @ 3.00GHz Memory: 8GB General Server Information The servers PID is 15477, on host BACKUPPC-Server, version 3.1.0, started at 3/29 14:35. This status was generated at 4/7 11:29. The configuration was last loaded at 4/7 11:29. PCs will be next queued at 4/7 12:00. Other info: 9 pending backup requests from last scheduled wakeup, 0 pending user backup requests, 10 pending command requests, Uncompressed pool: Pool is 357.26GB comprising 469442 files and 4369 directories (as of 4/6 I have a system with about the same amount of data and far more files running on much less hardware. I, too, found backupPC_nightly taking a LONG time. I switched from EXT3 to EXT4 (with extents enabled) and my problems were basically solved. My system was also 32-bit (I've switched to 64- bit since then to get better ext4 stability... long story, RedHat stinks) What filesystem are you using? -Josh -- Joshua Malone Systems Administrator (jmal...@nrao.edu)NRAO Charlottesville 434-296-0263 www.cv.nrao.edu 434-249-5699 (mobile) BOFH excuse #426: internet is needed to catch the etherbunny --- --- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ No virus found in this incoming message. Checked by AVG - www.avg.com Version: 9.0.800 / Virus Database: 271.1.1/2795 - Release Date: 04/06/10 14:32:00 -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On Wed, Apr 7, 2010 at 5:29 AM, Tino Schwarze backuppc.li...@tisc.de wrote: On Wed, Apr 07, 2010 at 12:11:31PM +0200, Norbert Schulze wrote: [SNIP] OS is Ubuntu 9.04 32Bit IMHO it is better to migrate to a 64Bit-System!? I don't see an urgent reason to migrate to 64 bit... I would have installed this machine 64 bit at the beginning, just because it's a 64 bit machine. You'll lose some performance, but it might be barely noticeable. I'm not so sure that's the case. My understanding is that a 32-bit OS can only address a little over 3GB of physical memory, since the system has 8GB, I would think you would want to upgrade to a 64-bit OK. Richard -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On Wed, 7 Apr 2010 16:47:55 -0400, David Williams dwilli...@dtw-consulting.com wrote: Josh, Interesting that. Is there an easy way to convert from ext3 to ext4? Or do you need to reformat? Also, did you change all your hard drives to ext4 or just the drive backuppc backs up to? It can be migrated (#include always-backup.h) umount /dev/foo tune4fs -O extents,uninit_bg,dir_index /dev/foo fsck -pDf /dev/foo I did not convert the / filesystem to ext3... just out of lazyness (as I'd have to boot from CD to do it, etc.) -Josh -- Joshua Malone Systems Administrator (jmal...@nrao.edu)NRAO Charlottesville 434-296-0263 www.cv.nrao.edu 434-249-5699 (mobile) BOFH excuse #426: internet is needed to catch the etherbunny -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] BackupPC_nightly takes too much time
On Wed, 7 Apr 2010 15:56:19 -0500, Richard Shaw hobbes1...@gmail.com wrote: On Wed, Apr 7, 2010 at 5:29 AM, Tino Schwarze backuppc.li...@tisc.de wrote: On Wed, Apr 07, 2010 at 12:11:31PM +0200, Norbert Schulze wrote: [SNIP] OS is Ubuntu 9.04 32Bit IMHO it is better to migrate to a 64Bit-System!? I don't see an urgent reason to migrate to 64 bit... I would have installed this machine 64 bit at the beginning, just because it's a 64 bit machine. You'll lose some performance, but it might be barely noticeable. I'm not so sure that's the case. My understanding is that a 32-bit OS can only address a little over 3GB of physical memory, since the system has 8GB, I would think you would want to upgrade to a 64-bit OK. Richard Using PAE, you can have 3.5 G of usable ram on a system. HOWEVER, each individual process only has a 4GB virtual address space, so only 4G of ram per process. If you have 1 memory-intensive process you can make use of 8G of ram on a 32-bit system. -Josh -- Joshua Malone Systems Administrator (jmal...@nrao.edu)NRAO Charlottesville 434-296-0263 www.cv.nrao.edu 434-249-5699 (mobile) BOFH excuse #426: internet is needed to catch the etherbunny -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
