Re: [BackupPC-users] very slow backup speed
On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Lets hope this doesnt wrap around... as you can see load is in 0.1-0.01 range. 1 usersLoad 0.12 0.05 0.01 Mar 27 07:30 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act 260203592 144912 6868 12384 count All 2497845456 232789611800 pages It wrapped pretty badly, but let me see if I'm interpreting this right (I'm no BSD expert, either): 1. Your server has ~250MB of memory. 2. Load average during backups is only 0.1-0.01? Does BSD calculate load average differently than Linux? Linux calculates load average by looking at the number of runnable tasks - this means if you have a single process waiting on disk IO you will have a load average of 1. If BSD calculates the load average the same way, then that means your server is not waiting on disk, but waiting for the clients. What's the load like on the clients you are backing up? -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Lets hope this doesnt wrap around... as you can see load is in 0.1-0.01 range. 1 usersLoad 0.12 0.05 0.01 Mar 27 07:30 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act 260203592 144912 6868 12384 count All 2497845456 232789611800 pages It wrapped pretty badly, but let me see if I'm interpreting this right (I'm no BSD expert, either): 1. Your server has ~250MB of memory. 2. Load average during backups is only 0.1-0.01? Does BSD calculate load average differently than Linux? Linux calculates load average by looking at the number of runnable tasks - this means if you have a single process waiting on disk IO you will have a load average of 1. If BSD calculates the load average the same way, then that means your server is not waiting on disk, but waiting for the clients. What's the load like on the clients you are backing up? Under FreeBSD if a task is blocked it doesn't contribute to load no matter what it's waiting for - so disk bound tasks don't add to load average. John - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
brien dieterle wrote: Jason Hughes wrote: Evren Yurtesen wrote: Jason Hughes wrote: That drive should be more than adequate. Mine is a 5400rpm 2mb buffer clunker. Works fine. Are you running anything else on the backup server, besides BackupPC? What OS? What filesystem? How many files total? FreeBSD, UFS2+softupdates, noatime. There are 4 hosts that have been backed up, for a total of: * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.45GB (prior to pooling and compression). # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 07:16), today's max is 10% (3/27 01:00) and yesterday's max was 10%. Host User #Full Full Age (days) Full Size (GB) Speed (MB/s) #Incr Incr Age (days) Last Backup (days) State Last attempt host1 4 5.4 3.88 0.22 6 0.4 0.4 idle idle host2 4 5.4 2.10 0.06 6 0.4 0.4 idle idle host3 4 5.4 7.57 0.14 6 0.4 0.4 idle idle host4 4 5.4 5.56 0.10 6 0.4 0.4 idle idle Hmm. This is a tiny backup setup, even smaller than mine. However, it appears that the average size of your file is only 22KB, which is quite small. For comparison sake, this is from my own server: Pool is 172.91GB comprising 217311 files and 4369 directories (as of 3/26 01:08), The fact that you have tons of little files will probably give significantly higher overhead when doing file-oriented work, simply because the inodes must be fetched for each file before seeking to the file itself. If we assume no files are shared between hosts (very conservative), and you have an 8ms access time, you will have 190132 files per host and two seeks per file, neglecting actual i/o time, gives you 50 minutes. Just to seek them all. If you have a high degree of sharing, it can be up to 4x worse. Realize, the same number of seeks must be made on the server as well as the client. Are you sure you need to be backing up everything that you're putting across the network? Maybe excluding some useless directories, maybe temp files or logs that haven't been cleaned up? Perhaps you can archive big chunks of it with a cron job? I'd start looking for ways to cut down the number of files, because the overhead of per-file accesses are probably eating you alive. I'm also no expert on UFS2 or FreeBSD, so it may be worthwhile to research its behavior with hard links and small files. JH For what it's worth, I have a server that backs up 8.6 million files averaging 10k in size from one host. It takes a full 10 hours for a full backup via tar over NFS ( 2.40MB/s for 87GB). CPU usage is low, around 10-20%, however IOwait is a pretty steady 25%. Server info: HP DL380 G4 debian sarge dual processor 3.2ghz xeon 2GB Ram 5x10k rpm scsi disks, raid5 128MB battery backed cache (50/50 r/w) ext3 filesystems brien You are distributing the reads and writes on 5 disks here. Dont you think that 2.40mb/s is a little bit slow compared to the horsepower you have? If you scale it down to my system, I think 1/10 performance is normal... Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Craig Barratt wrote:
Evren writes:
HostUser#Full Full Age (days) Full Size (GB)
Speed
(MB/s)#Incr Incr Age (days) Last Backup (days)
State
Last attempt
host14 5.4 3.880.226 0.4
0.4 idleidle
host24 5.4 2.100.066 0.4
0.4 idleidle
host34 5.4 7.570.146 0.4
0.4 idleidle
host44 5.4 5.560.106 0.4
0.4 idleidle
These xfer rates are certainly very low.
I don't think I caught which XferMethod you are using. Also,
what is your BackupPC version? If you are using 3.0, what is
the value of $Conf{IncrLevels}?
Craig
I use rsync with 3.0.0 but it was the same speed with 2.x.x versions.
$Conf{IncrLevels} = [1];
I wonder, since I have:
$Conf{IncrKeepCnt} = 6;
Wouldnt it make more sense to use this?:
$Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
or does this make things even more slower?
I have concluded that to get a decent performance from backuppc one must
use RAID and put a lot of ram to the machine because BackupPC is using
the disk intensively!
Thanks,
Evren
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: Server info: HP DL380 G4 debian sarge dual processor 3.2ghz xeon 2GB Ram 5x10k rpm scsi disks, raid5 128MB battery backed cache (50/50 r/w) ext3 filesystems brien You are distributing the reads and writes on 5 disks here. Dont you think that 2.40mb/s is a little bit slow compared to the horsepower you have? Raid5 doesn't distribute disk activity - it puts the drives in lockstep and is slower than a single drive, especially on small writes where it has to do extra reads to re-compute parity on the existing data. If you scale it down to my system, I think 1/10 performance is normal... Its the RAM and disk cache that makes this system work. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: That doesn't sound difficult at all. I suspect your real problem is that you are running a *bsd UFS filesystem with it's default sync metadata handling which is going to wait for the physical disk action to complete on every directory operation. I think there are other options but I haven't kept up with them. I gave up on UFS long ago when I needed to make an application that frequently truncated and rewrote a data file work on a machine that crashed frequently. The sync-metadata 'feature' statistically ensured that there was never any data in the file after recovering since the truncation was always forced to disk immediately but the data write was buffered so with a fast cycle the on-disk copy was nearly always empty. Is anyone else running a *bsd? The problem is the reads, not writes. It takes long time for BackupPC to figure out if the file should be backed up or not. Backups take very long time even when there are few files which are backed up. But I have to disagree, again :) When you make writes and filesystem is mounted with 'sync' option which is default on FreeBSD then all operations are done sync. Your program must have been doing something between truncate and the actual write operation which caused file to be empty for a period of time. From FreeBSD mount manual - syncAll I/O to the file system should be done synchronously. - async All I/O to the file system should be done asynchronously. This is a dangerous flag to set, and should not be used unless you are prepared to recreate the file system should your system crash. This 'feature' as you call it, exists in linux also From Linux mount manual - sync All I/O to the file system should be done synchronously - async All I/O to the file system should be done asynchronously. The only difference is that FreeBSD uses sync by default because they claim(there is no point to argue if it is true or not, there are enough threads about those if you google :) ) that it is safer while Linux uses async. You could easily enable 'async' IO on FreeBSD to have the same behaviour. Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. I cant see how you can reach to this conclusion. Observation... I run hundreds of servers, many of which are 5 or more years old. The disk failures have had no correlation to the server activity. Hard drive manufacturers disagree with you. They do consider the duty-cycles when calculating the MTBF values. http://www.digit-life.com/articles/storagereliability/ Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Lets hope this doesnt wrap around... as you can see load is in 0.1-0.01 range. 1 usersLoad 0.12 0.05 0.01 Mar 27 07:30 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act 260203592 144912 6868 12384 count All 2497845456 232789611800 pages It wrapped pretty badly, but let me see if I'm interpreting this right (I'm no BSD expert, either): 1. Your server has ~250MB of memory. 2. Load average during backups is only 0.1-0.01? Does BSD calculate load average differently than Linux? Linux calculates load average by looking at the number of runnable tasks - this means if you have a single process waiting on disk IO you will have a load average of 1. If BSD calculates the load average the same way, then that means your server is not waiting on disk, but waiting for the clients. What's the load like on the clients you are backing up? -Dave 2- The process is idle when it is waiting data from the disk (not in active state) So FreeBSD doesnt count it towards the load. I dont see why it should actually? Are processes which are waiting data from disk use CPU in Linux? 1- You are right, I will increase the memory a little bit. I will return back the results when I accomplish that. :) Lets hope it will decrease disk reads. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: Server info: HP DL380 G4 debian sarge dual processor 3.2ghz xeon 2GB Ram 5x10k rpm scsi disks, raid5 128MB battery backed cache (50/50 r/w) ext3 filesystems brien You are distributing the reads and writes on 5 disks here. Dont you think that 2.40mb/s is a little bit slow compared to the horsepower you have? Raid5 doesn't distribute disk activity - it puts the drives in lockstep and is slower than a single drive, especially on small writes where it has to do extra reads to re-compute parity on the existing data. I am confused, when a write is done the data is distributed in the disks depending on the stripe size you are using. When you start reading the file, you are reading from 5 different disks. So you get way better performance for sure on reads. Agreed, writes might be slower depending on the raid controller, amount of cache it has etc. but my problem here is the reads. In my machine BackupPC server is working slow even when it is backing up a very small amount of files. If you scale it down to my system, I think 1/10 performance is normal... Its the RAM and disk cache that makes this system work. Agreed, I will add more RAM and report the results. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: The problem is the reads, not writes. It takes long time for BackupPC to figure out if the file should be backed up or not. Backups take very long time even when there are few files which are backed up. That's only true with rsync and especially rsync fulls where the contents are compared as well as the timestamps. But I thought your complaint was about the hardlinks and data storage. The transfer speed value doesn't make a lot of sense in the rsync case anyway because you often spend a lot of time computing to avoid having to transfer anything - and you have the entire directory in RAM while doing it which may reduce your disk buffering. What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? But I have to disagree, again :) When you make writes and filesystem is mounted with 'sync' option which is default on FreeBSD then all operations are done sync. Your program must have been doing something between truncate and the actual write operation which caused file to be empty for a period of time. Yes, it was writing a small amount to the file and the data write was always deferred with this kind of problem: http://www.complang.tuwien.ac.at/anton/sync-metadata-updates.html The only difference is that FreeBSD uses sync by default because they claim(there is no point to argue if it is true or not, there are enough threads about those if you google :) ) that it is safer while Linux uses async. It is safer for the file system if you are concerned that fsck can't fix it. It is not safer for the data of any particular file. Anyway crashes are pretty much a thing of the past and I protect against most kinds of failures by raid-mirroring to an external drive that is rotated off-site weekly. I'm running a journaled reiserfs because it is very fast at creating and deleting files, but ext3 with the right options should also work. Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. I cant see how you can reach to this conclusion. Observation... I run hundreds of servers, many of which are 5 or more years old. The disk failures have had no correlation to the server activity. Hard drive manufacturers disagree with you. They do consider the duty-cycles when calculating the MTBF values. http://www.digit-life.com/articles/storagereliability/ Vendor MTBF values are very unrealistic. The trade rags seem to have caught on to this recently: http://www.eweek.com/article2/0%2C1895%2C2105551%2C00.asp If you want to know what the vendor really expects, look at the warranty instead. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On Mon, 26 Mar 2007, David Rees wrote: No kidding! My backuppc stats are like this: For my biggest server, my stats are: There are 36 hosts that have been backed up, for a total of: 503 full backups of total size 4200.30GB (prior to pooling and compression), 247 incr backups of total size 444.52GB (prior to pooling and compression). Pool is 776.90GB comprising 1900158 files and 4369 directories Dual Opteron 265(4x1.8ghz cores), 1 GB ram, 3ware 9550SX-16ML w/15 Maxtor MaxLine III 300 (7200 rpm, ATA, 8 meg cache), ext3 w/noatime. I'd change it to XFS, but backing up 1.7TB would take weeks/months because of the hardlinks. Runs BackupPC 2.1.2-3 on Debian Sarge. Backups take about 5-6 hours depending on incremental/full counts, I run 4 at a time.. Most of the workstations are on 100mbit, it's backing up about 330 gig total of files. And yeah, I love BackupPC too :) Mike - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: The problem is the reads, not writes. It takes long time for BackupPC to figure out if the file should be backed up or not. Backups take very long time even when there are few files which are backed up. That's only true with rsync and especially rsync fulls where the contents are compared as well as the timestamps. But I thought your complaint was about the hardlinks and data storage. The transfer speed value doesn't make a lot of sense in the rsync case anyway because you often spend a lot of time computing to avoid having to transfer anything - and you have the entire directory in RAM while doing it which may reduce your disk buffering. What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. But I have to disagree, again :) When you make writes and filesystem is mounted with 'sync' option which is default on FreeBSD then all operations are done sync. Your program must have been doing something between truncate and the actual write operation which caused file to be empty for a period of time. Yes, it was writing a small amount to the file and the data write was always deferred with this kind of problem: http://www.complang.tuwien.ac.at/anton/sync-metadata-updates.html You shouldnt be trusting this article. The article is flawed in certain ways. For example it says as demonstrated by soft updates (see below), which don't even require fsck upon crashing. but this information is wrong. Softupdates means that you can run the fsck in background but you must run it on a crash (system does it automatically anyhow). Also, about the problem in this article, if async updates would have been used then the file would have been lost anyhow as the writes would have been buffered. The information given is vague. I am not an expert on this but the problem about async updates that the BSD guys are scared of is probably not the same kind of inconsistency described here. In either case you could just enable sync or async updates on UFS to change this behaviour. You didnt have to change the filesystem because of a deficiency in UFS which is causing the problem. The only difference is that FreeBSD uses sync by default because they claim(there is no point to argue if it is true or not, there are enough threads about those if you google :) ) that it is safer while Linux uses async. It is safer for the file system if you are concerned that fsck can't fix it. It is not safer for the data of any particular file. Anyway crashes are pretty much a thing of the past and I protect against most kinds of failures by raid-mirroring to an external drive that is rotated off-site weekly. I'm running a journaled reiserfs because it is very fast at creating and deleting files, but ext3 with the right options should also work. If you check namesys.com benchmarks, you will see that they only tested reiserfs against ext2/ext3/xfs/jfs and conveniently forgot to test against ufs2. You can see in the end of the page that slight performance increase in reiserfs is also bringing twice the cpu usage! (plus extX is faster in certain operations even) http://www.namesys.com/benchmarks.html Another test results from sun, little bit old (from 2004) http://www.sun.com/software/whitepapers/solaris10/fs_performance.pdf I am not saying one is faster over another or anything like that. Sun's results do look too good to be true anyhow, they probably tweaked some stuff or I dont know, can they lie? wouldnt it be bad reputation? Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. I cant see how you can reach to this conclusion. Observation... I run hundreds of servers, many of which are 5 or more years old. The disk failures have had no correlation to the server activity. Hard drive manufacturers disagree with you. They do consider the duty-cycles when calculating the MTBF values. http://www.digit-life.com/articles/storagereliability/ Vendor MTBF values are very unrealistic. The trade rags seem to have caught on to this recently: http://www.eweek.com/article2/0%2C1895%2C2105551%2C00.asp If you want to know what the vendor really expects, look at the warranty instead. It is a logical fact that more movement = more wear. Thanks, Evren
Re: [BackupPC-users] Unable to connect to BackupPC server error
That spare hard drive wouldn't happen to be an external USB or Firewire, would it? Aaron Ciarlotta Linux/UNIX Systems Guru -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Winston Chan Sent: Monday, March 26, 2007 8:14 PM To: backuppc-users@lists.sourceforge.net Subject: [BackupPC-users] Unable to connect to BackupPC server error I had been running BackupPC on an Ubuntu computer for several months to back the computer to a spare hard drive without problem. About the time I added a new host (Windows XP computer using Samba), I started getting the following behavior: BackupPC backs both hosts properly onto the spare hard drive once or twice after I reboot the Ubuntu server. Then I get a Error: Unable to connect to BackupPC server error when I attempt to go the web interface. When I restart BackupPC with /etc/init.d/backuppc restart, I get a message Can't create LOG file /var/lib/backuppc/log/LOG at /usr/share/backuppc/bin/BackupPC line 1735. I have made sure that the backuppc is the owner of /var/lib/backuppc/log/LOG. It seems the only way to get backuppc to work again is to reboot the Ubuntu server. I then can see on the web interface that the last successful backup (of both hosts) occurred 1 or 2 days after the previous reboot. Then backupPC works for 1 or 2 backups and the cycle starts again. What is the cause of this and how can I fix it? Winston - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDE V ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ The information in this electronic mail message is sender's business Confidential and may be legally privileged. It is intended solely for the addressee(s). Access to this Internet electronic mail message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. The sender believes that this E-mail and any attachments were free of any virus, worm, Trojan horse, and/or malicious code when sent. This message and its attachments could have been infected during transmission. By reading the message and opening any attachments, the recipient accepts full responsibility for taking protective and remedial action about viruses and other defects. Travelport Inc. is not liable for any loss or damage arising in any way from this message or its attachments. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. From the perspective of the backup directories, it doesn't matter whether or not there are additional links to a file. It just looks at the directory entry to find the file it wants. It may matter that the inodes and file contents end up splattered all over the place because they were written at different times, though. But I have to disagree, again :) When you make writes and filesystem is mounted with 'sync' option which is default on FreeBSD then all operations are done sync. Your program must have been doing something between truncate and the actual write operation which caused file to be empty for a period of time. Yes, it was writing a small amount to the file and the data write was always deferred with this kind of problem: http://www.complang.tuwien.ac.at/anton/sync-metadata-updates.html You shouldnt be trusting this article. The article is flawed in certain ways. The fact is that 90+% of the time, my data files were empty after a crash while they always appeared to have data when running. That tells me that the truncate happened on disk immediately while the data write happened when a buffer was flushed which was consistent with the way the bsd sync-metadata was described. For example it says as demonstrated by soft updates (see below), which don't even require fsck upon crashing. but this information is wrong. Softupdates means that you can run the fsck in background but you must run it on a crash (system does it automatically anyhow). This was before softupdates was an option. I don't think reiserfs was around then either. Also, about the problem in this article, if async updates would have been used then the file would have been lost anyhow as the writes would have been buffered. But then you'd have about a 99% chance of still having the previous contents which would have been much more useful than an empty file. In either case you could just enable sync or async updates on UFS to change this behaviour. You didnt have to change the filesystem because of a deficiency in UFS which is causing the problem. I just got tired of the freebsd people telling me that their way was better and it was correct behavior to lose my data and moved on. The only difference is that FreeBSD uses sync by default because they claim(there is no point to argue if it is true or not, there are enough threads about those if you google :) ) that it is safer while Linux uses async. It is safer for the file system if you are concerned that fsck can't fix it. It is not safer for the data of any particular file. Anyway crashes are pretty much a thing of the past and I protect against most kinds of failures by raid-mirroring to an external drive that is rotated off-site weekly. I'm running a journaled reiserfs because it is very fast at creating and deleting files, but ext3 with the right options should also work. If you check namesys.com benchmarks, you will see that they only tested reiserfs against ext2/ext3/xfs/jfs and conveniently forgot to test against ufs2. You can see in the end of the page that slight performance increase in reiserfs is also bringing twice the cpu usage! (plus extX is faster in certain operations even) http://www.namesys.com/benchmarks.html There's a benchmark called 'postmark' that is the best I've seen for showing how well creating/deleting a lot of files works. A few years ago when I changed last, reiserfs was the best. It seems to have disappeared from it's original location from NetApp, but the last time I mentioned it someone said it was packaged for Debian so the source is still available. I'll dig it up again the next time I am interested in changing (maybe when zfs looks mature). Perhaps, but there is a difference if they are moving 10 times or 10 times. It is a logical fact that more movement = more wear. Yes, but more wear doesn't matter until you exceed the design limits. Worry about your air conditioner instead - it has more effect on the drive life. I've had as much trouble with drives that have been powered off for much of their lives as ones that stay busy. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net
Re: [BackupPC-users] very slow backup speed
On 03/27 11:48 , Les Mikesell wrote: Worry about your air conditioner instead - it has more effect on the drive life. I've had as much trouble with drives that have been powered off for much of their lives as ones that stay busy. as an interesting data point, I have a 512MB drive that ran my firewall for several years, and before that was in a desktop machine at a government office. The date on it is 1993; and it still works just fine, despite what I'm guessing is fairly regular and continuous use since that time. I would really like to see hard drives made to be more reliable, rather than just bigger. (Where the mfg. actually touts the reliability features more than the size). Maxtor sort of tried that a while ago with their single-platter 15GB drives; but AFAIK there wasn't anything special about those other than fewer platters and heads (no better bearings, gas-absorption material, etc). We've long since passed the point where big drives are necessary (at least for OS partitions on Linux boxen). I want reliable storage. to that end, this is kind of an attractive product: http://www.elandigitalsystems.com/adapter/p312.php we've had good luck with these units: http://www.mach5products.com/PRODCF/CFproducts.htm My ~1TB BackupPC server is using less than 1GB on its OS drive (the backuppc data is all on a separate drive array, for ease of management). While consumer-grade CF cards are pretty crappy things compared to the military-grade solid-state IDE or SCSI disks that places like M-Systems sells; they are cheap enough and big enough to compete with spinny storage when the space requirements take second place to the reliability of no moving parts. (I've seen CF cards die... I'm not sure that a spinny drive wouldn't have died in the same conditions tho). -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/27/07, Les Mikesell [EMAIL PROTECTED] wrote: Evren Yurtesen wrote: What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. From the perspective of the backup directories, it doesn't matter whether or not there are additional links to a file. It just looks at the directory entry to find the file it wants. It may matter that the inodes and file contents end up splattered all over the place because they were written at different times, though. Yep, Lee is right here. Unless BSD handles hard-links in some crazy manner. If you check namesys.com benchmarks, you will see that they only tested reiserfs against ext2/ext3/xfs/jfs and conveniently forgot to test against ufs2. You can see in the end of the page that slight performance increase in reiserfs is also bringing twice the cpu usage! (plus extX is faster in certain operations even) http://www.namesys.com/benchmarks.html When your overall speed is limited by the speed of your disks and your CPU spends all it's time twiddling it's thumbs waiting for disk, who cares if CPU doubles and still spends 90% of it's time waiting as long as it gets the job done faster? To summarize Evran's setup: FreeBSD, 250MB ram, CPU unknown, 1 7200rpm Seagate ATA disk Filesystem: ufs2, sync, noatime Pool is 17.08GB comprising 760528 files (avg file size ~22KB) BackupPC reports backup speeds between 0.06 - 0.22 MB/s Total backup time per host: Unknown CPU is 99% idle during backups Disk shows ~75 IO/s during load and low transfer rate Says even small backup w/small number of files is slow. Can you try mounting the backup partition async so we can see if it really is read performance or write performance that is killing backup performance? I would also highly recommend that you limit backups to 1 concurrent backup at a time. I must wonder if ufs2 is really bad at storing inodes on disk... BTW, how does BackupPC calculate speed? I think it calculates backup speed by reporting files transferred over time, so if you don't have many files that change, won't BackupPC report a very low backup speed. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren writes:
I use rsync with 3.0.0 but it was the same speed with 2.x.x versions.
$Conf{IncrLevels} = [1];
I wonder, since I have:
$Conf{IncrKeepCnt} = 6;
Wouldnt it make more sense to use this?:
$Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
or does this make things even more slower?
Yes, that would make it slower since several incrementals have
to be merged (ie: more server disk seeks for every file) to get
the latest baseline, which in 3.x is used as the starting point
for each rsync backup.
Craig
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/27/07, David Rees [EMAIL PROTECTED] wrote: Can you try mounting the backup partition async so we can see if it really is read performance or write performance that is killing backup performance? I must wonder if ufs2 is really bad at storing inodes on disk... I went and did some research on ufs filesystem performance and found this paper: http://www.bsdcan.org/2006/papers/FilesystemPerformance.pdf There appears to be 4 different mount options related to data integrity: sync: slowest, all writes synchronous noasync: (default?) data asynchronous, metadata synchronous soft updates: dependency ordering of writes to ensure on-disk consistency async: fastest, all writes asynchronous noasync seems to be the default. Evran, can you confirm that your filesystem is mounted this way? On Linux using ext3, the default mount option is data=ordered which should be similar to soft updates in terms of performance. If you can mount your backup partition in soft updates mode that should perform best, much better than the default noasync mode for the type of writes BackupPC does. I wouldn't recommend mounting a partition async permanently because of the risk to file system integrity. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Carl Wilhelm Soderstrom wrote: I would really like to see hard drives made to be more reliable, rather than just bigger. I'm not sure that can be improved enough to matter. A failure of an inexpensive part once in five years isn't a big deal other than the side effects it might cause, and unless they can be 100% reliable (probably impossible) you have to be prepared for those side effects anyway. I want reliable storage. Run mirrored hot-swap drives. If one dies, replace it at some convenient time, sync it up and keep going. I have one machine last booted in mid-2003 that has had a couple of it's drives replaced that way. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 03/27 01:40 , Les Mikesell wrote: Carl Wilhelm Soderstrom wrote: I would really like to see hard drives made to be more reliable, rather than just bigger. I'm not sure that can be improved enough to matter. A failure of an inexpensive part once in five years isn't a big deal other than the side effects it might cause, and unless they can be 100% reliable (probably impossible) you have to be prepared for those side effects anyway. on a small scale, perhaps. when you get to an organization the size of (TBS|Hotmail|Yahoo)tho; it may make sense to spend a little more money in order to spend less labor swapping parts. the market seems to agree with you tho... which is a pretty good indictment of central planning notions (i.e. Me) vs. what really is efficient. I want reliable storage. Run mirrored hot-swap drives. If one dies, replace it at some convenient time, sync it up and keep going. I have one machine last booted in mid-2003 that has had a couple of it's drives replaced that way. Your point is well-taken. I do keep in mind tho that I've seen multiple drives fail simultaneously or in rapid succession, and that the process of replacing drives costs time and effort. It is not as trivial as you might think, once you factor in the time to detect the failure, find the replacement (if you can), replace the drive (which may involve removing the old drive from the carrier and replacing it with the new one), and make sure it's working properly. In an ideal world it's 5 minutes of work, in the real world I usually expect to lose an hour or two dealing with a failed drive. This can be accounted as several hundred dollars of lost productivity in many cases; so it's worthwhile to spend more money on a better-quality drive sometimes. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Timing out full backups
Hi Les, Apologies for the late reply. Michael Mansour wrote: Hi, Hi, I have the following host summary: server1 username1 20.63.360.772 10.4 10.4idledone server2 username1 20.63.23 0.743 10.4 10.4idledone server3username2 20.68.52 1.894 18.6 18.6 idledone server4username2 20.63.01 1.415 10.4 10.4idledone server5username2 20.63.38 1.244 18.6 18.6 idledone server6username2 20.62.86 1.445 10.4 10.4idledone This is on a test BackupPC server still using Beta 3.0 (I'll soon build a production BackupPC server to replace this box). I'm wondering why the full backups numbered 2 are not going back down to 1 to free up some space on the server? In the glbal Schedule, I have the following: FullPeriod: 6.97 FullKeepCnt: 1 FullKeepCntMin: 1 FullAgeMax: 7 and it's my understanding that backuppc should cycle those full backups above to trash because of the above globals yes? Did anyone have any ideas on this? Full backups that incrementals are based from are not deleted until after the corresponding incremental runs have been removed, and your last full will not be removed until the next one is complete. I understand, but the stats are: There are 6 hosts that have been backed up, for a total of: 10 full backups of total size 44.55GB (prior to pooling and compression), 15 incr backups of total size 8.44GB (prior to pooling and compression). I have servers which have multiple full backups and incrementals, but neither is removed. Is there a way I can just manually delete some of the backups (incrementals and fulls) to start this process again? as I can't seem to do anything while the disks are full and the timeout processes are not working. Thanks. Michael. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Timing out full backups
Michael Mansour wrote: I have servers which have multiple full backups and incrementals, but neither is removed. I don't think they are removed until the replacements are completed. Is there a way I can just manually delete some of the backups (incrementals and fulls) to start this process again? as I can't seem to do anything while the disks are full and the timeout processes are not working. Yes, you can go to the pc directory and 'rm -rf backup_number' for what you want to delete. The space won't actually be released until BackupPC_nightly runs and then only for files that are not linked by other backup runs. Some of your web displays will be wrong for a while after deleting things this way but it will eventually correct itself. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Timing out full backups
Hi Les, Michael Mansour wrote: I have servers which have multiple full backups and incrementals, but neither is removed. I don't think they are removed until the replacements are completed. Is there a way I can just manually delete some of the backups (incrementals and fulls) to start this process again? as I can't seem to do anything while the disks are full and the timeout processes are not working. Yes, you can go to the pc directory and 'rm -rf backup_number' for what you want to delete. The space won't actually be released until BackupPC_nightly runs and then only for files that are not linked by other backup runs. Some of your web displays will be wrong for a while after deleting things this way but it will eventually correct itself. Ok, I have started deleting and will kick off some manual incrementals afterwards. Thanks. Michale. -- Les Mikesell [EMAIL PROTECTED] --- End of Original Message --- - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Timing out full backups
Michael writes:
Michael Mansour wrote:
I have servers which have multiple full backups and incrementals, but
neither
is removed.
I don't think they are removed until the replacements are completed.
Is there a way I can just manually delete some of the backups
(incrementals
and fulls) to start this process again? as I can't seem to do anything
while
the disks are full and the timeout processes are not working.
Yes, you can go to the pc directory and 'rm -rf backup_number' for
what you want to delete. The space won't actually be released until
BackupPC_nightly runs and then only for files that are not linked by
other backup runs. Some of your web displays will be wrong for a
while after deleting things this way but it will eventually correct itself.
Ok, I have started deleting and will kick off some manual incrementals
afterwards.
You should be *really* sure you know what you are doing when
you manually delete backups. At a miminum you should remove the
corresponding line in the backups file.
Your stats suggest you have 1 or 2 fulls per host. That's typically
the minimum number, since subsequent incrementals require the baseline
full to be kept. For example, even if you specify $Conf{FullKeepCnt} = 1,
you will often have two fulls because the subsequent incrementals need
it kept:
oldest - #0 Full
#1 Incr
#2 Incr
#4 Full
#5 Incr
#6 Incr
#7 Incr
newest - #8 Incr
This is a very common situation. If you remove full #0 then you have
rendered backups #1 and #2 invalid and useless.
The only three options to reduce the number of backups in this case is:
- remove incrementals #8, #7, #6, #5 (in that order)
- remove incrementals #2, #1 (in that order)
- remove full #0, and incr #1 and #2 (all or nothing).
I'd really recommend against manually remove backups unless you either
really know what you are doing or you are starting over after running
a temporary or test setup.
You can accomplish the same effect as above by setting $Conf{IncrKeepCnt}
to 4 or less.
But even then there will still be cases where there are 2 full backups
kept. The additional storage is much less than you expect because of
hardlinking.
Craig
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Carl Wilhelm Soderstrom wrote: I would really like to see hard drives made to be more reliable, rather than just bigger. I'm not sure that can be improved enough to matter. A failure of an inexpensive part once in five years isn't a big deal other than the side effects it might cause, and unless they can be 100% reliable (probably impossible) you have to be prepared for those side effects anyway. on a small scale, perhaps. when you get to an organization the size of (TBS|Hotmail|Yahoo)tho; it may make sense to spend a little more money in order to spend less labor swapping parts. I think before you get to that that scale you'd be network booting diskless and mostly disposable motherboards. You still have to deal with the drives somewhere, but using larger ones instead of large numbers of them. the market seems to agree with you tho... which is a pretty good indictment of central planning notions (i.e. Me) vs. what really is efficient. I find complete-system redundancy with commodity boxes to be cheaper and more efficient than using military strength parts that cost 10x as much and fail half as often. In our operations the most common failure is in software anyway since we try to push out new capabilities as fast as possible so I like lots of redundancy and the ability to run multiple versions of things concurrently. I could see where that would be different for operations where everything had to be in a single database, though. I want reliable storage. Run mirrored hot-swap drives. If one dies, replace it at some convenient time, sync it up and keep going. I have one machine last booted in mid-2003 that has had a couple of it's drives replaced that way. Your point is well-taken. I do keep in mind tho that I've seen multiple drives fail simultaneously or in rapid succession, and that the process of replacing drives costs time and effort. Yes, your building might catch fire or be flooded too. Those are probably more likely than multiple cheap drives failing simultaneously from some cause that wouldn't also kill better drives. You need offsite backups to cover these unlikely events anyway. It is not as trivial as you might think, once you factor in the time to detect the failure, find the replacement (if you can), replace the drive (which may involve removing the old drive from the carrier and replacing it with the new one), and make sure it's working properly. In an ideal world it's 5 minutes of work, in the real world I usually expect to lose an hour or two dealing with a failed drive. The real killer is the time planning out the replacement operation so the only time wasted is the one person who does the work. In the whole-system scenario where the systems are load balanced anyway, yanking a machine out isn't anything special and you don't even need the hot-swap drive case. This can be accounted as several hundred dollars of lost productivity in many cases; so it's worthwhile to spend more money on a better-quality drive sometimes. In practice I don't see any relationship between price and reliability. I'm inclined to think that they all come off the same production line these days. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
[BackupPC-users] drive quality (was: Re: very slow backup speed)
On 03/27 02:49 , Les Mikesell wrote: I find complete-system redundancy with commodity boxes to be cheaper and more efficient than using military strength parts that cost 10x as much and fail half as often. That's Google's solution, and it works for them. Nothing fancy... rebooting servers means a dude with a broomstick walking around pushing power buttons. The real killer is the time planning out the replacement operation so the only time wasted is the one person who does the work. In the whole-system scenario where the systems are load balanced anyway, yanking a machine out isn't anything special and you don't even need the hot-swap drive case. it's a nice ideal circumstance. :) In practice I've found that clustering systems often increases administrative headaches exponentially. There's a lot more that *can* go wrong and therefore *does* go wrong with a cluster. There's a place for them, like everything else... but they are far from a panacea. In practice I don't see any relationship between price and reliability. I'm inclined to think that they all come off the same production line these days. some are still better than others... how much better, I don't know. I've seen some really expensive hardware go belly-up at times too. Here's what I did with some of the aforementioned bad expensive hardware: http://www.redchrome.org/shooting_computers/shooting_computers_index.html (the Sparc 10, specifically) -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/27/07, Les Mikesell [EMAIL PROTECTED] wrote: Evren Yurtesen wrote: What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. I sent the status of the backups earlier today to mailing list? # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 22:40), today's max is 10% (3/27 01:00) and yesterday's max was 10%. * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.60GB (prior to pooling and compression). This is from 1 machine. Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 112 full149290 1957.8 0.10149251 1937.9 460120.7 224 full151701 2022.8 0.09151655 2004.7 100 18.1 238 full152170 2099.5 0.06152131 2081.6 115 17.9 244 incr214 48.90.00165 22.978 26.0 245 full152228 2095.2 0.06152177 2076.9 108 18.3 246 incr118 17.30.0076 0.2 69 17.1 247 incr159 21.40.00111 3.1 75 18.4 248 incr181 22.10.00132 2.5 79 19.7 249 incr186 24.00.00146 7.6 54 16.4 250 incr206 25.50.00159 6.7 70 18.8 From the perspective of the backup directories, it doesn't matter whether or not there are additional links to a file. It just looks at the directory entry to find the file it wants. It may matter that the inodes and file contents end up splattered all over the place because they were written at different times, though. Yep, Lee is right here. Unless BSD handles hard-links in some crazy manner. I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. For example this guy is using Linux (problem is OS independent) http://forum.psoft.net/showpost.php?p=107808postcount=16 If you check namesys.com benchmarks, you will see that they only tested reiserfs against ext2/ext3/xfs/jfs and conveniently forgot to test against ufs2. You can see in the end of the page that slight performance increase in reiserfs is also bringing twice the cpu usage! (plus extX is faster in certain operations even) http://www.namesys.com/benchmarks.html When your overall speed is limited by the speed of your disks and your CPU spends all it's time twiddling it's thumbs waiting for disk, who cares if CPU doubles and still spends 90% of it's time waiting as long as it gets the job done faster? Whatever, this is not the problem here.The fact is that, according to reiserfs developers reiserfs is more or less the same speed with ext2. I dont think the problem is related to any filesystem as it occurs on both Linux and FreeBSD To summarize Evran's setup: FreeBSD, 250MB ram, CPU unknown, 1 7200rpm Seagate ATA disk Filesystem: ufs2, sync, noatime Pool is 17.08GB comprising 760528 files (avg file size ~22KB) BackupPC reports backup speeds between 0.06 - 0.22 MB/s Total backup time per host: Unknown CPU is 99% idle during backups Disk shows ~75 IO/s during load and low transfer rate Says even small backup w/small number of files is slow. Can you try mounting the backup partition async so we can see if it really is read performance or write performance that is killing backup performance? I would also highly recommend that you limit backups to 1 concurrent backup at a time. I must wonder if ufs2 is really bad at storing inodes on disk... On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. BTW, how does BackupPC calculate speed? I think it calculates backup speed by reporting files transferred over time, so if you don't have many files that change, won't BackupPC report a very low backup speed. This is like the 'Contact'
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 245 full152228 2095.2 0.06152177 2076.9 108 18.3 246 incr118 17.30.0076 0.2 69 17.1 Can you post the duration that these backups took? All that these stats tell us is how much of your data is churning, and how big your average file size is. I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. For example this guy is using Linux (problem is OS independent) http://forum.psoft.net/showpost.php?p=107808postcount=16 Er, RAID-5 is slower than a single disk except on sustained reads, which is not the typical case with BackupPC. I only use a single disk and get between 0.5mb/s and 10mb/s, depending on the client and its file size/file count distribution. The question is whether other people have gotten the system to work correctly under the same or worse conditions, and already that has been answered in the affirmative. Be prepared to accept that the software isn't broken just because you haven't gotten it working to your satisfaction in your specific situation... It could be rsync, UFS, BSD, or any number of other factors that are causing you grief. Whatever, this is not the problem here.The fact is that, according to reiserfs developers reiserfs is more or less the same speed with ext2. I dont think the problem is related to any filesystem as it occurs on both Linux and FreeBSD Your argument lacks logic. If a filesystem can be configured to be slow on multiple OS's, does that mean BackupPC is failing to do its job? No, it means multiple people have managed to set up a system that performs badly using it. That's not so uncommon. BackupPC does not exist in a vacuum: its performance is sensitive to the environment of the server and its clients. Many people are using it without issue, right here on this very list. The question you should be asking is, what makes your system perform badly? Start by picking out things that you do differently, or that definitely affect performance. Transport protocol (rsync, tar). UFS. BSD. Sync on your filesystem. Start changing those one at a time and measure the performance. On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. If you want people to help you, it's probably best if you refrain from blaming the program until you have proof. So far, you have argued with people who provide evidence that the system works fine. It puts people on the defensive, and you may find people less willing to help you in the future. We do know BackupPC behaves well on other file systems and operating systems... maybe UFS or BSD is doing something poorly--how would you know they aren't? We have less data points to draw from there. I suspect it has a lot more to do with what the MB/s stats really mean. Maybe Craig can give a precise definition? This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. With rsync in incremental backups, it still has to check the metadata for each file to determine the changed file set. With millions of files, it will take a while. If your host or server is low on memory for any reason, this may bog it down and start vm swapping. I would recommend trying out tar to see if the protocol behavior matters. Try different mount options that are for higher performance. Others have made similar suggestions as well. Good luck, JH - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: David Rees wrote: On 3/27/07, Les Mikesell [EMAIL PROTECTED] wrote: Evren Yurtesen wrote: What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. I sent the status of the backups earlier today to mailing list? # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 22:40), today's max is 10% (3/27 01:00) and yesterday's max was 10%. * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.60GB (prior to pooling and compression). This is from 1 machine. Totals Existing Files New Files Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB 112 full 149290 1957.8 0.10 149251 1937.9 4601 20.7 224 full 151701 2022.8 0.09 151655 2004.7 100 18.1 238 full 152170 2099.5 0.06 152131 2081.6 115 17.9 244 incr 214 48.9 0.00 165 22.9 78 26.0 245 full 152228 2095.2 0.06 152177 2076.9 108 18.3 246 incr 118 17.3 0.00 76 0.2 69 17.1 247 incr 159 21.4 0.00 111 3.1 75 18.4 248 incr 181 22.1 0.00 132 2.5 79 19.7 249 incr 186 24.0 0.00 146 7.6 54 16.4 250 incr 206 25.5 0.00 159 6.7 70 18.8 From the perspective of the backup directories, it doesn't matter whether or not there are additional links to a file. It just looks at the directory entry to find the file it wants. It may matter that the inodes and file contents end up splattered all over the place because they were written at different times, though. Yep, Lee is right here. Unless BSD handles hard-links in some crazy manner. I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. Well, I'd like to point out that I am using tar over nfs over a bonded 100mbit(x2) link. NFS is terrible for small file access. However, we have an xserve that I use xtar with, over ssh-- I get 5MB/s for that (which is the best I've seen). My other rsync clients get up to 4.2 MB/sec. So, I think the 25% [EMAIL PROTECTED]/s indicates I should never expect more than 10MB/s. Considering everything backuppc has to do with each file as it receives them, without writing them to disk first, I don't think we can expect it to be very fast. However, it is absurdly efficient with space. You can't have everything :-). Complexity, speed, efficiency; pick any two? brien - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: # Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB 245 full 152228 2095.2 0.06 152177 2076.910818.3 On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. 2MB/sec isn't bad when handling a lot of files (try unpacking a tar with hundreds of thousands of little files to see). The problem is that you are getting .06 on a drive that is capable of running a couple of sessions at 1.5MB/sec or so. I'd try to get up to normal speed before claiming that the software needs to be fixed to get more. More RAM will probably make a huge difference, but first are you sure that your IDE controller and cable are a match for the drive and that your OS is using an efficient DMA mode? I've seen even some of the 80-pin cables have a problem that would make things shift down to 33Mhz pio mode which will kill you. Does freebsd log the IDE mode detected and have some way to test throughput? BTW, how does BackupPC calculate speed? I think it calculates backup speed by reporting files transferred over time, so if you don't have many files that change, won't BackupPC report a very low backup speed. This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. If you look at the 'duration' in the backup summary and the Size/MB in the lower File Size summary you can compute your own rate based on what you are backing up. For fulls at least this seems to be based on the real target size, not what rsync had to transfer. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: Raid5 doesn't distribute disk activity - it puts the drives in lockstep and is slower than a single drive, especially on small writes where it has to do extra reads to re-compute parity on the existing data. I am confused, when a write is done the data is distributed in the disks depending on the stripe size you are using. When you start reading the file, you are reading from 5 different disks. So you get way better performance for sure on reads. The stripe effect only comes into play on files large enough to span them and not at all for directory/inode accesses which is most of what you are doing. Meanwhile you have another head tied up checking the parity and for writes of less than a block you have to read the existing contents before the write to re-compute the parity. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/27/07, Evren Yurtesen [EMAIL PROTECTED] wrote: David Rees wrote: Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. I sent the status of the backups earlier today to mailing list? Still missing wall-clock time. Though we can extrapolate that it may be taking over 6 hours for a full backup of the system below. Is that correct? This is from 1 machine. Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 112 full149290 1957.8 0.10149251 1937.9 460120.7 I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. As mentioned earlier, RAID 5 is horrible for random small read/write performance. It is often worse than a single disk. But still, I have a client which has 1.5 million files and 80-100GB of data. A full backup takes 4-6 hours which is reasonable. Full backups average 4.5-5MB/s. For example this guy is using Linux (problem is OS independent) http://forum.psoft.net/showpost.php?p=107808postcount=16 Your transfer rates are at least 5x slower than his and 10x slower than what most people here are getting. There is something wrong specifically with your setup. I suspect that if your backups were going 5x faster you'd be happy and 10x faster you'd be very happy. On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. With ext2 the default is async IO. With ext3 (the default system now) the default is ordered which is similar to BSD's soft updates. This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. Using the rsync transfer method, it does require at least a stat of every single file to be backed up. For 150k files and 2GB of data, you'd really expect this to be done within a hour with nearly any machine. Are you using the --checksum-seed option? That will significantly speed up (the 3rd and later) backups as well. Are you also sure the backup server isn't swapping as well? Can you get us some good logs from vmstat or similar during a backup? I also suspect that there is something else in this case slowing your machine down. Unless you give us the information to help track it down, we will not be able to figure it out. I feel like I am pulling teeth. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] RSync v. Tar
Hi, Les Mikesell wrote on 27.03.2007 at 01:03:32 [Re: [BackupPC-users] RSync v. Tar]: Jesse Proudman wrote: I've got one customer who's server has taken 3600 minutes to backup. 77 Gigs of Data. 1,972,859 small files. Would tar be better or make this faster? It's directly connected via 100 Mbit to ^^^ the backup box. ^^ If the files don't change frequently, tar incremental runs will be much faster because they are based only on the target timestamps while rsync will load the entire directory at both ends and compare them. if you ask me, regardless of how your data changes, tar is the way to go, not rsync, especially for *full* backups. With a direct 100 MBit connection, there's not much point in spending (lots of) CPU time for saving bandwidth - not with 2 million files. rsync is good for low bandwidth connections, where the link severely limits the transfer and speeding it up makes a real difference. In your case, your link speed is in the same order of magnitude as your disk I/O performance (considering our favorite topic, the seek times on the pool file system, the network link may in fact not even be the limiting factor - it clearly isn't, as 77 GB would take slightly more than 2 hours to transfer over a 100 MBit link, and rsync is not making it go faster than that ;-). rsync has additional benefits concerning finding and backing up new (or moved) files with old timestamps and deleted files on *incremental* backups, but keeping the list of 2 million files in memory will probably be a problem, as it possibly was with your full (?) backup (how much memory do the BackupPC server and the backed up host have?). (Les: if the files *do* change frequently, there's even less speedup to get from using rsync. Only frequent metadata changes without file content changes would give rsync an advantage - assuming it is faster to figure out that the file is identical than to simply send it over the network.) Regards, Holger P.S.: rsync checksum caching *might* make a difference starting from the third backup, but I read that it's less improvement than one might expect. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: Raid5 doesn't distribute disk activity - it puts the drives in lockstep and is slower than a single drive, especially on small writes where it has to do extra reads to re-compute parity on the existing data. I am confused, when a write is done the data is distributed in the disks depending on the stripe size you are using. When you start reading the file, you are reading from 5 different disks. So you get way better performance for sure on reads. The stripe effect only comes into play on files large enough to span them and not at all for directory/inode accesses which is most of what you are doing. Meanwhile you have another head tied up checking the parity and for writes of less than a block you have to read the existing contents before the write to re-compute the parity. Actually most (all?) raid 5 systems I've met don't check parity on read - they rely on the drive indicating a failed read.However the write splice penalty for raid 5 can still be pretty high (seek, read, insert new data, compute new parity, write data, seek, write parity) - and that's on top of the fact that the OS did it's own logical read, check permissions, insert, write to update directories and the like. John - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Unable to connect to BackupPC server error
On Mon, 2007-03-26 at 22:50 -0700, Craig Barratt wrote: Winston writes: I had been running BackupPC on an Ubuntu computer for several months to back the computer to a spare hard drive without problem. About the time I added a new host (Windows XP computer using Samba), I started getting the following behavior: BackupPC backs both hosts properly onto the spare hard drive once or twice after I reboot the Ubuntu server. Then I get a Error: Unable to connect to BackupPC server error when I attempt to go the web interface. When I restart BackupPC with /etc/init.d/backuppc restart, I get a message Can't create LOG file /var/lib/backuppc/log/LOG at /usr/share/backuppc/bin/BackupPC line 1735. Perhaps the /var/lib file system is full? If not, does the backuppc user have permissions to write in /var/lib/backuppc/log? Craig I hadn't thought about the file system being full. After checking just now, this is not the answer. /var/lib has 48G available on my main hard drive. /var/lib/backuppc, to which the spare hard drive is mounted, has 59G available. The directory /var/lib/backuppc/log is owned by backuppc and is in the backuppc group. Its permission is 750. All files in it also are owned by backuppc and are in the backuppc group. They all have 640 permission. Winston - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Unable to connect to BackupPC server error
It is a SATA drive as is the main drive. Winston On Tue, 2007-03-27 at 09:54 -0600, Ciarlotta, Aaron wrote: That spare hard drive wouldn't happen to be an external USB or Firewire, would it? Aaron Ciarlotta Linux/UNIX Systems Guru -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Winston Chan Sent: Monday, March 26, 2007 8:14 PM To: backuppc-users@lists.sourceforge.net Subject: [BackupPC-users] Unable to connect to BackupPC server error I had been running BackupPC on an Ubuntu computer for several months to back the computer to a spare hard drive without problem. About the time I added a new host (Windows XP computer using Samba), I started getting the following behavior: BackupPC backs both hosts properly onto the spare hard drive once or twice after I reboot the Ubuntu server. Then I get a Error: Unable to connect to BackupPC server error when I attempt to go the web interface. When I restart BackupPC with /etc/init.d/backuppc restart, I get a message Can't create LOG file /var/lib/backuppc/log/LOG at /usr/share/backuppc/bin/BackupPC line 1735. I have made sure that the backuppc is the owner of /var/lib/backuppc/log/LOG. It seems the only way to get backuppc to work again is to reboot the Ubuntu server. I then can see on the web interface that the last successful backup (of both hosts) occurred 1 or 2 days after the previous reboot. Then backupPC works for 1 or 2 backups and the cycle starts again. What is the cause of this and how can I fix it? Winston - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDE V ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ The information in this electronic mail message is sender's business Confidential and may be legally privileged. It is intended solely for the addressee(s). Access to this Internet electronic mail message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. The sender believes that this E-mail and any attachments were free of any virus, worm, Trojan horse, and/or malicious code when sent. This message and its attachments could have been infected during transmission. By reading the message and opening any attachments, the recipient accepts full responsibility for taking protective and remedial action about viruses and other defects. Travelport Inc. is not liable for any loss or damage arising in any way from this message or its attachments. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Unable to connect to BackupPC server error
Winston Chan wrote: I had been running BackupPC on an Ubuntu computer for several months to back the computer to a spare hard drive without problem. About the time I added a new host (Windows XP computer using Samba), I started getting the following behavior: BackupPC backs both hosts properly onto the spare hard drive once or twice after I reboot the Ubuntu server. Then I get a Error: Unable to connect to BackupPC server error when I attempt to go the web interface. When I restart BackupPC with /etc/init.d/backuppc restart, I get a message Can't create LOG file /var/lib/backuppc/log/LOG at /usr/share/backuppc/bin/BackupPC line 1735. Perhaps the /var/lib file system is full? If not, does the backuppc user have permissions to write in /var/lib/backuppc/log? Craig I hadn't thought about the file system being full. After checking just now, this is not the answer. /var/lib has 48G available on my main hard drive. /var/lib/backuppc, to which the spare hard drive is mounted, has 59G available. The directory /var/lib/backuppc/log is owned by backuppc and is in the backuppc group. Its permission is 750. All files in it also are owned by backuppc and are in the backuppc group. They all have 640 permission. There's also a possibility of running out of inodes. Does 'df -i' look reasonable? -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] RSync v. Tar
I have a bit of hard data to offer on this subject, as I recently switched a backup from tar+ssh (over cygwin) to rsyncd. The backuppc server is on the same physical LAN, and connect to each other via a 192.168 address. All the cabling and switches support 100 MB full duplex communications, and the servers have gigabit NICs. The backuppc server is dumping the data to the /var partition which is on the 2 80Gb satas in the case, running in a RAID 1 software array, under mdadm on Debian stable. These are the current stats, using rsyncd - Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB 30http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=30 full 168480 41758.5 5.66 168333 41639.8 310 118.9 35http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=35 incr 730 191.7 0.19 564 103.6 223 88.2 36http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=36 incr 785 198.4 0.18 725 123.0 106 75.4 37http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=37 full 169010 41836.6 5.73 168876 41750.2 276 86.5 38http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=38 incr 0 0.0 0.00 0 0.0 29 0.0 39http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=39 incr 0 0.0 0.00 0 0.0 0 0.0 40http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=40 incr 155 89.4 0.04 42 5.5 169 83.9 41http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=41 incr 321 124.2 0.05 142 19.8 234 104.4 Backup# Type Filled Level Start Date Duration/mins Age/days Server Backup Path 30http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=30 full yes 0 3/16 18:00 122.9 11.2 /var/lib/backuppc/pc/sarah/30 35http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=35 incr no 1 3/21 18:00 16.9 6.2 /var/lib/backuppc/pc/sarah/35 36http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=36 incr no 1 3/22 18:00 17.9 5.2 /var/lib/backuppc/pc/sarah/36 37http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=37 full yes 0 3/23 18:00 121.7 4.2 /var/lib/backuppc/pc/sarah/37 38http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=38 incr no 1 3/24 18:00 16.5 3.2 /var/lib/backuppc/pc/sarah/38 39http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=39 incr no 1 3/25 18:00 15.4 2.2 /var/lib/backuppc/pc/sarah/39 40http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=40 incr no 1 3/26 18:00 33.1 1.2 /var/lib/backuppc/pc/sarah/40 41http://mail.stephanco.com/cgi-bin/BackupPC_Admin?action=browsehost=sarahnum=41 incr no 1 3/27 18:00 38.6 0.2 When it was doing tar, the full backups took far longer, in the neighborhood of 600 minutes. The incremental backups took around an hour most days. So I clearly made out better with rsyncd. Just as additional info, the server being backed up is a file server for a small company. It is backing up the directory where they store .jpg images of the products they sell. They organize it by date, so obviously everything in the current day's directory is new, but previous directories aren't modified most of the time. I hope that helps. Peace, Jim On 3/27/07, Holger Parplies [EMAIL PROTECTED] wrote: Hi, Les Mikesell wrote on 27.03.2007 at 01:03:32 [Re: [BackupPC-users] RSync v. Tar]: Jesse Proudman wrote: I've got one customer who's server has taken 3600 minutes to backup. 77 Gigs of Data. 1,972,859 small files. Would tar be better or make this faster? It's directly connected via 100 Mbit to ^^^ the backup box. ^^ If the files don't change frequently, tar incremental runs will be much faster because they are based only on the target timestamps while rsync will load the entire directory at both ends and compare them. if you ask me, regardless of how your data changes, tar is the way to go, not rsync, especially for *full* backups. With a direct 100 MBit connection, there's not much point in spending (lots of) CPU time for saving bandwidth - not with 2 million files. rsync is good for low bandwidth connections, where the link severely limits the transfer and speeding it up makes a real difference. In your case, your link speed is in the same order of magnitude as your disk I/O performance (considering our favorite topic, the seek times on the pool file system, the network link may in fact not even be the limiting factor - it clearly isn't, as 77 GB would take slightly more than 2 hours to transfer over a 100 MBit link, and rsync is not making it go faster than that ;-). rsync has additional benefits concerning finding and backing up new (or moved) files with old timestamps and deleted files on *incremental* backups, but keeping the list of
Re: [BackupPC-users] very slow backup speed
I found a strange characteristic of IDE drive. When I connected CDROM and IDE hdd on same channel the data transfer with hdparm -t command is @37 MB. As I removed CDROM from that channel now the data transfer rate is @60 MB. This seems to work with 4 to 5 linux pc. -- Nilesh Vaghela ElectroMech Redhat Channel Partner and Training Partner 74, Nalanda Complex, Satellite Rd, Ahmedabad 25, The Emperor, Fatehgunj, Baroda. www.electromech.info - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] Unable to connect to BackupPC server error
Winston writes: I hadn't thought about the file system being full. After checking just now, this is not the answer. /var/lib has 48G available on my main hard drive. /var/lib/backuppc, to which the spare hard drive is mounted, has 59G available. The directory /var/lib/backuppc/log is owned by backuppc and is in the backuppc group. Its permission is 750. All files in it also are owned by backuppc and are in the backuppc group. They all have 640 permission. Is that file system out of inodes? Use df -i /var/lib/backuppc to check. Craig - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: # Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB 245 full 152228 2095.2 0.06 152177 2076.910818.3 On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. 2MB/sec isn't bad when handling a lot of files (try unpacking a tar with hundreds of thousands of little files to see). The problem is that you Yes it is terrible. I get much better performance if I do the tar option with the same files. As a matter of fact I was using a smaller script for taking backups earlier. (which I still use on some servers) and transfer files over NFS. It works way faster, especially incremental backups take 5-10 minutes compared to 400 minutes with backuppc are getting .06 on a drive that is capable of running a couple of sessions at 1.5MB/sec or so. I'd try to get up to normal speed before claiming that the software needs to be fixed to get more. More RAM will probably make a huge difference, but first are you sure that your IDE controller and cable are a match for the drive and that your OS is using an efficient DMA mode? I've seen even some of the 80-pin cables have a problem that would make things shift down to 33Mhz pio mode which will kill you. Does freebsd log the IDE mode detected and have some way to test throughput? ad2: 238475MB ST3250824A/3.AAH [484521/16/63] at ata1-master UDMA100 DMA is working just fine. You are omitting the fact that 2mb/sec is very bad for a raid setup. I agree, handling a lot of files might be slow but this depends on how you handle the files. But I was handling the same files before and it wasnt taking this long. BTW, how does BackupPC calculate speed? I think it calculates backup speed by reporting files transferred over time, so if you don't have many files that change, won't BackupPC report a very low backup speed. This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. If you look at the 'duration' in the backup summary and the Size/MB in the lower File Size summary you can compute your own rate based on what you are backing up. For fulls at least this seems to be based on the real target size, not what rsync had to transfer. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/27/07, Evren Yurtesen [EMAIL PROTECTED] wrote: David Rees wrote: Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. I sent the status of the backups earlier today to mailing list? Still missing wall-clock time. Though we can extrapolate that it may be taking over 6 hours for a full backup of the system below. Is that correct? That is true, full backups take about 500-600 minutes and incrementals take 200-300minutes. This is from 1 machine. Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 112 full149290 1957.8 0.10149251 1937.9 460120.7 I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. As mentioned earlier, RAID 5 is horrible for random small read/write performance. It is often worse than a single disk. Parity information is not required for normal reads, so you get 4-5 times better performance with 5 disks. Plus usually those drives are 10K rpm. When I am making incremental or even full backups, backuppc does not write same files again so writes are very little. So you get a huge performance boost with raid5 and 10k rpm drives compared to a normal ide drive. But still, I have a client which has 1.5 million files and 80-100GB of data. A full backup takes 4-6 hours which is reasonable. Full backups average 4.5-5MB/s. For example this guy is using Linux (problem is OS independent) http://forum.psoft.net/showpost.php?p=107808postcount=16 Your transfer rates are at least 5x slower than his and 10x slower than what most people here are getting. There is something wrong specifically with your setup. I suspect that if your backups were going 5x faster you'd be happy and 10x faster you'd be very happy. I would be happy if backups were made with the same speed as other backup programs can do. On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. With ext2 the default is async IO. With ext3 (the default system now) the default is ordered which is similar to BSD's soft updates. This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. Using the rsync transfer method, it does require at least a stat of every single file to be backed up. For 150k files and 2GB of data, you'd really expect this to be done within a hour with nearly any machine. I am using rsync to sync 2GB of files from one machine to another and the number of files is about 100k and the operation takes 30seconds to 1 minute in average. Are you using the --checksum-seed option? That will significantly speed up (the 3rd and later) backups as well. No, it requires a patched rsync. Are you also sure the backup server isn't swapping as well? Can you get us some good logs from vmstat or similar during a backup? I can tell that it is not swapping because the disk where the swaps are idle while taking backup. ad0 is the disk with the swap. If there was such problem then it would be reading like crazy from swap. I have given this information before. I also suspect that there is something else in this case slowing your machine down. Unless you give us the information to help track it down, we will not be able to figure it out. I feel like I am pulling teeth. I have given you all the information you asked for(didnt I?), even tried async option. Incremental backup of 1 machine still took about 300 minutes. The machine is working fine. I was using another backup program which was working way faster to backup the same machines. So I dont think that I have a hardware problem or such. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net
Re: [BackupPC-users] very slow backup speed
nilesh vaghela wrote: I found a strange characteristic of IDE drive. When I connected CDROM and IDE hdd on same channel the data transfer with hdparm -t command is @37 MB. As I removed CDROM from that channel now the data transfer rate is @60 MB. This seems to work with 4 to 5 linux pc. I have no CDROMS installed on the machine and drives are on their own cables. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
I have given you all the information you asked for(didnt I?), even tried async option. Incremental backup of 1 machine still took about 300 minutes. The machine is working fine. I was using another backup program which was working way faster to backup the same machines. So I dont think that I have a hardware problem or such. I haven't read *every* email in this thread (it was getting quite repetitive seeing you complaining about how crap backuppc is), but anyway, could you please elaborate on the method that your 'other' backup solution used, ie, if your other solution used rsync, and you are using rsync with backuppc, then that might be helpful to know. If you used tar before, and now use rsync, that would obviously make a difference. I think overall, the attitude you have displayed is not conducive to resolving the problem. I also think it has been clearly shown that backuppc itself is not the problem, as other people are using it successfully, under similar conditions. What you need to do is basic diagnosis to find where the bottle neck is in your particular system. I'd suggest you use whatever tools are available under your chosen OS (I think it is freebsd), to measure: 1) CPU utilisation on both the client being backed up, and the backuppc server 2) Memory/swap utilisation on both client and server 3) Disk utilisation/bandwidth on both client and server 4) Network utilisation/bandwidth on both client and server Also, if you are using tar as your transfer method, then I'd suggest you try a 'manual' backup something like this (note, this command line is wrong, but you will need to adjust to get the right one). from the backuppc machine: cd /var/lib/backuppc;mkdir temp;cd temp;ssh client tar -cf - /backupdir | tar -xf - Try and copy the command line args from your backup log file Time how long that takes, and see how it compares to backuppc times if they are similar, then you know the issue is outside of backuppc. Of course, if you are using rsync, then do the same test with rsync data, but I'm sure it has already been said that you should use tar for your case with lots of small files. Just my 0.02c worth Regards, Adam -- Adam Goryachev Website Managers Ph: +61 2 8304 [EMAIL PROTECTED] Fax: +61 2 8304 0001www.websitemanagers.com.au - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
