Re: [BackupPC-users] very slow backup speed
On 3/29/07, Evren Yurtesen [EMAIL PROTECTED] wrote: I didnt blame anybody, just said BackupPC is working slow and it was working slow, very slow indeed. checksum-seeds option seems to be doing it's trick though. How long are full and incremental backups taking now? I am thankful to people who wrote suggestions here in this forum, I tried all of those suggestions one by one. I think that shows that I took them seriously even though some of them looked like long shots. Eventually one of the suggestions seems to be working. You only tried 2 things. Mounting the backup partition async and turning on checksum-seeds. Are you going to the 2 others? (Add memory and try tar instead of rsync) -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/29/07, Evren Yurtesen [EMAIL PROTECTED] wrote: I didnt blame anybody, just said BackupPC is working slow and it was working slow, very slow indeed. checksum-seeds option seems to be doing it's trick though. How long are full and incremental backups taking now? In one machine it went down from 900 minutes to 175 minutes. I expect better performance when more memory is added (today or tomorrow they will add it) and I dont think all files had checksums cached when this full was ran. Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 245 full280030 7570.4 0.14274205 6797.6 10578 776.3 252 full283960 8020.8 0.76276665 6959.3 12232 1065.0 Existing Files New Files Backup# TypeComp Level Size/MB Comp/MB Comp Size/MB Comp/MB Comp 245 full9 6797.6 3868.9 43.1% 776.3 368.7 52.5% 252 full9 6959.3 4056.9 41.7% 1065.0 539.0 49.4% I am thankful to people who wrote suggestions here in this forum, I tried all of those suggestions one by one. I think that shows that I took them seriously even though some of them looked like long shots. Eventually one of the suggestions seems to be working. You only tried 2 things. Mounting the backup partition async and turning on checksum-seeds. Are you going to the 2 others? (Add memory and try tar instead of rsync) The memory will be added. As I mentioned before the machine is at a remote location and the guys there should add it. I could try tar for testing purposes if you like? I think rsync will be sufficiently fast enough. I am guessing that with checksum-seeds the difference shouldnt be so much tar probably transfers much more data in full backups? Rsync can be faster perhaps if ignore-times was removed when taking full backups. I am thinking of removing ignore-times option from full backups with rsync and see how much it effects for seeing the difference. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On Fri, 30 Mar 2007, Sylvain MAURIN wrote: MaxLine III 300 (7200 rpm, ATA, 8 meg cache), ext3 w/noatime. I'd change it to XFS, but backing up 1.7TB would take weeks/months because of the hardlinks. Hi, I use your hardware clone except for HDs (which are seagate 750GB) and maybe for motherboard (Tyan) and RAM (4Gb) and choosed XFS for running backuppc on a amd64 sarge distrib with backported kernel. I purchased recently a large library to include backuppc and all other servers to amanda tape backup. You are telling us that xfsdump can't handle well hardlinks but I got around 5mb/s... In fact it's worse than the standard agregated 15~20mb/s I have while backuppc dumps but it goes accordingly with other servers dump/tar speeds... It just take 4~5 days for a 1.5Tb pool ! xfsdump should handle that fine, in fact i'd expect it to be relatively speedy. My weeks/months would be using rsync, as I can't just dump ext3 to xfs, I have to use something that can handle the hardlinks. ext3 to ext3 upgradesa are easy, just dd the partition and resize2fs it later.. and similar for xfs to xfs. I'm actually in the process of moving the system from ext3 to xfs, it's been 3 days so far just to copy the cpool, and then I'll start the pc/* directories using 3.0.x's backuppc_tarpccopy script. I'm hoping that'll be faster than rsync.. the last time i used rsync was on a 200 gig pool, and it took 4 days to run. I admit that I can't do a full restore for now ( I havn't another free TByted partition) but partials are fine with all hardlinks on small excerpt (50K harlinks, againt 50*100kb files pool). So, from where did you get your assumptions about xfs_dumps duration (monthes) ? Do I have have missed a point of configuration and just got luck in my small restore points ? Must I wait bad surprise like exponential augmentation of tape backup time ? Please help me and feel free to share your experience and my question by forwarding to backuppc ML. Sylvain PS : your 1gb isn't small for a 4TB partition ? Except for insane rsync runs, I don't have any problems with memory usage.. most of the backups are samba, so I don't need a lot of memory for rsync backups. I'm pondering putting another 4x256 memory in it anyways though, as ram is cheap. Mike - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/30/07, Evren Yurtesen [EMAIL PROTECTED] wrote: David Rees wrote: How long are full and incremental backups taking now? In one machine it went down from 900 minutes to 175 minutes. I expect better performance when more memory is added (today or tomorrow they will add it) and I dont think all files had checksums cached when this full was ran. Wow, that is a huge difference! I didn't expect performance to increase that much, apparently the checksum caching is really reducing the number of disk IOPs. I could try tar for testing purposes if you like? I think rsync will be sufficiently fast enough. I am guessing that with checksum-seeds the difference shouldnt be so much tar probably transfers much more data in full backups? Rsync can be faster perhaps if ignore-times was removed when taking full backups. I am thinking of removing ignore-times option from full backups with rsync and see how much it effects for seeing the difference. Tar is definitely worth a shot if it's short comings for incremental backups are acceptable and network bandwidth isn't an issue. Removing rsync ignore-times may also be an option if the reduction in possible data integrity is acceptible. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Hi, Evren Yurtesen wrote on 29.03.2007 at 08:24:43 [Re: [BackupPC-users] very slow backup speed]: Les Mikesell wrote: Evren Yurtesen wrote: Also I see the --ignore-times option in rsync args for full backups. Why is this necessary exactly? If you don't, you are trusting that every file that has the same name, timestamp and length on the previous backup still matches the contents on the target. It probably, mostly, usually, does... actually, if you don't, you are trusting that nobody/nothing wrote to a file, keeping its size the same, and then reset the modification time to the previous value. Technically, that's simple to do and does happen. Full backups are meant for clearing up such a situation. We're talking about backups after all. From time to time we want to be *absolutely* sure that our data is really on backup in unmodified form. A backup that perpetually assumes oh, everything will probably be fine for sake of speed is quite pointless. [...] Yes, this is causing a lot of extra processing on the server side. However most backup programs do not do this detailed file checking(I think?) and nobody seems to be complaining. There at least should be an option to only rely on the name/timestamp of a file when using rsync also. You've gone to incredible lengths to demonstrate that you don't know what rsync is all about, and you've spread that out over several threads. In a nutshell, it's save bandwidth at the cost of processing overhead. Within BackupPC, this comes at an additional cost, as files need to be uncompressed. BackupPC is not fine-tuned to your case of small files but rather handles huge files nicely, which it would be impossible to cache in uncompressed form. All of this is clearly documented, if you're only willing to read and understand it. Most backup programs take do a full backup to mean read *everything* over the network and write it to backup medium. BackupPC modifies this to mean read *everything* over the network and store *unchanged* contents efficiently by using hardlinks while taking the notion of unchanged very seriously. If previous backups have become corrupted for some reason, that is no good reason to feel fine with storing unchanged contents in unchangedly corrupted form for the future. That's not what backups are about. *rsync* further modifies the meaning (of a full backup!) to read only *changed* contents over the network, so we can do multi-gigabyte backups over a low bandwidth link (and store it efficiently as above) - low bandwidth meaning maybe modem links. Note that checksum caching already means a compromise: corruption in previous backups may go unnoticed and uncorrected. If you're not prepared to pay the price rsync costs, then don't use rsync. Use tar. That's what you've got the choice for. What I meant was, if tar is used then live comparisons can not be done. Tar relies on the modification time. However if rsync is used then the file location and modification time can be checked. If a file is renamed or moved keeping the same modification time then this can be detected using rsync even without checksum checks (which would give the same checksum anyway). Nonsense. If you previously had a file 'foo' with 32 KB of contents and modification time X and now have a file 'bar' with the same length and date, you would want to blindly assume they're identical without checking the contents? rsync doesn't do voodoo. On the remote end, you have the file 'bar' and no knowledge about the file 'foo'. Even with rsync, the file 'bar' is transfered in this case. It's BackupPC that re-establishes that the contents are identical while receiving it (and it does that even for a 1 GB file which could turn out to have only the last byte changed without needing to store an intermediate copy!). So checksum is saving bandwidth if a file modification time is different but the contents are same. That much is correct. Note that the main objective of rsync is to make sure that local and remote copy are *really the same*. You don't say 'rsync foo host:/bar' mainly to save bandwidth but to update the remote file (or local with reversed arguments). rsync means try to do that with low bandwidth requirements, skipping as much of the transfer as you can while still making sure the original goal is met. In any case, somebody else said that when checksum caching is enabled then file checksums are not calculated at it backup session by uncompressing the files. If this is true? then I can live with it but CPU usage could be lowered further if this was an option which could be disabled and better performance could be achieved. And that's where you completely miss the point. For taking a backup of a log file which has grown from 10 MB to 11 MB you either * need to transfer the new version completely (that's tar) or * figure out what has changed and transfer only that (that's rsync, and it needs block checksums to be calculated, because
Re: [BackupPC-users] very slow backup speed
Holger Parplies wrote: Also I see the --ignore-times option in rsync args for full backups. Why is this necessary exactly? If you don't, you are trusting that every file that has the same name, timestamp and length on the previous backup still matches the contents on the target. It probably, mostly, usually, does... actually, if you don't, you are trusting that nobody/nothing wrote to a file, keeping its size the same, and then reset the modification time to the previous value. Technically, that's simple to do and does happen. Actually, tar should be using ctime, not modification time. Ctime is the time of the last inode change, which will include the action of modifying mtime - or changing owner/permissions, etc. The only thing it will miss is a file which was itself unchanged but is under a directory that has been renamed since the full run. You will have a copy of these files in the full, just not in the right place. On the other hand, if the system time gets reset incorrectly so new files are getting timestamps earlier than the last full, they would all be skipped in the incrementals. And I think smb only sees mtime. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Hi, Evren Yurtesen wrote on 29.03.2007 at 17:04:05 [Re: [BackupPC-users] very slow backup speed]: Holger Parplies wrote: Most backup programs take do a full backup to mean read *everything* over the network and write it to backup medium. BackupPC modifies this to mean read *everything* over the network and store *unchanged* contents efficiently by using hardlinks while taking the notion of unchanged very seriously. If previous backups have become corrupted for some reason, that is no good reason to feel fine with storing unchanged contents in unchangedly [corrupted form] I am willing to take that risk, who are you to disagree? I am not asking that checksum checking should be removed. I am just asking that people who want to take that risk should be able to disable it. well, go ahead and implement it. If you want other people to do it for you, you'll at least have to convince them that your idea is not plain braindead. Isn't that obvious? And you'll have to accept others making a clear statement like: I wouldn't want to use backup software that can easily be misconfigured to make compromises concerning data security in favour of speed. You've made a convincing demonstration that it can already easily be misconfigured to be slow, and you've also shown that some people would try dubious things to speed it up, and, finally, you've shown that people would blame the author for their mistakes. That should compell the author to implement what you want, shouldn't it? Sure the way it works now fits to your usage doesnt mean that new features shouldnt be added. I'm not saying it fits my usage. I'm saying the concepts make sense. And you're not asking for new features, you're complaining that the impossible has not been achieved yet. Right. You're only asking for lowering bandwidth requirements at no extra cost. I wonder why nobody has thought of that before you. Regards, Holger - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Holger Parplies wrote: Hi, Evren Yurtesen wrote on 29.03.2007 at 17:04:05 [Re: [BackupPC-users] very slow backup speed]: Holger Parplies wrote: Most backup programs take do a full backup to mean read *everything* over the network and write it to backup medium. BackupPC modifies this to mean read *everything* over the network and store *unchanged* contents efficiently by using hardlinks while taking the notion of unchanged very seriously. If previous backups have become corrupted for some reason, that is no good reason to feel fine with storing unchanged contents in unchangedly [corrupted form] I am willing to take that risk, who are you to disagree? I am not asking that checksum checking should be removed. I am just asking that people who want to take that risk should be able to disable it. well, go ahead and implement it. If you want other people to do it for you, you'll at least have to convince them that your idea is not plain braindead. Isn't that obvious? And you'll have to accept others making a clear statement like: I wouldn't want to use backup software that can easily be misconfigured to make compromises concerning data security in favour of speed. You are missing something here please read below (but you might want to stop using backuppc :D) Right away when you enable 'tar' backuppc would be misconfigured by your definition, should tar support be removed? Tar can not check checksums anyway (plus has other flaws where it can miss newer/changed files) You've made a convincing demonstration that it can already easily be misconfigured to be slow, and you've also shown that some people would try dubious things to speed it up, and, finally, you've shown that people I guess everybody who is using 'tar' have been doing 'dubious things' :) would blame the author for their mistakes. That should compell the author to implement what you want, shouldn't it? I didnt blame anybody, just said BackupPC is working slow and it was working slow, very slow indeed. checksum-seeds option seems to be doing it's trick though. I am thankful to people who wrote suggestions here in this forum, I tried all of those suggestions one by one. I think that shows that I took them seriously even though some of them looked like long shots. Eventually one of the suggestions seems to be working. Not as fast as I would like but it is way better now and within acceptable levels (0.2mbyts/sec was really not acceptable). Sure the way it works now fits to your usage doesnt mean that new features shouldnt be added. I'm not saying it fits my usage. I'm saying the concepts make sense. And you're not asking for new features, you're complaining that the impossible has not been achieved yet. Right. You're only asking for lowering bandwidth There has been a misunderstanding here. I was only talking about backuppc comparing checksums for deciding if the file should be backed up or not and it is not impossible to disable this, it is disabled when you use tar for example. I have nothing against it using checksums when storing the file. (which would be impossible to disable) requirements at no extra cost. I wonder why nobody has thought of that before you. Actually on the contrary, what I suggest might or might not increase/decrease bandwidth requirements depending on the situation while making backups (especially full backups) faster and use less cpu as there wont be checksum checks done for each and every file. That is why people are suggesting to use tar to get better speed even though it is flawed. Even when this is disabled, the checksums are compared when any file with non-matching attribes is found. So it should be possible to find if there is a corruption in backed up data. After all when you enable checksum caching only 1% of the checksums in the backed up data is re-checked (by default) (this setting is what speeded up things for me, if you are using it, you must disable it for better file checks) Now that it is ok to suggest people to use tar even though it has the same flaw and more but making something better than tar but perhaps slight worse than current rsync implementation in catching changed files is not a good idea? How come? From BackupPC Info: - For SMB and tar, BackupPC uses the modification time (mtime) to determine which files have changed since the last lower-level backup. That mean SMB and tar incrementals are not able to detect deleted files, renamed files or new files whose modification time is prior to the last lower-level backup. - Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
Re: [BackupPC-users] very slow backup speed
Hi, Evren Yurtesen wrote on 30.03.2007 at 01:05:48 [Re: [BackupPC-users] very slow backup speed]: Right away when you enable 'tar' backuppc would be misconfigured by your definition, should tar support be removed? Tar can not check checksums anyway (plus has other flaws where it can miss newer/changed files) [...] There has been a misunderstanding here. I was only talking about backuppc comparing checksums for deciding if the file should be backed up or not and it is not impossible to disable this, it is disabled when you use tar for example. yep, there have been several misunderstandings here. Most of us have noticed, I believe. 1.) Rsync checksums are not a feature of BackupPC. They are a feature of the rsync protocol. 2.) Tar full backups do *not* miss files. And with that, my troll-sitting quota is used up. See you. Regards, Holger - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: From BackupPC Info: - For SMB and tar, BackupPC uses the modification time (mtime) to determine which files have changed since the last lower-level backup. That mean SMB and tar incrementals are not able to detect deleted files, renamed files or new files whose modification time is prior to the last lower-level backup. - When the documentation and the source differ, trust the source... Look at what your config.pl is actually using for the tar options - and you can change it if you want. You might also be able to take out the --ignore-timestamp on the rsync full arguments if you want but since the receiving side is built-in I'm not sure if it would work. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Holger Parplies wrote: Hi, Evren Yurtesen wrote on 30.03.2007 at 01:05:48 [Re: [BackupPC-users] very slow backup speed]: Right away when you enable 'tar' backuppc would be misconfigured by your definition, should tar support be removed? Tar can not check checksums anyway (plus has other flaws where it can miss newer/changed files) [...] There has been a misunderstanding here. I was only talking about backuppc comparing checksums for deciding if the file should be backed up or not and it is not impossible to disable this, it is disabled when you use tar for example. yep, there have been several misunderstandings here. Most of us have noticed, I believe. 1.) Rsync checksums are not a feature of BackupPC. They are a feature of the rsync protocol. BackupPC compares files in the pool using the checksum to find if the exact same file is in the pool or not. This is not a feature of rsync. Rsync has checksum checking too, where it is checking for if the file should be backed up or not and apply it to all files if --ignore-times option is used. I was first wrong and I thought this was done by BackupPC. Anyhow there is nothing to argue about. You are right, the thing I am talking about is not from backuppc. I could remove the --ignore-times option for better performance perhaps. 2.) Tar full backups do *not* miss files. Thats true, I forgot about it but you still might have wrong incremental backups. And with that, my troll-sitting quota is used up. See you. Sorry, and thanks. Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: From BackupPC Info: - For SMB and tar, BackupPC uses the modification time (mtime) to determine which files have changed since the last lower-level backup. That mean SMB and tar incrementals are not able to detect deleted files, renamed files or new files whose modification time is prior to the last lower-level backup. - When the documentation and the source differ, trust the source... Look at what your config.pl is actually using for the tar options - and you can change it if you want. You might also be able to take out the --ignore-timestamp on the rsync full arguments if you want but since the receiving side is built-in I'm not sure if it would work. Thank you, I realized about --ignore-timestamp a little bit too late. The problem was that I was wrong at first and thought checksum checking was a feature of backuppc. (and yes it also checks the checksum when a file is backed up.). Thanks Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
rsync is saving bandwidth at the cost of processing overhead doesnt mean that rsync is just for this purpose, it is a way to think about it. Dont be so single minded. As a matter of fact the rsync page says 'rsync is an open source utility that provides fast incremental file transfer.' and I want speed which coincides with the statement in rsync web pages. Just something that needs pointing out here, is that rsync is only a fast incremental file transfer program when transferring large files over a slow link. The rsync protocol reads the files at both ends, compares them blocks at a time with checksums, then writes out a whole new copy of each file. For a slow link this is great, as the data transferred is a little bigger than the differences in the file, so a 1Gb file with 1 byte changed at the end only has to transfer that 1 byte plus all the checksums etc. For a fast link this is actually slower, as both ends have to read the file, compute checksums, communicate them to each other, transfer chunks of the file down the line, then the destination server has to write out the file after applying the difference patch. For fast links it is much faster for the system to read at one end and write at the other, transferring large chunks without using any cpu power. And for completely new or completely changed files (think a gzip'd file) this actually means rsync is slower and transfers more data! So in the case where is says it's a fast incremental file transfer, it's still saying you're saving bandwidth (read that as time on a slow link and therefore is faster) at the expense of processing overhead. Just had to clear that misunderstanding up. Josh. P.S. If you're using rsync over ssh then make sure you use the blowfish cipher - it's much much faster than the default. I personally use rsyncd as it's on a trusted network and don't want the encryption overhead of ssh. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: 1.) Rsync checksums are not a feature of BackupPC. They are a feature of the rsync protocol. BackupPC compares files in the pool using the checksum to find if the exact same file is in the pool or not. This is not a feature of rsync. I think you are confusing 2 different things here. When backuppc transfers in a new file, it computes a hash of some portion of the file to use as the pool filename and a quick check to see if there is already a match in the pool. However, this is not done during the rsync comparison. Rsync just follows the links in the per-pc backup directory of the last full. Any name not matched from there is transferred in full, then matched or added to the pool with the filename hashing step. I just thought of one other thing that might make your transfers appear to be very slow. Do you happen to have any very large sparse files (dbm type) on the target machines? For example, a 64-bit linux machine that uses -1 as the nfsnobody uid will have a 1.2TB /var/log/lastlog containing only a few blocks of data. While backuppc will compress this to something small in the archive, rsync and tar don't handle the transfer very well and it is a good idea to exclude it. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Josh Marshall wrote: Just something that needs pointing out here, is that rsync is only a fast incremental file transfer program when transferring large files over a slow link. Or - in the incremental case - when you skip files that match in length and timestamp and there aren't many changes. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: 1.) Rsync checksums are not a feature of BackupPC. They are a feature of the rsync protocol. BackupPC compares files in the pool using the checksum to find if the exact same file is in the pool or not. This is not a feature of rsync. I think you are confusing 2 different things here. When backuppc transfers in a new file, it computes a hash of some portion of the file to use as the pool filename and a quick check to see if there is already a match in the pool. However, this is not done during the rsync comparison. Rsync just follows the links in the per-pc backup directory of the last full. Any name not matched from there is transferred in full, then matched or added to the pool with the filename hashing step. You are right, I was confusing them. I just thought of one other thing that might make your transfers appear to be very slow. Do you happen to have any very large sparse files (dbm type) on the target machines? For example, a 64-bit linux machine that uses -1 as the nfsnobody uid will have a 1.2TB /var/log/lastlog containing only a few blocks of data. While backuppc will compress this to something small in the archive, rsync and tar don't handle the transfer very well and it is a good idea to exclude it. I dont have very large files, perhaps largest is 200mb or so. But very few and they dont change often. Anyhow, thanks for all the help and patience. I learned a lot in the process. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Adam Goryachev wrote: I have given you all the information you asked for(didnt I?), even tried async option. Incremental backup of 1 machine still took about 300 minutes. The machine is working fine. I was using another backup program which was working way faster to backup the same machines. So I dont think that I have a hardware problem or such. I haven't read *every* email in this thread (it was getting quite repetitive seeing you complaining about how crap backuppc is), but I am saying that it is slow. I am not complaining that it is crap. I think when something is really slow, I should have right to say it right? anyway, could you please elaborate on the method that your 'other' backup solution used, ie, if your other solution used rsync, and you are using rsync with backuppc, then that might be helpful to know. If you used tar before, and now use rsync, that would obviously make a difference. I have already said it earlier, the other backup was taking backup over NFS. I also have machines were for example 2GB data with 100k files is synced to another using rsync, it takes 30-60 seconds to go through 100k files if there are not so many files to sync. I think overall, the attitude you have displayed is not conducive to resolving the problem. I also think it has been clearly shown that backuppc itself is not the problem, as other people are using it successfully, under similar conditions. What you need to do is basic diagnosis to find where the bottle neck is in your particular system. I disagree, I have given all the information requested from me. I have tried different things like mounting filesystems with async even though I know that it is not the problem just to make you guys happy. Now you say that my attitude is not helping the resolution? What makes you say so? Other people have been using it with dual-processor systems with 2-3GB of memory and raid setups. I would hardly consider this 'similar' conditions. BackupPC is very inefficent at handling files so you guys have to use very fast hardware solutions to speed it up. So you are covering up the problem from another side and say that there is no problem. I'd suggest you use whatever tools are available under your chosen OS (I think it is freebsd), to measure: 1) CPU utilisation on both the client being backed up, and the backuppc server I have already given this information the machines are almost idle. 2) Memory/swap utilisation on both client and server I have already given this information, the disk where the swap is idle while taking backups even though there is some data in swap (perhaps idle data) 3) Disk utilisation/bandwidth on both client and server I have sent this information also. I didnt send it on client actually but server is almost idle diskwise. The main disk load is on the server. 4) Network utilisation/bandwidth on both client and server The network links are idle. I can send this information if you dont believe me but there is maybe 200-300kbits/sec other usage while taking backups. Also, if you are using tar as your transfer method, then I'd suggest you try a 'manual' backup something like this (note, this command line is wrong, but you will need to adjust to get the right one). This is a good idea, I will try it later and return back to you. (even though I am using rsync) from the backuppc machine: cd /var/lib/backuppc;mkdir temp;cd temp;ssh client tar -cf - /backupdir | tar -xf - Try and copy the command line args from your backup log file Time how long that takes, and see how it compares to backuppc times if they are similar, then you know the issue is outside of backuppc. Of course, if you are using rsync, then do the same test with rsync data, but I'm sure it has already been said that you should use tar for your case with lots of small files. Just my 0.02c worth I will do the test with both actually, I will copy all files from the client to server using both rsync and tar and return back the results. By the way, if you have read all the mails, you would have realized that I have given all the information requested. The fact that backuppc saturates the backup drive while taking backups just makes me believe that the problem is there. But anyhow, I am open to try other things also. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: I am saying that it is slow. I am not complaining that it is crap. I think when something is really slow, I should have right to say it right? There is such a thing as tact. Many capable and friendly people have been patient with you, and you fail to show any form of respect. You're the one with the busted system; you should be nicer to the important people who are giving you their experience for free. I disagree, I have given all the information requested from me. I have tried different things like mounting filesystems with async even though I know that it is not the problem just to make you guys happy. Now you say that my attitude is not helping the resolution? What makes you say so? If you read the way you write, you're argumentative, combative, and rude. You have not been forthcoming in describing your situation, which wastes a lot of time and energy that is free to you, but not free to the rest of us trying to help you. In short, you're taking advantage of a friendly group of people, and show no gratitude whatsoever. In fact, you act as if you blame us for setting up the same software and being satisfied with its performance. It's very acceptable in my experience. Other people have been using it with dual-processor systems with 2-3GB of memory and raid setups. I would hardly consider this 'similar' conditions. BackupPC is very inefficent at handling files so you guys have to use very fast hardware solutions to speed it up. So you are covering up the problem from another side and say that there is no problem. I have told you several times that I get considerably better performance with a single slow 5400 rpm 2mb buffer IDE drive on a 450mhz P3. The hardware is *NOT* as big a factor as you make it out to be. 3) Disk utilisation/bandwidth on both client and server I have sent this information also. I didnt send it on client actually but server is almost idle diskwise. The main disk load is on the server. How do you know your clients aren't the bottleneck, then? What if they're set up in PIO mode? Or are swapping like mad? You need to diagnose the problem on both ends. 4) Network utilisation/bandwidth on both client and server The network links are idle. I can send this information if you dont believe me but there is maybe 200-300kbits/sec other usage while taking backups. Have you checked with ttcp that your network is configured properly, and you can get the speeds you expect? A misconfigured switch or multiple MACs associated with the same IP can do a lot of damage to performance through collisions. Same with improper cabling from the IDE controller to the drive. Same with badly set jumpers if both drives are on the same IDE port. There are many problems that could exist. You need to measure the device's raw performance, then measure the protocol on that device to /dev/nul, then over the network to server's /dev/nul, then to the server's hard drive. Then check out your Perl install and make sure you haven't got something going on there. There are a hundred things you could try. Don't wait for us to come up with them all. JH JH - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Jason Hughes wrote: Evren Yurtesen wrote: I am saying that it is slow. I am not complaining that it is crap. I think when something is really slow, I should have right to say it right? There is such a thing as tact. Many capable and friendly people have been patient with you, and you fail to show any form of respect. You're the one with the busted system; you should be nicer to the important people who are giving you their experience for free. Ok folks - tomorrow I'm going to run some benchmarks on my FreeBSD box I plan to run the following: ./bonnie++ -d . -s 3072 -n 10:10:10:10 -x 1 on the following filesystems FreeBSD 6.2 ufs2 1) an 80 Gb IDE drive 2) a mirror set from 2 300 gb IDE drives 3) a raid 10 1.5 tb made from 6 WD 500gb sata 300 drives on a 3ware 9500S-12 controller with battery backup I'll run the benchmark sync, soft updates and async This is the same benchmark run on the namesys page so it should give a reasonable comparison to RaiserFS. As a data point I have three backups running right now on the 9500S-12 raid 10 partition - I'm seeing about 300 disk operations a second - when I used to run this on one disk I would see about 100 disk ops a second and on an IDE raid 5 (6 disks on a hightpoint controller) I'd see about 150 ops a second. John - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/27/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Yes it is terrible. I get much better performance if I do the tar option with the same files. As a matter of fact I was using a smaller script for taking backups earlier. (which I still use on some servers) and transfer files over NFS. It works way faster, especially incremental backups take 5-10 minutes compared to 400 minutes with backuppc So have you tried using tar for backups using BackupPC? I've said it before, there is something wrong with your setup because on my server incrementals also only take 5-10 minutes as you'd expect. And my server isn't that different than yours disk-wise, just RAID1 instead of no raid, it's even the exact same disk. On 3/27/07, Evren Yurtesen [EMAIL PROTECTED] wrote: David Rees wrote: That is true, full backups take about 500-600 minutes and incrementals take 200-300minutes. Is that from all 4 clients being backed up? Are all similar in having ~150k files and ~2GB data to back up? Your transfer rates are at least 5x slower than his and 10x slower than what most people here are getting. There is something wrong specifically with your setup. I suspect that if your backups were going 5x faster you'd be happy and 10x faster you'd be very happy. I would be happy if backups were made with the same speed as other backup programs can do. What other backup programs are you comparing BackupPC to? I am using rsync to sync 2GB of files from one machine to another and the number of files is about 100k and the operation takes 30seconds to 1 minute in average. Is this using the same backup server and client in your earlier data example? Are you using the --checksum-seed option? That will significantly speed up (the 3rd and later) backups as well. No, it requires a patched rsync. You must have a very old version of rsync on your machine. --checksum-seed has been available for quite some time in the official rsync tree. It significantly improves rsync performance, I recommend that you installed a recent rsync if possible. You need rsync 2.6.3 (released 30 Sep 2004) or later. Are you also sure the backup server isn't swapping as well? Can you get us some good logs from vmstat or similar during a backup? I can tell that it is not swapping because the disk where the swaps are idle while taking backup. ad0 is the disk with the swap. If there was such problem then it would be reading like crazy from swap. I have given this information before. The information you provided before was just about impossible to read. Please provide additional data in a readable format. The machine is working fine. I was using another backup program which was working way faster to backup the same machines. So I dont think that I have a hardware problem or such. OK, so it is using the same machines? Earlier I sent the disk stats when the backup was working and I have pointed out that the disk with swap is not working at all while taking backup. We can easily conclude that swapping is not an issue. Can you provide stats during the entire backup? On all my production machines I make sure sysstat is running and then run `sar -A` the minute before midnight and pipe the output to email for analysis should the need arise. sar -A ? That provides detailed system statistics collected at specified intervals. Here's a sample from my backuppc server which has 3 disks while backups are being run. The backuppc partition uses 2 disks in RAID1 exactly the same as yours. The other disk is the system disk (also 7200rpm ATA). Hardware/OS: AthlonXP 2000+ 1GB RAM, Fedora Core 6. A max of 3 backups will run at a time on this machine. This same data can be generated using sar -b. 23:00:02 tps rtps wtps bread/s bwrtn/s 23:10:02 639.17219.11420.06 5440.01 8864.10 23:20:03 757.28215.81541.46 5422.98 11696.18 23:30:04 497.98211.21286.78 3367.85 5903.83 23:40:04 984.20322.85661.35 7436.95 14135.63 23:50:03 619.68391.74227.93 10355.83 5160.97 You can see how much higher this system's IO/s seems to be that yours from the only performance data you sent us. On 3/28/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Adam Goryachev wrote: anyway, could you please elaborate on the method that your 'other' backup solution used, ie, if your other solution used rsync, and you are using rsync with backuppc, then that might be helpful to know. If you used tar before, and now use rsync, that would obviously make a difference. I have already said it earlier, the other backup was taking backup over NFS. I also have machines were for example 2GB data with 100k files is synced to another using rsync, it takes 30-60 seconds to go through 100k files if there are not so many files to sync. Same machines or different machines? Please be very exact when you post information. I disagree, I have given all the information
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: 2MB/sec isn't bad when handling a lot of files (try unpacking a tar with hundreds of thousands of little files to see). The problem is that you Yes it is terrible. I get much better performance if I do the tar option with the same files. As a matter of fact I was using a smaller script for taking backups earlier. (which I still use on some servers) and transfer files over NFS. It works way faster, especially incremental backups take 5-10 minutes compared to 400 minutes with backuppc The big difference is that rsync loads the entire directory from the remote system into ram before starting - and the backuppc implementation does it in perl so the data structure is larger than you might expect. Then it compares what you have in your existing backup, again in perl code, and for fulls, uncompresses the files on the fly so it takes more cpu than you might expect. My guess about your system is that if it isn't actively swapping it is at least close and loading the directory takes all of the RAM that should be your disk buffers for efficient access. Then you won't be keeping inodes in cache and end up waiting for the seeks back for each of them. I don't think you've mentioned your processor speed either - even if it is waiting most of the time it may contribute to the slowness when it gets a chance to run. You probably shouldn't be running more than one backup concurrently. You could try backing up a target that contains only a few large files as a timing test to see how much loading a large directory listing affects the run. I agree, handling a lot of files might be slow but this depends on how you handle the files. But I was handling the same files before and it wasnt taking this long. With tar, you never need to read the existing files so you are doing much less work and you don't hold the remote directory in memory at all. But if tar works better on your system, why not use it? The only real drawback is that incremental runs won't catch old files in their new positions under renamed directories. The full runs will straighten this out. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: I am sorry but I have been patient enough with them also. I have done most of the suggestions about speeding it up and the tests etc. even though I knew that none of those suggestions will help and some were the most irrelevant things I have heard. Have you added RAM yet? Even better than speeding up the filesystem (which you seem not to be doing anyway)is letting the OS cache avoid actually using it for read operations. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/27/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Yes it is terrible. I get much better performance if I do the tar option with the same files. As a matter of fact I was using a smaller script for taking backups earlier. (which I still use on some servers) and transfer files over NFS. It works way faster, especially incremental backups take 5-10 minutes compared to 400 minutes with backuppc So have you tried using tar for backups using BackupPC? I've said it before, there is something wrong with your setup because on my server incrementals also only take 5-10 minutes as you'd expect. And my server isn't that different than yours disk-wise, just RAID1 instead of no raid, it's even the exact same disk. Are you using tar? I will give it a try soon if I cant solve the proble any other way. I will make a post about the results. On 3/27/07, Evren Yurtesen [EMAIL PROTECTED] wrote: David Rees wrote: That is true, full backups take about 500-600 minutes and incrementals take 200-300minutes. Is that from all 4 clients being backed up? Are all similar in having ~150k files and ~2GB data to back up? All 4 machines are quite similar. I have given very optimistic examples, the data is about 3-7gb and latest full backups have been taking about 900minutes in one machine. HostUser#Full Full Age (days) Full Size (GB) Speed (MB/s) clienthost 4 6.9 5.560.10 where backups Backup# TypeFilled Level Start Date Duration/mins 245 fullyes 0 3/21 20:00 913.9 6.9 246 incrno 1 3/22 20:00 280.4 5.9 Your transfer rates are at least 5x slower than his and 10x slower than what most people here are getting. There is something wrong specifically with your setup. I suspect that if your backups were going 5x faster you'd be happy and 10x faster you'd be very happy. I would be happy if backups were made with the same speed as other backup programs can do. What other backup programs are you comparing BackupPC to? I was running backup over nfs for 6 machines (I actually still run this few of the old machines) using afio to archive the files. Also feedback from some other people I know who were using http://www.r1soft.com/index.html I am using rsync to sync 2GB of files from one machine to another and the number of files is about 100k and the operation takes 30seconds to 1 minute in average. Is this using the same backup server and client in your earlier data example? No, it was another machine however I just ran rsync on the same machine and results are below.(the files were copied to the disk where backuppc is taking backups to) rsync -vaW -e 'ssh -c blowfish' --copy-dirlinks --keep-dirlinks --delete --exclude 'dev/*' --exclude 'proc/*' --exclude='usr/obj/*' Running 1st time to copy all files. sent 7938868 bytes received 6392356383 bytes 3283044.50 bytes/sec total size is 6365920146 speedup is 0.99 32m54.13s real 7m27.47s user 7m39.88s sys Eh I guess it is a good speed bump... Running 2nd time (I know this is not exactly same as incremental as there are very few changed files but it shows how long time it takes to compare files) sent 4432 bytes received 160088751 bytes 558789.47 bytes/sec total size is 6364731379 speedup is 39.76 5m3.21s real24.84s user 1m2.82s sys Are you using the --checksum-seed option? That will significantly speed up (the 3rd and later) backups as well. No, it requires a patched rsync. You must have a very old version of rsync on your machine. --checksum-seed has been available for quite some time in the official rsync tree. It significantly improves rsync performance, I recommend that you installed a recent rsync if possible. You need rsync 2.6.3 (released 30 Sep 2004) or later. rsync version 2.6.8 protocol version 29 In the conf file it says that you need a patched rsync, but now I see in manual that mine supports it also. Thanks for the heads up. I will try this and let you know how it goes. Are you also sure the backup server isn't swapping as well? Can you get us some good logs from vmstat or similar during a backup? I can do that when next backups come and return back to you. But I am sure that it is not swapping. As I told before the ad0 disk (where the swap is) is almost idle during backups. I can tell that it is not swapping because the disk where the swaps are idle while taking backup. ad0 is the disk with the swap. If there was such problem then it would be reading like crazy from swap. I have given this information before. The information you provided before was just about impossible to read. Please provide additional data in a readable format. I have sent this information earlier, Disks ad0 ad2 KB/t 4.00 25.50 tps 175 MB/s 0.00 1.87 % busy1
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: I am sorry but I have been patient enough with them also. I have done most of the suggestions about speeding it up and the tests etc. even though I knew that none of those suggestions will help and some were the most irrelevant things I have heard. Have you added RAM yet? Even better than speeding up the filesystem (which you seem not to be doing anyway)is letting the OS cache avoid actually using it for read operations. The machine is at a remote location. I have instructed the guys there to do that but I have to wait. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
I have run rsync manually to copy all files from a server where backups take 600 minutes. John Pettitt wrote: Jason Hughes wrote: Evren Yurtesen wrote: I am saying that it is slow. I am not complaining that it is crap. I think when something is really slow, I should have right to say it right? There is such a thing as tact. Many capable and friendly people have been patient with you, and you fail to show any form of respect. You're the one with the busted system; you should be nicer to the important people who are giving you their experience for free. I am sorry but I have been patient enough with them also. I have done most of the suggestions about speeding it up and the tests etc. even though I knew that none of those suggestions will help and some were the most irrelevant things I have heard. Ok folks - tomorrow I'm going to run some benchmarks on my FreeBSD box I plan to run the following: ./bonnie++ -d . -s 3072 -n 10:10:10:10 -x 1 Another good idea... I am trying to do this but it outputs the results in csv format? on the following filesystems FreeBSD 6.2 ufs2 1) an 80 Gb IDE drive 2) a mirror set from 2 300 gb IDE drives 3) a raid 10 1.5 tb made from 6 WD 500gb sata 300 drives on a 3ware 9500S-12 controller with battery backup I'll run the benchmark sync, soft updates and async This is the same benchmark run on the namesys page so it should give a reasonable comparison to RaiserFS. But you dont have exactly the same machine, unless you install linux and re-do the tests with reiserfs then results might be different By the way, I ran the benchmark on my supposedly flawed system which is causing the problem. As a data point I have three backups running right now on the 9500S-12 raid 10 partition - I'm seeing about 300 disk operations a second - when I used to run this on one disk I would see about 100 disk ops a second and on an IDE raid 5 (6 disks on a hightpoint controller) I'd see about 150 ops a second. That many operations would considerably slow down a non-raid system. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: 2MB/sec isn't bad when handling a lot of files (try unpacking a tar with hundreds of thousands of little files to see). The problem is that you Yes it is terrible. I get much better performance if I do the tar option with the same files. As a matter of fact I was using a smaller script for taking backups earlier. (which I still use on some servers) and transfer files over NFS. It works way faster, especially incremental backups take 5-10 minutes compared to 400 minutes with backuppc The big difference is that rsync loads the entire directory from the remote system into ram before starting - and the backuppc implementation does it in perl so the data structure is larger than you might expect. Then it compares what you have in your existing backup, again in perl code, and for fulls, uncompresses the files on the fly so it takes more cpu than you might expect. My guess about your system is that if it isn't actively swapping it is at least close and loading the directory takes all of the RAM that should be your disk buffers for efficient access. Then you won't be keeping inodes in cache and end up waiting for the seeks back for each of them. I don't think you've mentioned your processor speed either - even if it is waiting most of the time it may contribute to the slowness when it gets a chance to run. You probably shouldn't be running more than one backup concurrently. You could try backing up a target that contains only a few large files as a timing test to see how much loading a large directory listing affects the run. This makes a lot of sense. Anyhow, as I mentioned earlier. I will get the memory upgraded and I will see if it will solve the problem. Also I see the --ignore-times option in rsync args for full backups. Why is this necessary exactly? I agree, handling a lot of files might be slow but this depends on how you handle the files. But I was handling the same files before and it wasnt taking this long. With tar, you never need to read the existing files so you are doing much less work and you don't hold the remote directory in memory at all. But if tar works better on your system, why not use it? The only real drawback is that incremental runs won't catch old files in their new positions under renamed directories. The full runs will straighten this out. Is it still possible to pool the files for saving disk space when using tar? I wonder something, why cant backuppc set the backed up file modification times to match the client when backing up the files so it can compare the file modification time only at incrementals (and perhaps at full backups) like tar? In either case backuppc can do file by file checks when using rsync anyhow, but the limitations tar imposes would be removed, wouldnt it? Also it wouldnt need to decompress files only for checking if the one in client is modified or not. Is this impossible? Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: Also I see the --ignore-times option in rsync args for full backups. Why is this necessary exactly? If you don't, you are trusting that every file that has the same name, timestamp and length on the previous backup still matches the contents on the target. It probably, mostly, usually, does... If you posted a listing that showed the duration of fulls vs. incrementals, I think I missed it, but it would show the difference in the extra reading/comparison work on the server side. Even for incrementals you still have to wade through the metadata for each file to find the original length/timestamp/owner etc. which won't be the same as the compressed/pooled file itself because those may be different for different instances of the pooled contents. The other side effect of doing a full - and the one you really need - is that the complete directory structure is rebuilt as the base for subsequent incrementals. With tar, you never need to read the existing files so you are doing much less work and you don't hold the remote directory in memory at all. But if tar works better on your system, why not use it? The only real drawback is that incremental runs won't catch old files in their new positions under renamed directories. The full runs will straighten this out. Is it still possible to pool the files for saving disk space when using tar? Yes, there is no difference in the pool handling with any of the transfer methods. Tar and smb just have to transfer the file contents before it can be matched against the pool. If you have plenty of bandwidth this will happen faster than the rsync comparison. Also, tar and smb incrementals use only the timestamp on the target to determine what to back up so the server doesn't have to do extra work to compare. I wonder something, why cant backuppc set the backed up file modification times to match the client when backing up the files so it can compare the file modification time only at incrementals (and perhaps at full backups) like tar? In either case backuppc can do file by file checks when using rsync anyhow, but the limitations tar imposes would be removed, wouldnt it? Tar doesn't have a way to do live comparisons. GNUtar does have a way to detect renamed directories, but it requires saving a file on the target machine. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: Also I see the --ignore-times option in rsync args for full backups. Why is this necessary exactly? If you don't, you are trusting that every file that has the same name, timestamp and length on the previous backup still matches the contents on the target. It probably, mostly, usually, does... If you posted a listing that showed the duration of fulls vs. incrementals, I think I missed it, but it would show the difference in the extra reading/comparison work on the server side. Even for incrementals you still have to wade through the metadata for each file to find the original length/timestamp/owner etc. which won't be the same as the compressed/pooled file itself because those may be different for different instances of the pooled contents. Yes, this is causing a lot of extra processing on the server side. However most backup programs do not do this detailed file checking(I think?) and nobody seems to be complaining. There at least should be an option to only rely on the name/timestamp of a file when using rsync also. The other side effect of doing a full - and the one you really need - is that the complete directory structure is rebuilt as the base for subsequent incrementals. With tar, you never need to read the existing files so you are doing much less work and you don't hold the remote directory in memory at all. But if tar works better on your system, why not use it? The only real drawback is that incremental runs won't catch old files in their new positions under renamed directories. The full runs will straighten this out. Is it still possible to pool the files for saving disk space when using tar? Yes, there is no difference in the pool handling with any of the transfer methods. Tar and smb just have to transfer the file contents before it can be matched against the pool. If you have plenty of bandwidth this will happen faster than the rsync comparison. Also, tar and smb incrementals use only the timestamp on the target to determine what to back up so the server doesn't have to do extra work to compare. I wonder something, why cant backuppc set the backed up file modification times to match the client when backing up the files so it can compare the file modification time only at incrementals (and perhaps at full backups) like tar? In either case backuppc can do file by file checks when using rsync anyhow, but the limitations tar imposes would be removed, wouldnt it? Tar doesn't have a way to do live comparisons. GNUtar does have a way to detect renamed directories, but it requires saving a file on the target machine. What I meant was, if tar is used then live comparisons can not be done. Tar relies on the modification time. However if rsync is used then the file location and modification time can be checked. If a file is renamed or moved keeping the same modification time then this can be detected using rsync even without checksum checks (which would give the same checksum anyway). So checksum is saving bandwidth if a file modification time is different but the contents are same. In any case, somebody else said that when checksum caching is enabled then file checksums are not calculated at it backup session by uncompressing the files. If this is true? then I can live with it but CPU usage could be lowered further if this was an option which could be disabled and better performance could be achieved. Thanks for all the help and patience! :) Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Lets hope this doesnt wrap around... as you can see load is in 0.1-0.01 range. 1 usersLoad 0.12 0.05 0.01 Mar 27 07:30 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act 260203592 144912 6868 12384 count All 2497845456 232789611800 pages It wrapped pretty badly, but let me see if I'm interpreting this right (I'm no BSD expert, either): 1. Your server has ~250MB of memory. 2. Load average during backups is only 0.1-0.01? Does BSD calculate load average differently than Linux? Linux calculates load average by looking at the number of runnable tasks - this means if you have a single process waiting on disk IO you will have a load average of 1. If BSD calculates the load average the same way, then that means your server is not waiting on disk, but waiting for the clients. What's the load like on the clients you are backing up? -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Lets hope this doesnt wrap around... as you can see load is in 0.1-0.01 range. 1 usersLoad 0.12 0.05 0.01 Mar 27 07:30 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act 260203592 144912 6868 12384 count All 2497845456 232789611800 pages It wrapped pretty badly, but let me see if I'm interpreting this right (I'm no BSD expert, either): 1. Your server has ~250MB of memory. 2. Load average during backups is only 0.1-0.01? Does BSD calculate load average differently than Linux? Linux calculates load average by looking at the number of runnable tasks - this means if you have a single process waiting on disk IO you will have a load average of 1. If BSD calculates the load average the same way, then that means your server is not waiting on disk, but waiting for the clients. What's the load like on the clients you are backing up? Under FreeBSD if a task is blocked it doesn't contribute to load no matter what it's waiting for - so disk bound tasks don't add to load average. John - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
brien dieterle wrote: Jason Hughes wrote: Evren Yurtesen wrote: Jason Hughes wrote: That drive should be more than adequate. Mine is a 5400rpm 2mb buffer clunker. Works fine. Are you running anything else on the backup server, besides BackupPC? What OS? What filesystem? How many files total? FreeBSD, UFS2+softupdates, noatime. There are 4 hosts that have been backed up, for a total of: * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.45GB (prior to pooling and compression). # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 07:16), today's max is 10% (3/27 01:00) and yesterday's max was 10%. Host User #Full Full Age (days) Full Size (GB) Speed (MB/s) #Incr Incr Age (days) Last Backup (days) State Last attempt host1 4 5.4 3.88 0.22 6 0.4 0.4 idle idle host2 4 5.4 2.10 0.06 6 0.4 0.4 idle idle host3 4 5.4 7.57 0.14 6 0.4 0.4 idle idle host4 4 5.4 5.56 0.10 6 0.4 0.4 idle idle Hmm. This is a tiny backup setup, even smaller than mine. However, it appears that the average size of your file is only 22KB, which is quite small. For comparison sake, this is from my own server: Pool is 172.91GB comprising 217311 files and 4369 directories (as of 3/26 01:08), The fact that you have tons of little files will probably give significantly higher overhead when doing file-oriented work, simply because the inodes must be fetched for each file before seeking to the file itself. If we assume no files are shared between hosts (very conservative), and you have an 8ms access time, you will have 190132 files per host and two seeks per file, neglecting actual i/o time, gives you 50 minutes. Just to seek them all. If you have a high degree of sharing, it can be up to 4x worse. Realize, the same number of seeks must be made on the server as well as the client. Are you sure you need to be backing up everything that you're putting across the network? Maybe excluding some useless directories, maybe temp files or logs that haven't been cleaned up? Perhaps you can archive big chunks of it with a cron job? I'd start looking for ways to cut down the number of files, because the overhead of per-file accesses are probably eating you alive. I'm also no expert on UFS2 or FreeBSD, so it may be worthwhile to research its behavior with hard links and small files. JH For what it's worth, I have a server that backs up 8.6 million files averaging 10k in size from one host. It takes a full 10 hours for a full backup via tar over NFS ( 2.40MB/s for 87GB). CPU usage is low, around 10-20%, however IOwait is a pretty steady 25%. Server info: HP DL380 G4 debian sarge dual processor 3.2ghz xeon 2GB Ram 5x10k rpm scsi disks, raid5 128MB battery backed cache (50/50 r/w) ext3 filesystems brien You are distributing the reads and writes on 5 disks here. Dont you think that 2.40mb/s is a little bit slow compared to the horsepower you have? If you scale it down to my system, I think 1/10 performance is normal... Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Craig Barratt wrote:
Evren writes:
HostUser#Full Full Age (days) Full Size (GB)
Speed
(MB/s)#Incr Incr Age (days) Last Backup (days)
State
Last attempt
host14 5.4 3.880.226 0.4
0.4 idleidle
host24 5.4 2.100.066 0.4
0.4 idleidle
host34 5.4 7.570.146 0.4
0.4 idleidle
host44 5.4 5.560.106 0.4
0.4 idleidle
These xfer rates are certainly very low.
I don't think I caught which XferMethod you are using. Also,
what is your BackupPC version? If you are using 3.0, what is
the value of $Conf{IncrLevels}?
Craig
I use rsync with 3.0.0 but it was the same speed with 2.x.x versions.
$Conf{IncrLevels} = [1];
I wonder, since I have:
$Conf{IncrKeepCnt} = 6;
Wouldnt it make more sense to use this?:
$Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
or does this make things even more slower?
I have concluded that to get a decent performance from backuppc one must
use RAID and put a lot of ram to the machine because BackupPC is using
the disk intensively!
Thanks,
Evren
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: Server info: HP DL380 G4 debian sarge dual processor 3.2ghz xeon 2GB Ram 5x10k rpm scsi disks, raid5 128MB battery backed cache (50/50 r/w) ext3 filesystems brien You are distributing the reads and writes on 5 disks here. Dont you think that 2.40mb/s is a little bit slow compared to the horsepower you have? Raid5 doesn't distribute disk activity - it puts the drives in lockstep and is slower than a single drive, especially on small writes where it has to do extra reads to re-compute parity on the existing data. If you scale it down to my system, I think 1/10 performance is normal... Its the RAM and disk cache that makes this system work. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: That doesn't sound difficult at all. I suspect your real problem is that you are running a *bsd UFS filesystem with it's default sync metadata handling which is going to wait for the physical disk action to complete on every directory operation. I think there are other options but I haven't kept up with them. I gave up on UFS long ago when I needed to make an application that frequently truncated and rewrote a data file work on a machine that crashed frequently. The sync-metadata 'feature' statistically ensured that there was never any data in the file after recovering since the truncation was always forced to disk immediately but the data write was buffered so with a fast cycle the on-disk copy was nearly always empty. Is anyone else running a *bsd? The problem is the reads, not writes. It takes long time for BackupPC to figure out if the file should be backed up or not. Backups take very long time even when there are few files which are backed up. But I have to disagree, again :) When you make writes and filesystem is mounted with 'sync' option which is default on FreeBSD then all operations are done sync. Your program must have been doing something between truncate and the actual write operation which caused file to be empty for a period of time. From FreeBSD mount manual - syncAll I/O to the file system should be done synchronously. - async All I/O to the file system should be done asynchronously. This is a dangerous flag to set, and should not be used unless you are prepared to recreate the file system should your system crash. This 'feature' as you call it, exists in linux also From Linux mount manual - sync All I/O to the file system should be done synchronously - async All I/O to the file system should be done asynchronously. The only difference is that FreeBSD uses sync by default because they claim(there is no point to argue if it is true or not, there are enough threads about those if you google :) ) that it is safer while Linux uses async. You could easily enable 'async' IO on FreeBSD to have the same behaviour. Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. I cant see how you can reach to this conclusion. Observation... I run hundreds of servers, many of which are 5 or more years old. The disk failures have had no correlation to the server activity. Hard drive manufacturers disagree with you. They do consider the duty-cycles when calculating the MTBF values. http://www.digit-life.com/articles/storagereliability/ Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: Lets hope this doesnt wrap around... as you can see load is in 0.1-0.01 range. 1 usersLoad 0.12 0.05 0.01 Mar 27 07:30 Mem:KBREALVIRTUAL VN PAGER SWAP PAGER Tot Share TotShareFree in out in out Act 260203592 144912 6868 12384 count All 2497845456 232789611800 pages It wrapped pretty badly, but let me see if I'm interpreting this right (I'm no BSD expert, either): 1. Your server has ~250MB of memory. 2. Load average during backups is only 0.1-0.01? Does BSD calculate load average differently than Linux? Linux calculates load average by looking at the number of runnable tasks - this means if you have a single process waiting on disk IO you will have a load average of 1. If BSD calculates the load average the same way, then that means your server is not waiting on disk, but waiting for the clients. What's the load like on the clients you are backing up? -Dave 2- The process is idle when it is waiting data from the disk (not in active state) So FreeBSD doesnt count it towards the load. I dont see why it should actually? Are processes which are waiting data from disk use CPU in Linux? 1- You are right, I will increase the memory a little bit. I will return back the results when I accomplish that. :) Lets hope it will decrease disk reads. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: Server info: HP DL380 G4 debian sarge dual processor 3.2ghz xeon 2GB Ram 5x10k rpm scsi disks, raid5 128MB battery backed cache (50/50 r/w) ext3 filesystems brien You are distributing the reads and writes on 5 disks here. Dont you think that 2.40mb/s is a little bit slow compared to the horsepower you have? Raid5 doesn't distribute disk activity - it puts the drives in lockstep and is slower than a single drive, especially on small writes where it has to do extra reads to re-compute parity on the existing data. I am confused, when a write is done the data is distributed in the disks depending on the stripe size you are using. When you start reading the file, you are reading from 5 different disks. So you get way better performance for sure on reads. Agreed, writes might be slower depending on the raid controller, amount of cache it has etc. but my problem here is the reads. In my machine BackupPC server is working slow even when it is backing up a very small amount of files. If you scale it down to my system, I think 1/10 performance is normal... Its the RAM and disk cache that makes this system work. Agreed, I will add more RAM and report the results. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: The problem is the reads, not writes. It takes long time for BackupPC to figure out if the file should be backed up or not. Backups take very long time even when there are few files which are backed up. That's only true with rsync and especially rsync fulls where the contents are compared as well as the timestamps. But I thought your complaint was about the hardlinks and data storage. The transfer speed value doesn't make a lot of sense in the rsync case anyway because you often spend a lot of time computing to avoid having to transfer anything - and you have the entire directory in RAM while doing it which may reduce your disk buffering. What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? But I have to disagree, again :) When you make writes and filesystem is mounted with 'sync' option which is default on FreeBSD then all operations are done sync. Your program must have been doing something between truncate and the actual write operation which caused file to be empty for a period of time. Yes, it was writing a small amount to the file and the data write was always deferred with this kind of problem: http://www.complang.tuwien.ac.at/anton/sync-metadata-updates.html The only difference is that FreeBSD uses sync by default because they claim(there is no point to argue if it is true or not, there are enough threads about those if you google :) ) that it is safer while Linux uses async. It is safer for the file system if you are concerned that fsck can't fix it. It is not safer for the data of any particular file. Anyway crashes are pretty much a thing of the past and I protect against most kinds of failures by raid-mirroring to an external drive that is rotated off-site weekly. I'm running a journaled reiserfs because it is very fast at creating and deleting files, but ext3 with the right options should also work. Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. I cant see how you can reach to this conclusion. Observation... I run hundreds of servers, many of which are 5 or more years old. The disk failures have had no correlation to the server activity. Hard drive manufacturers disagree with you. They do consider the duty-cycles when calculating the MTBF values. http://www.digit-life.com/articles/storagereliability/ Vendor MTBF values are very unrealistic. The trade rags seem to have caught on to this recently: http://www.eweek.com/article2/0%2C1895%2C2105551%2C00.asp If you want to know what the vendor really expects, look at the warranty instead. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On Mon, 26 Mar 2007, David Rees wrote: No kidding! My backuppc stats are like this: For my biggest server, my stats are: There are 36 hosts that have been backed up, for a total of: 503 full backups of total size 4200.30GB (prior to pooling and compression), 247 incr backups of total size 444.52GB (prior to pooling and compression). Pool is 776.90GB comprising 1900158 files and 4369 directories Dual Opteron 265(4x1.8ghz cores), 1 GB ram, 3ware 9550SX-16ML w/15 Maxtor MaxLine III 300 (7200 rpm, ATA, 8 meg cache), ext3 w/noatime. I'd change it to XFS, but backing up 1.7TB would take weeks/months because of the hardlinks. Runs BackupPC 2.1.2-3 on Debian Sarge. Backups take about 5-6 hours depending on incremental/full counts, I run 4 at a time.. Most of the workstations are on 100mbit, it's backing up about 330 gig total of files. And yeah, I love BackupPC too :) Mike - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: The problem is the reads, not writes. It takes long time for BackupPC to figure out if the file should be backed up or not. Backups take very long time even when there are few files which are backed up. That's only true with rsync and especially rsync fulls where the contents are compared as well as the timestamps. But I thought your complaint was about the hardlinks and data storage. The transfer speed value doesn't make a lot of sense in the rsync case anyway because you often spend a lot of time computing to avoid having to transfer anything - and you have the entire directory in RAM while doing it which may reduce your disk buffering. What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. But I have to disagree, again :) When you make writes and filesystem is mounted with 'sync' option which is default on FreeBSD then all operations are done sync. Your program must have been doing something between truncate and the actual write operation which caused file to be empty for a period of time. Yes, it was writing a small amount to the file and the data write was always deferred with this kind of problem: http://www.complang.tuwien.ac.at/anton/sync-metadata-updates.html You shouldnt be trusting this article. The article is flawed in certain ways. For example it says as demonstrated by soft updates (see below), which don't even require fsck upon crashing. but this information is wrong. Softupdates means that you can run the fsck in background but you must run it on a crash (system does it automatically anyhow). Also, about the problem in this article, if async updates would have been used then the file would have been lost anyhow as the writes would have been buffered. The information given is vague. I am not an expert on this but the problem about async updates that the BSD guys are scared of is probably not the same kind of inconsistency described here. In either case you could just enable sync or async updates on UFS to change this behaviour. You didnt have to change the filesystem because of a deficiency in UFS which is causing the problem. The only difference is that FreeBSD uses sync by default because they claim(there is no point to argue if it is true or not, there are enough threads about those if you google :) ) that it is safer while Linux uses async. It is safer for the file system if you are concerned that fsck can't fix it. It is not safer for the data of any particular file. Anyway crashes are pretty much a thing of the past and I protect against most kinds of failures by raid-mirroring to an external drive that is rotated off-site weekly. I'm running a journaled reiserfs because it is very fast at creating and deleting files, but ext3 with the right options should also work. If you check namesys.com benchmarks, you will see that they only tested reiserfs against ext2/ext3/xfs/jfs and conveniently forgot to test against ufs2. You can see in the end of the page that slight performance increase in reiserfs is also bringing twice the cpu usage! (plus extX is faster in certain operations even) http://www.namesys.com/benchmarks.html Another test results from sun, little bit old (from 2004) http://www.sun.com/software/whitepapers/solaris10/fs_performance.pdf I am not saying one is faster over another or anything like that. Sun's results do look too good to be true anyhow, they probably tweaked some stuff or I dont know, can they lie? wouldnt it be bad reputation? Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. I cant see how you can reach to this conclusion. Observation... I run hundreds of servers, many of which are 5 or more years old. The disk failures have had no correlation to the server activity. Hard drive manufacturers disagree with you. They do consider the duty-cycles when calculating the MTBF values. http://www.digit-life.com/articles/storagereliability/ Vendor MTBF values are very unrealistic. The trade rags seem to have caught on to this recently: http://www.eweek.com/article2/0%2C1895%2C2105551%2C00.asp If you want to know what the vendor really expects, look at the warranty instead. It is a logical fact that more movement = more wear. Thanks, Evren
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. From the perspective of the backup directories, it doesn't matter whether or not there are additional links to a file. It just looks at the directory entry to find the file it wants. It may matter that the inodes and file contents end up splattered all over the place because they were written at different times, though. But I have to disagree, again :) When you make writes and filesystem is mounted with 'sync' option which is default on FreeBSD then all operations are done sync. Your program must have been doing something between truncate and the actual write operation which caused file to be empty for a period of time. Yes, it was writing a small amount to the file and the data write was always deferred with this kind of problem: http://www.complang.tuwien.ac.at/anton/sync-metadata-updates.html You shouldnt be trusting this article. The article is flawed in certain ways. The fact is that 90+% of the time, my data files were empty after a crash while they always appeared to have data when running. That tells me that the truncate happened on disk immediately while the data write happened when a buffer was flushed which was consistent with the way the bsd sync-metadata was described. For example it says as demonstrated by soft updates (see below), which don't even require fsck upon crashing. but this information is wrong. Softupdates means that you can run the fsck in background but you must run it on a crash (system does it automatically anyhow). This was before softupdates was an option. I don't think reiserfs was around then either. Also, about the problem in this article, if async updates would have been used then the file would have been lost anyhow as the writes would have been buffered. But then you'd have about a 99% chance of still having the previous contents which would have been much more useful than an empty file. In either case you could just enable sync or async updates on UFS to change this behaviour. You didnt have to change the filesystem because of a deficiency in UFS which is causing the problem. I just got tired of the freebsd people telling me that their way was better and it was correct behavior to lose my data and moved on. The only difference is that FreeBSD uses sync by default because they claim(there is no point to argue if it is true or not, there are enough threads about those if you google :) ) that it is safer while Linux uses async. It is safer for the file system if you are concerned that fsck can't fix it. It is not safer for the data of any particular file. Anyway crashes are pretty much a thing of the past and I protect against most kinds of failures by raid-mirroring to an external drive that is rotated off-site weekly. I'm running a journaled reiserfs because it is very fast at creating and deleting files, but ext3 with the right options should also work. If you check namesys.com benchmarks, you will see that they only tested reiserfs against ext2/ext3/xfs/jfs and conveniently forgot to test against ufs2. You can see in the end of the page that slight performance increase in reiserfs is also bringing twice the cpu usage! (plus extX is faster in certain operations even) http://www.namesys.com/benchmarks.html There's a benchmark called 'postmark' that is the best I've seen for showing how well creating/deleting a lot of files works. A few years ago when I changed last, reiserfs was the best. It seems to have disappeared from it's original location from NetApp, but the last time I mentioned it someone said it was packaged for Debian so the source is still available. I'll dig it up again the next time I am interested in changing (maybe when zfs looks mature). Perhaps, but there is a difference if they are moving 10 times or 10 times. It is a logical fact that more movement = more wear. Yes, but more wear doesn't matter until you exceed the design limits. Worry about your air conditioner instead - it has more effect on the drive life. I've had as much trouble with drives that have been powered off for much of their lives as ones that stay busy. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net
Re: [BackupPC-users] very slow backup speed
On 03/27 11:48 , Les Mikesell wrote: Worry about your air conditioner instead - it has more effect on the drive life. I've had as much trouble with drives that have been powered off for much of their lives as ones that stay busy. as an interesting data point, I have a 512MB drive that ran my firewall for several years, and before that was in a desktop machine at a government office. The date on it is 1993; and it still works just fine, despite what I'm guessing is fairly regular and continuous use since that time. I would really like to see hard drives made to be more reliable, rather than just bigger. (Where the mfg. actually touts the reliability features more than the size). Maxtor sort of tried that a while ago with their single-platter 15GB drives; but AFAIK there wasn't anything special about those other than fewer platters and heads (no better bearings, gas-absorption material, etc). We've long since passed the point where big drives are necessary (at least for OS partitions on Linux boxen). I want reliable storage. to that end, this is kind of an attractive product: http://www.elandigitalsystems.com/adapter/p312.php we've had good luck with these units: http://www.mach5products.com/PRODCF/CFproducts.htm My ~1TB BackupPC server is using less than 1GB on its OS drive (the backuppc data is all on a separate drive array, for ease of management). While consumer-grade CF cards are pretty crappy things compared to the military-grade solid-state IDE or SCSI disks that places like M-Systems sells; they are cheap enough and big enough to compete with spinny storage when the space requirements take second place to the reliability of no moving parts. (I've seen CF cards die... I'm not sure that a spinny drive wouldn't have died in the same conditions tho). -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/27/07, Les Mikesell [EMAIL PROTECTED] wrote: Evren Yurtesen wrote: What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. From the perspective of the backup directories, it doesn't matter whether or not there are additional links to a file. It just looks at the directory entry to find the file it wants. It may matter that the inodes and file contents end up splattered all over the place because they were written at different times, though. Yep, Lee is right here. Unless BSD handles hard-links in some crazy manner. If you check namesys.com benchmarks, you will see that they only tested reiserfs against ext2/ext3/xfs/jfs and conveniently forgot to test against ufs2. You can see in the end of the page that slight performance increase in reiserfs is also bringing twice the cpu usage! (plus extX is faster in certain operations even) http://www.namesys.com/benchmarks.html When your overall speed is limited by the speed of your disks and your CPU spends all it's time twiddling it's thumbs waiting for disk, who cares if CPU doubles and still spends 90% of it's time waiting as long as it gets the job done faster? To summarize Evran's setup: FreeBSD, 250MB ram, CPU unknown, 1 7200rpm Seagate ATA disk Filesystem: ufs2, sync, noatime Pool is 17.08GB comprising 760528 files (avg file size ~22KB) BackupPC reports backup speeds between 0.06 - 0.22 MB/s Total backup time per host: Unknown CPU is 99% idle during backups Disk shows ~75 IO/s during load and low transfer rate Says even small backup w/small number of files is slow. Can you try mounting the backup partition async so we can see if it really is read performance or write performance that is killing backup performance? I would also highly recommend that you limit backups to 1 concurrent backup at a time. I must wonder if ufs2 is really bad at storing inodes on disk... BTW, how does BackupPC calculate speed? I think it calculates backup speed by reporting files transferred over time, so if you don't have many files that change, won't BackupPC report a very low backup speed. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren writes:
I use rsync with 3.0.0 but it was the same speed with 2.x.x versions.
$Conf{IncrLevels} = [1];
I wonder, since I have:
$Conf{IncrKeepCnt} = 6;
Wouldnt it make more sense to use this?:
$Conf{IncrLevels} = [1, 2, 3, 4, 5, 6];
or does this make things even more slower?
Yes, that would make it slower since several incrementals have
to be merged (ie: more server disk seeks for every file) to get
the latest baseline, which in 3.x is used as the starting point
for each rsync backup.
Craig
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/backuppc-users
http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/27/07, David Rees [EMAIL PROTECTED] wrote: Can you try mounting the backup partition async so we can see if it really is read performance or write performance that is killing backup performance? I must wonder if ufs2 is really bad at storing inodes on disk... I went and did some research on ufs filesystem performance and found this paper: http://www.bsdcan.org/2006/papers/FilesystemPerformance.pdf There appears to be 4 different mount options related to data integrity: sync: slowest, all writes synchronous noasync: (default?) data asynchronous, metadata synchronous soft updates: dependency ordering of writes to ensure on-disk consistency async: fastest, all writes asynchronous noasync seems to be the default. Evran, can you confirm that your filesystem is mounted this way? On Linux using ext3, the default mount option is data=ordered which should be similar to soft updates in terms of performance. If you can mount your backup partition in soft updates mode that should perform best, much better than the default noasync mode for the type of writes BackupPC does. I wouldn't recommend mounting a partition async permanently because of the risk to file system integrity. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Carl Wilhelm Soderstrom wrote: I would really like to see hard drives made to be more reliable, rather than just bigger. I'm not sure that can be improved enough to matter. A failure of an inexpensive part once in five years isn't a big deal other than the side effects it might cause, and unless they can be 100% reliable (probably impossible) you have to be prepared for those side effects anyway. I want reliable storage. Run mirrored hot-swap drives. If one dies, replace it at some convenient time, sync it up and keep going. I have one machine last booted in mid-2003 that has had a couple of it's drives replaced that way. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 03/27 01:40 , Les Mikesell wrote: Carl Wilhelm Soderstrom wrote: I would really like to see hard drives made to be more reliable, rather than just bigger. I'm not sure that can be improved enough to matter. A failure of an inexpensive part once in five years isn't a big deal other than the side effects it might cause, and unless they can be 100% reliable (probably impossible) you have to be prepared for those side effects anyway. on a small scale, perhaps. when you get to an organization the size of (TBS|Hotmail|Yahoo)tho; it may make sense to spend a little more money in order to spend less labor swapping parts. the market seems to agree with you tho... which is a pretty good indictment of central planning notions (i.e. Me) vs. what really is efficient. I want reliable storage. Run mirrored hot-swap drives. If one dies, replace it at some convenient time, sync it up and keep going. I have one machine last booted in mid-2003 that has had a couple of it's drives replaced that way. Your point is well-taken. I do keep in mind tho that I've seen multiple drives fail simultaneously or in rapid succession, and that the process of replacing drives costs time and effort. It is not as trivial as you might think, once you factor in the time to detect the failure, find the replacement (if you can), replace the drive (which may involve removing the old drive from the carrier and replacing it with the new one), and make sure it's working properly. In an ideal world it's 5 minutes of work, in the real world I usually expect to lose an hour or two dealing with a failed drive. This can be accounted as several hundred dollars of lost productivity in many cases; so it's worthwhile to spend more money on a better-quality drive sometimes. -- Carl Soderstrom Systems Administrator Real-Time Enterprises www.real-time.com - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Carl Wilhelm Soderstrom wrote: I would really like to see hard drives made to be more reliable, rather than just bigger. I'm not sure that can be improved enough to matter. A failure of an inexpensive part once in five years isn't a big deal other than the side effects it might cause, and unless they can be 100% reliable (probably impossible) you have to be prepared for those side effects anyway. on a small scale, perhaps. when you get to an organization the size of (TBS|Hotmail|Yahoo)tho; it may make sense to spend a little more money in order to spend less labor swapping parts. I think before you get to that that scale you'd be network booting diskless and mostly disposable motherboards. You still have to deal with the drives somewhere, but using larger ones instead of large numbers of them. the market seems to agree with you tho... which is a pretty good indictment of central planning notions (i.e. Me) vs. what really is efficient. I find complete-system redundancy with commodity boxes to be cheaper and more efficient than using military strength parts that cost 10x as much and fail half as often. In our operations the most common failure is in software anyway since we try to push out new capabilities as fast as possible so I like lots of redundancy and the ability to run multiple versions of things concurrently. I could see where that would be different for operations where everything had to be in a single database, though. I want reliable storage. Run mirrored hot-swap drives. If one dies, replace it at some convenient time, sync it up and keep going. I have one machine last booted in mid-2003 that has had a couple of it's drives replaced that way. Your point is well-taken. I do keep in mind tho that I've seen multiple drives fail simultaneously or in rapid succession, and that the process of replacing drives costs time and effort. Yes, your building might catch fire or be flooded too. Those are probably more likely than multiple cheap drives failing simultaneously from some cause that wouldn't also kill better drives. You need offsite backups to cover these unlikely events anyway. It is not as trivial as you might think, once you factor in the time to detect the failure, find the replacement (if you can), replace the drive (which may involve removing the old drive from the carrier and replacing it with the new one), and make sure it's working properly. In an ideal world it's 5 minutes of work, in the real world I usually expect to lose an hour or two dealing with a failed drive. The real killer is the time planning out the replacement operation so the only time wasted is the one person who does the work. In the whole-system scenario where the systems are load balanced anyway, yanking a machine out isn't anything special and you don't even need the hot-swap drive case. This can be accounted as several hundred dollars of lost productivity in many cases; so it's worthwhile to spend more money on a better-quality drive sometimes. In practice I don't see any relationship between price and reliability. I'm inclined to think that they all come off the same production line these days. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/27/07, Les Mikesell [EMAIL PROTECTED] wrote: Evren Yurtesen wrote: What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. I sent the status of the backups earlier today to mailing list? # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 22:40), today's max is 10% (3/27 01:00) and yesterday's max was 10%. * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.60GB (prior to pooling and compression). This is from 1 machine. Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 112 full149290 1957.8 0.10149251 1937.9 460120.7 224 full151701 2022.8 0.09151655 2004.7 100 18.1 238 full152170 2099.5 0.06152131 2081.6 115 17.9 244 incr214 48.90.00165 22.978 26.0 245 full152228 2095.2 0.06152177 2076.9 108 18.3 246 incr118 17.30.0076 0.2 69 17.1 247 incr159 21.40.00111 3.1 75 18.4 248 incr181 22.10.00132 2.5 79 19.7 249 incr186 24.00.00146 7.6 54 16.4 250 incr206 25.50.00159 6.7 70 18.8 From the perspective of the backup directories, it doesn't matter whether or not there are additional links to a file. It just looks at the directory entry to find the file it wants. It may matter that the inodes and file contents end up splattered all over the place because they were written at different times, though. Yep, Lee is right here. Unless BSD handles hard-links in some crazy manner. I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. For example this guy is using Linux (problem is OS independent) http://forum.psoft.net/showpost.php?p=107808postcount=16 If you check namesys.com benchmarks, you will see that they only tested reiserfs against ext2/ext3/xfs/jfs and conveniently forgot to test against ufs2. You can see in the end of the page that slight performance increase in reiserfs is also bringing twice the cpu usage! (plus extX is faster in certain operations even) http://www.namesys.com/benchmarks.html When your overall speed is limited by the speed of your disks and your CPU spends all it's time twiddling it's thumbs waiting for disk, who cares if CPU doubles and still spends 90% of it's time waiting as long as it gets the job done faster? Whatever, this is not the problem here.The fact is that, according to reiserfs developers reiserfs is more or less the same speed with ext2. I dont think the problem is related to any filesystem as it occurs on both Linux and FreeBSD To summarize Evran's setup: FreeBSD, 250MB ram, CPU unknown, 1 7200rpm Seagate ATA disk Filesystem: ufs2, sync, noatime Pool is 17.08GB comprising 760528 files (avg file size ~22KB) BackupPC reports backup speeds between 0.06 - 0.22 MB/s Total backup time per host: Unknown CPU is 99% idle during backups Disk shows ~75 IO/s during load and low transfer rate Says even small backup w/small number of files is slow. Can you try mounting the backup partition async so we can see if it really is read performance or write performance that is killing backup performance? I would also highly recommend that you limit backups to 1 concurrent backup at a time. I must wonder if ufs2 is really bad at storing inodes on disk... On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. BTW, how does BackupPC calculate speed? I think it calculates backup speed by reporting files transferred over time, so if you don't have many files that change, won't BackupPC report a very low backup speed. This is like the 'Contact'
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 245 full152228 2095.2 0.06152177 2076.9 108 18.3 246 incr118 17.30.0076 0.2 69 17.1 Can you post the duration that these backups took? All that these stats tell us is how much of your data is churning, and how big your average file size is. I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. For example this guy is using Linux (problem is OS independent) http://forum.psoft.net/showpost.php?p=107808postcount=16 Er, RAID-5 is slower than a single disk except on sustained reads, which is not the typical case with BackupPC. I only use a single disk and get between 0.5mb/s and 10mb/s, depending on the client and its file size/file count distribution. The question is whether other people have gotten the system to work correctly under the same or worse conditions, and already that has been answered in the affirmative. Be prepared to accept that the software isn't broken just because you haven't gotten it working to your satisfaction in your specific situation... It could be rsync, UFS, BSD, or any number of other factors that are causing you grief. Whatever, this is not the problem here.The fact is that, according to reiserfs developers reiserfs is more or less the same speed with ext2. I dont think the problem is related to any filesystem as it occurs on both Linux and FreeBSD Your argument lacks logic. If a filesystem can be configured to be slow on multiple OS's, does that mean BackupPC is failing to do its job? No, it means multiple people have managed to set up a system that performs badly using it. That's not so uncommon. BackupPC does not exist in a vacuum: its performance is sensitive to the environment of the server and its clients. Many people are using it without issue, right here on this very list. The question you should be asking is, what makes your system perform badly? Start by picking out things that you do differently, or that definitely affect performance. Transport protocol (rsync, tar). UFS. BSD. Sync on your filesystem. Start changing those one at a time and measure the performance. On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. If you want people to help you, it's probably best if you refrain from blaming the program until you have proof. So far, you have argued with people who provide evidence that the system works fine. It puts people on the defensive, and you may find people less willing to help you in the future. We do know BackupPC behaves well on other file systems and operating systems... maybe UFS or BSD is doing something poorly--how would you know they aren't? We have less data points to draw from there. I suspect it has a lot more to do with what the MB/s stats really mean. Maybe Craig can give a precise definition? This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. With rsync in incremental backups, it still has to check the metadata for each file to determine the changed file set. With millions of files, it will take a while. If your host or server is low on memory for any reason, this may bog it down and start vm swapping. I would recommend trying out tar to see if the protocol behavior matters. Try different mount options that are for higher performance. Others have made similar suggestions as well. Good luck, JH - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: David Rees wrote: On 3/27/07, Les Mikesell [EMAIL PROTECTED] wrote: Evren Yurtesen wrote: What is wall clock time for a run and is it reasonable for having to read through both the client and server copies? I am using rsync but the problem is that it still has to go through a lot of hard links to figure out if files should be backed up or not. Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. I sent the status of the backups earlier today to mailing list? # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 22:40), today's max is 10% (3/27 01:00) and yesterday's max was 10%. * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.60GB (prior to pooling and compression). This is from 1 machine. Totals Existing Files New Files Backup# Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB 112 full 149290 1957.8 0.10 149251 1937.9 4601 20.7 224 full 151701 2022.8 0.09 151655 2004.7 100 18.1 238 full 152170 2099.5 0.06 152131 2081.6 115 17.9 244 incr 214 48.9 0.00 165 22.9 78 26.0 245 full 152228 2095.2 0.06 152177 2076.9 108 18.3 246 incr 118 17.3 0.00 76 0.2 69 17.1 247 incr 159 21.4 0.00 111 3.1 75 18.4 248 incr 181 22.1 0.00 132 2.5 79 19.7 249 incr 186 24.0 0.00 146 7.6 54 16.4 250 incr 206 25.5 0.00 159 6.7 70 18.8 From the perspective of the backup directories, it doesn't matter whether or not there are additional links to a file. It just looks at the directory entry to find the file it wants. It may matter that the inodes and file contents end up splattered all over the place because they were written at different times, though. Yep, Lee is right here. Unless BSD handles hard-links in some crazy manner. I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. Well, I'd like to point out that I am using tar over nfs over a bonded 100mbit(x2) link. NFS is terrible for small file access. However, we have an xserve that I use xtar with, over ssh-- I get 5MB/s for that (which is the best I've seen). My other rsync clients get up to 4.2 MB/sec. So, I think the 25% [EMAIL PROTECTED]/s indicates I should never expect more than 10MB/s. Considering everything backuppc has to do with each file as it receives them, without writing them to disk first, I don't think we can expect it to be very fast. However, it is absurdly efficient with space. You can't have everything :-). Complexity, speed, efficiency; pick any two? brien - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: # Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB 245 full 152228 2095.2 0.06 152177 2076.910818.3 On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. 2MB/sec isn't bad when handling a lot of files (try unpacking a tar with hundreds of thousands of little files to see). The problem is that you are getting .06 on a drive that is capable of running a couple of sessions at 1.5MB/sec or so. I'd try to get up to normal speed before claiming that the software needs to be fixed to get more. More RAM will probably make a huge difference, but first are you sure that your IDE controller and cable are a match for the drive and that your OS is using an efficient DMA mode? I've seen even some of the 80-pin cables have a problem that would make things shift down to 33Mhz pio mode which will kill you. Does freebsd log the IDE mode detected and have some way to test throughput? BTW, how does BackupPC calculate speed? I think it calculates backup speed by reporting files transferred over time, so if you don't have many files that change, won't BackupPC report a very low backup speed. This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. If you look at the 'duration' in the backup summary and the Size/MB in the lower File Size summary you can compute your own rate based on what you are backing up. For fulls at least this seems to be based on the real target size, not what rsync had to transfer. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: Raid5 doesn't distribute disk activity - it puts the drives in lockstep and is slower than a single drive, especially on small writes where it has to do extra reads to re-compute parity on the existing data. I am confused, when a write is done the data is distributed in the disks depending on the stripe size you are using. When you start reading the file, you are reading from 5 different disks. So you get way better performance for sure on reads. The stripe effect only comes into play on files large enough to span them and not at all for directory/inode accesses which is most of what you are doing. Meanwhile you have another head tied up checking the parity and for writes of less than a block you have to read the existing contents before the write to re-compute the parity. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/27/07, Evren Yurtesen [EMAIL PROTECTED] wrote: David Rees wrote: Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. I sent the status of the backups earlier today to mailing list? Still missing wall-clock time. Though we can extrapolate that it may be taking over 6 hours for a full backup of the system below. Is that correct? This is from 1 machine. Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 112 full149290 1957.8 0.10149251 1937.9 460120.7 I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. As mentioned earlier, RAID 5 is horrible for random small read/write performance. It is often worse than a single disk. But still, I have a client which has 1.5 million files and 80-100GB of data. A full backup takes 4-6 hours which is reasonable. Full backups average 4.5-5MB/s. For example this guy is using Linux (problem is OS independent) http://forum.psoft.net/showpost.php?p=107808postcount=16 Your transfer rates are at least 5x slower than his and 10x slower than what most people here are getting. There is something wrong specifically with your setup. I suspect that if your backups were going 5x faster you'd be happy and 10x faster you'd be very happy. On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. With ext2 the default is async IO. With ext3 (the default system now) the default is ordered which is similar to BSD's soft updates. This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. Using the rsync transfer method, it does require at least a stat of every single file to be backed up. For 150k files and 2GB of data, you'd really expect this to be done within a hour with nearly any machine. Are you using the --checksum-seed option? That will significantly speed up (the 3rd and later) backups as well. Are you also sure the backup server isn't swapping as well? Can you get us some good logs from vmstat or similar during a backup? I also suspect that there is something else in this case slowing your machine down. Unless you give us the information to help track it down, we will not be able to figure it out. I feel like I am pulling teeth. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: Raid5 doesn't distribute disk activity - it puts the drives in lockstep and is slower than a single drive, especially on small writes where it has to do extra reads to re-compute parity on the existing data. I am confused, when a write is done the data is distributed in the disks depending on the stripe size you are using. When you start reading the file, you are reading from 5 different disks. So you get way better performance for sure on reads. The stripe effect only comes into play on files large enough to span them and not at all for directory/inode accesses which is most of what you are doing. Meanwhile you have another head tied up checking the parity and for writes of less than a block you have to read the existing contents before the write to re-compute the parity. Actually most (all?) raid 5 systems I've met don't check parity on read - they rely on the drive indicating a failed read.However the write splice penalty for raid 5 can still be pretty high (seek, read, insert new data, compute new parity, write data, seek, write parity) - and that's on top of the fact that the OS did it's own logical read, check permissions, insert, write to update directories and the like. John - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
I found a strange characteristic of IDE drive. When I connected CDROM and IDE hdd on same channel the data transfer with hdparm -t command is @37 MB. As I removed CDROM from that channel now the data transfer rate is @60 MB. This seems to work with 4 to 5 linux pc. -- Nilesh Vaghela ElectroMech Redhat Channel Partner and Training Partner 74, Nalanda Complex, Satellite Rd, Ahmedabad 25, The Emperor, Fatehgunj, Baroda. www.electromech.info - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: # Type #Files Size/MB MB/sec #Files Size/MB #Files Size/MB 245 full 152228 2095.2 0.06 152177 2076.910818.3 On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. 2MB/sec isn't bad when handling a lot of files (try unpacking a tar with hundreds of thousands of little files to see). The problem is that you Yes it is terrible. I get much better performance if I do the tar option with the same files. As a matter of fact I was using a smaller script for taking backups earlier. (which I still use on some servers) and transfer files over NFS. It works way faster, especially incremental backups take 5-10 minutes compared to 400 minutes with backuppc are getting .06 on a drive that is capable of running a couple of sessions at 1.5MB/sec or so. I'd try to get up to normal speed before claiming that the software needs to be fixed to get more. More RAM will probably make a huge difference, but first are you sure that your IDE controller and cable are a match for the drive and that your OS is using an efficient DMA mode? I've seen even some of the 80-pin cables have a problem that would make things shift down to 33Mhz pio mode which will kill you. Does freebsd log the IDE mode detected and have some way to test throughput? ad2: 238475MB ST3250824A/3.AAH [484521/16/63] at ata1-master UDMA100 DMA is working just fine. You are omitting the fact that 2mb/sec is very bad for a raid setup. I agree, handling a lot of files might be slow but this depends on how you handle the files. But I was handling the same files before and it wasnt taking this long. BTW, how does BackupPC calculate speed? I think it calculates backup speed by reporting files transferred over time, so if you don't have many files that change, won't BackupPC report a very low backup speed. This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. If you look at the 'duration' in the backup summary and the Size/MB in the lower File Size summary you can compute your own rate based on what you are backing up. For fulls at least this seems to be based on the real target size, not what rsync had to transfer. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: On 3/27/07, Evren Yurtesen [EMAIL PROTECTED] wrote: David Rees wrote: Evren, I didn't see that you mentioned a wall clock time for your backups? I want to know how many files are in a single backup, how much data is in that backup and how long it takes to perform that backup. I sent the status of the backups earlier today to mailing list? Still missing wall-clock time. Though we can extrapolate that it may be taking over 6 hours for a full backup of the system below. Is that correct? That is true, full backups take about 500-600 minutes and incrementals take 200-300minutes. This is from 1 machine. Totals Existing Files New Files Backup# Type#Files Size/MB MB/sec #Files Size/MB #Files Size/MB 112 full149290 1957.8 0.10149251 1937.9 460120.7 I dont know if the problem is hard links. This is not a FreeBSD or Linux problem. It exists on both. Just that people using ultra fast 5 disk raid 5 setups are seeing 2mbytes/sec transfer rate means that backuppc is very very inefficient. As mentioned earlier, RAID 5 is horrible for random small read/write performance. It is often worse than a single disk. Parity information is not required for normal reads, so you get 4-5 times better performance with 5 disks. Plus usually those drives are 10K rpm. When I am making incremental or even full backups, backuppc does not write same files again so writes are very little. So you get a huge performance boost with raid5 and 10k rpm drives compared to a normal ide drive. But still, I have a client which has 1.5 million files and 80-100GB of data. A full backup takes 4-6 hours which is reasonable. Full backups average 4.5-5MB/s. For example this guy is using Linux (problem is OS independent) http://forum.psoft.net/showpost.php?p=107808postcount=16 Your transfer rates are at least 5x slower than his and 10x slower than what most people here are getting. There is something wrong specifically with your setup. I suspect that if your backups were going 5x faster you'd be happy and 10x faster you'd be very happy. I would be happy if backups were made with the same speed as other backup programs can do. On Linux with raid setup with async io etc. people are getting slightly better results. I think ufs2 is just fine. I wonder if there is something in my explanations...The problem is backuppc. People are getting ~2mbytes/sec(was it 2 or 5?) speed with raid5 and 5 drives, using Linux. It is a miracle that backup even finishes in 24 hours using a standart ide drive. With ext2 the default is async IO. With ext3 (the default system now) the default is ordered which is similar to BSD's soft updates. This is like the 'Contact' movie. The sphere took 30 seconds to download but there were 18 hours of recording. If what you said was true and backuppc would be backing up very small amount of files and skipping most, then backups would probably take less time than 2-4 hours each. Using the rsync transfer method, it does require at least a stat of every single file to be backed up. For 150k files and 2GB of data, you'd really expect this to be done within a hour with nearly any machine. I am using rsync to sync 2GB of files from one machine to another and the number of files is about 100k and the operation takes 30seconds to 1 minute in average. Are you using the --checksum-seed option? That will significantly speed up (the 3rd and later) backups as well. No, it requires a patched rsync. Are you also sure the backup server isn't swapping as well? Can you get us some good logs from vmstat or similar during a backup? I can tell that it is not swapping because the disk where the swaps are idle while taking backup. ad0 is the disk with the swap. If there was such problem then it would be reading like crazy from swap. I have given this information before. I also suspect that there is something else in this case slowing your machine down. Unless you give us the information to help track it down, we will not be able to figure it out. I feel like I am pulling teeth. I have given you all the information you asked for(didnt I?), even tried async option. Incremental backup of 1 machine still took about 300 minutes. The machine is working fine. I was using another backup program which was working way faster to backup the same machines. So I dont think that I have a hardware problem or such. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net
Re: [BackupPC-users] very slow backup speed
nilesh vaghela wrote: I found a strange characteristic of IDE drive. When I connected CDROM and IDE hdd on same channel the data transfer with hdparm -t command is @37 MB. As I removed CDROM from that channel now the data transfer rate is @60 MB. This seems to work with 4 to 5 linux pc. I have no CDROMS installed on the machine and drives are on their own cables. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
I have given you all the information you asked for(didnt I?), even tried async option. Incremental backup of 1 machine still took about 300 minutes. The machine is working fine. I was using another backup program which was working way faster to backup the same machines. So I dont think that I have a hardware problem or such. I haven't read *every* email in this thread (it was getting quite repetitive seeing you complaining about how crap backuppc is), but anyway, could you please elaborate on the method that your 'other' backup solution used, ie, if your other solution used rsync, and you are using rsync with backuppc, then that might be helpful to know. If you used tar before, and now use rsync, that would obviously make a difference. I think overall, the attitude you have displayed is not conducive to resolving the problem. I also think it has been clearly shown that backuppc itself is not the problem, as other people are using it successfully, under similar conditions. What you need to do is basic diagnosis to find where the bottle neck is in your particular system. I'd suggest you use whatever tools are available under your chosen OS (I think it is freebsd), to measure: 1) CPU utilisation on both the client being backed up, and the backuppc server 2) Memory/swap utilisation on both client and server 3) Disk utilisation/bandwidth on both client and server 4) Network utilisation/bandwidth on both client and server Also, if you are using tar as your transfer method, then I'd suggest you try a 'manual' backup something like this (note, this command line is wrong, but you will need to adjust to get the right one). from the backuppc machine: cd /var/lib/backuppc;mkdir temp;cd temp;ssh client tar -cf - /backupdir | tar -xf - Try and copy the command line args from your backup log file Time how long that takes, and see how it compares to backuppc times if they are similar, then you know the issue is outside of backuppc. Of course, if you are using rsync, then do the same test with rsync data, but I'm sure it has already been said that you should use tar for your case with lots of small files. Just my 0.02c worth Regards, Adam -- Adam Goryachev Website Managers Ph: +61 2 8304 [EMAIL PROTECTED] Fax: +61 2 8304 0001www.websitemanagers.com.au - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
[BackupPC-users] very slow backup speed
I am using backuppc but it is extremely slow. I narrowed it down to disk bottleneck. (ad2 being the backup disk). Also checked the archives of the mailing list and it is mentioned that this is happening because of too many hard links. Disks ad0 ad2 KB/t 4.00 25.50 tps 175 MB/s 0.00 1.87 % busy196 But I couldnt find any solution to this. Is there a way to get this faster without changing to faster disks? I guess I could 2 disks in mirror or something but it is stupid to waste the space I gain by backuppc algorithm by using multiple disks to get a decent performance :) I am feeling that this performance problem is extreme. It goes with snail speed even when I am backing up 1 machine only. Should I be adding disks for each machine I backup? :) Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: I am using backuppc but it is extremely slow. I narrowed it down to disk bottleneck. (ad2 being the backup disk). Also checked the archives of the mailing list and it is mentioned that this is happening because of too many hard links. [snip] The basic problem is backuppc is using the file system as a database - specifically using the hard link capability to store multiple references to an object and the link count to manage garbage collection. Many (all?) filesystems seem to get slow when you get into the millios of files with thousands of links range. Changing the way is works (say to use a real database) looks like a very non trivial task. Adding disk spindles will help (particularly if you have multiple backups going at once) but in the end it's still not going to be blazingly fast. John - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
John Pettitt wrote: Evren Yurtesen wrote: I am using backuppc but it is extremely slow. I narrowed it down to disk bottleneck. (ad2 being the backup disk). Also checked the archives of the mailing list and it is mentioned that this is happening because of too many hard links. [snip] The basic problem is backuppc is using the file system as a database - specifically using the hard link capability to store multiple references to an object and the link count to manage garbage collection. Many (all?) filesystems seem to get slow when you get into the millios of files with thousands of links range. Changing the way is works (say to use a real database) looks like a very non trivial task. Adding disk spindles will help (particularly if you have multiple backups going at once) but in the end it's still not going to be blazingly fast. John Well, so there are no plans to fix this problem? I found forum threads that in certain cases backups take over 24hours! Goodbye to daily incremental backups :) Do you know any alternatives to backuppc with web gui? which probably works faster? :P I wonder what is the mechanical stress this poses on the hard drive when it has to work 24/7 moving it's head like crazy. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: John Pettitt wrote: Evren Yurtesen wrote: I am using backuppc but it is extremely slow. I narrowed it down to disk bottleneck. (ad2 being the backup disk). Also checked the archives of the mailing list and it is mentioned that this is happening because of too many hard links. [snip] The basic problem is backuppc is using the file system as a database - specifically using the hard link capability to store multiple references to an object and the link count to manage garbage collection. Many (all?) filesystems seem to get slow when you get into the millios of files with thousands of links range. Changing the way is works (say to use a real database) looks like a very non trivial task. Adding disk spindles will help (particularly if you have multiple backups going at once) but in the end it's still not going to be blazingly fast. John Well, so there are no plans to fix this problem? I found forum threads that in certain cases backups take over 24hours! Goodbye to daily incremental backups :) If your filesystem isn't a good place to store files, there is not much an application can do about it. Perhaps it would help if you mentioned what kind of scale you are attempting with what server hardware. I know there are some people on the list handling what I would consider large backups with backuppc. If yours is substantially smaller perhaps they can help diagnose the problem. Maybe you are short on RAM and swapping memory to disk with large rsync targets. I wonder what is the mechanical stress this poses on the hard drive when it has to work 24/7 moving it's head like crazy. They'll die at some random time averaging around 4-5 years - just like any other hard drive. Disk heads are made to move... -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: John Pettitt wrote: The basic problem is backuppc is using the file system as a database - specifically using the hard link capability to store multiple references to an object and the link count to manage garbage collection. Many (all?) filesystems seem to get slow when you get into the millios of files with thousands of links range. Changing the way is works (say to use a real database) looks like a very non trivial task. Adding disk spindles will help (particularly if you have multiple backups going at once) but in the end it's still not going to be blazingly fast. Well, so there are no plans to fix this problem? I found forum threads that in certain cases backups take over 24hours! Goodbye to daily incremental backups :) Well, I just saw a proposal on linux-kernel which addresses inode allocation performance issues on ext3/4 by preallocating contiguous blocks of inodes for directories. I suspect this would help reduce the number of seeks required when performing backups. If there is another filesystem which does this I imagine it would perform better than ext3. Do you know any alternatives to backuppc with web gui? which probably works faster? :P BackupPC is the best. Most backups complete in a reasonable time, those that don't are backups which are either very large (lots of bandwidth) or have lots of files. My backup server is a simple Athlon XP 2000+ with a RAID1 consisting of 2 Seagate 250GB 7200rpm ATA drives. More spindles and/or disks with faster seek times is the way to go. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: John Pettitt wrote: Evren Yurtesen wrote: I am using backuppc but it is extremely slow. I narrowed it down to disk bottleneck. (ad2 being the backup disk). Also checked the archives of the mailing list and it is mentioned that this is happening because of too many hard links. [snip] The basic problem is backuppc is using the file system as a database - specifically using the hard link capability to store multiple references to an object and the link count to manage garbage collection. Many (all?) filesystems seem to get slow when you get into the millios of files with thousands of links range. Changing the way is works (say to use a real database) looks like a very non trivial task. Adding disk spindles will help (particularly if you have multiple backups going at once) but in the end it's still not going to be blazingly fast. John Well, so there are no plans to fix this problem? I found forum threads that in certain cases backups take over 24hours! Goodbye to daily incremental backups :) If your filesystem isn't a good place to store files, there is not much an application can do about it. Perhaps it would help if you mentioned what kind of scale you are attempting with what server hardware. I know there are some people on the list handling what I would consider large backups with backuppc. If yours is substantially smaller perhaps they can help diagnose the problem. Maybe you are short on RAM and swapping memory to disk with large rsync targets. I know that the bottleneck is the disk. I am using a single ide disk to take the backups, only 4 machines and 2 backups running at a time(if I am not remembering wrong). I see that it is possible to use raid to solve this problem to some extent but the real solution is to change backuppc in such way that it wont use so much disk operations. I wonder what is the mechanical stress this poses on the hard drive when it has to work 24/7 moving it's head like crazy. They'll die at some random time averaging around 4-5 years - just like any other hard drive. Disk heads are made to move... Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: I know that the bottleneck is the disk. I am using a single ide disk to take the backups, only 4 machines and 2 backups running at a time(if I am not remembering wrong). I see that it is possible to use raid to solve this problem to some extent but the real solution is to change backuppc in such way that it wont use so much disk operations. The whole purpose of live backup media is to use the media. What you may be noticing is that perhaps your drive is mounted with access time being tracked. You should check that your fstab has noatime as a parameter for your mounted data volume. This probably cuts the seeks down by nearly half or more. And, you could consider buying a faster drive, or one with a larger buffer. Some IDE drives have pathetically small buffers and slow rotation rates. That makes for a greater need for seeking, and worse seek performance. Also, if your server is a single-proc, you'll probably want to reduce it to 1 simultaneous backup, not 2. Heck, if you are seeing bad thrashing on the disk, it would have better coherence if you stick to 1 anyway. Increase your memory and you may see less virtual memory swapping as well. It seems that your setup is very similar to mine, and I'm not seeing the kind of performance problems you're reporting. Full backup using rsyncd over a slow wifi link of about 65gb is only taking about 100 minutes. Incrementals are about 35 minutes. Using SMB on a different machine with about 30gb, it takes 300 minutes for a full, even over gigabit, but only a couple of minutes for an incremental (because it doesn't detect as many changes as rsync). So it varies dramatically with the protocol and hardware. JH - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: I know that the bottleneck is the disk. I am using a single ide disk to take the backups, only 4 machines and 2 backups running at a time(if I am not remembering wrong). I see that it is possible to use raid to solve this problem to some extent but the real solution is to change backuppc in such way that it wont use so much disk operations. From what I can tell the issue is that each file requires a hard link - depending on your file system metadata like directory entries, had links etc get treated differently that regular data - on a BSD ufs2 system metadata updates are typically synchronous, that is the system doesn't return until the write has made it to the disk. This is good for reliability but really bad for performance since it prevents out of order writes which can save a lot of disk activity. Changing backuppc would be decidedly non-trivial - eyeballing it to hack in a real database to store the relationship between pool and individual files would touch almost just about every part of the system. What filesystem are you using and have you turned off atime - I found that makes a big difference. John - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Original Message Subject: Re:[BackupPC-users] very slow backup speed From: Evren Yurtesen [EMAIL PROTECTED] To: David Rees [EMAIL PROTECTED] Date: 26.03.2007 23:37 David Rees wrote: It is true that BackupPC is great, however backuppc is slow because it is trying to make backup of a single instance of each file to save space. Now we are wasting (perhaps even more?) space to make it fast when we do raid1. You can't be serious about that: let's say you have a handful of workstations full backup 200GB each and perform backups for a couple of weeks - in my case after a month 1,4 TB for the fulls and 179GB for the incrementals. After pooling and compression: 203 (!) GB TOTAL. Xfer time for a 130GB full: 50min. How fast are your tapes? But if you prefer changing tapes (and spending a lot more money on the drives) - go ahead ... so much for wasting space ;-) Regards, Bernhard - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
John Pettitt wrote: Evren Yurtesen wrote: I know that the bottleneck is the disk. I am using a single ide disk to take the backups, only 4 machines and 2 backups running at a time(if I am not remembering wrong). I see that it is possible to use raid to solve this problem to some extent but the real solution is to change backuppc in such way that it wont use so much disk operations. From what I can tell the issue is that each file requires a hard link - depending on your file system metadata like directory entries, had links etc get treated differently that regular data - on a BSD ufs2 system metadata updates are typically synchronous, that is the system doesn't return until the write has made it to the disk. This is good for reliability but really bad for performance since it prevents out of order writes which can save a lot of disk activity. Changing backuppc would be decidedly non-trivial - eyeballing it to hack in a real database to store the relationship between pool and individual files would touch almost just about every part of the system. What filesystem are you using and have you turned off atime - I found that makes a big difference. John I have noatime, I will try bumping up the memory and hope that the caching will help. I will let you know if it helps. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: If your filesystem isn't a good place to store files, there is not much an application can do about it. Perhaps it would help if you mentioned what kind of scale you are attempting with what server hardware. I know there are some people on the list handling what I would consider large backups with backuppc. If yours is substantially smaller perhaps they can help diagnose the problem. Maybe you are short on RAM and swapping memory to disk with large rsync targets. I know that the bottleneck is the disk. I am using a single ide disk to take the backups, only 4 machines and 2 backups running at a time(if I am not remembering wrong). That's still not very informative. Approximately how much data do those targets hold (number of files and total space used)? Are you using tar or rsync? If you are running Linux, what does 'hdparm -t -T' say about your disk speed (the smaller number)? And what filesystem are you using? I see that it is possible to use raid to solve this problem to some extent but the real solution is to change backuppc in such way that it wont use so much disk operations. First we should find out if your system is performing badly compared to others or if you are just expecting too much. As an example, one of my systems is backing up 20 machines and the summary says: Pool is 152.54GB comprising 2552606 files and 4369 directories This is a RAID1 (mirrored, so no faster than a single drive) on IDE drives and the backups always complete overnight. I wonder what is the mechanical stress this poses on the hard drive when it has to work 24/7 moving it's head like crazy. They'll die at some random time averaging around 4-5 years - just like any other hard drive. Disk heads are made to move... Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
John Pettitt wrote: Changing backuppc would be decidedly non-trivial - eyeballing it to hack in a real database to store the relationship between pool and individual files would touch almost just about every part of the system. And there's not much reason to think that a database could do this with atomic updates any better than the filesystem it sits on. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/26/07, Bernhard Ott [EMAIL PROTECTED] wrote: It is true that BackupPC is great, however backuppc is slow because it is trying to make backup of a single instance of each file to save space. Now we are wasting (perhaps even more?) space to make it fast when we do raid1. You can't be serious about that: let's say you have a handful of workstations full backup 200GB each and perform backups for a couple of weeks - in my case after a month 1,4 TB for the fulls and 179GB for the incrementals. After pooling and compression: 203 (!) GB TOTAL. Xfer time for a 130GB full: 50min. How fast are your tapes? But if you prefer changing tapes (and spending a lot more money on the drives) - go ahead ... so much for wasting space ;-) No kidding! My backuppc stats are like this: 18 hosts 76 full backups of total size 748.09GB (prior to pooling and compression) 113 incr backups of total size 134.11GB (prior to pooling and compression) Pool is 135.07GB comprising 2477803 files and 4369 directories 6.5:1 compression ratio is pretty good, I think. Athlon XP 2000+ 1GB RAM, software RAID 1 w/ 2 ST3250824A (7200rpm, ATA, 8MB cache). The machine was just built from leftover parts. Running on Fedora Core 6. I love BackupPC. :-) -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: And, you could consider buying a faster drive, or one with a larger buffer. Some IDE drives have pathetically small buffers and slow rotation rates. That makes for a greater need for seeking, and worse seek performance. Well this is a seagate barracuda 7200rpm drive with 8mb cache ST3250824A http://www.seagate.com/support/disc/manuals/ata/100389997c.pdf Same drive as I'm using, except mine are in RAID1 which doubles random read performance. I read your posts about wifi etc. on forum. The processor is not the problem however adding memory probably might help bufferwise. I think this idea can actually work.:) thanks! I am seeing swapping problems but the disk the swap is on is almost idle. The backup drive is working all the time. Please show us some more real data showing CPU utilization while a backup is running. Please also give us the real specs of the machine and what other jobs the machine performs. I have to say that slow performance with BackupPC is a known problem. I have heard it from several other people who are using BackupPC and it is the #1 reason of changing to another backup program from what I hear. Things must improve on this area. There are plenty of ways to speed up BackupPC. It really isn't slow in my experience. But you must tell us what you are actually doing and what is going on with your server for us to help instead of repeatedly saying it's slow, speed it up. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Let's start at the beginning: On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: I am using backuppc but it is extremely slow. I narrowed it down to disk bottleneck. (ad2 being the backup disk). Also checked the archives of the mailing list and it is mentioned that this is happening because of too many hard links. Disks ad0 ad2 KB/t 4.00 25.50 tps 175 MB/s 0.00 1.87 % busy196 What OS are you runnnig? What filesystem? What backup method (ssh+rsync, rsyncd, smb, tar, etc)? 75 tps seems to be a bit slow for a single disk. Do you have vmstat, iostat and/or top output while a backup is running? But I couldnt find any solution to this. Is there a way to get this faster without changing to faster disks? I guess I could 2 disks in mirror or something but it is stupid to waste the space I gain by backuppc algorithm by using multiple disks to get a decent performance :) A mirror will only help speed up random reads at best. This usually isn't a problem for actual backups, but will help speed up the nightly maintenance runs. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Jason Hughes wrote: Evren Yurtesen wrote: And, you could consider buying a faster drive, or one with a larger buffer. Some IDE drives have pathetically small buffers and slow rotation rates. That makes for a greater need for seeking, and worse seek performance. Well this is a seagate barracuda 7200rpm drive with 8mb cache ST3250824A http://www.seagate.com/support/disc/manuals/ata/100389997c.pdf Perhaps it is not the maximum amount of cache one can have on a drive but it is not that bad really. That drive should be more than adequate. Mine is a 5400rpm 2mb buffer clunker. Works fine. Are you running anything else on the backup server, besides BackupPC? What OS? What filesystem? How many files total? FreeBSD, UFS2+softupdates, noatime. There are 4 hosts that have been backed up, for a total of: * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.45GB (prior to pooling and compression). # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 07:16), today's max is 10% (3/27 01:00) and yesterday's max was 10%. Host User#Full Full Age (days) Full Size (GB) Speed (MB/s) #Incr Incr Age (days) Last Backup (days) State Last attempt host1 4 5.4 3.880.226 0.4 0.4 idleidle host2 4 5.4 2.100.066 0.4 0.4 idleidle host3 4 5.4 7.570.146 0.4 0.4 idleidle host4 4 5.4 5.560.106 0.4 0.4 idleidle I read your posts about wifi etc. on forum. The processor is not the problem however adding memory probably might help bufferwise. I think this idea can actually work.:) thanks! I am seeing swapping problems but the disk the swap is on is almost idle. The backup drive is working all the time. Hmm. That's a separate disk, not a separate partition of the same disk, right? If it's just a separate partition, I'm not sure how well the OS will be able to allocate wait states to logical devices sharing the same physical media... in other words, what looks like waiting on ad2 may be waiting on ad0. Someone more familiar with device drivers and linux internals would have chime in here. I'm not an expert. It is a separate disk. The disk is on FreeBSD not Linux. They are not waiting for each other, they can be used simultaneously. I have to say that slow performance with BackupPC is a known problem. I have heard it from several other people who are using BackupPC and it is the #1 reason of changing to another backup program from what I hear. Things must improve on this area. I did quite a lot of research and found only one other program that was near my needs, and it was substantially slower due to encryption overhead, and didn't have a central pool to combine backup data. I may have missed an app out there, though. What are these people switching to, if you don't mind? Re: what must improve is more people helping Craig. He's doing it all for free. I think if it's important enough to have fixed, it's important enough to pay for. Or dive into the code and start making those changes. It is open source, after all. I think we are already helping already by discussing the issue. Even if we wanted to pay, there is nothing to pay yet, as there is no agreed solution to this slowness. Thanks, Evren - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Les Mikesell wrote: Evren Yurtesen wrote: If your filesystem isn't a good place to store files, there is not much an application can do about it. Perhaps it would help if you mentioned what kind of scale you are attempting with what server hardware. I know there are some people on the list handling what I would consider large backups with backuppc. If yours is substantially smaller perhaps they can help diagnose the problem. Maybe you are short on RAM and swapping memory to disk with large rsync targets. I know that the bottleneck is the disk. I am using a single ide disk to take the backups, only 4 machines and 2 backups running at a time(if I am not remembering wrong). That's still not very informative. Approximately how much data do those targets hold (number of files and total space used)? Are you using tar or rsync? If you are running Linux, what does 'hdparm -t -T' say about your disk speed (the smaller number)? And what filesystem are you using? There are 4 hosts that have been backed up, for a total of: * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.45GB (prior to pooling and compression). Host User#Full Full Age (days) Full Size (GB) Speed (MB/s) #Incr Incr Age (days) Last Backup (days) State Last attempt host1 4 5.4 3.880.226 0.4 0.4 idleidle host2 4 5.4 2.100.066 0.4 0.4 idleidle host3 4 5.4 7.570.146 0.4 0.4 idleidle host4 4 5.4 5.560.106 0.4 0.4 idle idle # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 07:16), today's max is 10% (3/27 01:00) and yesterday's max was 10%. I see that it is possible to use raid to solve this problem to some extent but the real solution is to change backuppc in such way that it wont use so much disk operations. First we should find out if your system is performing badly compared to others or if you are just expecting too much. As an example, one of my systems is backing up 20 machines and the summary says: Pool is 152.54GB comprising 2552606 files and 4369 directories This is a RAID1 (mirrored, so no faster than a single drive) on IDE drives and the backups always complete overnight. I wonder what is the mechanical stress this poses on the hard drive when it has to work 24/7 moving it's head like crazy. They'll die at some random time averaging around 4-5 years - just like any other hard drive. Disk heads are made to move... Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. I cant see how you can reach to this conclusion. So you say that a car which was driven 10miles have the same possibility of breaking down compared to the same model car driven for 10miles? There are frictions involved when head moves in the hard drive. - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
David Rees wrote: Let's start at the beginning: On 3/26/07, Evren Yurtesen [EMAIL PROTECTED] wrote: I am using backuppc but it is extremely slow. I narrowed it down to disk bottleneck. (ad2 being the backup disk). Also checked the archives of the mailing list and it is mentioned that this is happening because of too many hard links. Disks ad0 ad2 KB/t 4.00 25.50 tps 175 MB/s 0.00 1.87 % busy196 What OS are you runnnig? What filesystem? What backup method (ssh+rsync, rsyncd, smb, tar, etc)? 75 tps seems to be a bit slow for a single disk. Do you have vmstat, iostat and/or top output while a backup is running? Well, 1.7MB/s random reads is not that bad really. There are 4 hosts that have been backed up, for a total of: vmstat procs memory pagedisks faults cpu r b w avmfre flt re pi po fr sr ad0 ad2 in sy cs us sy id 1 10 0 18 14124 112 0 1 1 245 280 0 0 438 361 192 6 3 91 iostat tty ad0 ad2 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 02 11.39 3 0.04 9.52 59 0.54 6 0 3 0 91 But I couldnt find any solution to this. Is there a way to get this faster without changing to faster disks? I guess I could 2 disks in mirror or something but it is stupid to waste the space I gain by backuppc algorithm by using multiple disks to get a decent performance :) A mirror will only help speed up random reads at best. This usually isn't a problem for actual backups, but will help speed up the nightly maintenance runs. -Dave - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/ - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Jason Hughes wrote: Evren Yurtesen wrote: Jason Hughes wrote: That drive should be more than adequate. Mine is a 5400rpm 2mb buffer clunker. Works fine. Are you running anything else on the backup server, besides BackupPC? What OS? What filesystem? How many files total? FreeBSD, UFS2+softupdates, noatime. There are 4 hosts that have been backed up, for a total of: * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.45GB (prior to pooling and compression). # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), # Pool hashing gives 38 repeated files with longest chain 6, # Nightly cleanup removed 10725 files of size 0.40GB (around 3/27 05:54), # Pool file system was recently at 10% (3/27 07:16), today's max is 10% (3/27 01:00) and yesterday's max was 10%. Host User #Full Full Age (days) Full Size (GB) Speed (MB/s) #Incr Incr Age (days) Last Backup (days) State Last attempt host1 4 5.4 3.88 0.22 6 0.4 0.4 idle idle host2 4 5.4 2.10 0.06 6 0.4 0.4 idle idle host3 4 5.4 7.57 0.14 6 0.4 0.4 idle idle host4 4 5.4 5.56 0.10 6 0.4 0.4 idle idle Hmm. This is a tiny backup setup, even smaller than mine. However, it appears that the average size of your file is only 22KB, which is quite small. For comparison sake, this is from my own server: Pool is 172.91GB comprising 217311 files and 4369 directories (as of 3/26 01:08), The fact that you have tons of little files will probably give significantly higher overhead when doing file-oriented work, simply because the inodes must be fetched for each file before seeking to the file itself. If we assume no files are shared between hosts (very conservative), and you have an 8ms access time, you will have 190132 files per host and two seeks per file, neglecting actual i/o time, gives you 50 minutes. Just to seek them all. If you have a high degree of sharing, it can be up to 4x worse. Realize, the same number of seeks must be made on the server as well as the client. Are you sure you need to be backing up everything that you're putting across the network? Maybe excluding some useless directories, maybe temp files or logs that haven't been cleaned up? Perhaps you can archive big chunks of it with a cron job? I'd start looking for ways to cut down the number of files, because the overhead of per-file accesses are probably eating you alive. I'm also no expert on UFS2 or FreeBSD, so it may be worthwhile to research its behavior with hard links and small files. JH For what it's worth, I have a server that backs up 8.6 million files averaging 10k in size from one host. It takes a full 10 hours for a full backup via tar over NFS ( 2.40MB/s for 87GB). CPU usage is low, around 10-20%, however IOwait is a pretty steady 25%. Server info: HP DL380 G4 debian sarge dual processor 3.2ghz xeon 2GB Ram 5x10k rpm scsi disks, raid5 128MB battery backed cache (50/50 r/w) ext3 filesystems brien - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
Re: [BackupPC-users] very slow backup speed
Evren Yurtesen wrote: There are 4 hosts that have been backed up, for a total of: * 16 full backups of total size 72.16GB (prior to pooling and compression), * 24 incr backups of total size 13.45GB (prior to pooling and compression). # Pool is 17.08GB comprising 760528 files and 4369 directories (as of 3/27 05:54), That doesn't sound difficult at all. I suspect your real problem is that you are running a *bsd UFS filesystem with it's default sync metadata handling which is going to wait for the physical disk action to complete on every directory operation. I think there are other options but I haven't kept up with them. I gave up on UFS long ago when I needed to make an application that frequently truncated and rewrote a data file work on a machine that crashed frequently. The sync-metadata 'feature' statistically ensured that there was never any data in the file after recovering since the truncation was always forced to disk immediately but the data write was buffered so with a fast cycle the on-disk copy was nearly always empty. Is anyone else running a *bsd? Perhaps, but there is a difference if they are moving 10 times or 10 times. Where the difference is that the possibility of failure due to mechanical problems increases 1 times. No, it doesn't make a lot of difference as long as the drive doesn't overheat. The head only moves so fast and it doesn't matter if it does it continuously. However, if your system has sufficient RAM, it will cache and optimize many of the things that might otherwise need an additional seek and access. I cant see how you can reach to this conclusion. Observation... I run hundreds of servers, many of which are 5 or more years old. The disk failures have had no correlation to the server activity. So you say that a car which was driven 10miles have the same possibility of breaking down compared to the same model car driven for 10miles? There are frictions involved when head moves in the hard drive. Cars run under less predictable conditions and need some periodic maintenance, but yes, I expect my cars to run 10 miles under their design conditions without breaking down. -- Les Mikesell [EMAIL PROTECTED] - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/backuppc-users http://backuppc.sourceforge.net/
