I am listing the observations and complaints of mine after using BackupPC for 5 months. My system is running Ubuntu Hardy and has been kept up with the latest updates and patches. I upgraded from 3.0.0 to 3.1.0 within my first few months and have kept this computer solely a BackupPC machine. I've since gone through a few hard drive configurations which I will list here: - 3x640GB in hardware RAID3 using an IDE RAID card with SATA ports - 3x250GB in Linux software RAID5 using mdadm and onboard SATA ports - 1x2TB using onboard SATA ports All of those configurations have assured me it is not the speed of the drives themselves which have been giving me any issues. The machine itself has a 3.4GHz P4HT which I have equally tested with and without HyperThreading to no apparent change in usage. The Ethernet NIC is a gigabit broadcom one, and I have both switched the cables the switches with no apparent change in performance as far as I can remember.
Through all my reading of the documentation and many articles about BackupPC which I found using Google, I was able to reconfigure my configuration files to fit the needs of the machines I had and their backup schedules. I was constantly tweaking and changing setting files, but I want to note, I did not see a change in performance, just less error messages such as excluding /dev for instance. Please feel free, from here on in, to question or assert my obervations and complaints so that we may all get a well-rounded understanding of the situation. ./01\. The first thing I noticed when using the program was that the defaults seemed a bit extreme for my system. I began my setup with two always-on Windows machines configured with Samba. I let one full backup go one night and the other the next. From here, I noticed the load of my computer consistently at or over 4, and I don't think I've ever seen the processor usage go under 100% unless BackupPC isn't doing anything. Either way, these backups seemed to go very smoothly and did not cause me any problems after the initial full backups. I did have one very pertinent issue, the speed of the full backups and incrementals never went over 10MB/s. On gigabit switches with all gigabit-linked computers, this was very strange. The Windows machines could transfer anywhere from 50-75MB/s to each other so I figured something else might be wrong. I understand many small files are being transferred, but I would expect the average to be higher when using Samba over Gigabit. ./02\. I later added in a few always-on Linux machines configured with Rsyncd. This is where I started noticing some things that were fishy. One of those machines was hooked up to the network using a USB Wi-Fi card and the other an Intel Gigabit Ethernet adapter. Unlike common sense would show, the computer on WLAN had a backup speed in MB/s was faster than that of the computer on LAN. Both machines had similar setups and both had their own public IP addresses hosting web pages publicly using a different interface than the one for backup purposes. The one main difference in the two setups was the operating system. Both were on Ubuntu, but one had been consistently upgraded so it was on Karmic (WLAN) while the other was on Hardy (LAN). Either way, this should not have affected much of anything. The LAN computer also has twice as much RAM and a slightly faster processor. Both had IDE drives at similar speeds. I can only think the speed difference might have been caused by the LAN computer having 4 more GB of data, possibly many small files, than the WLAN machine which had only 2.7GB in use at the time. but I cannot be sure. ./03\. I eventually ended up with 18 machines configured in BackupPC with more being added as I configure them. In those 18 are a few websites of mine for which the servers are remote. Those website I am backing up using the Rsync configuration. The Rsync configuration with these websites works exactly as it should. One of the sites had 11GB of data and while that took forever to download on my connection, incremental updates are speedy. The full updates are speedy too. Backup number 61 on this website took only 6.4 minutes for 2.5GB of files. This is because those 2.5GB were already backed up in the last full and therefore no longer needed to be redownloaded. It is my understanding that this is how BackupPC is supposed to work. While the very first full took around 160 minutes, the second was quite a huge difference, only taking a bit more than three times as long as the previous incrementals. ./04\. After some time, I noticed my Samba backups were missing files. I had begun to need older deleted files and noticed they were not backed up in the last full. It appeared as though BackupPC would get to a point and just give up but make sure to backup the root folder list first at least. There were times where certain folders were gone but the rest adjacent were filled. This last sentence may be wrong; I am judging based on my memory as those backups have already been cleaned. ./05\. It was at this point I decided it was time to setup Rsyncd for all my Windows machines too and then the majority of problems all became readily apparent. Whereas I could backup my laptops in under 15 minutes each day seemingly no matter what I downloaded or changed, backups began taking in excess of 40 minutes even up to 2 hours. Sure, everything I wanted to be backed up did get backed up, but it was becoming slower and slower as the days progressed. I noticed each incremental seemed to take longer than the one before it and that the two Linux boxes began to take longer and longer to backup as well. I even had a small embedded version of FreeBSD running solely in RAM on a machine and noticed even it taking almost up to 2 minutes to backup whereas it'd always take 0.4 to 0.6 minutes. The strain of Rsyncd started to really wain on the machine. It's now to the point where backups take up to 4 hours for most machines, it seems as though the CPU usage never changes from 100%, and laptops need to be left on overnight to backup at all. It's really annoying to constantly get administrative e-mails that certain machines weren't backed up because they are mobile machines and Rsyncd just isn't speedy enough. This never seemed to be a problem before until I converted all my Windows machines by the sheer need to have all things backed up which should have been backed up in the last full. I've been using DeltaCopy but I did update the DLLs and executable files. I even once tried the BackupPC Cygwin files but that ended up taking 2-3 times as long as the newer files I use. ./06\. The logs seem to suck really bad. I can't seem to find out which new files are being created and what's making the biggest impact on each backup operation every night. I'd really like to know where the 700MB came from even if I wasn't at the machine. It's one thing to backup the changes on my laptop, but there's no way I downloaded or changed 300MB-worth of new files all which I had to backup. My understanding is that these are probably a lot of text files because the compression ratio averages around 66% or more, but I still have to transfer those files over the network as slowly as 0.06MB/s can be. What's worse, no matter how much more stuff I put in my excludes, more data seems to be downloaded. The growth of the pool seems to have leveled with old incrementals going away and new ones coming in, but it was at a point where there was at least 1GB of growth a night from when it was a 160GB pool to a 210GB pool when it began jumping half-dozens of gigabytes during the period of a week. Even with only using the web browser on my laptop there was over a gigabyte of information last backed up with 950MB of it new files. The laptop is only using 22GB of space and if there really were 400MB of new files a night, I'd surely run out of room on this drive fast. I've done my best to pinpoint the issue, but no where do I see more than 100MB of changes in total so I'm lost as to what changes are actually being made and if they're necessary. Blocking all *.edb files should really have lowered the amount of new information being backed up. I have disabled indexing on all my machines and have no drives with compressed data. I have a decent list of excludes and try to get as much as I can as a just-in-case measure, but it looks as if there is a huge flaw here I am completely missing out on. I wish I could figure this out. Everything I'm seeing is attributed to Rsyncd. Samba was so fast and Rsync seems to work just fine but Rsyncd is entirely botched. All of my machines show a high amount of CPU usage whenever BackupPC beings an Rsyncd backup anywhere from 33% on my most-powerful rig (a quad-core) to 95-100% on my netbook. Tonight, to do a final test, I've set 8 backups to go in tandem. Four seemed to have dropped off very quickly leaving me with four final machines to watch. Through writing this post, not one of those four has finished. Full backups seem to average 1MB/s instead of at least 3MB/s like before. I recommend people write in explaining this phenomenon so there may be some resolution. I've done far too much testing to just let all of this slide. I'd really like for my machines to actually backup, not sit there and waste battery and use up CPU only for the backup to fail because I finished using the machine and 4 hours hadn't passed yet. +---------------------------------------------------------------------- |This was sent by saturn2...@gmail.com via Backup Central. |Forward SPAM to ab...@backupcentral.com. +---------------------------------------------------------------------- ------------------------------------------------------------------------------ _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/