On 7/6/2018 10:22 AM, G.W. Haywood via BackupPC-users wrote: > Hi there, > > On Fri, 6 Jul 2018, Bowie Bailey wrote: > >> I am trying to backup a large directory tree with BackupPC v4.? This >> directory is 660GB and contains over 25 million files with about 3 >> million hard links.? The initial backup ran for 2 weeks before dying >> with an rsync error.? It is showing as a partial backup, but it doesn't >> show a file count. > > This is not an unusually large task for BackupPC. I routinely back up > directories with similar volumes of contents, but there are caveats. > > My 'home' directory on my desktop machine at work for example is half > that size, and just over a million files, although it contains only a > few thousand hard links. Using rsync over a 100Mbit/s Ethernet link > between a couple of 2.4GHz dual Opteron machines, each with 8-16GBytes > RAM, I don't recall ever being surprised or disappointed by the time > it took to complete a backup. The first backup was 376G in ~1M files, > and took nearly 16 hours. > > Recently I added another user's home directory to the pool. To back > up 530GBytes of fresh data in 1.06M files between the same machines > took 23.5 hours for the first pass. A full backup with a few tens of > megabytes of new files takes half a day, an incremental takes only ten > to 25 minutes depending mostly on the numbers of new files. > > Your files average 26.4kBytes, my new user's average ~500kBytes, you > can expect some differences because of that. > > The problem description isn't very clear about the structure of the > data, and doesn't mention the type(s) of filesystem involved, CPU/RAM, > the transport mechanism(s). For example do you have huge numbers of > files at a single directory level? Putting even tens of thousands of > files, let alone millions, in a single ext[234] directory is likely to > cause performance problems. Are you using a 56k modem? :)
The directory is XFS. There is a single directory with 3 million files, which is where the hardlinks originate, but the rest are scattered through various subdirectories. I know it's not a good thing to have that many files in one directory, but it has grown over time and was never expected to get that large. The backup is local to the machine and is being done via rsync. The data and backup are on separate disks. This is a new system and these files were recently copied from another disk. A file-level rsync (rsync -aH /orig /new) did not have any problems copying the files and hard links (although it did take a day or two). > >> Is BackupPC going to be able to deal with this directory... > > Yes. Is the filesystem going to cause trouble? I don't know. > >> do I need to look for a different backup method? > > No, you need to find out what's going on. Make sure you're looking at > all the logs, and if there isn't enough information in the logs tell > them to collect more The problem is that the original backup took 2 weeks to fail with no indication of problems that I could see... it was just very slow. I posted a previous question about it on this list while it was running. I could not find any bottlenecks or problems. I'm reluctant to start it again without some idea of what I'm looking for. How would you suggest I go about collecting more info? Up the log level in BPC? Make rsync more verbose? Right now, the only error I've seen is the error that stopped the backup: rsync error: error in rsync protocol data stream (code 12) at io.c(1556) [generator=3.0.9.12] The main annoyance is that I have no way to track progress. While the backup is running, I can't tell if it's about to finish, or if it's bogged down and is likely to take a few more hours (or days). -- Bowie ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
