Hi there, On Fri, 6 Jul 2018, Bowie Bailey wrote:
I am trying to backup a large directory tree with BackupPC v4.? This directory is 660GB and contains over 25 million files with about 3 million hard links.? The initial backup ran for 2 weeks before dying with an rsync error.? It is showing as a partial backup, but it doesn't show a file count.
This is not an unusually large task for BackupPC. I routinely back up directories with similar volumes of contents, but there are caveats. My 'home' directory on my desktop machine at work for example is half that size, and just over a million files, although it contains only a few thousand hard links. Using rsync over a 100Mbit/s Ethernet link between a couple of 2.4GHz dual Opteron machines, each with 8-16GBytes RAM, I don't recall ever being surprised or disappointed by the time it took to complete a backup. The first backup was 376G in ~1M files, and took nearly 16 hours. Recently I added another user's home directory to the pool. To back up 530GBytes of fresh data in 1.06M files between the same machines took 23.5 hours for the first pass. A full backup with a few tens of megabytes of new files takes half a day, an incremental takes only ten to 25 minutes depending mostly on the numbers of new files. Your files average 26.4kBytes, my new user's average ~500kBytes, you can expect some differences because of that. The problem description isn't very clear about the structure of the data, and doesn't mention the type(s) of filesystem involved, CPU/RAM, the transport mechanism(s). For example do you have huge numbers of files at a single directory level? Putting even tens of thousands of files, let alone millions, in a single ext[234] directory is likely to cause performance problems. Are you using a 56k modem? :)
Is BackupPC going to be able to deal with this directory...
Yes. Is the filesystem going to cause trouble? I don't know.
do I need to look for a different backup method?
No, you need to find out what's going on. Make sure you're looking at all the logs, and if there isn't enough information in the logs tell them to collect more. -- 73, Ged. ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
