Jim Leonard wrote at about 21:18:16 -0500 on Sunday, August 30, 2009: > dan wrote: > > One of the biggest concerns with backuppc that is constantly discussed > > on this list is syncing the backup data between two or more servers. > > Simply reducing the file count by eliminating the hardlinks would allow > > rsync to be used reliably and effectively. > > It's almost as if you guys haven't heard of filesystem-specific dump > utilities. For such utils (vxdump, ufsdump, zfs send/receive, etc.) the > number of hardlinks isn't a problem. You can do both full and > incremental dumps, even across separate machines. This isn't a problem > that needs solving.
I think you are missing some key points. First, why should a program require it's own separate filesystem? This seems to me like an unnatural and kludgey type of requirement. Also, unless you use LVM, then you have difficulties growing or moving the filesystem -- and even so it always seems heavy-handed to me to have to worry about adding LVM volumes. Plus, as you add LVM volumes, you become more like a huge JBOD with the corresponding risk of corruption. I see lots of advantage in keeping the database portion relatively small, fast, replicable, and moveable. Then you can keep and distribute the files themselves wherever you want them spread across one or more separate filesystems. Then the database portion is optimized for what a database does best and the file-storing portion can be optimized for what a filesystem stores best. And both parts are easily moveable, replicable and not dependent or limited by hardlinks or other filesystem-dependent functionality. Don't get me wrong - Backuppc is great and hardlinks are a great kludge to at first glance get something for nothing. I'm just saying that hardlinks while "easy" bring some longer-term limitations and that there comes the time when it may be worth investing in going beyond them. Personally I would like to see Backuppc evolve to combine the pooling functionality, leveraging of rsync, and relative simplicity of the existing Backuppc with the expandability, portability, and flexibility of the database-based systems like Bacula. I believe that the combination of a database to store the file attributes and metadata together with a filesystem to store the pool would be an ideal hybrid. > > I feel like the whole "we need an SQL/hybrid" solution discussion is > happening because you aren't aware of better ways to do things. Just > because a filesystem is a database doesn't mean it would be "better" to > replace it with a "better" database. Well, did you ever consider that maybe there is a reason that people keep returning to the same issues and are not satisfied with the current Backuppc approach? One issue is the pool replication issue that you cite and many of us are not satisfied with the answer of "just use ZFS" or "just block copy the filesystem". Another equally important limitation is that expanding the attrib files to include Linux extended attributes or Windows ACLs (or any ACLs) is kludgey which is probably why it hasn't been done. And I don't see how you can have a complete backup solution if you can't back up all the associated file attributes. > > For anyone thinking that working with giant multi-gigabyte BLOBs in a > database is the right way to go, I suggest you actually attempt it > yourself and see what happens. I'm backing up my HD video production > rig with BackupPC, and although such a machine (Windows, 16T of storage, > most video files are at least 50G in size) is outside of the intent of > BackupPC, it actually works. If BackupPC were to rely on an SQL > database, it would greatly shrink the potential userbase. You are attacking a straw man. No one has ever suggested "multi-gigabyte BLOBS in a database." The database would only consist of the filenames, links, attrib data, and other backup-related metadata. I would imagine in most cases this would be at most a couple of gigabytes, assuming you have millions of files in your pool. > -- > Jim Leonard ([email protected]) http://www.oldskool.org/ > Help our electronic games project: http://www.mobygames.com/ > Or check out some trippy MindCandy at http://www.mindcandydvd.com/ > A child borne of the home computer wars: http://trixter.wordpress.com/ > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus > on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > BackupPC-users mailing list > [email protected] > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ BackupPC-users mailing list [email protected] List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
