Les Mikesell wrote at about 12:35:49 -0500 on Monday, August 31, 2009: > Jeffrey J. Kosowsky wrote: > > > > > It's almost as if you guys haven't heard of filesystem-specific dump > > > utilities. For such utils (vxdump, ufsdump, zfs send/receive, etc.) > > the > > > number of hardlinks isn't a problem. You can do both full and > > > incremental dumps, even across separate machines. This isn't a problem > > > that needs solving. > > > > I think you are missing some key points. > > First, why should a program require it's own separate filesystem? This > > seems to me like an unnatural and kludgey type of requirement. > > It doesn't, unless you hit some limit on the filesystem you use. The > usual limit is how many times and how far you have to move the disk head > since that's the slow operation. > > > I see lots of advantage in keeping the database portion relatively > > small, fast, replicable, and moveable. Then you can keep and > > distribute the files themselves wherever you want them spread across > > one or more separate filesystems. Then the database portion is > > optimized for what a database does best and the file-storing > > portion can be optimized for what a filesystem stores best. And both > > parts are easily moveable, replicable and not dependent or limited by > > hardlinks or other filesystem-dependent functionality. > > But the parts aren't independent. How do you propose keeping them in > sync or fixing them when they inevitably differ?
Perhaps analogously to the way BackupPC_Nightly now makes sure that pool is in synch. More generally, we would need to consider two things: 1. What are the normal ways in which the two could get out of synch and then address each of those cases 2. If necessary, create a repair tool (similar to what I created for the current system) if something breaks in non-standard ways (e.g., due to crashes, disk failures, etc.) I guess I can't answer your question without knowing what use cases you are worried about. > > > Don't get me wrong - Backuppc is great and hardlinks are a great > > kludge to at first glance get something for nothing. I'm just saying > > that hardlinks while "easy" bring some longer-term limitations and > > that there comes the time when it may be worth investing in going > > beyond them. > > > > Personally I would like to see Backuppc evolve to combine the pooling > > functionality, leveraging of rsync, and relative simplicity of the > > existing Backuppc with the expandability, portability, and flexibility > > of the database-based systems like Bacula. I believe that the > > combination of a database to store the file attributes and metadata > > together with a filesystem to store the pool would be an ideal hybrid. > > But someone has yet to establish that this would be faster if you don't > add the requirement of putting the sql tables on different drives - and > weren't you just saying that applications shouldn't have requirements > like that? > > > > > > > For anyone thinking that working with giant multi-gigabyte BLOBs in a > > > database is the right way to go, I suggest you actually attempt it > > > yourself and see what happens. I'm backing up my HD video production > > > rig with BackupPC, and although such a machine (Windows, 16T of > > storage, > > > most video files are at least 50G in size) is outside of the intent of > > > BackupPC, it actually works. If BackupPC were to rely on an SQL > > > database, it would greatly shrink the potential userbase. > > > > You are attacking a straw man. No one has ever suggested > > "multi-gigabyte BLOBS in a database." The database would only consist > > of the filenames, links, attrib data, and other backup-related metadata. I > > would imagine in most cases this would be at most a couple of > > gigabytes, assuming you have millions of files in your pool. > > If you don't put them in the database you can't enforce atomic > operations - something you get in the filesystem for the price of a seek > over to the inode to bump the link count with the entry locked. > > -- > Les Mikesell > [email protected] > > > > ------------------------------------------------------------------------------ > Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day > trial. Simplify your report design, integration and deployment - and focus > on > what you do best, core application coding. Discover what's new with > Crystal Reports now. http://p.sf.net/sfu/bobj-july > _______________________________________________ > BackupPC-users mailing list > [email protected] > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users > Wiki: http://backuppc.wiki.sourceforge.net > Project: http://backuppc.sourceforge.net/ > ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ BackupPC-users mailing list [email protected] List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
