> In normal circumstances, databases are more efficient at > handling lookups than filesystems. > > In your image application database, use a timestamp field > that is updated whenever images are added or updated. > > Generate your backup jobs based on queries to this database > instead of requiring rsync to do its differencing thing. For > example you can automate a process that queries the database > for images that have been updated or added since the last > time it ran and generate a file list or backup job that only > copies over new or updated images based on the timestamp. > You would have to somehow map within the database the actual > physical location of the files if you are not already doing > it, in addition to using squid/apache to translate to the client. > > That is the first step.
MIkey's right here. You cannot expect your filesystem to be able to return the query results you need. You need to take this "out of band". You could even store your data base on a separate filesystem so you don't use I/O in the disk array that you need for the backups. > The second step is to ditch storing everything on a single > 9TB system that cannot be backed up efficiently. Distribute > the storage of the images on clusters or whatever. For > example peel of 1TB of images onto a single server, then > update the database (or apache/squid mapping) to point to the > new location. 9 1TB boxes would be far less prone to > catastrophic failure and much easier to > replicate/mirror/backup than a single 9TB box. This is what > I call the "google approach" ;) Use cheap commodity hardware > and smart implementation to distribute/scale the load. Many years ago, I was at a Progress database conference and one very useful presentation was about the effect on performance of a large data store without increasing the bandwidth available to that data store. The speaker's example showed how your performance decreases when you have one large database but still only a single channel for access. His point was to increase the number of channels along with the size of the store, otherwise you actually lose performance. This is tantamount to MIkey's discussion above. Spread out your disks if possible. Your problem is that you cannot get more channels into your backup store, so you'll have to think about either a separate local backup SAN or a provider with more bandwidth. Bill -- [email protected] mailing list
