On Mon, Jun 27, 2005 at 05:49:40PM -0400, Jeffrey Hutzelman wrote: > > > On Monday, June 27, 2005 04:03:10 PM -0500 Troy Benjegerdes > <[EMAIL PROTECTED]> wrote: > > >Has anyone looked into making volserver/vos dump go any faster? Is there > >any way to coalesce multiple small files into one RPC call, or > >pre-buffer files from disk? > > This is not about multiple RPC's. A volume dump is transferred as part of > one large RPC.
Is there any analog to the '-chunksize' argument to afsd? I'd guess not if it's one big rpc. > >I'm seeing around an average of 10MB/sec, and performance goes in the > >toilet when the volume has lots of small files. > > Volume dumps are created on the fly. Especially with lots of small files, > much of the data is metadata emitted by the volserver based on the contents > of the vnode index, rather than actual file data. There will also be > delays related to opening files and reading their contents, which will be > more prominent when lots of small files are involved. The test volume was approximately 1GB.. which fits fully in the buffer cache on the (relatively unloaded) server machine I was testing, so this is in no way I/O bound. I'd like to get to somewhere near 50MB/sec (which is about half of the throughput gigabit ethernet can support).. I'm guessing it's time for some oprofile runs.. > >This has basically prevented vos backup/etc from ever being able to go > >fast enough to get my DLT8000 drive to stream, and I think probably > >prematurely wore out one of my drives. > > While volume dump performance is slow and certainly something we'd like to > improve, it is unreasonable to expect the volserver to keep up a transfer > rate sufficient for tape streaming given variability in network > performance, fileserver load, and other factors. You should consider a > backup system which spools volume dumps to disk, and writes spooled dumps > to tape as part of a separate process which is not tightly coupled to the > volume dump process. How is amanda-afs integration these days? When I first set up my cell I used the integrated openafs backup system because it didn't appear anything else had a good way to deal with AFS ACL's, and I didn't feel like paying for some big and complicated enterprise backup system that I didn't understand. -------------------------------------------------------------------------- Troy Benjegerdes 'da hozer' [EMAIL PROTECTED] Somone asked my why I work on this free (http://www.fsf.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer: "Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life." -- Charles Shultz _______________________________________________ OpenAFS-devel mailing list [email protected] https://lists.openafs.org/mailman/listinfo/openafs-devel
