Hi Ian. This is not really that surprising. Unfortunately, moving large numbers of small files always seems to have this problem. As I see it, the problems stem from the time needed for per file transactional overhead at multiple layers, including the filesystem ,protocol(nfs, cifs), and network(connection setups and teardowns+slow start). Straight writing of bits is at each level is just plain easier. A streaming tar using netcat could save some on network connections and protocol operations, but not on the filesystem. There may be some filesystem options that you can tweak as well.
I have seen many pieces of software that claim to accelerate small file transfers, but haven't really found anything that is that great at it. rgt On 07/22/2010 01:25 PM, Ian Stokes-Rees wrote: > > > I have a question regarding expectations for file movement between > disks on adjacent servers. > > Due to a sub-optimal file system layout, I regularly have to move lots > of files between file systems. The servers are in the same rack, or at > least in racks next to each other, and I am fairly certain they are all > connected to the same GB switch. > > Moving blocks of ~300k files totaling about 5-10 GB takes hours to > complete. Yesterday afternoon I started a move of 1.3 million files > totaling about 94 GB. 20 hours later the transfer seems to be less than > half done. > > Does this surprise anyone? Any hints as to what might be wrong or what > might speed it up? I'm at a loss to know where to start looking. > > Regards, > > Ian > > > > More details, for those who are interested: > > The files at the origin are on RAID1 SATA disks (1 TB, ext3, Seagate > Barracuda 7200 RPM), and I have a ganglia snapshot of the 24 hour status > (you can see the start of the transfer about 20 hours ago): > > http://dl.dropbox.com/u/1561496/shared/abitibi-origin.pdf > > The destination is an Apple X-RAID array (4TB) connected to an Apple > XServe. The corresponding ganglia snapshot is here: > > http://dl.dropbox.com/u/1561496/shared/macintel-destination.pdf > > > > > _______________________________________________ > bblisa mailing list > [email protected] > http://www.bblisa.org/mailman/listinfo/bblisa _______________________________________________ bblisa mailing list [email protected] http://www.bblisa.org/mailman/listinfo/bblisa
