freebsd-performance  

Re: Migrate large amount of small files

Mike Tancsa
Sun, 11 Oct 2009 14:07:09 -0700

At 11:46 AM 10/10/2009, Ckcheng wrote:
Hi all,
Currently, I have a directory with over 5M small files (1~32K). Now,
I want to transfer this directory to another machine and found
that it's extremely slow and painful process. I tried the following
method:

It might help if you mount all -onoatime as well as bump up vfs.ufs.dirhash_maxmem to 4x the default size

        ---Mike

1. rsync
2. tar via ssh
3. tar via nc
(all take hours and hours to finish)

None of them is able to give me a reasonable migration time. So, I'm
here for asking help. Any suggestion is extremely welcomed. Thank you.

Btw, here is brief information of my server. (both machines are the same)

OS: FreeBSD 6.4-Stable 64Bit
CPU: 2 x Xeon L5420 2.50GHz
RAM: 2 x 2G ECC DDR2-667 (full buffered)
DISK: Seagate Barracuda ES 16MB (SATA 300)
Network: 1Gbps (Broadcom BCM5708)
Filesystem: UFS2

Regards,
_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscr...@freebsd.org"

--------------------------------------------------------------------
Mike Tancsa,                                      tel +1 519 651 3400
Sentex Communications,                            m...@sentex.net
Providing Internet since 1994                    www.sentex.net
Cambridge, Ontario Canada                         www.sentex.net/mike

_______________________________________________
freebsd-performance@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscr...@freebsd.org"