Re: UFS2 optimization for many small files
We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd be happy if anyone can suggest what does fragment (block/8) in the ufs2 mean and how this parameter works. I know It's better to read the full ufs2 specification, but hope that someone here can give a hint. Please advice with optimizations or tricks. Thank you very much. Try zfs on amd64 unless your app doesn't work well with zfs or your organization doesn't allow current. Current is remarkably stable taking into account zfs is fairly new and ported from solaris and running on current. I'm using it on a 8.2 TB nexsan storage and no crashes during testing and a limited time in production. Some years ago I used FreeBSD (5.2) as nfs-server (using ufs2) on approx. 15 partitions ranging from 400 GB to 2 TB in size. If the server for some reason had crashed the webservers were unable to access the nfs-mounted partitions during the period the server did a snapshot of a partition, in order to perform a background-fsck and thus our website was down. So ufs2 does not scale well. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
Nguyen Tam Chinh wrote: Greetings, We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd be happy if anyone can suggest what does fragment (block/8) in the ufs2 mean and how this parameter works. I know It's better to read the full ufs2 specification, but hope that someone here can give a hint. Please advice with optimizations or tricks. Thank you very much. If all else fails; try divide and conquer by having one filesystem per subdirectory. Unless you plan on having all files in a single dir?! Also look at how the squid proxy stores its files using a hashed dir structure. Another alternative is storing the data in a database if you don't need direct random RW file access.. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
approx. 15 partitions ranging from 400 GB to 2 TB in size. If the server for some reason had crashed the webservers were unable to the question is about the reason it crashed... access the nfs-mounted partitions during the period the server did a snapshot of a partition, in order to perform a background-fsck and thus our website was down. So ufs2 does not scale well. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
I have tried using a 4K/0.5K UFS1 filesystem in the past and found the performance was very poor. UFS2 was based on 16K/2K and I would expect it to perform even worse with 4K/0.5K. I would suggest you try 8K/1K. not for small files. you are light with large files but it's not THAT bad as you say. i reagularly use 4K/0.5 UFS but not for everything if i require good fast speed for big files. for really big files i make 32/4 filesystem with very little inodes ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd be happy if anyone can suggest what does fragment (block/8) in the ufs2 mean and how this parameter works. I know It's better to read the exactly as a block/cluster in windows. fragment is the smallest allocation block. block is a group of 8 fragments to make allocation faster and smarter. full ufs2 specification, but hope that someone here can give a hint. Please advice with optimizations or tricks. please DO NOT make single partition like that. try to divide it to 3-4 partitions. it will work on a single one but waiting for fsck will kill you ;) AFAIK fsck time grows nonlinearly with fs size to some extent.. options for newfs will be like that newfs -m A -i B -b 4096 -f 512 -U /dev/partition where A is space left. with mostly small files and huge partition don't worry to set it 1 or even 0. B - size of disk(bytes)/amount of inodes default is probably 2048, you may use 1024 or 4096 for your case - make rough estimate how much files will you have (you told between 4 and 0.5k, but what average?). making too much inodes=wasted space (128 bytes/inode), making too little=big problem :) another question - HOW do you plan to make backups of such data? with dump rsync tar etc. it's clearly mission impossible. feel free to mail me i had such cases not 5E8 but over 1E8 files :) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
Thank you very much. Try zfs on amd64 unless your app doesn't work well with zfs or your does zfs have RELIABLE and USABLE software allowing to efficiently backup large filesystems to other media? (DVD's, tapes, other hard discs) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
approx. 15 partitions ranging from 400 GB to 2 TB in size. If the server for some reason had crashed the webservers were unable to the question is about the reason it crashed... access the nfs-mounted partitions during the period the server did a snapshot of a partition, in order to perform a background-fsck and thus our website was down. So ufs2 does not scale well. Reasons not related to the nfs-server itself. FreeBSD itself was rock-solid. It was firmware-related on the storage-side. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
Try zfs on amd64 unless your app doesn't work well with zfs or your does zfs have RELIABLE and USABLE software allowing to efficiently backup large filesystems to other media? (DVD's, tapes, other hard discs) Zfs has send/receive where you can do snapshots and send them to a different host. This could be your backup-host. I'm considering this solution myself where FreeBSD and zfs is my primary host and my nightly backups will be send to my solaris-host. Solaris has the required lto-3-drivers. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
snapshot of a partition, in order to perform a background-fsck and thus our website was down. So ufs2 does not scale well. Reasons not related to the nfs-server itself. FreeBSD itself was rock-solid. It was firmware-related on the storage-side. i always use software mirror concat or both in FreeBSD. always works, 10 times cheaper, fully portable and (yes true) comparable in speed. in some cases - faster. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
On Wednesday, 27 June 2007 at 14:11:19 +0400, Nguyen Tam Chinh wrote: Greetings, We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd be happy if anyone can suggest what does fragment (block/8) in the ufs2 mean and how this parameter works. I know It's better to read the full ufs2 specification, but hope that someone here can give a hint. Please advice with optimizations or tricks. Thank you very much. -- With best regards,| The Power to Serve Nguyen Tam Chinh | http://www.FreeBSD.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED] I am not aware of any ZFS results on such tasks, may be you will be the one who share them ;) However RaiserFS whould be the best choise on such spesific case. It's not available on FreeBSD currently. I don't think UFS can handle a huge amount of small files effectively. Of course gjournal could be an option for fsck problems, but how do you plan to backup or sync this storage? -- == - Best regards, Nikolay Pavlov. --- == ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
On 7/2/07, Nikolay Pavlov [EMAIL PROTECTED] wrote:
On Wednesday, 27 June 2007 at 14:11:19 +0400, Nguyen Tam Chinh wrote:
Greetings,
We're going to build a server with some 1Tb of over 500 million small
files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle
this kind of system well. From newfs(8) the min block size is 4k. This
is not optimal in our case, a 1k or 0,5k block is more effective IMHO.
I'd be happy if anyone can suggest what does fragment (block/8) in the
ufs2 mean and how this parameter works. I know It's better to read the
full ufs2 specification, but hope that someone here can give a hint.
Please advice with optimizations or tricks.
Thank you very much.
--
With best regards,| The Power to Serve
Nguyen Tam Chinh | http://www.FreeBSD.org
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
I am not aware of any ZFS results on such tasks, may be you will be the
one who share them ;) However RaiserFS whould be the best choise on such
spesific case. It's not available on FreeBSD currently.
I don't think UFS can handle a huge amount of small files effectively.
Of course gjournal could be an option for fsck problems, but how do you
plan to backup or sync this storage?
I'm aware of the fsck/backup problems. In our case there's no need for
backup so i went with ufs2. The current configuration is 4x250Gb disks
with bloc/frag ratio 4k/512b. We're generating files with the average
size of 6k ('cause the compress procedure does not work as well as we
estimated).
After a week I think we could collect some statistics in production.
Anyway, in this case a 8k/1k would be more effective for us. Hope that
I can test this in the next server.
--
With best regards,| The Power to Serve
Nguyen Tam Chinh | http://www.FreeBSD.org
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: UFS2 optimization for many small files
On 2007-Jun-27 14:11:19 +0400, Nguyen Tam Chinh [EMAIL PROTECTED] wrote: We're going to build a server with some 1Tb of over 500 million small files with size from 0,5k to 4k. I'm wonder if the ufs2 can handle this kind of system well. Short answer: No. Longer answer: FreeBSD and UFS2 have been tweaked to support large numbers of files in larger filesystems and there are no hard limits that you will exceed by having 500,000,000 files in a 1TB FS. However, you will not be able to fsck the FS on an i386 system and will need a lot of RAM+SWAP on amd64 or SPARC64. fsck will also take a _long_ time (hours) to run. Depending on how the files are organised, you may run into severe performance problems with directory searching. From newfs(8) the min block size is 4k. This is not optimal in our case, a 1k or 0,5k block is more effective IMHO. I'd be happy if anyone can suggest what does fragment (block/8) in the ufs2 mean and how this parameter works. I suggest you read /usr/share/doc/smm/05.fastfs/paper.ascii.gz Whilst this paper discusses UFS1, the basics remain the same. I have tried using a 4K/0.5K UFS1 filesystem in the past and found the performance was very poor. UFS2 was based on 16K/2K and I would expect it to perform even worse with 4K/0.5K. I would suggest you try 8K/1K. BTW, in sizing your system, you will need to allow for both the last space when the file sizes are rounded up to a multiple of the fragment size, as well as the inode size (256 bytes). If you have 1TB of data, it's likely that you will have another 0.5-1TB of overheads. Overall, I suggest you look at an alternative way to store the data. -- Peter Jeremy pgpIAT77fjtea.pgp Description: PGP signature
Re: UFS2 optimization for many small files
Nguyen Tam Chinh wrote: [snipped] Please advice with optimizations or tricks. [...] Did you already looked at 'man 7 tuning'? HTH, Philipp -- www.familie-ost.info/~pj ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
