Re: [zfs-discuss] Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.
I have just (re)installed FreeBSD amd64 current with gcc 4.2 with src from May. 21'st on a dual Dell PE 2850. Does the post-gcc-4-2 current include all your zfs-optimizations? I have commented out INVARIANTS, INVARIANTS_SUPPORT, WITNESS and WITNESS_SKIPSPIN in my kernel and recompiled with CPUTYPE=nocona. A default install solaris fares better io-wise compared to a default FreeBSD where writes could pass 100 MB/s (zpool iostat 1) and FreeBSD would write 30-40 MB/s. After adding the following to /boot/loader.conf writes peak at 90-95 MB/s: vm.kmem_size_max=2147483648 vfs.zfs.arc_max=1610612736 Now FreeBSD seems to perfom almost as good as solaris io-wise although I don't have any numbers to justify my statement. I did not import postgresql in solaris as one thing. This patch also improve concurrency in VFS: http://people.freebsd.org/~pjd/patches/vfs_shared.patch I applied the patch and it seems to speed up my reads and writes. Watching zpool iostat I saw reads at 155 MB/s and writes at 111 MB/s. But it also seems to introduce some minor complete stops accessing the zpool lasting for 10-20 secs. I apologize I'm not very specific but I only had time to test disk-io but not digg into the issues. When you want to operate on mmap(2)ed files, you should disable ZIL and remote file systems: # sysctl vfs.zfs.zil_disable=1 # zpool export name # zpool import name Won't disabling ZIL minimize the chance of a consistent zfs-filesystem if - for some reason - the server did an unplanned reboot? -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.
Won't disabling ZIL minimize the chance of a consistent zfs- filesystem if - for some reason - the server did an unplanned reboot? ZIL in ZFS is only used to speed-up various workloads, it has nothing to do with file system consistency. ZFS is always consistent on disk no matter if you use ZIL or not. But it can cause NFS client corruption and you no longer gets synchronous write semantics (see if your app depend on that): http://blogs.sun.com/erickustarz/entry/zil_disable I highly recommend *against* setting zil_disable. Disabling zil did improve the postgresql-import from 1 h. 45 min. to 1. h. 35 min. I get a 10 min. speedup but as the link point out disabling has it's disadvantages. So I'll revert to the old setting. -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.
On Thu, May 24, 2007 at 11:20:44AM +0100, Darren J Moffat wrote: Pawel Jakub Dawidek wrote: Hi. I'm all set for doing performance comparsion between Solaris/ZFS and FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I think I'm ready. The machine is 1xQuad-core DELL PowerEdge 1950, 2GB RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the links to disks are the bottleneck, so I'm going to use not more than 4 disks, probably. I do know how to tune FreeBSD properly, but I don't know much about Solaris tunning. I just upgraded Solaris to: SunOS lab14.wheel.pl 5.11 opensol-20070521 i86pc i386 i86pc I took upgrades from: http://dlc.sun.com/osol/on/downloads/current/ I believe this is a version with some debugging options turned on. How can I turn debug off? Can I or do I need to install something else? What other tunnings should I apply? Don't install from bfu archives instead install a Solaris Express directly from a DVD image. Or if you do want to use bfu because you really want to match your source code revisions up to a given day then you will need to build the ON consolidation yourself and you an the install the non debug bfu archives (note you will need to download the non debug closed bins to do that). Easiest way is to just use a DVD install. Ha, I originally installed from sol-nv-b55b-x86-dvd-iso-[a-e].zip, but then upgraded to OpenSolaris. -- Pawel Jakub Dawidek http://www.wheel.pl [EMAIL PROTECTED] http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! pgpi5q9PL2OoS.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.
I'm all set for doing performance comparsion between Solaris/ZFS and FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I think I'm ready. The machine is 1xQuad-core DELL PowerEdge 1950, 2GB RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the links to disks are the bottleneck, so I'm going to use not more than 4 disks, probably. I do know how to tune FreeBSD properly, but I don't know much about Solaris tunning. I just upgraded Solaris to: I have just (re)installed FreeBSD amd64 current with gcc 4.2 with src from May. 21'st on a dual Dell PE 2850. Does the post-gcc-4-2 current include all your zfs-optimizations? I have commented out INVARIANTS, INVARIANTS_SUPPORT, WITNESS and WITNESS_SKIPSPIN in my kernel and recompiled with CPUTYPE=nocona. A few weeks ago I installed FreeBSD but it panicked when I used iozone. So I installed solaris 10 on this box and wanted to keep it that way. But solaris lacks FreeBSD ports ;-) so when current upgraded gcc to 4.2 I re-installed FreeBSD and the box is so far very stable. I have imported a 3.9 GB compressed postgresql dump five times to tune io-performance, have copied 66 GB of data from another server using nfs, installed 117 packages from the ports-collection and it's *very* stable. A default install solaris fares better io-wise compared to a default FreeBSD where writes could pass 100 MB/s (zpool iostat 1) and FreeBSD would write 30-40 MB/s. After adding the following to /boot/loader.conf writes peak at 90-95 MB/s: vm.kmem_size_max=2147483648 vfs.zfs.arc_max=1610612736 Now FreeBSD seems to perfom almost as good as solaris io-wise although I don't have any numbers to justify my statement. I did not import postgresql in solaris as one thing. Copying the 3.9 GB dump from $HOME to a subdir takes 1 min. 13 secs. which is approx. 55 MB/s. Reads peaked at 115 MB/s. The storage is a atabeast with two raid-controllers connected via two qlogic 2300 hba's. Each controller have four raid5-arrays with five 400 GB disks each. zetta~#zpool status pool: disk1 state: ONLINE scrub: scrub completed with 0 errors on Thu May 24 21:39:46 2007 config: NAMESTATE READ WRITE CKSUM disk1 ONLINE 0 0 0 raidz1ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 raidz1ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 errors: No known data errors The atabeast is not the fastest storage-provider around but on this machine will primarily be a file- and mail-server. Are there any other tunables on FreeBSD I can look at? -- regards Claus When lenity and cruelty play for a kingdom, the gentlest gamester is the soonest winner. Shakespeare ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.
On Thu, May 24, 2007 at 01:16:32PM +0200, Claus Guttesen wrote: I'm all set for doing performance comparsion between Solaris/ZFS and FreeBSD/ZFS. I spend last few weeks on FreeBSD/ZFS optimizations and I think I'm ready. The machine is 1xQuad-core DELL PowerEdge 1950, 2GB RAM, 15 x 74GB-FC-10K accesses via 2x2Gbit FC links. Unfortunately the links to disks are the bottleneck, so I'm going to use not more than 4 disks, probably. I do know how to tune FreeBSD properly, but I don't know much about Solaris tunning. I just upgraded Solaris to: I have just (re)installed FreeBSD amd64 current with gcc 4.2 with src from May. 21'st on a dual Dell PE 2850. Does the post-gcc-4-2 current include all your zfs-optimizations? I have commented out INVARIANTS, INVARIANTS_SUPPORT, WITNESS and WITNESS_SKIPSPIN in my kernel and recompiled with CPUTYPE=nocona. A few weeks ago I installed FreeBSD but it panicked when I used iozone. So I installed solaris 10 on this box and wanted to keep it that way. But solaris lacks FreeBSD ports ;-) so when current upgraded gcc to 4.2 I re-installed FreeBSD and the box is so far very stable. I have imported a 3.9 GB compressed postgresql dump five times to tune io-performance, have copied 66 GB of data from another server using nfs, installed 117 packages from the ports-collection and it's *very* stable. A default install solaris fares better io-wise compared to a default FreeBSD where writes could pass 100 MB/s (zpool iostat 1) and FreeBSD would write 30-40 MB/s. After adding the following to /boot/loader.conf writes peak at 90-95 MB/s: vm.kmem_size_max=2147483648 vfs.zfs.arc_max=1610612736 Now FreeBSD seems to perfom almost as good as solaris io-wise although I don't have any numbers to justify my statement. I did not import postgresql in solaris as one thing. Copying the 3.9 GB dump from $HOME to a subdir takes 1 min. 13 secs. which is approx. 55 MB/s. Reads peaked at 115 MB/s. The storage is a atabeast with two raid-controllers connected via two qlogic 2300 hba's. Each controller have four raid5-arrays with five 400 GB disks each. zetta~#zpool status pool: disk1 state: ONLINE scrub: scrub completed with 0 errors on Thu May 24 21:39:46 2007 config: NAMESTATE READ WRITE CKSUM disk1 ONLINE 0 0 0 raidz1ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 raidz1ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 errors: No known data errors The atabeast is not the fastest storage-provider around but on this machine will primarily be a file- and mail-server. Are there any other tunables on FreeBSD I can look at? There is probably not much you can do to tune sequential I/Os. I'd suggest starting investigation from benchmarking drivers on both systems, by using raw disks (without ZFS). There are some other things you could try to improve different workloads. To improve concurrency you should use shared locks for VFS lookups: # sysctl vfs.lookup_shared=1 This patch also improve concurrency in VFS: http://people.freebsd.org/~pjd/patches/vfs_shared.patch When you want to operate on mmap(2)ed files, you should disable ZIL and remote file systems: # sysctl vfs.zfs.zil_disable=1 # zpool export name # zpool import name I think ZIL should be dataset property, as differences depending on the workload are huge. For example fsx test is like 15 _times_ faster when ZIL is disabled. There are still some things to optimize, like using UMA for memory allocations, but we run out of KVA too fast then. Benchmarking file system is not easy, as there are other subsystems involved, like namecache or VM. fsstress test, which mostly operates on metadata (creates, removes files, directories, renames them, etc.) is 3 times faster on FreeBSD/ZFS than on Solaris/ZFS, but I believe it's mostly because of namecache implementation. Solaris guys should seriously look at improving DNLC or replacing it. Another possibility is VFS, but Solaris VFS is much cleaner, and I somehow don't believe it's slower. fsx is about 20% faster on FreeBSD, this could be VM's fault. Don't take this numbers too seriously - those were only first tries to see where my port is and I was using OpenSolaris for comparsion, which has debugging turned on. -- Pawel Jakub Dawidek http://www.wheel.pl [EMAIL PROTECTED] http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! pgpylZji1JT6R.pgp Description: PGP signature
Re: [zfs-discuss] Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.
On Thu, May 24, 2007 at 01:16:32PM +0200, Claus Guttesen wrote: iozone. So I installed solaris 10 on this box and wanted to keep it that way. But solaris lacks FreeBSD ports ;-) so when current upgraded Not entirely. :) I don't know about FreeBSD PORTS, but NetBSD's ports system works very well on solaris. The only thing I didn't like about it is it considers gcc a dependency to certain things, so even though I have Studio 11 installed, it would insist on installing gcc, which kinda irritated me. :) -brian -- Perl can be fast and elegant as much as J2EE can be fast and elegant. In the hands of a skilled artisan, it can and does happen; it's just that most of the shit out there is built by people who'd be better suited to making sure that my burger is cooked thoroughly. -- Jonathan Patschke ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.
On 24/05/07, Brian Hechinger [EMAIL PROTECTED] wrote: I don't know about FreeBSD PORTS, but NetBSD's ports system works very well on solaris. The only thing I didn't like about it is it considers gcc a dependency to certain things, so even though I have Studio 11 installed, it would insist on installing gcc, which kinda irritated me. :) pkgsrc is ok, but I found it got very messy very quickly once you get a reasonably sized dependency tree. portugrade. FreeBSD portupgrade (a layer on top of ports, but easy to install) handled that much better. To be fair I haven't used NetBSD in a couple of years - but pkgsrc was the reason I left :) -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Preparing to compare Solaris/ZFS and FreeBSD/ZFS performance.
Don't take this numbers too seriously - those were only first tries to see where my port is and I was using OpenSolaris for comparsion, which has debugging turned on. Yeah, ZFS does a lot of extra work with debugging on (such as verifying checksums in the ARC), so always do serious performance comparisons with non-debug bits. Nice to see ZFS performing quite well on FreeBSD! eric ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
