Re: [zfs-discuss] ZFS + OpenSolaris for home NAS?
So I have just finished building something similar to this... I'm finally replacing my Pentium II 400Mhz fileserver! My setup is: Opensolaris 2008.11 http://www.newegg.com/Product/Product.aspx?Item=N82E16813138117 http://www.newegg.com/Product/Product.aspx?Item=N82E16820145184 http://www.newegg.com/Product/Product.aspx?Item=N82E16819103255 http://www.newegg.com/Product/Product.aspx?Item=N82E16811129025 (there is no power supply here I had one of those too) I ended up paying $278 The case and everything makes it super quiet, and it runs very cool. Still have a power test to run and see but I'm guessing 125 - 150 watts. I had hard drives available, and using a USB DVD/CD drive for installation I am using 2 ATA drives mirrored for the root pool I am using 6 500GB (hitachi) sata drives in raidz for data with the onboard interface. The initial iozone test I ran, under ideal block sizes etc, I could sustain enough to saturate the 1GB interface. Real world, I haven't been able to see what my real utilization is. I use this mostly to store media (movies, mp3s, etc) which I play with a mythtv box (just using NFS as the share). Also planning to use this as a backup server for the other machines in the house (considering running Amanda at home for this) I'm usually not doing heavy file I/O, so I've also put a virtual machine on here with virtual box for casual use for other stuff. During the process I swapped the whole thing to ubuntu on a usb flash and ran the opensolaris fileserver as a VM with VMWare...but I ended up just not happy with that in the end. The usb flash I had was OK, but it's not a hard drive. With opensolaris, I can still play flash stuff from Hulu fullscreen in HD with the onboard video. I've been trying to remember why I still need windows for a desktop and have been debating just killing that PC as well. Most of the things that I used windows for (ripping a DVD, etc) can be done with a VM with virtualbox...it's slow, but it works. I also don't re-encode video at all. I don't know if mythtv will actually run on opensolaris...that would be cool if it did. As far as a fileserver that has capacity to do other stuff, I'm very happy with this setup. As for the person who suggested a Sun x4150. Those are extremely loud because of the fans. There is no way I could run one of those at my house because of that. It was designed for a datacenter (and I have run them there). -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Performance bake off vxfs/ufs/zfs need some help
So to give a little background on this, we have been benchmarking Oracle RAC on Linux vs. Oracle on Solaris. In the Solaris test, we are using vxvm and vxfs. We noticed that the same Oracle TPC benchmark at roughly the same transaction rate was causing twice as many disk I/O's to the backend DMX4-1500. So we concluded this is pretty much either Oracle is very different in RAC, or our filesystems may be the culprits. This testing is wrapping up (it all gets dismantled Monday), so we took the time to run a simulated disk I/O test with an 8K IO size. vxvm with vxfs we achieved 2387 IOPS vxvm with ufs we achieved 4447 IOPS ufs on disk devices we achieved 4540 IOPS zfs we achieved 1232 IOPS The only zfs tunings we have done are setting set zfs:zfs_nocache=1 in /etc/system and changing the recordsize to be 8K to match the test. I think the files we are using in the test were created before we changed the recordsize, so I deleted them and recreated them and have started the other test...but does anyone have any other ideas? This is my first experience with ZFS with a comercial RAID array and so far it's not that great. For those interested, we are using the iorate command from EMC for the benchmark. For the different test, we have 13 luns presented. Each one is its own volume and filesystem and a singel file on those filesystems. We are running 13 iorate processes in parallel (there is no cpu bottleneck in this either). For zfs, we put all those luns in a pool with no redundancy and created 13 filesystems and still running 13 iorate processes. we are running Solaris 10U6 -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Performance bake off vxfs/ufs/zfs need some help
that should be set zfs:zfs_nocacheflush=1 in the post above...that was my typo in the post. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Performance bake off vxfs/ufs/zfs need some help
zfs with the datafiles recreated after the recordsize change was 3079 IOPS So now we are at least in the ballpark. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Performance bake off vxfs/ufs/zfs need some help
Right now we are not using Oracle...we are using iorate so we don't have separate logs. When the testing was with Oracle the logs were separate. This test represents the 13 data luns that we had during those test. The reason it wasn't striped with vxvm is that the original comparison test was vxvm + vxfs compared to Oracle RAC on linux with ocfs. On the linux side we don't have a volume manager, so the database has to do the striping across the separate datafiles. The only way I could mimic that with zfs would be to create 13 separate zpools and that sounded pretty painful. Again, the thing that led us down this path was the the Oracle RAC on Linux accompished slightly more transactions but only required 1/2 the I/O's to the array to do so. The Sun test, actually bottlenecked on the backend disk and had plenty of CPU left on the host. So if the I/O bottleneck is actually the vxfs filesystem causing more I/O to the backend, and we can fix that with a different filesystem, then the Sun box may beat the Linux RAC. But our initial testing has shown that vxfs is all it's cracked up to be with respect to databases (yes we tried the database edition too and the performance actually got slightly worse). -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] OpenStorage GUI
Do you have any info on this upgrade path? I can't seem to find anything about this... I would also like to throw in my $0.02 worth that I would like to see the software offered to existing sun X4540 (or upgraded X4500) customers. Chris G. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS + OpenSolaris for home NAS?
I've been looking at this board myself for the same thing The blog below is regarding the D945GCLF but looking at the two, it looks like the processor is the only thing that is different (single core vs. dual core). http://blogs.sun.com/PotstickerGuru/entry/solaris_running_on_intel_atom -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] volname
iostat -En will show you device and serial number (at least on the Sun hardware I've used). My old way of finding drives is to run format and run a disk read test that isn't destructive and look for the drive with the access light going crazy. Most drives still have a very small LED on them to reflect activity. You may or may not be able to see it. Chris G. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Strategies to avoid single point of failure w/ X45x0 Servers?
If you are having touble booting to the mirrored drive, the following is what
we had to do to correctly boot off the mirrored drive in a Thumper mirrored
with disksuite. The root drive is c5t0d0 and the mirror is c5t4d0. The BIOS
will try those 2 drives.
Just a note, if it ever switches to c5t4d0 as the primary boot device, the BIOS
will not swap back autoamtically. You will have to change this back by hand in
the BIOS. Mostly likely you'll find this out on your next OS upgrade when you
upgrade c5t0d0 only to still be booting off c5t4d0.
The vtoc.out file was created with prtvtoc on the primary root drive.
and the partitions were added to SVM and synced one by one in a for loop.
MIRROR=c5t4d0
echo y | /usr/sbin/fdisk -B /dev/rdsk/${MIRROR}s2
echo 5 | /usr/sbin/fdisk -b /usr/lib/fs/ufs/mboot /dev/rdsk/${MIRROR}p0
/usr/sbin/fmthard -s /tmp/disk-vtoc.out /dev/rdsk/${MIRROR}s2
/sbin/installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/${MIRROR}s0
--
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] An slog experiment (my NAS can beat up your NAS)
I was using EMC's iorate for the comparison. ftp://ftp.emc.com/pub/symm3000/iorate/ I had 4 processes running on the pool in parallel do 4K sequential writes. I've also been playing around with a few other benchmark tools (i just had results from other storage test with this same iorate test). -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] An slog experiment (my NAS can beat up your NAS)
I currently have a traditional NFS cluster hardware setup in the lab (2 host with FC attached JBOD storage) but no cluster software yet. I've been wanting to try out the separate ZIL to see what it might do to boost performance. My problem is that I don't have any cool SSD devices, much less ones that I could have shared between two host. Commercial arrays have custom hardware with mirrored cache which got me thinking about a way to do this with regular hardware. So I tried this experiment this week... On each host (OpenSolaris 2008.05), I created an 8GB ramdisk with ramdiskadm. I shared this ramdisk on each host via the iscsi target and initiator over a 1GB crossconnect cable (jumbo frames enabled). I added these as mirrored slog devices in a zpool. The end result was a pool that I could import and export between host, and it can survive one of the host dying. I also copied a dd image of my ramdisk device to stable storage with the pool exported (thus flushed), which allowed me to shut the entire cluster down, and power 1 node up, recreate the ramdisk and dd the image back and re-import the pool. I'm not sure I could survive a crash of both nodes, going to try and test some more. The big thing here is I ended up getting a MASSIVE boost in performance even with the overhead of the 1GB link, and iSCSI. The iorate test I was using went from 3073 IOPS on 90% sequential writes to 23953 IOPS with the RAM slog added. The service time was also significantly better than the physical disk. It also boosted the reads significantly and I'm guessing this is because of updating the access time on the files was completely cached. So what are the downsides to this? If both nodes were to crash and I used the same technique to recreate the ramdisk I would lose any transactions in the slog at the time of the crash, but the physical disk image is still in a consistent state right (just not from my apps point of view)? Anyone have any idea what difference infiniband might make for the cross connect? In some test, I did completely saturate the 1GB link between the boxes. So is this idea completely crazy? It also brings up questions of correctly sizing your slog in relation to the physical disk on the backend. It looks like if the ZIL can handle significantly more I/O than the physical disk the effect will be short lived as the system will have to slow things down as it spends more time flushing from the slog to physical disk. The 8GB looked like overkill in my case, because in a lot of the test, it drove the individual disk in the system to 100% and was causing service times on the physical disk in the 900 - 1000ms range (although my app never saw that because of the slog). -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Cheap Array Enclosure for ZFS pool?
We've used enclosures manufactured by Xyratex (http://www.xyratex.com/). Several RAID vendors have used these disk in their systems. One reseller is listed below (the one we used got bought out). I've been very happy with these enclosures and a Qlogic HBA. As we have retired some of the RAID arrays I'm redeploying the drives in ZFS pools. We have the fiber channel and the SATA versions of their enclosure and both work well. http://pacdata.com/index.php?page=766437.txt This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
