I thought this might be useful for others dipping their toes into SmartOS
Installing Smartos with raidz2 or 3 ------------------------------ Background: I have been running USB drives off a Linux Xen/KVM setup for many years. This year on the advice of my attorney (long story) I sold my motorcycle and had some extra cash so I decided to build a software RAID based system. I fully intended to stay with Centos and MD when I found out about Smartos. This looked intriguing since its design was similar to ESXi with software RAID. I was a bit tentative going down this road since I no nothing about Solaris and it meant learning one more thing. This drove me to go with Centos 7 with ZFS. I moved off this due to one major and one minor issue. At this time libvirt on Linux does not directly support VMs running on ZFS. You have to create a block device on ZFS, present it to Linux, format it as XFS, ext4, or whatever is your FS choice. You can then drop your VMs on this files system. The second and minor issue is Centos has a bug and doesn’t currently support Logitec wireless keyboards and mice. You can get around this by using one of the EPEL kernels but getting ZFS to work on it is a chore. So these issues drove me to SmartOS. I am by no means an expert, if you have comments on how to do this in a simpler manner please chime in As any good hacker does I started my Google searches, read, and came up with a plan. These are the articles I am basing this Howto on. Each article had good information but none had the step by step I was looking for. - Aaron Toponce’s intro to ZFS on Linux <https://pthree.org/2012/04/17/install-zfs-on-debian-gnulinux/> - Sam Smoot’s how to install raidz2, 3 on SmartOS Youtube video <http://www.youtube.com/watch?v=ZLcJb1u0yKQ> - Christopher Houges’ Wed, 11 Sep 2013 20:29:13 -0700 <http://tinyurl.com/zpool-change-smos> SmartOS-discuss list-serve post (compression) - Nathan Coad’s 19 Oct 14 Blog post <http://blog.nathancoad.com/item/2014/10/19/> on sharing rpool with L2ARC (understanding disk partitioning in Solaris) Needed - System running Intell processor with VT extentions (any recent i3 and above should work) - Disks that will make up your raidz array - 2 scratch disks, 1 for the initial setup and at least 1 for a dump device - if you are interested in SLOG and L2ARC cache have 2 disks available - SmartOS on a USB key What I have to work with: - Fractal Design Arc Midi R2 8 3.5” disk-bay case - Antec NeoECO C NeoECO 620C 620W ATX12V 80 PLUS BRONZE power supply - ASRock C226 WS ATX Server Motherboard LGA 1150 - QTY 2 Crucial 16GB (2 x 8GB) 240-Pin DDR3 1333 (PC3 10600) Server Memory Model CT2KIT102472BD1339 (32gb RAM) - QTY 6 Seagate NAS HDD ST4000VN000 4TB 64MB Cache SATA 6.0Gb/s Internal Hard Drive - Intel E3-1276 v3 3.6GHz 8MB L3 Cache 84W Server Processor BX80646E31276V3 (Haswell XEON) - StarTech.com 4 Drive 2.5in Removable SAS SATA Mobile Rack Backplane - IBM m1015 RAID card flashed to LSI IT - Addonics AD4SAHMS Dual Hyper HDD-mSATA SSD hybrid PCIe Controller - Samsung Electronics 840 EVO mSATA 0.85-Inch Solid State Drive MZ-MTE250BW - Corsair Force LX Series CSSD-F256GBLX 2.5" 256GB SATA III MLC Internal Solid State Drive (SSD) - Crucial CT120M50-MU03 (had laying around) - ADATA SP600 60G SSD (had laying around) SmartOS install Goal ------------------------------ - Compressed RAIDZ2 using 6 4TB drives with SLOG, L2ARC and dump on SSD partitions - There are valid arguments on why SLOG and L2ARC should not be used with the SSDs I have. I was more interested with how to accomplish the task than their operational merits Installing SMartOS: Setting up your Disks ------------------------------ - Download and Prep USB key with SmartOS appliance - Boot system with SmartOS and select the noinstall option - Log in as root/root - at root prompt issue format - This will present you with all your disks and their addresses. Write down the device to disk names so you can choose the correct disk for the install - quit format by issuing: q - I had lots of trouble with partitioning the SSDs. The lack of Solaris smarts caused lots of hours to be burned on getting disk partitions/slices setup how I wanted. What worked for me was to create zpools on each SSD, delete the zpool, then use format to carve out slices for SLOG, L2ARC and dump - from root prompt issue: - zpool create temp1 c0t0d0 - zpool create temp2 c0t1d0 - zpool destroy temp1 - zpool destroy temp2 - format - disk - <use the number corresponding to the disk you want to carve up, in my case its zero> 0 - partition (carve things up as you like). See output examples in the appendix - This took me a while to figure this out since I have never used Solaris or BSD. The structures are different than Linux but the concepts do map once you understand things. Even though Nathan Coad’s post wasn’t specifically about BSD partitions it did instruct on the use of format and how disks are carved up - Now that you know how your disks are assigned and slices are created you can reboot and install Install SmartOS ------------------------------ - Follow the prompts for setting up NIC, IP, Host Name, etc (watch Sam’s video) - Once you get to the disk selection, pick the one disk you set aside as the initial install drive. In my case it is - c3t0d0 <ATA-ADATA SP600-2.4-59.63GB> /pci@0,0/pci1849,8c02@1f ,2/disk@0,0 - Sam mentions that no swap device is created. This is fixed now and you do have a swap device. This changes his procedure a bit. I will explain later - I don’t know the merits of moving the dump device. I do know the change is not persistent across reboots. So if you do move it you have to make a change to persist across reboots. I don’t know how to do this yet - create your new pool with all the disks you plan to put in production - zpool create backup raidz2 c1t2d0 c3t1d0 c3t2d0 c3t3d0 c3t4d0 c3t5d0 - if you do a zpool list you can see the two pools - [root@bc-5f-f4-c7-d5-65 ~]# zpool list - NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT - backup 21.8T 852K 21.7T - 0% 0% 1.00x ONLINE - - zones 59.5G 3.25M 59.5G - 1% 0% 1.00x ONLINE - - add compression to zpool backup (optional from Chris H.’s list-serve post) - zpool set feature@lz4_compress=enabled backup - zfs set compression=lz4 backup && zfs get compression backup - verify compression is turned on - zpool get all backup - backup feature@lz4_compress active - or zpool get feature@lz4_compress backup - backup feature@lz4_compress active - snapshot zones pool - zfs snapshot -r zones@backup - verify snap - zfs list -t snapshot - send the zones snap to backup, this creates SmartOS on backup - zfs send -R zones@backup | zfs recv -Fd backup - check everything is copied - zfs list - reboot system, and select the noinstall option - login root/root - import your two pools - zpool import -f zones - zpool import -f backup Install SmartOS continued: ------------------------------ - destroy zpool zones - zpool destroy -f zones - export zpool backup and import it as zones - zpool export backup - zpool import backup zones - zpool list should now shows only one zpool named zones - at this point reboot - log in as root and the password you set at install - zpool list will now show zones on your new raidz<x> device - remove backup created earlier - zfs destroy -r zones@backup Optional ------------------------------ - add dedicated dump device from partition created - dumpadm -d /dev/dsk/c0t1d0s2 - destroy dump device on zones - zpool destroy -r zones/dump - add SLOG - zpool add zones log mirror /dev/dsk/c0t0d0s0 /dev/dsk/c0t1d0s0 - check with zpool status - mirror-1 ONLINE - c0t0d0s0 ONLINE - c0t1d0s0 ONLINE - you can check useage with - zpool iostat -v zones - add L2ARC - zpool add zones cache /dev/dsk/c0t0d0s1 /dev/dsk/c0t1d0s1 - check with zpool status - cache - c0t0d0s1 ONLINE - c0t1d0s1 ONLINE - to remove cache devices - zpool remove zones c0t1d0s1 - zpool remove zones c0t0d0s1 - to remove SLOG - zpool remove zones mirror-1 Appendix: ------------------------------ Disks and partitons used for this: [root@bc-5f-f4-c7-d5-65 ~]# format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t0d0 <ATA-CORSAIR FORCE LX-7A-238.47GB> /pci@0,0/pci8086,8c10@1c/pci1849,9172@0/disk@0,0 1. c0t1d0 <ATA-Crucial_CT120M50-MU03-111.79GB> /pci@0,0/pci8086,8c10@1c/pci1849,9172@0/disk@1,0 2. c1t0d0 <ATA-Samsung SSD 840-1B6Q-232.89GB> /pci@0,0/pci8086,8c18@1c,4/pci1b4b,9230@0/disk@0,0 3. c1t2d0 <ATA-ST4000VN000-1H41-SC44-3.64TB> /pci@0,0/pci8086,8c18@1c,4/pci1b4b,9230@0/disk@2,0 4. c3t0d0 <ATA-ADATA SP600-2.4-59.63GB> /pci@0,0/pci1849,8c02@1f,2/disk@0,0 5. c3t1d0 <ATA-ST4000VN000-1H41-SC44-3.64TB> /pci@0,0/pci1849,8c02@1f,2/disk@1,0 6. c3t2d0 <ATA-ST4000VN000-1H41-SC44-3.64TB> /pci@0,0/pci1849,8c02@1f,2/disk@2,0 7. c3t3d0 <ATA-ST4000VN000-1H41-SC44-3.64TB> /pci@0,0/pci1849,8c02@1f,2/disk@3,0 8. c3t4d0 <ATA-ST4000VN000-1H41-SC44-3.64TB> /pci@0,0/pci1849,8c02@1f,2/disk@4,0 9. c3t5d0 <ATA-ST4000VN000-1H41-SC44-3.64TB> /pci@0,0/pci1849,8c02@1f,2/disk@5,0 partition table of c0t0d0 Current partition table (original): Total disk sectors available: 500101741 + 16384 (reserved sectors) Part Tag Flag First Sector Size Last Sector 0 usr wm 256 5.00GB 10486015 (SLOG) 1 usr wm 10486016 60.00GB 136315135 (L2ARC) 2 unassigned wm 0 0 0 3 unassigned wm 0 0 0 4 unassigned wm 0 0 0 5 unassigned wm 0 0 0 6 unassigned wm 0 0 0 8 reserved wm 500101775 8.00MB 500118158 partition table of c0t1d0 Current partition table (original): Total disk sectors available: 234425197 + 16384 (reserved sectors) Part Tag Flag First Sector Size Last Sector 0 usr wm 256 5.00GB 10486015 (SLOG) 1 usr wm 10486016 60.00GB 136315135 (L2ARC) 2 usr wm 136315136 45.00GB 230686975 (dump) 3 unassigned wm 0 0 0 4 unassigned wm 0 0 0 5 unassigned wm 0 0 0 6 unassigned wm 0 0 0 8 reserved wm 234425231 8.00MB 234441614 -- Greg http://www.linkedin.com/in/gregtreantos ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
