I think I'm finally going to get around to putting together the illumos
home/hobby server I've been thinking about for the past few years :),
and would appreciate a little feedback on parts/compatibility/design.
The box is intended to be both a storage server (music/video/etc media,
documents, whatever) with content available via both NFS and CIFS, as
well as a virtualization server using kvm to run some number of linux
instances (the most heavyweight of which will probably be the mythtv
instance, but there will be a number of other miscellaneous things going
on). I'm thinking of using two SSD's with a partition mirrored for
rpool, a 2nd separate partition as L2ARC on each, and possibly a third
mirrored for slog (or potentially a separate SSD just for slog), and a
storage pool consisting of 2 6 disk raidz2 vdevs.
For the case, I'm looking at the Supermicro 836BA-R920B rackmount
chassis, which has 16 3.5" hot-swap bays on the front, and 2 2.5"
hot-swap bays on the back, along with dual redundant 960w 80+ platinum
certified power supplies. This particular model has all 16 front bays
direct attached, with four SFF-8087 connectors. There are two other
models available with either one or two SAS expanders; however, from
what I understand hooking up SATA drives on the other side of a SAS
expander is a bad idea. If I went with near-line SAS, I could get the
model with the expanders, which would reduce my cost in terms of SAS
controllers, but the pricing on near-line SAS is ridiculous compared to
SATA, and the extra cost in SAS controller should be outweighed by
reduced cost in drives (I'm already looking at a way higher budget than
I'd like for a hobby project, but I have few vices, and electronics are
one of them ;) ).
For the motherboard, I'm looking at the Supermicro X9DRD-7LN4F-JBOD,
which is a dual LGA 2011 socket board with 16 DIMM slots, 2 x SATA3, 4 x
SATA2, and 8 x SAS (LSI 2308 controller onboard) along with 4 intel i350
based gig nics. My understanding is that illumos is perfectly happy with
the LSI 2308 in IT mode. The -JBOD version of this motherboard comes
from the factory with IT firmware. It doesn't seem readily available
though, if I went with the regular version the LSI controller comes with
RAID firmware, it's possible to reflash with IT but from what I've read
it's a bit of a pain (you need to do it from the EFI shell). It also
looks like illumos works with the intel i350 gig nics, and I assume
there should be no issue with the onboard Intel AHCI SATA controller?
CPU, 2 x Intel Xeon E5-2620. The hex core is a bit pricier than the
quads, but I've just got my heart set on 12 cores, and no one said a
hobby had to be cost effective ;). These are Sandy Bridge Xeons, I know
there were some Sandy Bridge issues in the past, but I think there were
workarounds, and it looks like Joyent recently fixed them
(https://github.com/joyent/illumos-joyent/commit/4d86fb7f59410be72e467483b74e2eebff6052b2),
so I'm hoping they will work well.
I haven't really spec'd specific RAM, although I'm partial to crucial,
it takes 1333MHz registered ECC DDR3. I think I want at least 32GB for
the storage server side, and I'm not sure yet how much more I'll add in
on top of that for virtualization.
8 of the 16 3.5" bays will be covered by the onboard LSI controller, I
need to get an additional PCIe controller with 2 x SFF-8087 connectors
to cover the rest. Seems there are a fair number of options, although
I'm not sure if there's a clear winner among them. Any favorites?
Hard drives are the parts I'm least confident in 8-/. I'd like to go 2TB
or 3TB, that's cost prohibitive for near-line SAS, and pretty darn pricy
for "enterprise" SATA. I don't really want to go with desktop class
drives though.
Is there any opinion yet on the new WD Red "NAS" drives? They're only
$170 for a 3TB drive, which is pretty cheap. On the plus side, they're
engineered for 7x24 operation, have a three year warranty, and are
supposed to be low power/low heat (both would be good; while I installed
a 4.5kw solar power system a few years ago when I remodeled our house,
and have been net negative powerwise since, I anticipate that to change
when this beast starts running. I also set up a dedicated wiring closet
with a separate 8000btu wall air conditioner, but still less heat = less
cooling = less power utilization). They come out-of-the-box with 7
second TLER, plus the ability to tune that however you'd like. On the
downside, while WD doesn't specify it, they evidently run at 5400rpm
(where I suppose the low power low heat comes from), and aren't exactly
screamers (streaming isn't too bad, but random IO leaves a bit to be
desired).
My mythtv vm will potentially be recording 4 HD ATSC streams
(originating from network connected HD homeruns), reading all 4 back
from disk at the same time (for commercial flagging) and potentially
reading a different two streams for playback on the two front ends I
currently have connected to TVs. Arguably worst case for an ATSC
transport stream is about 18Mbps, so it's not really that much. But then
all of the vm's will be doing their thing, plus whatever NFS/CIFS
clients are up to. Sizing for IO is black magic to me <sigh>, on the one
hand I want to maximize my storage for the cost, but on the other I
don't want to have recordings that skip and stutter and vm's that lag
and are unresponsive...
I also don't really have a good handle on what SSD's to go with. As I
mentioned, I'm thinking of getting two for rpool/l2arc, and hook them up
to the onboard SATA3 controller. If I can find ones that are
appropriate, I'd carve out a third partition on them for a mirrored
slog; otherwise I'd get a separate third one and stick it in a 3.5" bay
to be dedicated slog. I don't think I'd bother to mirror the slog if it
is on a separate SSD, I believe there are no longer any critical failure
modes from slog failure, worst-case being it fails when the pool is
off-line and you need to manually import it. Any suggestions on good
rpool/l2arc/slog SSD's, or rpool/l2arc SSD's with a different model slog
SSD would be greatly appreciated.
Thanks much for reading so far :), I realize I've gone on for quite a
bit... Any comments/feedback/suggestions on compatibility or design
issues with what I've laid out would be very welcome. Thanks again...
_______________________________________________
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss
