> In general, mixing SATA and SAS directly behind expanders (eg without
> SAS/SATA intereposers) seems to be bad juju that an OS can't fix.
In general I'd agree. Just mixing them on the same box can be problematic,
I've noticed - though I think as much as anything that the firmware
on the 3G/s expanders just isn't as well-tuned as the firmware
on the 6G/s expanders, and for all I know there's a firmware update
that will make things better.
SSDs seem to be an exception, however. Several boxes have a mix of
Crucial C300, OCZ Vertex Pro, and OCZ Vertex-3 SSDs for the usual
purposes on the expander with the constellations, or in one case,
Cheetah 15ks. One box has SSDs and Cheetah 15ks/constellations on the same
expander under massive loads - the aforementioned box suffering from
80k ZIO queues - with nary a blip. (The SSDs are swap drives, and
we were force-swapping processes out to disk as part of task management.
Meanwhile, the Java processes are doing batch import processing using
the Cheetahs as staging area, so those two expanders are under constant
heavy load. Yes that is as ugly as it sounds, don't ask, and don't do
this yourself. This is what happens when you develop a database without
clear specs and have to just throw hardware underneath it guessing
all the way. But to give you an idea of the load they were/are under.)
The SSDs were chosen with an eye towards expander-friendliness, and tested
relatively extensively before use. YMMV of course and this is nowhere
to skimp on a-data or Kingston; buy what Anand says to buy and you
seem to do very well.
I would say, never do it on LSI 3G/s expanders. Be careful with using
SATA spindles. Test the hell out of any SSD you use first. But you seem
to be able to get away with the better consumer-class SATA SSDs.
(I realize that many here would say that if you are going to use
SSD in an enterprise config, you shouldn't be messing with anything
short of Deneva or the SAS-based SSDs. I'd say there are simply
a bunch of caveats with the consumer MLC SSDs in such situations
to consider and if you are very clear about them up front, then
they can be just fine.
I suspect the real difficulty in these situations is in having
a management chain that is capable of both grokking the caveats up
front and remembering that they agreed to them when something
does go wrong. :) As in this case I am the management chain,
it's not an issue. This is of course not the usual case.)
zfs-discuss mailing list