I'm preparing to replicate about 200TB of data between two data centers using
zfs send. We have ten 10TB zpools that are further broken down into zvols of
various sizes in each data center. One DC is primary and the other will be the
replication target and there is plenty of bandwidth between
We tried all combinations of OCZ SSDs including their PCI based SSDs and they
do NOT work as a ZIL. After a very short time performance degrades horribly and
for the OCZ drives they eventually fail completely. We also tried Intel which
performed a little better and didn't flat out fail over
They have been incredibly reliable with zero downtime or issues. As a result,
we use 2 in every system striped. For one application outside of VDI, we use a
pair of them mirrored, but that is very unusual and driven by the customer and
not us.
--
This message posted from opensolaris.org
Let me frame this in the context specifically of VMWare ESXi 4.x. If I create a
zvol and give it to ESXi via iSCSI our experience has been that it is very fast
and guest response is excellent. If we use NFS without a zil (we use DDRdrive
X1==awesome) because VMWare uses sync (Stable = FSYNC)
We tried this in our environment and found that it didn't work out. The more
partitions we used, the slower it went. We decided just to use the entire SSD
as a read cache and it worked fine. Still has the TRIM issue of course until
the next version.
--
This message posted from opensolaris.org
The NFS client in this case was VMWare ESXi 4.1 release build. What happened is
that the file uploader behavior was changed in 4.1 to prevent I/O contention
with the VM guests. That means when you go to upload something to the
datastore, it only sends chunks of the file instead of streaming it
The NFS client that we're using always uses O_SYNC, which is why it was
critical for us to use the DDRdrive X1 as the ZIL. I was unclear on the entire
system we're using, my apologies. It is:
OpenSolaris SNV_134
Motherboard: SuperMicro X8DAH
RAM: 72GB
CPU: Dual Intel 5503 @ 2.0GHz
ZIL: DDRdrive
Figured it out - it was the NFS client. I used snoop and then some dtrace magic
to prove that the client (which was using O_SYNC) was sending very bursty
requests to the system. I tried a number of other NFS clients with O_SYNC as
well and got excellent performance when they were configured
I have a 24 x 1TB system being used as an NFS file server. Seagate SAS disks
connected via an LSI 9211-8i SAS controller, disk layout 2 x 11 disk RAIDZ2 + 2
spares. I am using 2 x DDR Drive X1s as the ZIL. When we write anything to it,
the writes are always very bursty like this:
ool
As I said, please by all means try it and post your benchmarks for first hour,
first day and first week and then first month. The data will be of interest to
you. On a subjective basis, if you feel that an SSD is working just fine as
your ZIL, run with it. Good luck!
--
This message posted
I can't think of an easy way to measure pages that have not been consumed since
it's really an SSD controller function which is obfuscated from the OS, and add
the variable of over provisioning on top of that. If anyone would like to
really get into what's going on inside of an SSD that makes
Saso is correct - ESX/i always uses F_SYNC for all writes and that is for sure
your performance killer. Do a snoop | grep sync and you'll see the sync write
calls from VMWare. We use DDRdrives in our production VMWare storage and they
are excellent for solving this problem. Our cluster supports
David asked me what I meant by filled up. If you make the unwise decision to
use an SSD as your ZIL, at some point days to weeks after you install it, all
of the pages will be allocated and you will suddenly find the device to be
slower than a conventional disk drive. This is due to the way
By all means please try it to validate it yourself and post your results from
hour one, day one and week one. In a ZIL use case, although the data set is
small it is always writing a small ever changing (from the SSDs perspective)
data set. The SSD does not know to release previously written
Don't waste your time with something other than the DDRdrive for NFS ZIL. If
it's RAM based it might work, but why risk it and if it's an SSD forget it. No
SSD will work well for the ZIL long term. Short term the only SSD to consider
would be Intel, but again long term even that will not work
We are doing NFS in VMWare 4.0U2 production, 50K users using OpenSolaris
SNV_134 on SuperMicro boxes with SATA drives. Yes, I am crazy. Our experience
has been that iSCSI for ESXi 4.x is fast and works well with minimal fussing
until there is a problem. When that problem happens, getting to
Our experience has been that a new out of the box SSD works well for the ZIL
but as soon as it's completely full, performance drops to slower than a regular
SAS hard drive due to the write performance penalty in their fundamental
design, their LBA map strategy and the not yet released (to me at
OpenSolaris snv_131 (problem also in snv_132 still) on an X4500, bug #6924390
Victor,
I see in researching this issue that you know ZFS really well. I would
appreciate your help so much and this problem seems interesting.
I created a large zpool named xpool and then created 3 filesystems on
18 matches
Mail list logo
