On 2013-04-19T16:27:14, Ulrich Windl <[email protected]> wrote:
> Hello,
>
> Using OCFS2 on top of a cLVM-mirrored LV is an absolute no-go for SLES11 SP2:
Note that this is unrelated to OCFS2; cLVM2 mirroring is rather slow,
since it communicates over the network to keep the dirty bitmaps and
locks in sync.
> First, while mirroring the LV ("only" 300GB) access to any of the involved
> devices is very slow, but it's not the I/O that's the limit, but something
> else; "communication" I suspect.
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ P COMMAND
> 14883 root RT 0 535m 11m 7808 R 42 0.0 34:35.24 0 corosync
> 16493 root 20 0 33804 2168 1752 R 14 0.0 10:32.29 0 cmirrord
>
> Shouldn't a busy mirroring job have a "D" state instead of an "R" state,
> burning CPU?
They're not in "D" because they are not waiting on disk IO, but have
a lot of network IO and data structure maintenance to handle.
> So besides of the inefficient I/O with cLVM there are more issues:
> 1) LVM should load-balance between the mirror legs
It doesn't, because this simplifies the dirty logging. It always will
write to leg 1 first, hence all read requests can always be satisfied
from leg 1 without the need to cluster-wide sync if leg 1 and leg 2 are
already in sync in the IO paths.
> 2) LVM should use a leg-internal bitmap to resynchronize the mirror in a
> non-stupid way
It does use a bitmap for syncing, if you created the lvmirror with a
persistent mirrorlog.
> 3) LVM should mirror the more recent mirror leg to the outdated mirror leg,
> not use a fixed direction.
The only situation where this matters is a split brain combined with
split IO. That's a situation that even DRBD doesn't handle well, and
the resolution that LVM2 mirroring implements is as valid as any.
> So my advice is: Don't use it (for SLES11 SP2).
You should not use it if performance is your primary goal for using it,
no.
> I'm somewhat
> displeased about the situation, because I had a support request asking
> exactly whether this setup is a supported configuration, and it was
> confirmed.
It *is* supported, but cLVM2 mirroring has constraints, especially with
regard to performance and flexibility.
If you can avoid the need for a concurrent cluster mirror, do so: use
SAN-based mirroring, use md raid1 if active/passive access is
sufficient, consider building an iSCSI server using Raid1 to re-export
via iSCSI for your concurrent IO needs, consider using DRBD, use cLVM2
mirroring but with local activation, etc. They all, alas, have
trade-offs.
Cluster concurrency is a hard problem. cLVM2 mirroring performance is
certainly pretty close to the top of our priority lists, but the battle
is not won in a day.
> Now the first proposal regarding the terrible performance was to _try_
> SLES11 SP3 beta...
The CPU overhead will have improved some, but the basic design of cLVM2
mirroring hasn't changed a lot.
This is the same upstream and in all distributions, it is not SLES
specific.
Regards,
Lars
--
Architect Storage/HA
SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB
21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde
_______________________________________________
Linux-HA mailing list
[email protected]
http://lists.linux-ha.org/mailman/listinfo/linux-ha
See also: http://linux-ha.org/ReportingProblems
