Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Il 15-09-2017 17:12 David Bruzos ha scritto:

Hi Danti,
The behavior you are experiencing is normal.  As I pointed out
previously, I've been using this setup for many years and I've seen
the same thing.  You will encounter that when the filesystem is
written on the DRBD device without the use of a partition table.
As a side note, I've had some nasty stability issues with the DRBD
version in the kernel (4.4/4.9 kernels) when running on ZFS, but DRBD
8.4.10 and ZFS 0.6.5.11 seem to be running great.  I also run the
storage as part of dom0, which many admins don't recommend, but
generally it works alright.  The stability issues were typically rare
and happened under high I/O loads and were DRBD related deadlock type
crashes.  Again, those problems seem to be resolve in the latest DRBD
8 RELEASE.

David


Hi David, thanks again for taking the time to report your findings.
I plan to use CentOS, which has no build-in DRBD support, so I will use 
ELRepo's DRBD packages + the official ZFS 0.7.x repository.


Regards.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[David Bruzos]

Hi Danti,
The behavior you are experiencing is normal.  As I pointed out previously, 
I've been using this setup for many years and I've seen the same thing.  You 
will encounter that when the filesystem is written on the DRBD device without 
the use of a partition table.
As a side note, I've had some nasty stability issues with the DRBD version 
in the kernel (4.4/4.9 kernels) when running on ZFS, but DRBD 8.4.10 and ZFS 
0.6.5.11 seem to be running great.  I also run the storage as part of dom0, 
which many admins don't recommend, but generally it works alright.  The 
stability issues were typically rare and happened under high I/O loads and were 
DRBD related deadlock type crashes.  Again, those problems seem to be resolve 
in the latest DRBD 8 RELEASE.

David

-- 
David Bruzos (Systems Administrator)
Jacksonville Port Authority
2831 Talleyrand Ave.
Jacksonville, FL  32206
Cell: (904) 625-0969
Office: (904) 357-3069
Email: david.bru...@jaxport.com

On Wed, Sep 06, 2017 at 03:56:35PM +0200, Gionatan Danti wrote:
> On 06/09/2017 15:31, Yannis Milios wrote:
> > If your topology is like the following:  HDD -> ZFS (ZVOL) -> DRBD -> 
> > XFS then I believe it should make sense to always mount at the DRBD 
> > level and not at the ZVOL level which happens to be the underlying 
> > blockdev for DRBD.
> Sure! Directly mounting the DRBD-backing ZVOL would, at the bare 
> minumum, ruin the replication with the peer.
> 
> I was speaking about mounting ZVOLs *snapshots* to access previous data 
> version.
> 
> Regards.
> 
> -- 
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.da...@assyoma.it - i...@assyoma.it
> GPG public key ID: FF5F32A8
> ___
> drbd-user mailing list
> drbd-user@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

-- 


Please note that under Florida's public records law (F.S. 668.6076), most 
written communications 
to or from the Jacksonville Port Authority are public records, available to 
the public and media 
upon request. Your email communications may therefore be subject to public 
disclosure. If you have 
received this email in error, please notify the sender by return email and 
delete immediately 
without forwarding to others.
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Il 06-09-2017 16:22 David Bruzos ha scritto:

I've used DRBD devices on top of ZFS zvols for years now and have been
very satisfied with the performance and possibilities that that
configuration allows for.  I use DRBD 8.x on ZFS latest mainly on Xen
hypervisors running a mix a Linux and Windows VMs with both SSD and
mechanical drives.  I've also done similar things in the past with
DRBD and LVM.  The DRBD on ZFS conbination is the most flexible and
elegant.  You can use snapshotting and streams to do data migrations
across the Internet with minimal down time while getting storage level
redundancy and integrity from ZFS and realtime replication from DRBD.
A few scripts can automate the creation/removal of devices and
coordinate VM migrations and things will just work.  Also, you can
then use ZFS streams for offsite backups (if you need that kind of
thing).
Another thing is that you may not need the realtime replication for
some workloads, so in those cases you can just run directly on ZFS and
omit the DRBD device.  At least for me, that great flexibility is what
makes running my own configuration worth it.

Just my 25 cents!

David


Hi David,
thank you for your input. It was greatly appreciated.

Regards.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Il 06-09-2017 16:03 Yannis Milios ha scritto:

...I mean by cloning it first, since snapshot does not appear as
blockdev to the system but the clone does.


Hi, this is incorrect: ZVOL snapshots surely can appear as regular block 
devices. You simply need to set the "snapdev=visible" property.


Regards.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[David Bruzos]

I've used DRBD devices on top of ZFS zvols for years now and have been very 
satisfied with the performance and possibilities that that configuration allows 
for.  I use DRBD 8.x on ZFS latest mainly on Xen hypervisors running a mix a 
Linux and Windows VMs with both SSD and mechanical drives.  I've also done 
similar things in the past with DRBD and LVM.  The DRBD on ZFS conbination is 
the most flexible and elegant.  You can use snapshotting and streams to do data 
migrations across the Internet with minimal down time while getting storage 
level redundancy and integrity from ZFS and realtime replication from DRBD.
A few scripts can automate the creation/removal of devices and coordinate VM 
migrations and things will just work.  Also, you can then use ZFS streams for 
offsite backups (if you need that kind of thing).
Another thing is that you may not need the realtime replication for some 
workloads, so in those cases you can just run directly on ZFS and omit the DRBD 
device.  At least for me, that great flexibility is what makes running my own 
configuration worth it.

Just my 25 cents!

David

-- 
David Bruzos (Systems Administrator)
Jacksonville Port Authority
2831 Talleyrand Ave.
Jacksonville, FL  32206
Cell: (904) 625-0969
Office: (904) 357-3069
Email: david.bru...@jaxport.com

On Wed, Sep 06, 2017 at 03:03:32PM +0100, Yannis Milios wrote:
> ...I mean by cloning it first, since snapshot does not appear as blockdev
> to the system but the clone does.
> 
> On Wed, Sep 6, 2017 at 2:58 PM, Yannis Milios 
> wrote:
> 
> > Even in that case I would prefer to assemble a new DRBD device ontop of
> > the ZVOL snapshot and then mount the DRBD device instead :)
> >
> > On Wed, Sep 6, 2017 at 2:56 PM, Gionatan Danti  wrote:
> >
> >> On 06/09/2017 15:31, Yannis Milios wrote:
> >>
> >>> If your topology is like the following:  HDD -> ZFS (ZVOL) -> DRBD ->
> >>> XFS then I believe it should make sense to always mount at the DRBD level
> >>> and not at the ZVOL level which happens to be the underlying blockdev for
> >>> DRBD.
> >>>
> >> Sure! Directly mounting the DRBD-backing ZVOL would, at the bare minumum,
> >> ruin the replication with the peer.
> >>
> >> I was speaking about mounting ZVOLs *snapshots* to access previous data
> >> version.
> >>
> >> Regards.
> >>
> >>
> >> --
> >> Danti Gionatan
> >> Supporto Tecnico
> >> Assyoma S.r.l. - www.assyoma.it
> >> email: g.da...@assyoma.it - i...@assyoma.it
> >> GPG public key ID: FF5F32A8
> >>
> >
> >

> ___
> drbd-user mailing list
> drbd-user@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user


-- 


Please note that under Florida's public records law (F.S. 668.6076), most 
written communications 
to or from the Jacksonville Port Authority are public records, available to 
the public and media 
upon request. Your email communications may therefore be subject to public 
disclosure. If you have 
received this email in error, please notify the sender by return email and 
delete immediately 
without forwarding to others.
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Yannis Milios]

...I mean by cloning it first, since snapshot does not appear as blockdev
to the system but the clone does.

On Wed, Sep 6, 2017 at 2:58 PM, Yannis Milios 
wrote:

> Even in that case I would prefer to assemble a new DRBD device ontop of
> the ZVOL snapshot and then mount the DRBD device instead :)
>
> On Wed, Sep 6, 2017 at 2:56 PM, Gionatan Danti  wrote:
>
>> On 06/09/2017 15:31, Yannis Milios wrote:
>>
>>> If your topology is like the following:  HDD -> ZFS (ZVOL) -> DRBD ->
>>> XFS then I believe it should make sense to always mount at the DRBD level
>>> and not at the ZVOL level which happens to be the underlying blockdev for
>>> DRBD.
>>>
>> Sure! Directly mounting the DRBD-backing ZVOL would, at the bare minumum,
>> ruin the replication with the peer.
>>
>> I was speaking about mounting ZVOLs *snapshots* to access previous data
>> version.
>>
>> Regards.
>>
>>
>> --
>> Danti Gionatan
>> Supporto Tecnico
>> Assyoma S.r.l. - www.assyoma.it
>> email: g.da...@assyoma.it - i...@assyoma.it
>> GPG public key ID: FF5F32A8
>>
>
>
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Yannis Milios]

Even in that case I would prefer to assemble a new DRBD device ontop of the
ZVOL snapshot and then mount the DRBD device instead :)

On Wed, Sep 6, 2017 at 2:56 PM, Gionatan Danti  wrote:

> On 06/09/2017 15:31, Yannis Milios wrote:
>
>> If your topology is like the following:  HDD -> ZFS (ZVOL) -> DRBD -> XFS
>> then I believe it should make sense to always mount at the DRBD level and
>> not at the ZVOL level which happens to be the underlying blockdev for DRBD.
>>
> Sure! Directly mounting the DRBD-backing ZVOL would, at the bare minumum,
> ruin the replication with the peer.
>
> I was speaking about mounting ZVOLs *snapshots* to access previous data
> version.
>
> Regards.
>
>
> --
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.da...@assyoma.it - i...@assyoma.it
> GPG public key ID: FF5F32A8
>
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

On 06/09/2017 15:31, Yannis Milios wrote:
If your topology is like the following:  HDD -> ZFS (ZVOL) -> DRBD -> 
XFS then I believe it should make sense to always mount at the DRBD 
level and not at the ZVOL level which happens to be the underlying 
blockdev for DRBD.
Sure! Directly mounting the DRBD-backing ZVOL would, at the bare 
minumum, ruin the replication with the peer.


I was speaking about mounting ZVOLs *snapshots* to access previous data 
version.


Regards.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Hi,

On 06/09/2017 13:28, Jan Schermer wrote:

Not sure you can mount snapshot (I always create a clone).


the only difference is that snapshots are read-only, while clones are 
read-write. This is why I used the "-o ro,norecovery" option while 
mounting XFS.



However I never saw anything about “drbd” filesystem - what distribution is 
this? Apparently it tries to be too clever…


It is a CentOS 7.3 x86_64. Actually, I *really* like what the mount 
command is doing: by checking at the device end and discovering the DRBD 
metadata, it prevent accidental double mounts of the main (DRBD-backing) 
block device.


I was only wondering if it is something that only happens to me, or it 
is "normal" to specify the mounting filesystem when using snapshot 
volumes with DRBD.


Regards.


--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Yannis Milios]

If your topology is like the following:  HDD -> ZFS (ZVOL) -> DRBD -> XFS
then I believe it should make sense to always mount at the DRBD level and
not at the ZVOL level which happens to be the underlying blockdev for DRBD.

On Wed, Sep 6, 2017 at 12:28 PM, Jan Schermer  wrote:

> Not sure you can mount snapshot (I always create a clone).
> However I never saw anything about “drbd” filesystem - what distribution
> is this? Apparently it tries to be too clever…
> Try creating a clone and mounting it instead, it’s safer anyway (saw bug
> in issue tracker that ZFS panics if you try to write to the snapshot or
> something like that…)
>
> Other than that - yes, this should work fine.
>
> Jan
>
>
> > On 6 Sep 2017, at 13:23, Gionatan Danti  wrote:
> >
> > On 19/08/2017 10:24, Yannis Milios wrote:
> >> Option (b) seems more suitable for a 2 node drbd8 cluster in a
> primary/secondary setup. Haven't tried it so I cannot tell if there are any
> clurpits. My only concern in such setup would be if drbd corrupts silently
> the data on the lower level and zfs is not aware of that.Also, if you are
> *not* going to use live migration, and you can affort loosing some seconds
> of data on the secondary node in favor of better performance in the primary
> node, then you could consider using protocol A instead of C for the
> replication link.
> >
> > Hi all,
> > I "revive" this old thread to let you know I settled to use DRBD 8.4 on
> top of ZVOLs.
> >
> > I have a question for anyone using DRBD on top of a snapshot-capable
> backend (eg: ZFS, LVM, etc)...
> >
> > When snapshotting a DRBD block device, trying to mount it (the snapshot,
> not the original volume!) results in the following error message:
> >
> > [root@master7 tank]# mount /dev/zvol/tank/vol1\@snap1 /mnt/
> > mount: unknown filesystem type 'drbd'
> >
> > To successfully mount the snapshot volume, I need to specify the volume
> filesystem, for example (the other options are xfs-specific):
> >
> > [root@master7 tank]# mount -t xfs /dev/zvol/tank/vol1\@snap1 /mnt/ -o
> ro,norecovery,nouuid
> >
> > Is that the right approach? Or I am missing something?
> > Thanks.
> >
> > --
> > Danti Gionatan
> > Supporto Tecnico
> > Assyoma S.r.l. - www.assyoma.it
> > email: g.da...@assyoma.it - i...@assyoma.it
> > GPG public key ID: FF5F32A8
> > ___
> > drbd-user mailing list
> > drbd-user@lists.linbit.com
> > http://lists.linbit.com/mailman/listinfo/drbd-user
>
>
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Jan Schermer]

Not sure you can mount snapshot (I always create a clone).
However I never saw anything about “drbd” filesystem - what distribution is 
this? Apparently it tries to be too clever…
Try creating a clone and mounting it instead, it’s safer anyway (saw bug in 
issue tracker that ZFS panics if you try to write to the snapshot or something 
like that…)

Other than that - yes, this should work fine.

Jan


> On 6 Sep 2017, at 13:23, Gionatan Danti  wrote:
> 
> On 19/08/2017 10:24, Yannis Milios wrote:
>> Option (b) seems more suitable for a 2 node drbd8 cluster in a 
>> primary/secondary setup. Haven't tried it so I cannot tell if there are any 
>> clurpits. My only concern in such setup would be if drbd corrupts silently 
>> the data on the lower level and zfs is not aware of that.Also, if you are 
>> *not* going to use live migration, and you can affort loosing some seconds 
>> of data on the secondary node in favor of better performance in the primary 
>> node, then you could consider using protocol A instead of C for the 
>> replication link.
> 
> Hi all,
> I "revive" this old thread to let you know I settled to use DRBD 8.4 on top 
> of ZVOLs.
> 
> I have a question for anyone using DRBD on top of a snapshot-capable backend 
> (eg: ZFS, LVM, etc)...
> 
> When snapshotting a DRBD block device, trying to mount it (the snapshot, not 
> the original volume!) results in the following error message:
> 
> [root@master7 tank]# mount /dev/zvol/tank/vol1\@snap1 /mnt/
> mount: unknown filesystem type 'drbd'
> 
> To successfully mount the snapshot volume, I need to specify the volume 
> filesystem, for example (the other options are xfs-specific):
> 
> [root@master7 tank]# mount -t xfs /dev/zvol/tank/vol1\@snap1 /mnt/ -o 
> ro,norecovery,nouuid
> 
> Is that the right approach? Or I am missing something?
> Thanks.
> 
> -- 
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.da...@assyoma.it - i...@assyoma.it
> GPG public key ID: FF5F32A8
> ___
> drbd-user mailing list
> drbd-user@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

On 19/08/2017 10:24, Yannis Milios wrote:
Option (b) seems more suitable for a 2 node drbd8 cluster in a 
primary/secondary setup. Haven't tried it so I cannot tell if there are 
any clurpits. My only concern in such setup would be if drbd corrupts 
silently the data on the lower level and zfs is not aware of that.Also, 
if you are *not* going to use live migration, and you can affort loosing 
some seconds of data on the secondary node in favor of better 
performance in the primary node, then you could consider using protocol 
A instead of C for the replication link.


Hi all,
I "revive" this old thread to let you know I settled to use DRBD 8.4 on 
top of ZVOLs.


I have a question for anyone using DRBD on top of a snapshot-capable 
backend (eg: ZFS, LVM, etc)...


When snapshotting a DRBD block device, trying to mount it (the snapshot, 
not the original volume!) results in the following error message:


[root@master7 tank]# mount /dev/zvol/tank/vol1\@snap1 /mnt/
mount: unknown filesystem type 'drbd'

To successfully mount the snapshot volume, I need to specify the volume 
filesystem, for example (the other options are xfs-specific):


[root@master7 tank]# mount -t xfs /dev/zvol/tank/vol1\@snap1 /mnt/ -o 
ro,norecovery,nouuid


Is that the right approach? Or I am missing something?
Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Yannis Milios]

Hello,

Personally I'm using option (a) on a 3 node proxmox cluster and drbd9.
Replica count per VM is 2 and all 3 nodes act as both drbd control volumes
and satellite nodes.I can live migrate VM over all 3 nodes without issues.
Snapshots are also possible via drbdmanage + zfs snapshot + clones
capability.

Option (b) seems more suitable for a 2 node drbd8 cluster in a
primary/secondary setup. Haven't tried it so I cannot tell if there are any
clurpits. My only concern in such setup would be if drbd corrupts silently
the data on the lower level and zfs is not aware of that.Also, if you are
*not* going to use live migration, and you can affort loosing some seconds
of data on the secondary node in favor of better performance in the primary
node, then you could consider using protocol A instead of C for the
replication link.

Yannis


On Fri, 18 Aug 2017 at 15:16, Gionatan Danti  wrote:

> Il 18-08-2017 12:58 Julien Escario ha scritto:
> > If you design with a signle big ressource, a simple split brain and
> > you're screwed.
> >
> > Julien
>
> Hi, I plan to use a primary/secondary setup, with manual failover.
> In other words, split brain should not be possible at all.
>
> Thanks.
>
> --
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.da...@assyoma.it - i...@assyoma.it
> GPG public key ID: FF5F32A8
> ___
> drbd-user mailing list
> drbd-user@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
-- 
Sent from Gmail Mobile
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Il 18-08-2017 17:22 Veit Wahlich ha scritto:


Yes, I regard qemu -> DRBD -> volume management [-> RAID] -> disk the
most recommendable solution for this scenario.

I personally go with LVM thinp for volume management, but ZVOLs should
do the trick, too.

With named ressources (named after VMs) and multiple volumes per
ressource (for multiple VM disks), this works very well for us for
hundreds of VMs.

Having a cluster-wide unified system for numbering VMs is very
advisable, as it allows to calculate the ports and minor numbers for
both DRBD and KVM/qemu configuration.

Example:
* numbering VMs from 0 to 999 as , padded with leading zeros
* numbering volumes from 0 to 99 as , padded with leading zeros
* DRBD ressource port: 10
* VNC/SPICE unencrypted port: 11
* SPICE TLS port: 12
* DRBD minor: 

Let's say your VM gets number 123, it has 3 virtual disks and uses VNC:
* DRBD ressource port: 10123
* VNC port: 11123
* DRBD minor of volume/VM disk 0: 12300
* DRBD minor of volume/VM disk 1: 12301
* DRBD minor of volume/VM disk 2: 12302

Best regards,
// Veit


Hi Veit, excellent advises!
Thank you.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Il 18-08-2017 17:09 Yannis Milios ha scritto:

Personally I'm using option (a) on a 3 node proxmox cluster and drbd9.
Replica count per VM is 2 and all 3 nodes act as both drbd control
volumes and satellite nodes.I can live migrate VM between all nodes
and snapshot them by using drbdmanage utility (which is using zfs
snapshot+clones).


Hi Yannis, thank you for describing your setup!


Option (b) seems more suitable for a 2 node drbd8 cluster in a
primary/secondary setup. Haven't tried it so I cannot tell if there
are any clurpits. My only concern in such setup would be if drbd
corrupts silently the data on the lower level and zfs is not aware of
that.


I think that such a silent corruption will be better caught by ZFS when 
it happens at the lower layer (ie: the ZOOL, at VDEV level) rather than 
when it happens at upper layers (ie: DRBD on ZVOL). This is the stronger 
argument why on the ZFS list I was adviced to use DRBD on RAW device + 
ZFS on higher layer.


From what I read on this list, however, basically no-one is using ZFS 
over DRBD over RAW disks, so I am somewhat worried about some potential, 
hidden pitfalls.



Also, if you are *not* going to use live migration, and you can
afford loosing some seconds of data on the secondary node in favor of
better performance on the primary node, then you could consider using
protocol A instead of C for the replication link.


Sure. On other installation, I am using protocol B with great success.

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Veit Wahlich]

Am Freitag, den 18.08.2017, 15:46 +0200 schrieb Gionatan Danti:
> Il 18-08-2017 14:40 Veit Wahlich ha scritto:
> > VM live migration requires primary/primary configuration of the DRBD
> > ressource accessed by the VM, but only during migration. The ressource
> > can be reconfigured for allow-two-primaries and revert this setting
> > afterwards on the fly.
> 
> Hi Veit, this is interesting.
> So you suggest to use DRBD on top of a ZVOLs?

Yes, I regard qemu -> DRBD -> volume management [-> RAID] -> disk the
most recommendable solution for this scenario.

I personally go with LVM thinp for volume management, but ZVOLs should
do the trick, too. 

With named ressources (named after VMs) and multiple volumes per
ressource (for multiple VM disks), this works very well for us for
hundreds of VMs.

Having a cluster-wide unified system for numbering VMs is very
advisable, as it allows to calculate the ports and minor numbers for
both DRBD and KVM/qemu configuration.

Example:
* numbering VMs from 0 to 999 as , padded with leading zeros
* numbering volumes from 0 to 99 as , padded with leading zeros
* DRBD ressource port: 10
* VNC/SPICE unencrypted port: 11
* SPICE TLS port: 12
* DRBD minor: 

Let's say your VM gets number 123, it has 3 virtual disks and uses VNC:
* DRBD ressource port: 10123
* VNC port: 11123
* DRBD minor of volume/VM disk 0: 12300
* DRBD minor of volume/VM disk 1: 12301
* DRBD minor of volume/VM disk 2: 12302

Best regards,
// Veit

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Yannis Milios]

Hello,

Personally I'm using option (a) on a 3 node proxmox cluster and drbd9.
Replica count per VM is 2 and all 3 nodes act as both drbd control volumes
and satellite nodes.I can live migrate VM between all nodes and snapshot
them by using drbdmanage utility (which is using zfs snapshot+clones).

Option (b) seems more suitable for a 2 node drbd8 cluster in a
primary/secondary setup. Haven't tried it so I cannot tell if there are any
clurpits. My only concern in such setup would be if drbd corrupts silently
the data on the lower level and zfs is not aware of that.Also, if you are
*not* going to use live migration, and you can afford loosing some seconds
of data on the secondary node in favor of better performance on the primary
node, then you could consider using protocol A instead of C for the
replication link.

Yannis


On Fri, 18 Aug 2017 at 15:16, Gionatan Danti  wrote:

> Il 18-08-2017 12:58 Julien Escario ha scritto:
> > If you design with a signle big ressource, a simple split brain and
> > you're screwed.
> >
> > Julien
>
> Hi, I plan to use a primary/secondary setup, with manual failover.
> In other words, split brain should not be possible at all.
>
> Thanks.
>
> --
> Danti Gionatan
> Supporto Tecnico
> Assyoma S.r.l. - www.assyoma.it
> email: g.da...@assyoma.it - i...@assyoma.it
> GPG public key ID: FF5F32A8
> ___
> drbd-user mailing list
> drbd-user@lists.linbit.com
> http://lists.linbit.com/mailman/listinfo/drbd-user
>
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Il 18-08-2017 14:40 Veit Wahlich ha scritto:

To clarify:

Am Freitag, den 18.08.2017, 14:34 +0200 schrieb Veit Wahlich:
hosts simultaniously, enables VM live migration and your hosts may 
even


VM live migration requires primary/primary configuration of the DRBD
ressource accessed by the VM, but only during migration. The ressource
can be reconfigured for allow-two-primaries and revert this setting
afterwards on the fly.


Hi Veit, this is interesting.
So you suggest to use DRBD on top of a ZVOLs?

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Veit Wahlich]

To clarify:

Am Freitag, den 18.08.2017, 14:34 +0200 schrieb Veit Wahlich:
> hosts simultaniously, enables VM live migration and your hosts may even

VM live migration requires primary/primary configuration of the DRBD
ressource accessed by the VM, but only during migration. The ressource
can be reconfigured for allow-two-primaries and revert this setting
afterwards on the fly.

___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Veit Wahlich]

Hi,

Am Freitag, den 18.08.2017, 14:16 +0200 schrieb Gionatan Danti:
> Hi, I plan to use a primary/secondary setup, with manual failover.
> In other words, split brain should not be possible at all.
> 
> Thanks.

having one DRBD ressource per VM also allows you to run VMs on both
hosts simultaniously, enables VM live migration and your hosts may even
go into (planned or unplanned) DRBD disconnected situations without
interruption of service and with automatic recovery on reconnect.

This might be worth to consider.

Best regards,
// Veit


___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Il 18-08-2017 12:58 Julien Escario ha scritto:

If you design with a signle big ressource, a simple split brain and
you're screwed.

Julien


Hi, I plan to use a primary/secondary setup, with manual failover.
In other words, split brain should not be possible at all.

Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

Re: [DRBD-user] DRBD over ZFS - or the other way around?

[Julien Escario]

Le 17/08/2017 à 16:48, Gionatan Danti a écrit :
> Hi list,
> I am discussing how to have a replicated ZFS setup on the ZoL mailing list, 
> and
> DRBD is obviously on the radar ;)
> 
> It seems that three possibilities exist:
> 
> a) DRBD over ZVOLs (with one DRBD resource per ZVOL);
> b) ZFS over DRBD over the RAW disks (with DRBD resource per disk);
> c) ZFS over DRBD over a single huge and sparse ZVOL (see for an example:
> http://v-optimal.nl/index.php/2016/02/04/ha-zfs/)
> 
> What option do you feel is the better one? On the ZoL list seems to exists a
> preference for option b - create a DRBD resource for each disk and let ZFS
> manage the DRBD devices.
> 
> Any thought on that?
> Thanks.
> 

Hello,
I didn't play with ZFS and DRBD, only LVM for now but similar questions occurs.

May I suggest you to let drbdmanage be in charge of this question ? I don't know
which option it will choose but claerly, drbdmanage simplifies A LOT management
of volumes.

The only thing important is to have one DRBD ressource per (I assume) VM disk.
As such, in caseof split brain, you can still have primary on every node, with
or without a synchronized peer.

If you design with a signle big ressource, a simple split brain and you're 
screwed.

Julien



smime.p7s
Description: Signature cryptographique S/MIME
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user

[DRBD-user] DRBD over ZFS - or the other way around?

[Gionatan Danti]

Hi list,
I am discussing how to have a replicated ZFS setup on the ZoL mailing 
list, and DRBD is obviously on the radar ;)


It seems that three possibilities exist:

a) DRBD over ZVOLs (with one DRBD resource per ZVOL);
b) ZFS over DRBD over the RAW disks (with DRBD resource per disk);
c) ZFS over DRBD over a single huge and sparse ZVOL (see for an example: 
http://v-optimal.nl/index.php/2016/02/04/ha-zfs/)


What option do you feel is the better one? On the ZoL list seems to 
exists a preference for option b - create a DRBD resource for each disk 
and let ZFS manage the DRBD devices.


Any thought on that?
Thanks.

--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.da...@assyoma.it - i...@assyoma.it
GPG public key ID: FF5F32A8
___
drbd-user mailing list
drbd-user@lists.linbit.com
http://lists.linbit.com/mailman/listinfo/drbd-user