Dear DRBD Users! I have setup a proxmox 6.2 cluster linstore 1.9, linstore-proxmox with two nodes Hydra3/4 two months ago. Works like a charm.
Now I added a third node Hydra1 (proxmox 6.3, linstore 1.9, linstore-proxmox). [This project is a Proxmox 4->6 migration. I migrated all VMs from Hydra1/2 (Proxmox4) by hand to our new Hydra3/4 (Proxmox6) servers and now reintegrate Hydra1/2] Also no problem. But all the volumes are now and remain in diskless state on Hydra1? DRBDTOP v0.2.3-4-g5fa4b14 (kernel: 9.0.25; utils: 9.15.1; host: hydra1) - Details for vm-118-disk-1 Resource: vm-118-disk-1: (Overall danger score: 0) │ Local Disc(*Secondary*): │ volume 0 (/dev/drbd1023): *Diskless*(detached from backing disk) │ │ Connection to hydra3(*Primary*): Connected(connected to hydra3) │ volume 0: │ UpToDate(normal disk state) │ │ Connection to hydra4(Secondary): Connected(connected to hydra4) │ volume 0: │ UpToDate(normal disk state) The 3-Node cluster is fully operational, creation of VMs and live migration between all three node works perfectly. So when I migrate e.g. VM118 from Hydra3 to the new Hydra1 the VMs switches really fine. But in the end the DRBD-Device is still diskless on Hydra1 │Resource: vm-118-disk-1: (Overall danger score: 0) │ Local Disc(*Primary*): │ volume 0 (/dev/drbd1023): *Diskless*(detached from backing disk) │ │ Connection to hydra3(*Secondary*): Connected(connected to hydra3) │ volume 0: │ UpToDate(normal disk state) │ │ Connection to hydra4(Secondary): Connected(connected to hydra4) │ volume 0: │ UpToDate(normal disk state) Stupid question: How do I get the volumes also DRBD-replicated onto the third node? I thought that this would be done in the moment when I start a migration. /etc/pve/storage.cfg drbd: drbdstorage content images, rootdir controller 10.1.3.18 resourcegroup ResGrpDRBD root@hydra1:~# linstor rg list ╭──────────────────────────────────────────────────────────────────╮ ┊ ResourceGroup ┊ SelectFilter ┊ VlmNrs ┊ Description ┊ ╞══════════════════════════════════════════════════════════════════╡ ┊ DfltRscGrp ┊ PlaceCount: 2 ┊ ┊ ┊ ╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡ ┊ ResGrpDRBD ┊ PlaceCount: 2 ┊ 0 ┊ ┊ ┊ ┊ StoragePool(s): pool_drbd ┊ ┊ ┊ ╰──────────────────────────────────────────────────────────────────╯ root@hydra1:~# linstor volume-definition list ╭───────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ VolumeNr ┊ VolumeMinor ┊ Size ┊ Gross ┊ State ┊ ╞═══════════════════════════════════════════════════════════════════╡ ┊ vm-100-disk-1 ┊ 0 ┊ 1000 ┊ 50 GiB ┊ ┊ ok ┊ ┊ vm-100-disk-2 ┊ 0 ┊ 1001 ┊ 900 GiB ┊ ┊ ok ┊ ┊ vm-101-disk-1 ┊ 0 ┊ 1002 ┊ 100 GiB ┊ ┊ ok ┊ ┊ vm-102-disk-1 ┊ 0 ┊ 1003 ┊ 16 GiB ┊ ┊ ok ┊ ┊ vm-103-disk-1 ┊ 0 ┊ 1004 ┊ 32 GiB ┊ ┊ ok ┊ ┊ vm-104-disk-1 ┊ 0 ┊ 1005 ┊ 32 GiB ┊ ┊ ok ┊ ┊ vm-105-disk-1 ┊ 0 ┊ 1006 ┊ 50 GiB ┊ ┊ ok ┊ ┊ vm-105-disk-2 ┊ 0 ┊ 1007 ┊ 100 GiB ┊ ┊ ok ┊ ┊ vm-106-disk-1 ┊ 0 ┊ 1008 ┊ 50 GiB ┊ ┊ ok ┊ ┊ vm-107-disk-1 ┊ 0 ┊ 1009 ┊ 32 GiB ┊ ┊ ok ┊ ┊ vm-107-disk-2 ┊ 0 ┊ 1010 ┊ 80 GiB ┊ ┊ ok ┊ ┊ vm-108-disk-1 ┊ 0 ┊ 1011 ┊ 32 GiB ┊ ┊ ok ┊ ┊ vm-109-disk-1 ┊ 0 ┊ 1012 ┊ 100 GiB ┊ ┊ ok ┊ ┊ vm-110-disk-1 ┊ 0 ┊ 1013 ┊ 100 GiB ┊ ┊ ok ┊ ┊ vm-110-disk-2 ┊ 0 ┊ 1014 ┊ 1.46 TiB ┊ ┊ ok ┊ ┊ vm-111-disk-1 ┊ 0 ┊ 1015 ┊ 50 GiB ┊ ┊ ok ┊ ┊ vm-112-disk-1 ┊ 0 ┊ 1016 ┊ 32 GiB ┊ ┊ ok ┊ ┊ vm-113-disk-1 ┊ 0 ┊ 1017 ┊ 50 GiB ┊ ┊ ok ┊ ┊ vm-114-disk-1 ┊ 0 ┊ 1018 ┊ 50 GiB ┊ ┊ ok ┊ ┊ vm-115-disk-1 ┊ 0 ┊ 1019 ┊ 22 GiB ┊ ┊ ok ┊ ┊ vm-115-disk-2 ┊ 0 ┊ 1020 ┊ 100 GiB ┊ ┊ ok ┊ ┊ vm-116-disk-1 ┊ 0 ┊ 1021 ┊ 32 GiB ┊ ┊ ok ┊ ┊ vm-117-disk-1 ┊ 0 ┊ 1022 ┊ 32 GiB ┊ ┊ ok ┊ ┊ vm-118-disk-1 ┊ 0 ┊ 1023 ┊ 32 GiB ┊ ┊ ok ┊ ╰───────────────────────────────────────────────────────────────────╯ root@hydra1:~# linstor resource-definition list ╭──────────────────────────────────────────────╮ ┊ ResourceName ┊ Port ┊ ResourceGroup ┊ State ┊ ╞══════════════════════════════════════════════╡ ┊ vm-100-disk-1 ┊ 7000 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-100-disk-2 ┊ 7001 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-101-disk-1 ┊ 7002 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-102-disk-1 ┊ 7003 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-103-disk-1 ┊ 7004 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-104-disk-1 ┊ 7005 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-105-disk-1 ┊ 7006 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-105-disk-2 ┊ 7007 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-106-disk-1 ┊ 7008 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-107-disk-1 ┊ 7009 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-107-disk-2 ┊ 7010 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-108-disk-1 ┊ 7011 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-109-disk-1 ┊ 7012 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-110-disk-1 ┊ 7013 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-110-disk-2 ┊ 7014 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-111-disk-1 ┊ 7015 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-112-disk-1 ┊ 7016 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-113-disk-1 ┊ 7017 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-114-disk-1 ┊ 7018 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-115-disk-1 ┊ 7019 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-115-disk-2 ┊ 7020 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-116-disk-1 ┊ 7021 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-117-disk-1 ┊ 7022 ┊ ResGrpDRBD ┊ ok ┊ ┊ vm-118-disk-1 ┊ 7023 ┊ ResGrpDRBD ┊ ok ┊ ╰──────────────────────────────────────────────╯ root@hydra1:~# linstor resource list ╭─────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊ ╞═════════════════════════════════════════════════════════════════════════════════╡ ┊ vm-100-disk-1 ┊ hydra3 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04 18:53:37 ┊ ┊ vm-100-disk-1 ┊ hydra4 ┊ 7000 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-04 18:53:37 ┊ ┊ vm-100-disk-2 ┊ hydra3 ┊ 7001 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04 19:49:55 ┊ ┊ vm-100-disk-2 ┊ hydra4 ┊ 7001 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-04 19:49:55 ┊ ┊ vm-101-disk-1 ┊ hydra3 ┊ 7002 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04 20:00:09 ┊ ┊ vm-101-disk-1 ┊ hydra4 ┊ 7002 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-04 20:00:08 ┊ ┊ vm-102-disk-1 ┊ hydra3 ┊ 7003 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04 20:04:28 ┊ ┊ vm-102-disk-1 ┊ hydra4 ┊ 7003 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-04 20:04:27 ┊ ┊ vm-103-disk-1 ┊ hydra3 ┊ 7004 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20 17:54:11 ┊ ┊ vm-103-disk-1 ┊ hydra4 ┊ 7004 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20 17:54:11 ┊ ┊ vm-104-disk-1 ┊ hydra3 ┊ 7005 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20 21:06:05 ┊ ┊ vm-104-disk-1 ┊ hydra4 ┊ 7005 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20 21:06:05 ┊ ┊ vm-105-disk-1 ┊ hydra3 ┊ 7006 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20 21:09:24 ┊ ┊ vm-105-disk-1 ┊ hydra4 ┊ 7006 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20 21:09:22 ┊ ┊ vm-105-disk-2 ┊ hydra3 ┊ 7007 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20 21:09:58 ┊ ┊ vm-105-disk-2 ┊ hydra4 ┊ 7007 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20 21:09:56 ┊ ┊ vm-106-disk-1 ┊ hydra3 ┊ 7008 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20 22:12:26 ┊ ┊ vm-106-disk-1 ┊ hydra4 ┊ 7008 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20 22:12:26 ┊ ┊ vm-107-disk-1 ┊ hydra3 ┊ 7009 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20 22:52:51 ┊ ┊ vm-107-disk-1 ┊ hydra4 ┊ 7009 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20 22:52:51 ┊ ┊ vm-107-disk-2 ┊ hydra3 ┊ 7010 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-20 22:53:27 ┊ ┊ vm-107-disk-2 ┊ hydra4 ┊ 7010 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-20 22:53:27 ┊ ┊ vm-108-disk-1 ┊ hydra3 ┊ 7011 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24 08:18:52 ┊ ┊ vm-108-disk-1 ┊ hydra4 ┊ 7011 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24 08:18:52 ┊ ┊ vm-109-disk-1 ┊ hydra3 ┊ 7012 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24 08:20:33 ┊ ┊ vm-109-disk-1 ┊ hydra4 ┊ 7012 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24 08:20:34 ┊ ┊ vm-110-disk-1 ┊ hydra3 ┊ 7013 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24 11:17:06 ┊ ┊ vm-110-disk-1 ┊ hydra4 ┊ 7013 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24 11:17:06 ┊ ┊ vm-110-disk-2 ┊ hydra3 ┊ 7014 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24 11:17:44 ┊ ┊ vm-110-disk-2 ┊ hydra4 ┊ 7014 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24 11:17:52 ┊ ┊ vm-111-disk-1 ┊ hydra3 ┊ 7015 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24 13:14:20 ┊ ┊ vm-111-disk-1 ┊ hydra4 ┊ 7015 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24 13:14:22 ┊ ┊ vm-112-disk-1 ┊ hydra3 ┊ 7016 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-24 15:46:42 ┊ ┊ vm-112-disk-1 ┊ hydra4 ┊ 7016 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-24 15:46:44 ┊ ┊ vm-113-disk-1 ┊ hydra3 ┊ 7017 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-26 11:18:34 ┊ ┊ vm-113-disk-1 ┊ hydra4 ┊ 7017 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-26 11:18:34 ┊ ┊ vm-114-disk-1 ┊ hydra3 ┊ 7018 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-26 22:47:51 ┊ ┊ vm-114-disk-1 ┊ hydra4 ┊ 7018 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-26 22:47:51 ┊ ┊ vm-115-disk-1 ┊ hydra3 ┊ 7019 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-27 12:47:43 ┊ ┊ vm-115-disk-1 ┊ hydra4 ┊ 7019 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-27 12:47:42 ┊ ┊ vm-115-disk-2 ┊ hydra3 ┊ 7020 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-11-27 12:58:01 ┊ ┊ vm-115-disk-2 ┊ hydra4 ┊ 7020 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-11-27 12:58:01 ┊ ┊ vm-116-disk-1 ┊ hydra3 ┊ 7021 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2020-12-02 18:42:30 ┊ ┊ vm-116-disk-1 ┊ hydra4 ┊ 7021 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02 18:42:30 ┊ ┊ vm-117-disk-1 ┊ hydra3 ┊ 7022 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02 19:31:31 ┊ ┊ vm-117-disk-1 ┊ hydra4 ┊ 7022 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02 19:31:31 ┊ ┊ vm-118-disk-1 ┊ hydra1 ┊ 7023 ┊ InUse ┊ Ok ┊ Diskless ┊ 2020-12-02 19:40:00 ┊ ┊ vm-118-disk-1 ┊ hydra3 ┊ 7023 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02 19:40:04 ┊ ┊ vm-118-disk-1 ┊ hydra4 ┊ 7023 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2020-12-02 19:40:02 ┊ ╰─────────────────────────────────────────────────────────────────────────────────╯ root@hydra1:~# linstor storage-pool list ╭───────────────────────────────────────────────────────────────────────────────────────────────────────────╮ ┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ ╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════╡ ┊ DfltDisklessStorPool ┊ hydra1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ DfltDisklessStorPool ┊ hydra3 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ DfltDisklessStorPool ┊ hydra4 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ ┊ pool_drbd ┊ hydra1 ┊ LVM ┊ vg_drbd ┊ 7.28 TiB ┊ 7.28 TiB ┊ False ┊ Ok ┊ ┊ pool_drbd ┊ hydra3 ┊ LVM ┊ vg_drbd ┊ 14.70 TiB ┊ 18.19 TiB ┊ False ┊ Ok ┊ ┊ pool_drbd ┊ hydra4 ┊ LVM ┊ vg_drbd ┊ 14.70 TiB ┊ 18.19 TiB ┊ False ┊ Ok ┊ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯ Any pointers appreciated Volker -- ========================================================= inqbus Scientific Computing Dr. Volker Jaenisch Hungerbichlweg 3 +49 (8860) 9222 7 92 86977 Burggen https://inqbus.de =========================================================
_______________________________________________ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list drbd-user@lists.linbit.com https://lists.linbit.com/mailman/listinfo/drbd-user
