Re: [PVE-User] CEPH performance

[Mark Adams via pve-user]

--- Begin Message ---
The simplest thing to set also is to make sure you are using writeback
cache in your vms with ceph. It makes a huge difference in performance.

On Wed, 10 Jun 2020, 07:31 Eneko Lacunza,  wrote:

> Hi Marco,
>
> El 9/6/20 a las 19:46, Marco Bellini escribió:
> > Dear All,
> > I'm trying to use proxmox on a 4 nodes cluster with ceph.
> > every node has a 500G NVME drive, with dedicated 10G ceph network with
> 9000bytes MTU.
> >
> > despite off nvme warp speed I can reach when used as lvm volume, as soon
> as I convert it into a 4-osd ceph, performance are very very poor.
> >
> > is there any trick to have ceph intro proxmox working fast?
> >
> What is "very very poor"? What specs have the Proxmox nodes (CPU, RAM)?
>
> AFAIK, it will be a challenge to get more that 2000 IOPS from one VM
> using Ceph...
>
> How are you performing the benchmark?
>
> Cheers
>
> --
> Zuzendari Teknikoa / Director Técnico
> Binovo IT Human Project, S.L.
> Telf. 943569206
> Astigarragako bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
> www.binovo.es
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] VM Power Issue

[Mark Adams via pve-user]

--- Begin Message ---
Sivakumar - This is a "known issue" as far as I am aware, usually when you
are allocating quite a bit of memory (although 16G is not a lot in your
case, but maybe the server doesn't have much ram?) when starting a vm with
a PCI device passed through to it. It also only seems to happen when you
are nearing "peak" ram usage, so getting close to running out. It never
happens on a fresh boot.

I don't know if it has been acknowledged or even reported to redhat, or
whether simply the timeout should be longer in proxmox.

I wrote to this list about it not long ago and never received a response,
and I have seen at least 1 forum post about it.

Anyway to cut a long story short, just start it manually on the cli, which
has no timeout. "qm showcmd VMID | bash" should start it fine. IE "qm
showcmd 101 | bash"

Regards,
Mark

On Mon, 8 Jun 2020 at 15:31, Arjen via pve-user 
wrote:

>
>
>
> -- Forwarded message --
> From: Arjen 
> To: PVE User List 
> Cc:
> Bcc:
> Date: Mon, 08 Jun 2020 14:30:57 +
> Subject: Re: [PVE-User] VM Power Issue
> On Monday, June 8, 2020 10:14 AM, Sivakumar SARAVANAN <
> sivakumar.saravanan.jv@valeo-siemens.com> wrote:
>
> > Hello,
> >
> > I am not able to start the VM after adding the PCI device to VM.
> > I can see the below error message.
>
> Maybe your system is very busy? Maybe it takes a while to allocate the
> memory?
> Maybe you could give more information about the VM configuration and your
> PVE setup?
>
> Can you try running the command below from the command line of your PVE
> host, to see if it works and how long it takes?
> Sometimes (often memory-size related), it just works but takes longer than
> the time-out.
>
> > TASK ERROR: start failed: command '/usr/bin/kvm -id 175 -name
> > HIL-System096Planned -chardev
> > 'socket,id=qmp,path=/var/run/qemu-server/175.qmp,server,nowait' -mon
> > 'chardev=qmp,mode=control' -chardev
> > 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon
> > 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/175.pid
> > -daemonize -smbios 'type=1,uuid=1ab2409d-4b67-4d3c-822a-7a024d05d9bf'
> -smp
> > '4,sockets=2,cores=2,maxcpus=4' -nodefaults -boot
> >
> 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg'
> > -vnc unix:/var/run/qemu-server/175.vnc,password -no-hpet -cpu
> >
> 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi'
> > -m 32768 -object 'memory-backend-ram,id=ram-node0,size=16384M' -numa
> > 'node,nodeid=0,cpus=0-1,memdev=ram-node0' -object
> > 'memory-backend-ram,id=ram-node1,size=16384M' -numa
> > 'node,nodeid=1,cpus=2-3,memdev=ram-node1' -device
> > 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device
> > 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device
> > 'vmgenid,guid=c98f392f-13af-43d9-b26e-ca070177f6bb' -device
> > 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device
> > 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device
> > 'vfio-pci,host=:1b:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device
> > 'VGA,id=vga,bus=pci.0,addr=0x2' -chardev
> > 'socket,path=/var/run/qemu-server/175.qga,server,nowait,id=qga0' -device
> > 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device
> > 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi
> > 'initiator-name=iqn.1993-08.org.debian:01:626ca038d6c7' -drive
> > 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device
> > 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive
> >
> 'file=/dev/zvol/SSD-Storage-PRX018/vm-175-disk-0,if=none,id=drive-virtio0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on'
> > -device
> >
> 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100'
> > -drive
> >
> 'file=/dev/zvol/HDD-Storage-PRX018/vm-175-disk-1,if=none,id=drive-virtio1,cache=writethrough,format=raw,aio=threads,detect-zeroes=on'
> > -device
> 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb'
> > -netdev
> >
> 'type=tap,id=net0,ifname=tap175i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown'
> > -device
> >
> 'e1000,mac=F2:3F:4D:48:7B:68,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'
> > -rtc 'driftfix=slew,base=localtime' -machine 'type=pc+pve1' -global
> > 'kvm-pit.lost_tick_policy=discard'' failed: got timeout
> >
> > Appreciating your suggestion.
> >
> > Best regards,
> >
> > SK
> >
> >
> 

Re: [PVE-User] no iommu detected please activate it.see documentation for further information

[Mark Adams via pve-user]

--- Begin Message ---
Have you enabled IOMMU in the BIOS? Assuming your server hardware supports
it?

On Fri, 15 May 2020 at 15:03, Sivakumar SARAVANAN <
sivakumar.saravanan.jv@valeo-siemens.com> wrote:

> Hello,
>
> I am unable to add the PCI device to VM's, where I am getting below error
> message.
>
> 'no iommu detected please activate it.see documentation for further
> information '
>
> I also updated the grub as below. But not luck.
>
> / etc / defaults / grub
>
> GRUB_CMDLINE_LINUX_DEFAULT = "quiet intel_iommu = on"
>
> Appreciating your support.
>
> Best regards
> SK
>
> --
> *This e-mail message is intended for the internal use of the intended
> recipient(s) only.
> The information contained herein is
> confidential/privileged. Its disclosure or reproduction is strictly
> prohibited.
> If you are not the intended recipient, please inform the sender
> immediately, do not disclose it internally or to third parties and destroy
> it.
>
> In the course of our business relationship and for business purposes
> only, Valeo may need to process some of your personal data.
> For more
> information, please refer to the Valeo Data Protection Statement and
> Privacy notice available on Valeo.com
> *
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Some VM's are not able to start

[Mark Adams via pve-user]

--- Begin Message ---
The info is in the first line of the log

"kvm: -device vfio-pci,host=:5e:00.0,id=hostpci0,bus=pci.0,addr=0x10:
vfio :5e:00.0: failed to open /dev/vfio/46: Device or resource busy"

This means the device is already passed through to another running VM, or
it is being locked by something else.


On Thu, 14 May 2020 at 18:22, Sivakumar SARAVANAN <
sivakumar.saravanan.jv@valeo-siemens.com> wrote:

> Hello,
>
> Thank you for your support.
>
> But I am getting below error message after removing the 'hugepages =2'
> line in the VM config file.
>
> kvm: -device vfio-pci,host=:5e:00.0,id=hostpci0,bus=pci.0,addr=0x10:
> vfio :5e:00.0: failed to open /dev/vfio/46: Device or resource busy
> TASK ERROR: start failed: command '/usr/bin/kvm -id 145 -name
> HIL-System151 -chardev
> 'socket,id=qmp,path=/var/run/qemu-server/145.qmp,server,nowait' -mon
> 'chardev=qmp,mode=control' -chardev
> 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon
> 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/145.pid
> -daemonize -smbios 'type=1,uuid=750b5d38-2fa0-4a9b-8b0f-c8b2d5ef769d' -smp
> '4,sockets=2,cores=2,maxcpus=4' -nodefaults -boot
> 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg'
> -vnc unix:/var/run/qemu-server/145.vnc,password -no-hpet -cpu
> 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi'
> -m 32768 -object 'memory-backend-ram,id=ram-node0,size=16384M' -numa
> 'node,nodeid=0,cpus=0-1,memdev=ram-node0' -object
> 'memory-backend-ram,id=ram-node1,size=16384M' -numa
> 'node,nodeid=1,cpus=2-3,memdev=ram-node1' -device
> 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device
> 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device
> 'vmgenid,guid=e256bf4d-f862-47d0-a1ab-25821f586542' -device
> 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device
> 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device
> 'vfio-pci,host=:5e:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device
> 'VGA,id=vga,bus=pci.0,addr=0x2' -chardev
> 'socket,path=/var/run/qemu-server/145.qga,server,nowait,id=qga0' -device
> 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device
> 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi
> 'initiator-name=iqn.1993-08.org.debian:01:29191a75ef5e' -drive
> 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device
> 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive
> 'file=/dev/zvol/SSD-Storage-PRX002/vm-145-disk-0,if=none,id=drive-virtio0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on'
> -device
> 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100'
> -drive
> 'file=/dev/zvol/HDD-Storage-PRX002/vm-145-disk-1,if=none,id=drive-virtio1,cache=writethrough,format=raw,aio=threads,detect-zeroes=on'
> -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb'
> -netdev
> 'type=tap,id=net0,ifname=tap145i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown'
> -device
> 'e1000,mac=3E:DE:DC:87:CE:75,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'
> -rtc 'driftfix=slew,base=localtime' -machine 'type=pc+pve1' -global
> 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
>
>
> Best regards
> SK
>
>
> On Thu, May 14, 2020 at 6:37 PM Mark Adams via pve-user <
> pve-user@pve.proxmox.com> wrote:
>
>>
>>
>>
>> -- Forwarded message --
>> From: Mark Adams 
>> To: PVE User List 
>> Cc:
>> Bcc:
>> Date: Thu, 14 May 2020 17:36:19 +0100
>> Subject: Re: [PVE-User] Some VM's are not able to start
>> Remove the hugepages line from your vmid.conf (ie 100.conf)
>>
>> On Thu, 14 May 2020, 17:24 Sivakumar SARAVANAN, <
>> sivakumar.saravanan.jv@valeo-siemens.com> wrote:
>>
>> > Thank you so much.
>> >
>> > What is the steps to disable the hugepage ?
>> >
>> >
>> > Best regards
>> > SK
>> >
>> > On Thu, May 14, 2020 at 6:20 PM Mark Adams via pve-user <
>> > pve-user@pve.proxmox.com> wrote:
>> >
>> > >
>> > >
>> > >
>> > > -- Forwarded message --
>> > > From: Mark Adams 
>> > > To: PVE User List 
>> > > Cc:
>> > > Bcc:
>> > > Date: Thu, 14 May 2020 17:19:09 +0100
>> > > Subject: Re: [PVE-User] Some VM's are not able to start
>> > > Do you really need hugepages? if not disable it.
>> > >
>> > > On Thu, 14 May 2020 at 17:17, Sivakumar SARAVANA

Re: [PVE-User] Some VM's are not able to start

[Mark Adams via pve-user]

--- Begin Message ---
Remove the hugepages line from your vmid.conf (ie 100.conf)

On Thu, 14 May 2020, 17:24 Sivakumar SARAVANAN, <
sivakumar.saravanan.jv@valeo-siemens.com> wrote:

> Thank you so much.
>
> What is the steps to disable the hugepage ?
>
>
> Best regards
> SK
>
> On Thu, May 14, 2020 at 6:20 PM Mark Adams via pve-user <
> pve-user@pve.proxmox.com> wrote:
>
> >
> >
> >
> > -- Forwarded message --
> > From: Mark Adams 
> > To: PVE User List 
> > Cc:
> > Bcc:
> > Date: Thu, 14 May 2020 17:19:09 +0100
> > Subject: Re: [PVE-User] Some VM's are not able to start
> > Do you really need hugepages? if not disable it.
> >
> > On Thu, 14 May 2020 at 17:17, Sivakumar SARAVANAN <
> > sivakumar.saravanan.jv@valeo-siemens.com> wrote:
> >
> > > Hello Daniel,
> > >
> > > Thanks for coming back.
> > >
> > > I mean, I am unable to power ON the VM until shutdown the other VM's in
> > the
> > > same host.
> > >
> > > There are 6 VM's running on each Host, But sometimes all 6 VM's would
> run
> > > without any issue. But Sometimes if I stop ( Shutdown) and Power ON (
> > > Start) getting an error saying as below. But each will have 32 GB
> memory.
> > >
> > > start failed: hugepage allocation failed at
> > > /usr/share/perl5/PVE/QemuServer/Memory.pm line 541.
> > >
> > > Appreciating your suggestion.
> > >
> > >
> > >
> > >
> > > Best regards
> > > SK
> > >
> > > On Thu, May 14, 2020 at 5:46 PM Daniel Berteaud <
> > > dan...@firewall-services.com> wrote:
> > >
> > > >
> > > >
> > > > - Le 14 Mai 20, à 17:38, Sivakumar SARAVANAN
> > > > sivakumar.saravanan.jv@valeo-siemens.com a écrit :
> > > >
> > > > > Hello,
> > > > >
> > > > > We have implemented the Proxmox VE in our environment.
> > > > >
> > > > > So each server will have a maximum 6 VM. But not able to start the
> > few
> > > > VM's
> > > > > ON until we bring down the 1 or 2 VM's in the same Host.
> > > > >
> > > >
> > > > Please describe what you mean by "not able to start"
> > > >
> > > > Cheers,
> > > > Daniel
> > > >
> > > > --
> > > > [ https://www.firewall-services.com/ ]
> > > > Daniel Berteaud
> > > > FIREWALL-SERVICES SAS, La sécurité des réseaux
> > > > Société de Services en Logiciels Libres
> > > > Tél : +33.5 56 64 15 32
> > > > Matrix: @dani:fws.fr
> > > > [ https://www.firewall-services.com/ |
> > https://www.firewall-services.com
> > > ]
> > > >
> > > > ___
> > > > pve-user mailing list
> > > > pve-user@pve.proxmox.com
> > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > >
> > >
> > > --
> > > *This e-mail message is intended for the internal use of the intended
> > > recipient(s) only.
> > > The information contained herein is
> > > confidential/privileged. Its disclosure or reproduction is strictly
> > > prohibited.
> > > If you are not the intended recipient, please inform the sender
> > > immediately, do not disclose it internally or to third parties and
> > destroy
> > > it.
> > >
> > > In the course of our business relationship and for business purposes
> > > only, Valeo may need to process some of your personal data.
> > > For more
> > > information, please refer to the Valeo Data Protection Statement and
> > > Privacy notice available on Valeo.com
> > > <https://www.valeo.com/en/ethics-and-compliance/#principes>*
> > > ___
> > > pve-user mailing list
> > > pve-user@pve.proxmox.com
> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> >
> >
> > -- Forwarded message --
> > From: Mark Adams via pve-user 
> > To: PVE User List 
> > Cc: Mark Adams 
> > Bcc:
> > Date: Thu, 14 May 2020 17:19:09 +0100
> > Subject: Re: [PVE-User] Some VM's are not able to start
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
>
> --
> *This e-mail message is intended for the internal use of the intended
> recipient(s) only.
> The information contained herein is
> confidential/privileged. Its disclosure or reproduction is strictly
> prohibited.
> If you are not the intended recipient, please inform the sender
> immediately, do not disclose it internally or to third parties and destroy
> it.
>
> In the course of our business relationship and for business purposes
> only, Valeo may need to process some of your personal data.
> For more
> information, please refer to the Valeo Data Protection Statement and
> Privacy notice available on Valeo.com
> <https://www.valeo.com/en/ethics-and-compliance/#principes>*
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Some VM's are not able to start

[Mark Adams via pve-user]

--- Begin Message ---
Do you really need hugepages? if not disable it.

On Thu, 14 May 2020 at 17:17, Sivakumar SARAVANAN <
sivakumar.saravanan.jv@valeo-siemens.com> wrote:

> Hello Daniel,
>
> Thanks for coming back.
>
> I mean, I am unable to power ON the VM until shutdown the other VM's in the
> same host.
>
> There are 6 VM's running on each Host, But sometimes all 6 VM's would run
> without any issue. But Sometimes if I stop ( Shutdown) and Power ON (
> Start) getting an error saying as below. But each will have 32 GB memory.
>
> start failed: hugepage allocation failed at
> /usr/share/perl5/PVE/QemuServer/Memory.pm line 541.
>
> Appreciating your suggestion.
>
>
>
>
> Best regards
> SK
>
> On Thu, May 14, 2020 at 5:46 PM Daniel Berteaud <
> dan...@firewall-services.com> wrote:
>
> >
> >
> > - Le 14 Mai 20, à 17:38, Sivakumar SARAVANAN
> > sivakumar.saravanan.jv@valeo-siemens.com a écrit :
> >
> > > Hello,
> > >
> > > We have implemented the Proxmox VE in our environment.
> > >
> > > So each server will have a maximum 6 VM. But not able to start the few
> > VM's
> > > ON until we bring down the 1 or 2 VM's in the same Host.
> > >
> >
> > Please describe what you mean by "not able to start"
> >
> > Cheers,
> > Daniel
> >
> > --
> > [ https://www.firewall-services.com/ ]
> > Daniel Berteaud
> > FIREWALL-SERVICES SAS, La sécurité des réseaux
> > Société de Services en Logiciels Libres
> > Tél : +33.5 56 64 15 32
> > Matrix: @dani:fws.fr
> > [ https://www.firewall-services.com/ | https://www.firewall-services.com
> ]
> >
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
>
> --
> *This e-mail message is intended for the internal use of the intended
> recipient(s) only.
> The information contained herein is
> confidential/privileged. Its disclosure or reproduction is strictly
> prohibited.
> If you are not the intended recipient, please inform the sender
> immediately, do not disclose it internally or to third parties and destroy
> it.
>
> In the course of our business relationship and for business purposes
> only, Valeo may need to process some of your personal data.
> For more
> information, please refer to the Valeo Data Protection Statement and
> Privacy notice available on Valeo.com
> *
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] critical HA problem on a PVE6 cluster

[Mark Adams via pve-user]

--- Begin Message ---
As Eneko already said, this really sounds like a network problem - if your
hosts lose connectivity to each other they will reboot themselves, and it
sounds like this is what happened to you.

You are sure there has been no changes to your network around the time this
happened? Have you checked your switch config is still right (maybe it
reset?)

Maybe the switches have bugged out and need a reboot? check the logs on
them for errors.

On Mon, 11 May 2020 at 18:13, Herve Ballans 
wrote:

> Hi again, (sorry for the spam!).
>
> I just found logs just before the crash of one of the nodes (time of
> crash : 18:36:36). It could be more useful than logs sent
> previously...(I deleted here normal events)
>
> First, several messages like that (first one at 11:00 am):
>
> May  6 18:33:25 inf-proxmox7 corosync[2648]:   [TOTEM ] Token has not
> been received in 2212 ms
> May  6 18:33:26 inf-proxmox7 corosync[2648]:   [TOTEM ] A processor
> failed, forming new configuration.
>
> Then:
>
> May  6 18:34:14 inf-proxmox7 corosync[2648]:   [MAIN  ] Completed
> service synchronization, ready to provide service.
> May  6 18:34:14 inf-proxmox7 pvesr[3342642]: error with cfs lock
> 'file-replication_cfg': got lock request timeout
> May  6 18:34:14 inf-proxmox7 systemd[1]: pvesr.service: Main process
> exited, code=exited, status=17/n/a
> May  6 18:34:14 inf-proxmox7 systemd[1]: pvesr.service: Failed with
> result 'exit-code'.
> May  6 18:34:14 inf-proxmox7 systemd[1]: Failed to start Proxmox VE
> replication runner.
> May  6 18:34:14 inf-proxmox7 pmxcfs[2602]: [status] notice:
> cpg_send_message retry 30
> May  6 18:34:14 inf-proxmox7 pmxcfs[2602]: [status] notice:
> cpg_send_message retried 30 times
>
> Then again a series of processor failed messages (in totally 147 before
> the crash):
>
> May  6 18:35:03 inf-proxmox7 corosync[2648]:   [TOTEM ] Token has not
> been received in 2212 ms
> May  6 18:35:04 inf-proxmox7 corosync[2648]:   [TOTEM ] A processor
> failed, forming new configuration.
>
> Then:
>
> May  6 18:35:40 inf-proxmox7 pmxcfs[2602]: [dcdb] notice: start cluster
> connection
> May  6 18:35:40 inf-proxmox7 pmxcfs[2602]: [dcdb] crit: cpg_join failed: 14
> May  6 18:35:40 inf-proxmox7 pmxcfs[2602]: [dcdb] crit: can't initialize
> service
> May  6 18:35:40 inf-proxmox7 pve-ha-lrm[5528]: lost lock
> 'ha_agent_inf-proxmox7_lock - cfs lock update failed - Device or
> resource busy
> May  6 18:35:40 inf-proxmox7 pve-ha-crm[5421]: status change slave =>
> wait_for_quorum
> May  6 18:35:41 inf-proxmox7 corosync[2648]:   [TOTEM ] A new membership
> (1.e60) was formed. Members joined: 1 3 4 5
>
> Then:
>
> May  6 18:35:41 inf-proxmox7 pmxcfs[2602]: [status] notice: node has quorum
> May  6 18:35:42 inf-proxmox7 pmxcfs[2602]: [status] notice:
> cpg_send_message retried 1 times
> May  6 18:35:42 inf-proxmox7 pmxcfs[2602]: [status] notice: received
> sync request (epoch 1/2592/0031)
> May  6 18:35:42 inf-proxmox7 pmxcfs[2602]: [status] notice: received
> sync request (epoch 1/2592/0032)
> May  6 18:35:42 inf-proxmox7 pmxcfs[2602]: [dcdb] crit: cpg_send_message
> failed: 9
> May  6 18:35:42 inf-proxmox7 pmxcfs[2602]: [dcdb] crit: cpg_send_message
> failed: 9
> May  6 18:35:42 inf-proxmox7 pmxcfs[2602]: [status] notice: received all
> states
> May  6 18:35:42 inf-proxmox7 pmxcfs[2602]: [status] notice: all data is
> up to date
> May  6 18:35:42 inf-proxmox7 pmxcfs[2602]: [status] notice:
> dfsm_deliver_queue: queue length 144
>
> Then:
>
> May  6 18:35:57 inf-proxmox7 corosync[2648]:   [TOTEM ] A new membership
> (1.e64) was formed. Members left: 3 4
> May  6 18:35:57 inf-proxmox7 corosync[2648]:   [TOTEM ] Failed to
> receive the leave message. failed: 3 4
>
> And finally crash after this last logs:
>
> May  6 18:36:36 inf-proxmox7 pve-ha-crm[5421]: status change
> wait_for_quorum => slave
> May  6 18:36:36 inf-proxmox7 systemd[1]: pvesr.service: Main process
> exited, code=exited, status=17/n/a
> May  6 18:36:36 inf-proxmox7 systemd[1]: pvesr.service: Failed with
> result 'exit-code'.
> May  6 18:36:36 inf-proxmox7 systemd[1]: Failed to start Proxmox VE
> replication runner.
> May  6 18:36:36 inf-proxmox7 pve-ha-crm[5421]: loop take too long (51
> seconds)
> May  6 18:36:36 inf-proxmox7 systemd[1]: watchdog-mux.service: Succeeded.
> May  6 18:36:36 inf-proxmox7 kernel: [1292969.953131] watchdog:
> watchdog0: watchdog did not stop!
> May  6 18:36:36 inf-proxmox7 pvestatd[2894]: status update time (5.201
> seconds)
> ^@^@^@^@^@^@
>
> following by a binary part...
>
> Thank you again,
> Hervé
>
> On 11/05/2020 10:39, Eneko Lacunza wrote:
> >>> Hi Hervé,
> >>>
> >>> This seems a network issue. What is the network setup in this
> >>> cluster? What logs in syslog about corosync and pve-cluster?
> >>>
> >>> Don't enable HA until you have a stable cluster quorum.
> >>>
> >>> Cheers
> >>> Eneko
> >>>
> >>> El 11/5/20 a las 10:35, Herve Ballans escribió:
>  Hi everybody,
> 
>  I would like to 

[PVE-User] VM Start timeout with pci passthrough

[Mark Adams via pve-user]

--- Begin Message ---
Hi All,

I am having the issue that is detailed in this forum post:

https://forum.proxmox.com/threads/vm-start-timeout-with-pci-gpu.45843/

I thought I would take it to the mailing list to see if anyone here has any
ideas?

VM's boot fine the first time the machine starts up, and even a few reboots
work fine - but eventually the timeout start happening.

If you start the vm manually it works fine.

Also if you lower the ram allocation (for instance from 64 -> 32) then it
also seems to boot fine.

Any thoughts on this very welcome!

Regards,
Mark
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] lzo files conundrum

[Mark Adams via pve-user]

--- Begin Message ---
Is the data inside the VM's different? maybe the data on the bigger one is
not as compressible?

On Wed, 11 Mar 2020, 08:07 Renato Gallo via pve-user, <
pve-user@pve.proxmox.com> wrote:

>
>
>
> -- Forwarded message --
> From: Renato Gallo 
> To: g noto 
> Cc: Alwin Antreich , pve-user <
> pve-user@pve.proxmox.com>
> Bcc:
> Date: Wed, 11 Mar 2020 09:07:42 +0100 (CET)
> Subject: Re: [PVE-User] lzo files conundrum
> Hello,
>
> Might be I was not clear, I apologize.
> it's not that I want both backups to be equal.
> Both machines are set up as scsi and have equal storage configuration.
> Both machines sit on the same proxmox server.
> I would like to know why the backups gigas are so different one from
> another.
>
> Renato Gallo
>
>
>
> - Original Message -
> From: "Alwin Antreich" 
> To: "PVE User List" 
> Cc: "renato" , "Gaspare Noto" <
> g.n...@way-interactive.com>
> Sent: Wednesday, March 11, 2020 8:55:13 AM
> Subject: Re: [PVE-User] lzo files conundrum
>
> Hello Renato,
>
> On Wed, Mar 11, 2020 at 07:35:21AM +0100, Renato Gallo via pve-user wrote:
> > Date: Wed, 11 Mar 2020 07:35:21 +0100 (CET)
> > From: Renato Gallo 
> > To: pve-user@pve.proxmox.com
> > Cc: g noto 
> > Subject: lzo files conundrum
> > X-Mailer: Zimbra 8.8.15_GA_3829 (ZimbraWebClient - FF68
> >  (Linux)/8.8.15_GA_3829)
> >
> > Hello,
> >
> > we have two machines same system 100 Gb of virtual disk both have the
> disks on a zfspool.
> > from the df -h point of view the disks use is quite similar.
> > we backup those machine regularly during the weekend using the scheduled
> backup of proxmox from the datastore menu.
> > we cannot understand why one machine have a .lzo file of 9.something
> gigas and the other has a backup .lzo file of around 40 gigas.
> > can you explain this to us ?
> Possibly thin-provisioning.
>
> In the storage definition the _Thin provision_ needs to be set, to
> create thin-provisioned zVOLs from the start. And OFC, you will need to
> have the VM disks setup with SCSI and discard on to run a TRIM inside
> the VM.
>
> --
> Cheers,
> Alwin
>
>
>
> -- Forwarded message --
> From: Renato Gallo via pve-user 
> To: g noto 
> Cc: Renato Gallo , pve-user <
> pve-user@pve.proxmox.com>
> Bcc:
> Date: Wed, 11 Mar 2020 09:07:42 +0100 (CET)
> Subject: Re: [PVE-User] lzo files conundrum
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Creating VM with thin provisioning (please help me understand it).

[Mark Adams via pve-user]

--- Begin Message ---
REF: "Thin provisioning is set on the storage, it is a checkbox and of
course it has to be a storage type than can be thin provisioned (ie
lvmthin, zfs, ceph etc)."

I have to correct myself on this sorry, it's been a long time since I used
lvmthin This checkbox option is only available on zfs storage as I
guess you have the option for both, where as lvmthin should always be thin?

On Thu, 5 Mar 2020 at 16:43, Mark Adams  wrote:

> Thin provisioning is set on the storage, it is a checkbox and of course it
> has to be a storage type than can be thin provisioned (ie lvmthin, zfs,
> ceph etc).
>
> Then every virtual disk that is created on that storage type is thin
> provisioned.
>
> Regards,
> Mark
>
> On Thu, 5 Mar 2020, 16:28 Leandro Roggerone, 
> wrote:
>
>> Hi guys , here i am trying to get used to pve. (I came from VM ware EXSi).
>> Im used to use "thin provisioning" option in vmware.
>> It allowed me to set maximum drive space allocation but vm will use
>> minimum and dynamically grow as guest OS needs
>> .
>> How should I accomplish same on pve  ?
>> I created a VM and set 200GB at the storage capacity but can not set any
>> flag about thin provisioning.
>> Now , If I list lslbl , it  shows that created VM is taking the 200GB.
>> root@pve:~# lsblk
>> NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
>> sda8:00  5.5T  0 disk
>> ├─sda1 8:10 1007K  0 part
>> ├─sda2 8:20  512M  0 part
>> └─sda3 8:30  3.9T  0 part
>>   ├─pve-swap 253:008G  0 lvm  [SWAP]
>>   ├─pve-root 253:10  1.2T  0 lvm  /
>>   ├─pve-data_tmeta   253:20 15.8G  0 lvm
>>   │ └─pve-data-tpool 253:40  1.9T  0 lvm
>>   │   ├─pve-data 253:50  1.9T  0 lvm
>>   │   └─pve-vm--100--disk--0 253:60  200G  0 lvm
>>   └─pve-data_tdata   253:30  1.9T  0 lvm
>> └─pve-data-tpool 253:40  1.9T  0 lvm
>>   ├─pve-data 253:50  1.9T  0 lvm
>>   └─pve-vm--100--disk--0 253:60  200G  0 lvm
>> sr0   11:01 1024M  0 rom
>>
>>
>> I can see from gui , that is is possible to expand disk.
>> So ... perhaps I should assign a little storage at beginning and then
>> manually expand if needed.
>> Please share your idea about this.
>> I need to begin creating some VMs and work on them , I dont want to create
>> in a wrong way and have to destroy later.
>>
>> <
>> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
>> >
>> Libre
>> de virus. www.avast.com
>> <
>> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
>> >
>> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>> ___
>> pve-user mailing list
>> pve-user@pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
>
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Creating VM with thin provisioning (please help me understand it).

[Mark Adams via pve-user]

--- Begin Message ---
Atila - just to follow up on Giannis discard notes, depending on what OS
and filesystems you use inside of your VMs, you may need to run fstrim,
mount with different options, or run specific commands (ie zpool trim for
zfs) to get it all working correctly.

Regards,
Mark

On Thu, 5 Mar 2020, 22:22 Atila Vasconcelos,  wrote:

> WOW!
>
> I was searching for this piece of information for a long long time!!!
>
> Thank you Gianni, you changed my life! For real! Thanks a lot!
>
>
> ABV
>
> On 2020-03-05 1:32 p.m., Gianni Milo wrote:
> >
> > One last thing, normally the guest OS won't release the occupied disk
> space
> > on the host when you delete data within it. For example, let's say that
> you
> > have a VM which is using 30GB of data and then you delete 10GB worth of
> > data. That 10GB of space won't be released from the VM ThinLV volume,
> > unless you use virtio-scsi as storage controller for the VM and the
> discard
> > option is enabled.
> >
> > More on LVM:
> >
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/logical_volume_manager_administration/index
> >
> >
> > G.
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Creating VM with thin provisioning (please help me understand it).

[Mark Adams via pve-user]

--- Begin Message ---
Gianni provided a great reply (please do go and read the admin
documentation), but let me just follow up to one of your points below.

On Thu, 5 Mar 2020, 19:46 Leandro Roggerone, 
wrote:

> Dear Mark, you said:
> "Then every virtual disk that is created on that storage type is thin
> provisioned."
> Ok ... good to hear that.
>
> So ... Lets suppose I created 5 VMs about 200Gb each on a 1TB space.
> Im using less than 50% on each VM and now I need  to add a 6th VM ?  Even
> Im using thin storage , I will have no free space.
>

If you have configured everything correctly, then in your 50% scenario you
will have used 500G of your 1T. You should have space for more VMs but like
Gianni said make sure you keep a close eye on utilisation, as running out
of space will be very bad for you.

Also note that you can't retrospectively enable thin provisioning (as a
general rule... there are some manual exceptions). This means that if you
created these 5 vms before you ticked that thin provisioning box, they are
thick provisioned and not thin. Which from what you say about being out of
space with 5 x 200G disks sounds like what you did.

Regards,
Mark

What can I do ?
> a) Should assign minimun capacity during creating time and enlarge later if
> needed ?
> b ) Should I look for unused space among existing  VMs and reduce the disk
> ?
> Please let me know what is your criteria to assign storage to new VM.
> Leandro.
>
>
>
>
> El jue., 5 mar. 2020 a las 13:44, Mark Adams via pve-user (<
> pve-user@pve.proxmox.com>) escribió:
>
> >
> >
> >
> > -- Forwarded message --
> > From: Mark Adams 
> > To: PVE User List 
> > Cc:
> > Bcc:
> > Date: Thu, 5 Mar 2020 16:43:08 +
> > Subject: Re: [PVE-User] Creating VM with thin provisioning (please help
> me
> > understand it).
> > Thin provisioning is set on the storage, it is a checkbox and of course
> it
> > has to be a storage type than can be thin provisioned (ie lvmthin, zfs,
> > ceph etc).
> >
> > Then every virtual disk that is created on that storage type is thin
> > provisioned.
> >
> > Regards,
> > Mark
> >
> > On Thu, 5 Mar 2020, 16:28 Leandro Roggerone, 
> > wrote:
> >
> > > Hi guys , here i am trying to get used to pve. (I came from VM ware
> > EXSi).
> > > Im used to use "thin provisioning" option in vmware.
> > > It allowed me to set maximum drive space allocation but vm will use
> > > minimum and dynamically grow as guest OS needs
> > > .
> > > How should I accomplish same on pve  ?
> > > I created a VM and set 200GB at the storage capacity but can not set
> any
> > > flag about thin provisioning.
> > > Now , If I list lslbl , it  shows that created VM is taking the 200GB.
> > > root@pve:~# lsblk
> > > NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> > > sda8:00  5.5T  0 disk
> > > ├─sda1 8:10 1007K  0 part
> > > ├─sda2 8:20  512M  0 part
> > > └─sda3 8:30  3.9T  0 part
> > >   ├─pve-swap 253:008G  0 lvm  [SWAP]
> > >   ├─pve-root 253:10  1.2T  0 lvm  /
> > >   ├─pve-data_tmeta   253:20 15.8G  0 lvm
> > >   │ └─pve-data-tpool 253:40  1.9T  0 lvm
> > >   │   ├─pve-data 253:50  1.9T  0 lvm
> > >   │   └─pve-vm--100--disk--0 253:60  200G  0 lvm
> > >   └─pve-data_tdata   253:30  1.9T  0 lvm
> > > └─pve-data-tpool 253:40  1.9T  0 lvm
> > >   ├─pve-data 253:50  1.9T  0 lvm
> > >   └─pve-vm--100--disk--0 253:60  200G  0 lvm
> > > sr0   11:01 1024M  0 rom
> > >
> > >
> > > I can see from gui , that is is possible to expand disk.
> > > So ... perhaps I should assign a little storage at beginning and then
> > > manually expand if needed.
> > > Please share your idea about this.
> > > I need to begin creating some VMs and work on them , I dont want to
> > create
> > > in a wrong way and have to destroy later.
> > >
> > > <
> > >
> >
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
> > > >
> > > Libre
> > > de virus. www.avast.com
> > > <
> > >
> >
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=we

Re: [PVE-User] Creating VM with thin provisioning (please help me understand it).

[Mark Adams via pve-user]

--- Begin Message ---
Thin provisioning is set on the storage, it is a checkbox and of course it
has to be a storage type than can be thin provisioned (ie lvmthin, zfs,
ceph etc).

Then every virtual disk that is created on that storage type is thin
provisioned.

Regards,
Mark

On Thu, 5 Mar 2020, 16:28 Leandro Roggerone, 
wrote:

> Hi guys , here i am trying to get used to pve. (I came from VM ware EXSi).
> Im used to use "thin provisioning" option in vmware.
> It allowed me to set maximum drive space allocation but vm will use
> minimum and dynamically grow as guest OS needs
> .
> How should I accomplish same on pve  ?
> I created a VM and set 200GB at the storage capacity but can not set any
> flag about thin provisioning.
> Now , If I list lslbl , it  shows that created VM is taking the 200GB.
> root@pve:~# lsblk
> NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> sda8:00  5.5T  0 disk
> ├─sda1 8:10 1007K  0 part
> ├─sda2 8:20  512M  0 part
> └─sda3 8:30  3.9T  0 part
>   ├─pve-swap 253:008G  0 lvm  [SWAP]
>   ├─pve-root 253:10  1.2T  0 lvm  /
>   ├─pve-data_tmeta   253:20 15.8G  0 lvm
>   │ └─pve-data-tpool 253:40  1.9T  0 lvm
>   │   ├─pve-data 253:50  1.9T  0 lvm
>   │   └─pve-vm--100--disk--0 253:60  200G  0 lvm
>   └─pve-data_tdata   253:30  1.9T  0 lvm
> └─pve-data-tpool 253:40  1.9T  0 lvm
>   ├─pve-data 253:50  1.9T  0 lvm
>   └─pve-vm--100--disk--0 253:60  200G  0 lvm
> sr0   11:01 1024M  0 rom
>
>
> I can see from gui , that is is possible to expand disk.
> So ... perhaps I should assign a little storage at beginning and then
> manually expand if needed.
> Please share your idea about this.
> I need to begin creating some VMs and work on them , I dont want to create
> in a wrong way and have to destroy later.
>
> <
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
> >
> Libre
> de virus. www.avast.com
> <
> https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail
> >
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
--- End Message ---
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Reboot on psu failure in redundant setup

[Mark Adams]

Hi Daniel, Thomas,

When I looked back at the docs after reading Daniels email I saw exactly
what your saying Thomas, that its hardware watchdogs only which are
disabled and need to be manually enabled, and that pve-ha-crm has a
software watchdog enabled by default.

Thanks for both your responses and clearing this up for me.

Cheers,
Mark


On Fri, 8 Nov 2019 at 15:48, Thomas Lamprecht 
wrote:

> On 11/8/19 4:22 PM, Mark Adams wrote:
> > I didn't configure it to do
> > this myself, so is this an automatic feature? Everything I have read says
> > it should be configured manually.
>
> Maybe my previous mail did not answered this point in a good way.
>
> You need to configure *hardware-based* Watchdogs manually. But the
> fallback will *always* be the Linux Kernel Softdog (which is very
> reliable, from experience ^^) - else, without fencing, HA recovery
> could never be done in a safe way (double resource usage).
>
> cheers,
> Thomas
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] Reboot on psu failure in redundant setup

[Mark Adams]

Hi All,

This cluster is on 5.4-11.

This is most probably a hardware issue either with ups or server psus, but
wanted to check if there is any default watchdog or auto reboot in a
proxmox HA cluster.

Explanation of what happened:

All servers have redundant psu, being fed from separate ups in
separate racks on separate feeds. One of the UPS went out, and when it did
all nodes rebooted. They were functioning normally after the reboot, but I
wasn't expecting the reboot to occur.

When the UPS went down, it also took down all of the core network because
the power was not connected up in a redundant fashion. Ceph and "LAN"
traffic was blocked because of this. Did a watchdog reboot each node
because it lost contact with its cluster peers? I didn't configure it to do
this myself, so is this an automatic feature? Everything I have read says
it should be configured manually.

Thanks in advance.

Cheers,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] SQL server 2014 poor performances

[Mark Adams]

There is a WD Blue SSD - but it is a desktop drive, you probably shouldn't
use it in a server.

Are you using the virtio-scsi blockdev and the newest virtio drivers? also,
have you tried with writeback enabled?

Have you tested the performance of your ssd zpool from the command line on
the host?

On Mon, 28 Oct 2019 at 15:46, Michael Rasmussen via pve-user <
pve-user@pve.proxmox.com> wrote:

>
>
>
> -- Forwarded message --
> From: Michael Rasmussen 
> To: pve-user@pve.proxmox.com
> Cc:
> Bcc:
> Date: Mon, 28 Oct 2019 16:46:23 +0100
> Subject: Re: [PVE-User] SQL server 2014 poor performances
> On Mon, 28 Oct 2019 15:47:18 +0100 (CET)
> Fabrizio Cuseo  wrote:
>
> > Hello.
> > I have a customer with proxmox 5.X, 4 x SSD (WD blue) in raid-10 ZFS
> > configuration, Poweredge R710 dual xeon and 144Gbyte RAM. Same
> > problem with 4 x SAS 15k rpm drives.
> >
> Are you sure it is SSD? I don't recollect that WD has produced WD blue
> as SSD.
>
> --
> Hilsen/Regards
> Michael Rasmussen
>
> Get my public GnuPG keys:
> michael  rasmussen  cc
> https://pgp.key-server.io/pks/lookup?search=0xD3C9A00E
> mir  datanom  net
> https://pgp.key-server.io/pks/lookup?search=0xE501F51C
> mir  miras  org
> https://pgp.key-server.io/pks/lookup?search=0xE3E80917
> --
> /usr/games/fortune -es says:
> Follow each decision as closely as possible with its associated action.
> - The Elements of Programming Style (Kernighan & Plaugher)
>
>
>
> -- Forwarded message --
> From: Michael Rasmussen via pve-user 
> To: pve-user@pve.proxmox.com
> Cc: Michael Rasmussen 
> Bcc:
> Date: Mon, 28 Oct 2019 16:46:23 +0100
> Subject: Re: [PVE-User] SQL server 2014 poor performances
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] AMD ZEN 2 (EPYC 7002 aka "rome") kernel requirements

[Mark Adams]

Thanks for your responses Thomas and Fabian.

On Fri, 27 Sep 2019 at 09:37, Fabian Grünbichler 
wrote:

> On September 27, 2019 10:30 am, Mark Adams wrote:
> > Hi All,
> >
> > I'm trying out one of these new processors, and it looks like I need at
> > least 5.2 kernel to get some support, preferably 5.3.
> >
> > At present the machine will boot in to proxmox, but IOMMU does not work,
> > and I can see ECC memory is not working.
> >
> > So my question is, whats the recommended way to get a newer kernel than
> is
> > provided by the pve-kernel package? I understand that pve-kernel uses the
> > newer ubuntu kernel rather than the debian buster one, but are you
> building
> > anything else in to it? Will proxmox work ok if I install the ubuntu 5.3
> > kernel?
>
> these are the patches we currently ship on-top of Ubuntu Disco's kernel:
>
>
> https://git.proxmox.com/?p=pve-kernel.git;a=tree;f=patches/kernel;hb=refs/heads/master
>
> another thing we add are the ZFS modules. not sure which version Ubuntu
> Eoan ships there.
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] AMD ZEN 2 (EPYC 7002 aka "rome") kernel requirements

[Mark Adams]

Hi All,

I'm trying out one of these new processors, and it looks like I need at
least 5.2 kernel to get some support, preferably 5.3.

At present the machine will boot in to proxmox, but IOMMU does not work,
and I can see ECC memory is not working.

So my question is, whats the recommended way to get a newer kernel than is
provided by the pve-kernel package? I understand that pve-kernel uses the
newer ubuntu kernel rather than the debian buster one, but are you building
anything else in to it? Will proxmox work ok if I install the ubuntu 5.3
kernel?

Cheers,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] PVE 5.4: cannot move disk image to Ceph

[Mark Adams]

Is it potentially an issue with having the same pool name on 2 different
ceph clusters?

is there a vm-112-disk-0 on vdisks_cluster2?

On Fri, 6 Sep 2019, 12:45 Uwe Sauter,  wrote:

> Hello Alwin,
>
> Am 06.09.19 um 11:32 schrieb Alwin Antreich:
> > Hello Uwe,
> >
> > On Fri, Sep 06, 2019 at 10:41:18AM +0200, Uwe Sauter wrote:
> >> Hi,
> >>
> >> I'm having trouble moving a disk image to Ceph. Moving between local
> disks and NFS share is working.
> >>
> >> The error given is:
> >>
> >> 
> >> create full clone of drive scsi0
> (aurel-cluster1-VMs:112/vm-112-disk-0.qcow2)
> >> rbd: create error: (17) File exists
> >> TASK ERROR: storage migration failed: error with cfs lock
> 'storage-vdisks_vm': rbd create vm-112-disk-0' error: rbd: create error:
> >> (17) File exists
> >> 
> > Can you see anything in the ceph logs? And on what version (pveversion
> > -v) are you on?
>
> Nothing obvious in the logs. The cluster is healthy
>
> root@px-bravo-cluster:~# ceph status
>   cluster:
> id: 982484e6-69bf-490c-9b3a-942a179e759b
> health: HEALTH_OK
>
>   services:
> mon: 3 daemons, quorum
> px-alpha-cluster,px-bravo-cluster,px-charlie-cluster
> mgr: px-alpha-cluster(active), standbys: px-bravo-cluster,
> px-charlie-cluster
> osd: 9 osds: 9 up, 9 in
>
>   data:
> pools:   1 pools, 128 pgs
> objects: 14.76k objects, 56.0GiB
> usage:   163GiB used, 3.99TiB / 4.15TiB avail
> pgs: 128 active+clean
>
>   io:
> client:   2.31KiB/s wr, 0op/s rd, 0op/s wr
>
> I'm on a fully up-to-date PVE 5.4 (all three nodes).
>
> root@px-bravo-cluster:~# pveversion -v
> proxmox-ve: 5.4-2 (running kernel: 4.15.18-20-pve)
> pve-manager: 5.4-13 (running version: 5.4-13/aee6f0ec)
> pve-kernel-4.15: 5.4-8
> pve-kernel-4.15.18-20-pve: 4.15.18-46
> pve-kernel-4.15.18-19-pve: 4.15.18-45
> ceph: 12.2.12-pve1
> corosync: 2.4.4-pve1
> criu: 2.11.1-1~bpo90
> glusterfs-client: 3.8.8-1
> ksm-control-daemon: 1.2-2
> libjs-extjs: 6.0.1-2
> libpve-access-control: 5.1-12
> libpve-apiclient-perl: 2.0-5
> libpve-common-perl: 5.0-54
> libpve-guest-common-perl: 2.0-20
> libpve-http-server-perl: 2.0-14
> libpve-storage-perl: 5.0-44
> libqb0: 1.0.3-1~bpo9
> lvm2: 2.02.168-pve6
> lxc-pve: 3.1.0-6
> lxcfs: 3.0.3-pve1
> novnc-pve: 1.0.0-3
> proxmox-widget-toolkit: 1.0-28
> pve-cluster: 5.0-38
> pve-container: 2.0-40
> pve-docs: 5.4-2
> pve-edk2-firmware: 1.20190312-1
> pve-firewall: 3.0-22
> pve-firmware: 2.0-7
> pve-ha-manager: 2.0-9
> pve-i18n: 1.1-4
> pve-libspice-server1: 0.14.1-2
> pve-qemu-kvm: 3.0.1-4
> pve-xtermjs: 3.12.0-1
> qemu-server: 5.0-54
> smartmontools: 6.5+svn4324-1
> spiceterm: 3.0-5
> vncterm: 1.5-3
> zfsutils-linux: 0.7.13-pve1~bpo2
>
>
>
> >>
> >> but this is not true:
> >>
> >> 
> >> root@px-bravo-cluster:~# rbd -p vdisks ls
> >> vm-106-disk-0
> >> vm-113-disk-0
> >> vm-113-disk-1
> >> vm-113-disk-2
> >> vm-118-disk-0
> >> vm-119-disk-0
> >> vm-120-disk-0
> >> vm-125-disk-0
> >> vm-125-disk-1
> >> 
> > Can you create the image by hand (rbd -p rbd create vm-112-disk-0 --size
> > 1G)? And (rbd -p rbd rm vm-112-disk-0) for delete, ofc.
>
> root@px-bravo-cluster:~# rbd -p vdisks create vm-112-disk-0 --size 1G
> rbd: create error: (17) File exists
> 2019-09-06 11:35:20.943998 7faf704660c0 -1 librbd: rbd image vm-112-disk-0
> already exists
>
> root@px-bravo-cluster:~# rbd -p vdisks create test --size 1G
>
> root@px-bravo-cluster:~# rbd -p vdisks ls
> test
> vm-106-disk-0
> vm-113-disk-0
> vm-113-disk-1
> vm-113-disk-2
> vm-118-disk-0
> vm-119-disk-0
> vm-120-disk-0
> vm-125-disk-0
> vm-125-disk-1
>
> root@px-bravo-cluster:~# rbd -p vdisks rm test
> Removing image: 100% complete...done.
>
> root@px-bravo-cluster:~# rbd -p vdisks rm vm-112-disk-0
> 2019-09-06 11:36:07.570749 7eff7cff9700 -1 librbd::image::OpenRequest:
> failed to retreive immutable metadata: (2) No such file or
> directory
> Removing image: 0% complete...failed.
> rbd: delete error: (2) No such file or directory
>
>
> >
> >>
> >> Here is the relevant part of my storage.cfg:
> >>
> >> 
> >> nfs: aurel-cluster1-VMs
> >>  export /backup/proxmox-infra/VMs
> >>  path /mnt/pve/aurel-cluster1-VMs
> >>  server X.X.X.X
> >>  content images
> >>  options vers=4.2
> >>
> >>
> >> rbd: vdisks_vm
> >>  content images
> >>  krbd 0
> >>  pool vdisks
> >> 
> > Is this the complete storage.cfg?
>
> No, only the parts that are relevant for this particular move. Here's the
> complete file:
>
> 
> rbd: vdisks_vm
> content images
> krbd 0
> pool vdisks
>
> dir: local-hdd
> path /mnt/local
> content images,iso
> nodes px-alpha-cluster,px-bravo-cluster,px-charlie-cluster
> shared 0
>
> nfs: aurel-cluster1-daily
> export /backup/proxmox-infra/daily
> path /mnt/pve/aurel-cluster1-daily
> server X.X.X.X
> content backup
> maxfiles 30
> options vers=4.2
>
> nfs: 

Re: [PVE-User] Move VM's HDD incl. snapshots from one Ceph to another

[Mark Adams]

On Mon, 19 Aug 2019 at 11:59, Uwe Sauter  wrote:

> Hi,
>
> @Eneko
>
> Both clusters are hyper-converged PVE clusters each running its own Ceph
> cluster. On the older PVE 5 cluster I created a new RBD
> storage configuration for the PVE6 Ceph. So I can access the new Ceph from
> the old Cluster.
>
> I also migrated some VMs via the GUI from one Ceph to the other but those
> VMs didn't have snapshots.
>
> My question is: how can I move a RBD image with snapshots from one cluster
> to the other while keeping the history (snapshots)?
>
>
> @Mark:
>
> Do you think that this would work?
>
> rbd --cluster ceph --pool vdisks export --export-format 2 vm-101-disk-0 -
> | \
> rbd --cluster cluster2 --pool vdisks import --export-format 2 -
> vm-101-disk-0
>

Hi Uwe, I don't remember the exact command without checking the ceph docs -
but I believe it was something straight forward like rbd export
ceph/vm-101-disk-0  | ssh user@server  'rbd import - cluster2/vm-101-disk-0'

I -think- that sends all the snapshots as well, though again I can't
remember exactly (it's been a while since I did this...) - you can use rbd
export-diff and import-diff to send just the changes between snapshots
though in case it doesn't and you want to build it up from the earliest
snapshot.

Sorry that's not more exact!

Regards,
Mark


>
> Regards,
>
> Uwe
>
>
>
> Am 19.08.19 um 12:30 schrieb Mark Adams:
> > It is relatively straight forward using cli, use the rbd export (and
> > export-diff) command over ssh.
> >
> > On Mon, 19 Aug 2019, 12:26 Eneko Lacunza,  wrote:
> >
> >> Hi Uwe,
> >>
> >> El 19/8/19 a las 10:14, Uwe Sauter escribió:
> >>> is it possible to move a VM's disks from one Ceph cluster to another,
> >> including all snapshots that those disks have? The GUI
> >>> doesn't let me do it but is there some commandline magic that will move
> >> the disks and all I have to do is edit the VM's config file?
> >>>
> >>>
> >>> Background: I have two PVE clusters, one of which is already PVE 6. Now
> >> I need to move the VMs from the older to the newer cluster
> >>> so I can re-install those machines with PVE 6. I know that an upgrade
> is
> >> possible but I'd like the cleaner approach of re-installing.
> >> What storage are you using? (the snapshot thing can be a problem).
> >>
> >> Cheers
> >> Eneko
> >>
> >> --
> >> Zuzendari Teknikoa / Director Técnico
> >> Binovo IT Human Project, S.L.
> >> Telf. 943569206
> >> Astigarraga bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
> >> www.binovo.es
> >>
> >> ___
> >> pve-user mailing list
> >> pve-user@pve.proxmox.com
> >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >>
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Move VM's HDD incl. snapshots from one Ceph to another

[Mark Adams]

It is relatively straight forward using cli, use the rbd export (and
export-diff) command over ssh.

On Mon, 19 Aug 2019, 12:26 Eneko Lacunza,  wrote:

> Hi Uwe,
>
> El 19/8/19 a las 10:14, Uwe Sauter escribió:
> > is it possible to move a VM's disks from one Ceph cluster to another,
> including all snapshots that those disks have? The GUI
> > doesn't let me do it but is there some commandline magic that will move
> the disks and all I have to do is edit the VM's config file?
> >
> >
> > Background: I have two PVE clusters, one of which is already PVE 6. Now
> I need to move the VMs from the older to the newer cluster
> > so I can re-install those machines with PVE 6. I know that an upgrade is
> possible but I'd like the cleaner approach of re-installing.
> What storage are you using? (the snapshot thing can be a problem).
>
> Cheers
> Eneko
>
> --
> Zuzendari Teknikoa / Director Técnico
> Binovo IT Human Project, S.L.
> Telf. 943569206
> Astigarraga bidea 2, 2º izq. oficina 11; 20180 Oiartzun (Gipuzkoa)
> www.binovo.es
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Storage Server Recommendations Please (JR Richardson)

[Mark Adams]

Do you need your storage to be High-Availability? This also has a bearing
on what might be a good solution.

If you don't need HA, then a zfs backed linux server sharing nfs would be
very straight forward. But even better would be using the zfs over iscsi
features of proxmox so that each of your disks for your vms could be a
separate iscsi lun.

If you want a web GUI on it then a freebsd based "off the shelf" like
freenas as mj suggested would be the simplest.

Hardware wise, Personally I would always go with supermicro, asus, asrock,
gigabyte, intel etc just because of cost - but I guess this depends on your
list of suppliers/if you can build it yourself.

Regards,
Mark

On Sun, 4 Aug 2019 at 13:52, mj  wrote:

> Hi,
>
> Just our 2 cts: You should check out freenas/truenas as well. It will do
> what you require (and a bit more) :-)
>
> MJ
>
> On 8/3/19 3:02 PM, JR Richardson wrote:
> > Gilberto,
> >
> > I have 14 hypervisor nodes hitting 6 storage servers, dell 610's, and a
> > couple of backup servers. All VMs, 200 Linux servers and about 20 Win
> > servers use NFS shared storage. I'd like to consolidate storage down to
> two
> > servers.
> >
> > Nada suggested Synology RS818RP+ it is turnkey solution, I'll check into
> it.
> >
> > Thanks.
> >
> > JR
> >
> >
> > how much servers do you have?
> > ---
> > Gilberto Nunes Ferreira
> >>
> >> Hi All,
> >>
> >> I'm thinking about upgrading to 10 Gig storage network for my
> >> clusters. Current servers are dell 610 with H700 Raid controller, 2
> >> drives in Raid1 for OS and 4 drives in Raid10 for NFS shared storage
> >> to the hypervisor nodes. Network is 2xGigabit up links to all servers
> >> and storage. Load from the hypervisors is are about 200 Linux servers
> >> and about 20 Win servers. I'm not seeing any issue currently in normal
> >> operations, but I can see disk IO wait states on storage servers
> >> periodically when some of the VMs are booting or tasking storage,
> >> creating new VM disks. All storage array disks are currently 15K SAS
> >> spinners, no SSDs.
> >>
> >> I'm considering upgrading to SSD caching server with conventional
> >> storage disk back end. Main thing is to get network up to 10Gig.
> >> Building servers from scratch is OK but I'd also like to hear about
> >> some turnkey options. Storage requirement is currently under 10TB.
> >>
> >> Any suggestions are welcome.
> >>
> >> Thanks.
> >>
> >> JR
> >> --
> >> JR Richardson
> >> Engineering for the Masses
> >> Chasing the Azeotrope
> >
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs 0.7 no discard

[Mark Adams]

Ah right! thanks for that info, I had assumed it would use fstrim. Looks
like zpool trim is what I need to be running once upgraded.

Cheers,
Mark

On Tue, 9 Jul 2019 at 21:19, Gianni Milo  wrote:

> On that case, I would upgrade the VM to zfs 0.8, leaving the host on zfs
> 0.7 with compression enabled.
> Then I would run zfs trim command inside the VM and see if the space is
> reclaimed back on the host.
>
> Note: fstrim command only works on specific filesystems, not in zfs.
>
> Gianni
>
>
>
> On Tue, 9 Jul 2019 at 21:01, Mark Adams  wrote:
>
> > Hi GIanni,
> >
> > Thanks for your response, what I forgot to mention is I am also using zfs
> > inside of the VM, which I think is the main problem - I need it to be 0.8
> > to actually send the discard/trim command to the underlying zfs... what
> I'm
> > not clear on is if I need both to be 0.8.
> >
> > Also, I assume this can be done retrospectively ? so if I upgrade then I
> > can run fstrim and it will clear the space in the host...? Maybe that
> > question is better posed to the zfsonlinux list though...
> >
> > Cheers,
> > Mark
> >
> > On Tue, 9 Jul 2019 at 20:53, Gianni Milo 
> wrote:
> >
> > > Haven't moved to zfs 0.8 but you could have similar results by enabling
> > > compression on zfs and by periodically executing fstrim (linux) or
> > sdelete
> > > (windows) inside VMs to reclaim unused space ?
> > >
> > > On Linux VMs, adding "discard" in fstab mount options (ext4) may have
> > > similar results as when executing fstrim.
> > >
> > > On Windows 10/2012+ VMs, defrag command can be used to perform free
> space
> > > consolidation.
> > >
> > > Make sure you enable "discard" option on the VM disk options tab (which
> > you
> > > seem to have done already).
> > >
> > > Check this article on wiki as well...
> > > https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files
> > >
> > > Gianni
> > >
> > >
> > > On Tue, 9 Jul 2019 at 09:49, Mark Adams  wrote:
> > >
> > > > Hi All,
> > > >
> > > > Currently having an issue on a few servers where more space is being
> > > "used"
> > > > in the host (zfs), than is actually being used inside the VM. Discard
> > is
> > > > enabled, but zfs 0.7 does not have support for it.
> > > >
> > > > zfs 0.8 has brought in discard support, so I was wondering if anyone
> > else
> > > > has upgraded to it to resolve this kind of issue?
> > > >
> > > > I realise it's in proxmox 6, but I don't want to go there until it's
> > out
> > > of
> > > > beta.
> > > >
> > > > Any comments appreciated!
> > > >
> > > > Cheers,
> > > > Mark
> > > > _
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] zfs 0.7 no discard

[Mark Adams]

Hi GIanni,

Thanks for your response, what I forgot to mention is I am also using zfs
inside of the VM, which I think is the main problem - I need it to be 0.8
to actually send the discard/trim command to the underlying zfs... what I'm
not clear on is if I need both to be 0.8.

Also, I assume this can be done retrospectively ? so if I upgrade then I
can run fstrim and it will clear the space in the host...? Maybe that
question is better posed to the zfsonlinux list though...

Cheers,
Mark

On Tue, 9 Jul 2019 at 20:53, Gianni Milo  wrote:

> Haven't moved to zfs 0.8 but you could have similar results by enabling
> compression on zfs and by periodically executing fstrim (linux) or sdelete
> (windows) inside VMs to reclaim unused space ?
>
> On Linux VMs, adding "discard" in fstab mount options (ext4) may have
> similar results as when executing fstrim.
>
> On Windows 10/2012+ VMs, defrag command can be used to perform free space
> consolidation.
>
> Make sure you enable "discard" option on the VM disk options tab (which you
> seem to have done already).
>
> Check this article on wiki as well...
> https://pve.proxmox.com/wiki/Shrink_Qcow2_Disk_Files
>
> Gianni
>
>
> On Tue, 9 Jul 2019 at 09:49, Mark Adams  wrote:
>
> > Hi All,
> >
> > Currently having an issue on a few servers where more space is being
> "used"
> > in the host (zfs), than is actually being used inside the VM. Discard is
> > enabled, but zfs 0.7 does not have support for it.
> >
> > zfs 0.8 has brought in discard support, so I was wondering if anyone else
> > has upgraded to it to resolve this kind of issue?
> >
> > I realise it's in proxmox 6, but I don't want to go there until it's out
> of
> > beta.
> >
> > Any comments appreciated!
> >
> > Cheers,
> > Mark
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] zfs 0.7 no discard

[Mark Adams]

Hi All,

Currently having an issue on a few servers where more space is being "used"
in the host (zfs), than is actually being used inside the VM. Discard is
enabled, but zfs 0.7 does not have support for it.

zfs 0.8 has brought in discard support, so I was wondering if anyone else
has upgraded to it to resolve this kind of issue?

I realise it's in proxmox 6, but I don't want to go there until it's out of
beta.

Any comments appreciated!

Cheers,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Shutting down Windows 10, 2016 and 2019 VMs

[Mark Adams]

I sent you the information you needed an hour after you posted to this list.

If you simply followed the instructions in the link I sent to install the
agent correctly your issue would have been resolved.

On Tue, 14 May 2019, 01:24 Saint Michael,  wrote:

> Wrong: I did not get the information from this list, even a post in the
> support forum that lasted 24 hours and nobody had the answer, that includes
> Proxmox engineers. I was astounded that nobody had any idea about what I
> was going through. The list offered as a solution in the forum, is
> incomplete,
>
> https://forum.proxmox.com/threads/windows-vm-fails-to-shutdown-from-proxmox-web.54233/#post-249883
>
> By the way, somebody should fix it.
> I got the solution from Redhat.
>
>
> On Mon, May 13, 2019 at 6:21 PM Mark Adams  wrote:
>
> > So the appropriate thing for you to do now is apologise about your rude
> > list post and say thanks for the link that told you how to configure it
> > correctly!
> >
> > Regards
> >
> > On Mon, 13 May 2019, 23:12 Saint Michael,  wrote:
> >
> > > when I did this,
> > >  qm shutdown 104
> > > the command was ignored. The VM had the qemu-agent installed. It turned
> > out
> > > that is not enough, you need to install the virtio-serial driver.
> > >
> > >
> > > On Mon, May 13, 2019 at 5:41 PM LHProjects Bugs Dump <
> > > bugs+proxmoxl...@lhpmail.us>  wrote:
> > >
> > > > What do you mean by shutdown? I've had Windows Server 2019 with
> > > > guest-agent, it suppose to work? No VMs I use with 'shutdown' ever
> > > worked?
> > > >
> > > > On 5/13/19 9:10 AM, Andreas Heinlein wrote:
> > > > > Hello,
> > > > >
> > > > > I can confirm that this is working here with Windows 10 and 2016.
> > > > >
> > > > > However, the guest agent from the latest stable VirtIO ISO is *not*
> > the
> > > > > latest one. You may need the version from
> > > > >
> > > >
> > >
> >
> https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-qemu-ga/
> > > > >
> > > > > Bye,
> > > > >
> > > > > Andreas
> > > > >
> > > > > Am 13.05.19 um 14:51 schrieb Mark Adams:
> > > > >> I haven't tried 2019 as yet, but windows 10 and 2016 work fine for
> > me.
> > > > >>
> > > > >> Make sure your following this correctly!
> > > > >> https://pve.proxmox.com/wiki/Qemu-guest-agent
> > > > >>
> > > > >> Regards
> > > > >>
> > > > >> On Mon, 13 May 2019 at 12:52, Saint Michael 
> > > wrote:
> > > > >>
> > > > >>> I have not been able to shut down from Proxmox a Windows VM from
> > any
> > > > >>> version after W2012, and that is a critical issue. If you need to
> > > > reboot a
> > > > >>> server and have dozens of Windows virtual machines, it takes a
> long
> > > > time
> > > > >>> for a user to login to each VM and shut it down.
> > > > >>>
> > > > >>> Is there is a way to do this? The documentation provided in
> > > > >>>
> > > > >>>
> > > >
> > >
> >
> https://pve.proxmox.com/wiki/Windows_2012_guest_best_practices#Shutdown_Guest_From_Web_UI
> > > > >>>
> > > > >>>
> > > > >>> is useless, it refers to a version of Windows that has largely
> > > > disappeared
> > > > >>> from datacenters.
> > > > >>> ___
> > > > >>> pve-user mailing list
> > > > >>> pve-user@pve.proxmox.com
> > > > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > > >>>
> > > > >> ___
> > > > >> pve-user mailing list
> > > > >> pve-user@pve.proxmox.com
> > > > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > > > ___
> > > > > pve-user mailing list
> > > > > pve-user@pve.proxmox.com
> > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > > ___
> > > > pve-user mailing list
> > > > pve-user@pve.proxmox.com
> > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > >
> > > ___
> > > pve-user mailing list
> > > pve-user@pve.proxmox.com
> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > >
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Shutting down Windows 10, 2016 and 2019 VMs

[Mark Adams]

So the appropriate thing for you to do now is apologise about your rude
list post and say thanks for the link that told you how to configure it
correctly!

Regards

On Mon, 13 May 2019, 23:12 Saint Michael,  wrote:

> when I did this,
>  qm shutdown 104
> the command was ignored. The VM had the qemu-agent installed. It turned out
> that is not enough, you need to install the virtio-serial driver.
>
>
> On Mon, May 13, 2019 at 5:41 PM LHProjects Bugs Dump <
> bugs+proxmoxl...@lhpmail.us>  wrote:
>
> > What do you mean by shutdown? I've had Windows Server 2019 with
> > guest-agent, it suppose to work? No VMs I use with 'shutdown' ever
> worked?
> >
> > On 5/13/19 9:10 AM, Andreas Heinlein wrote:
> > > Hello,
> > >
> > > I can confirm that this is working here with Windows 10 and 2016.
> > >
> > > However, the guest agent from the latest stable VirtIO ISO is *not* the
> > > latest one. You may need the version from
> > >
> >
> https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-qemu-ga/
> > >
> > > Bye,
> > >
> > > Andreas
> > >
> > > Am 13.05.19 um 14:51 schrieb Mark Adams:
> > >> I haven't tried 2019 as yet, but windows 10 and 2016 work fine for me.
> > >>
> > >> Make sure your following this correctly!
> > >> https://pve.proxmox.com/wiki/Qemu-guest-agent
> > >>
> > >> Regards
> > >>
> > >> On Mon, 13 May 2019 at 12:52, Saint Michael 
> wrote:
> > >>
> > >>> I have not been able to shut down from Proxmox a Windows VM from any
> > >>> version after W2012, and that is a critical issue. If you need to
> > reboot a
> > >>> server and have dozens of Windows virtual machines, it takes a long
> > time
> > >>> for a user to login to each VM and shut it down.
> > >>>
> > >>> Is there is a way to do this? The documentation provided in
> > >>>
> > >>>
> >
> https://pve.proxmox.com/wiki/Windows_2012_guest_best_practices#Shutdown_Guest_From_Web_UI
> > >>>
> > >>>
> > >>> is useless, it refers to a version of Windows that has largely
> > disappeared
> > >>> from datacenters.
> > >>> ___
> > >>> pve-user mailing list
> > >>> pve-user@pve.proxmox.com
> > >>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > >>>
> > >> ___
> > >> pve-user mailing list
> > >> pve-user@pve.proxmox.com
> > >> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > ___
> > > pve-user mailing list
> > > pve-user@pve.proxmox.com
> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Shutting down Windows 10, 2016 and 2019 VMs

[Mark Adams]

I haven't tried 2019 as yet, but windows 10 and 2016 work fine for me.

Make sure your following this correctly!
https://pve.proxmox.com/wiki/Qemu-guest-agent

Regards

On Mon, 13 May 2019 at 12:52, Saint Michael  wrote:

> I have not been able to shut down from Proxmox a Windows VM from any
> version after W2012, and that is a critical issue. If you need to reboot a
> server and have dozens of Windows virtual machines, it takes a long time
> for a user to login to each VM and shut it down.
>
> Is there is a way to do this? The documentation provided in
>
> https://pve.proxmox.com/wiki/Windows_2012_guest_best_practices#Shutdown_Guest_From_Web_UI
>
>
> is useless, it refers to a version of Windows that has largely disappeared
> from datacenters.
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] ceph rebalance/ raw vs pool usage

[Mark Adams]

Doing some more research this evening, it turns out the big divergence
between the POOLS %USED and GLOBAL %RAW USED I've had is because the pool
numbers are based on the amount of space that the most full OSD has left.

So if you have 1 OSD that is disproportionately full, the %USED for POOLS
will only show you the capacity you have until that overweight OSD is full.

I've done quite a bit of reweighting and the %USED (POOLS) and %RAW USED
(GLOBAL) are now much closer together.

Cheers for your help so far Alwin - If you have any suggestions to improve
things based on my current tunables I would love to have your input.

Cheers,
Mark

On Wed, 8 May 2019 at 11:53, Mark Adams  wrote:

>
>
> On Wed, 8 May 2019 at 11:34, Alwin Antreich 
> wrote:
>
>> On Wed, May 08, 2019 at 09:34:44AM +0100, Mark Adams wrote:
>> > Thanks for getting back to me Alwin. See my response below.
>> >
>> >
>> > I have the same size and count in each node, but I have had a disk
>> failure
>> > (has been replaced) and also had issues with osds dropping when that
>> memory
>> > allocation bug was around just before last christmas (Think it was when
>> > they made some bluestore updates, then the next release they increased
>> the
>> > default memory allocation to rectify the issue) so that could have
>> messed
>> > up the balance.
>> Ok, that can impact the distribution of PGs. Could you please post the
>> crush tunables too? Maybe there could be something to tweak, besides the
>> reweight-by-utilization.
>>
>
>   "choose_local_tries": 0,
> "choose_local_fallback_tries": 0,
> "choose_total_tries": 50,
> "chooseleaf_descend_once": 1,
> "chooseleaf_vary_r": 1,
> "chooseleaf_stable": 1,
> "straw_calc_version": 1,
> "allowed_bucket_algs": 54,
> "profile": "jewel",
> "optimal_tunables": 1,
> "legacy_tunables": 0,
> "minimum_required_version": "jewel",
> "require_feature_tunables": 1,
> "require_feature_tunables2": 1,
> "has_v2_rules": 0,
> "require_feature_tunables3": 1,
> "has_v3_rules": 0,
> "has_v4_buckets": 1,
> "require_feature_tunables5": 1,
> "has_v5_rules": 0
>
>
>> >
>> > ceph osd df tree:
>> >
>> > ID CLASS WEIGHTREWEIGHT SIZEUSE AVAIL   %USE  VAR  PGS TYPE
>> > NAME
>> > -1   209.58572-  210TiB  151TiB 58.8TiB 71.92 1.00   - root
>> > default
>> > -369.86191- 69.9TiB 50.2TiB 19.6TiB 71.91 1.00   -
>>  host
>> > prod-pve1
>> >  0   ssd   6.98619  0.90002 6.99TiB 5.70TiB 1.29TiB 81.54 1.13 116
>> >  osd.0
>> >  1   ssd   6.98619  1.0 6.99TiB 5.49TiB 1.49TiB 78.65 1.09 112
>> >  osd.1
>> >  2   ssd   6.98619  1.0 6.99TiB 4.95TiB 2.03TiB 70.88 0.99 101
>> >  osd.2
>> >  4   ssd   6.98619  1.0 6.99TiB 4.90TiB 2.09TiB 70.11 0.97 100
>> >  osd.4
>> >  5   ssd   6.98619  1.0 6.99TiB 4.52TiB 2.47TiB 64.67 0.90  92
>> >  osd.5
>> >  6   ssd   6.98619  1.0 6.99TiB 5.34TiB 1.64TiB 76.50 1.06 109
>> >  osd.6
>> >  7   ssd   6.98619  1.0 6.99TiB 4.56TiB 2.42TiB 65.31 0.91  93
>> >  osd.7
>> >  8   ssd   6.98619  1.0 6.99TiB 4.91TiB 2.08TiB 70.21 0.98 100
>> >  osd.8
>> >  9   ssd   6.98619  1.0 6.99TiB 4.66TiB 2.32TiB 66.76 0.93  95
>> >  osd.9
>> > 30   ssd   6.98619  1.0 6.99TiB 5.20TiB 1.78TiB 74.49 1.04 106
>> >  osd.30
>> > -569.86191- 69.9TiB 50.3TiB 19.6TiB 71.93 1.00   -
>>  host
>> > prod-pve2
>> > 10   ssd   6.98619  1.0 6.99TiB 4.47TiB 2.52TiB 63.92 0.89  91
>> >  osd.10
>> > 11   ssd   6.98619  1.0 6.99TiB 4.86TiB 2.13TiB 69.53 0.97  99
>> >  osd.11
>> > 12   ssd   6.98619  1.0 6.99TiB 4.46TiB 2.52TiB 63.91 0.89  91
>> >  osd.12
>> > 13   ssd   6.98619  1.0 6.99TiB 4.71TiB 2.28TiB 67.43 0.94  96
>> >  osd.13
>> > 14   ssd   6.98619  1.0 6.99TiB 5.50TiB 1.49TiB 78.68 1.09 112
>> >  osd.14
>> > 15   ssd   6.98619  1.0 6.99TiB 5.20TiB 1.79TiB 74.38 1.03 106
>> >  osd.15
>> > 16   ssd   6.98619  1.0 6.99TiB 4.66TiB 2.32TiB 66.74 0.93  95
>> >  osd.16
>> > 17   ssd   6.98619  1.0 6.99TiB 5.51TiB 1.48TiB 78.84 1.10 112
>> >  osd.17
>> > 18   ssd   6.98619  1.0 6.99TiB

Re: [PVE-User] ceph rebalance/ raw vs pool usage

[Mark Adams]

On Wed, 8 May 2019 at 11:34, Alwin Antreich  wrote:

> On Wed, May 08, 2019 at 09:34:44AM +0100, Mark Adams wrote:
> > Thanks for getting back to me Alwin. See my response below.
> >
> >
> > I have the same size and count in each node, but I have had a disk
> failure
> > (has been replaced) and also had issues with osds dropping when that
> memory
> > allocation bug was around just before last christmas (Think it was when
> > they made some bluestore updates, then the next release they increased
> the
> > default memory allocation to rectify the issue) so that could have messed
> > up the balance.
> Ok, that can impact the distribution of PGs. Could you please post the
> crush tunables too? Maybe there could be something to tweak, besides the
> reweight-by-utilization.
>

  "choose_local_tries": 0,
"choose_local_fallback_tries": 0,
"choose_total_tries": 50,
"chooseleaf_descend_once": 1,
"chooseleaf_vary_r": 1,
"chooseleaf_stable": 1,
"straw_calc_version": 1,
"allowed_bucket_algs": 54,
"profile": "jewel",
"optimal_tunables": 1,
"legacy_tunables": 0,
"minimum_required_version": "jewel",
"require_feature_tunables": 1,
"require_feature_tunables2": 1,
"has_v2_rules": 0,
"require_feature_tunables3": 1,
"has_v3_rules": 0,
"has_v4_buckets": 1,
"require_feature_tunables5": 1,
"has_v5_rules": 0


> >
> > ceph osd df tree:
> >
> > ID CLASS WEIGHTREWEIGHT SIZEUSE AVAIL   %USE  VAR  PGS TYPE
> > NAME
> > -1   209.58572-  210TiB  151TiB 58.8TiB 71.92 1.00   - root
> > default
> > -369.86191- 69.9TiB 50.2TiB 19.6TiB 71.91 1.00   -
>  host
> > prod-pve1
> >  0   ssd   6.98619  0.90002 6.99TiB 5.70TiB 1.29TiB 81.54 1.13 116
> >  osd.0
> >  1   ssd   6.98619  1.0 6.99TiB 5.49TiB 1.49TiB 78.65 1.09 112
> >  osd.1
> >  2   ssd   6.98619  1.0 6.99TiB 4.95TiB 2.03TiB 70.88 0.99 101
> >  osd.2
> >  4   ssd   6.98619  1.0 6.99TiB 4.90TiB 2.09TiB 70.11 0.97 100
> >  osd.4
> >  5   ssd   6.98619  1.0 6.99TiB 4.52TiB 2.47TiB 64.67 0.90  92
> >  osd.5
> >  6   ssd   6.98619  1.0 6.99TiB 5.34TiB 1.64TiB 76.50 1.06 109
> >  osd.6
> >  7   ssd   6.98619  1.0 6.99TiB 4.56TiB 2.42TiB 65.31 0.91  93
> >  osd.7
> >  8   ssd   6.98619  1.0 6.99TiB 4.91TiB 2.08TiB 70.21 0.98 100
> >  osd.8
> >  9   ssd   6.98619  1.0 6.99TiB 4.66TiB 2.32TiB 66.76 0.93  95
> >  osd.9
> > 30   ssd   6.98619  1.0 6.99TiB 5.20TiB 1.78TiB 74.49 1.04 106
> >  osd.30
> > -569.86191- 69.9TiB 50.3TiB 19.6TiB 71.93 1.00   -
>  host
> > prod-pve2
> > 10   ssd   6.98619  1.0 6.99TiB 4.47TiB 2.52TiB 63.92 0.89  91
> >  osd.10
> > 11   ssd   6.98619  1.0 6.99TiB 4.86TiB 2.13TiB 69.53 0.97  99
> >  osd.11
> > 12   ssd   6.98619  1.0 6.99TiB 4.46TiB 2.52TiB 63.91 0.89  91
> >  osd.12
> > 13   ssd   6.98619  1.0 6.99TiB 4.71TiB 2.28TiB 67.43 0.94  96
> >  osd.13
> > 14   ssd   6.98619  1.0 6.99TiB 5.50TiB 1.49TiB 78.68 1.09 112
> >  osd.14
> > 15   ssd   6.98619  1.0 6.99TiB 5.20TiB 1.79TiB 74.38 1.03 106
> >  osd.15
> > 16   ssd   6.98619  1.0 6.99TiB 4.66TiB 2.32TiB 66.74 0.93  95
> >  osd.16
> > 17   ssd   6.98619  1.0 6.99TiB 5.51TiB 1.48TiB 78.84 1.10 112
> >  osd.17
> > 18   ssd   6.98619  1.0 6.99TiB 5.40TiB 1.59TiB 77.24 1.07 110
> >  osd.18
> > 19   ssd   6.98619  1.0 6.99TiB 5.50TiB 1.49TiB 78.66 1.09 112
> >  osd.19
> > -769.86191- 69.9TiB 50.2TiB 19.6TiB 71.93 1.00   -
>  host
> > prod-pve3
> > 20   ssd   6.98619  1.0 6.99TiB 4.22TiB 2.77TiB 60.40 0.84  86
> >  osd.20
> > 21   ssd   6.98619  1.0 6.99TiB 4.43TiB 2.56TiB 63.35 0.88  90
> >  osd.21
> > 22   ssd   6.98619  0.95001 6.99TiB 5.69TiB 1.30TiB 81.45 1.13 116
> >  osd.22
> > 23   ssd   6.98619  1.0 6.99TiB 4.67TiB 2.32TiB 66.79 0.93  95
> >  osd.23
> > 24   ssd   6.98619  0.95001 6.99TiB 5.74TiB 1.24TiB 82.20 1.14 117
> >  osd.24
> > 25   ssd   6.98619  1.0 6.99TiB 4.51TiB 2.47TiB 64.59 0.90  92
> >  osd.25
> > 26   ssd   6.98619  1.0 6.99TiB 4.90TiB 2.09TiB 70.15 0.98 100
> >  osd.26
> > 27   ssd   6.98619  1.0 6.99TiB 5.39TiB 1.59TiB 77.21 1.07 110
> >  osd.27
> > 28   ssd   6.98619  1.0 6.99TiB 5.69TiB 1.29TiB 81.47 1.13 116
> >  osd.28
&

Re: [PVE-User] ceph rebalance/ raw vs pool usage

[Mark Adams]

Thanks for getting back to me Alwin. See my response below.

On Wed, 8 May 2019 at 08:10, Alwin Antreich  wrote:

> Hello Mark,
>
> On Tue, May 07, 2019 at 11:26:17PM +0100, Mark Adams wrote:
> > Hi All,
> >
> > I would appreciate a little pointer or clarification on this.
> >
> > My "ceph" vm pool is showing 84.80% used. But the %RAW usage is only
> 71.88%
> > used. is this normal? there is nothing else on this ceph cluster apart
> from
> > this one pool.
> It is normal that the pools used-% is higher, then the raw-% usage of
> the cluster, because for one the bluestore OSDs (DB+WAL) occupy by
> default ~1.5GiB. And depending on the OSDs the pool resides on (class
> based rules), the amount may even further diverge.
>
> The general %-usage numbers of your cluster may not allow a recovery if
> a node or multiple OSDs will fail. Consider to put in more disks or
> reduce the data usage.


> >
> > Also, I notice some of my OSD's are out of balance. I've done some
> > reweighting using "ceph osd reweight-by-utilization" which has helped a
> > bit, but I think it needs to be tweaked some more (still some OSDs are
> over
> > 82% utilised, while some are around 65-70%).
> This is something that should be done with keeping an eye on the general
> cluster performance and how it fills the OSDs further. Could please post
> a 'ceph osd df tree'? It seems to me the cluster is uneven balanced in
> disk size and/or count.
>

I have the same size and count in each node, but I have had a disk failure
(has been replaced) and also had issues with osds dropping when that memory
allocation bug was around just before last christmas (Think it was when
they made some bluestore updates, then the next release they increased the
default memory allocation to rectify the issue) so that could have messed
up the balance.

ceph osd df tree:

ID CLASS WEIGHTREWEIGHT SIZEUSE AVAIL   %USE  VAR  PGS TYPE
NAME
-1   209.58572-  210TiB  151TiB 58.8TiB 71.92 1.00   - root
default
-369.86191- 69.9TiB 50.2TiB 19.6TiB 71.91 1.00   - host
prod-pve1
 0   ssd   6.98619  0.90002 6.99TiB 5.70TiB 1.29TiB 81.54 1.13 116
 osd.0
 1   ssd   6.98619  1.0 6.99TiB 5.49TiB 1.49TiB 78.65 1.09 112
 osd.1
 2   ssd   6.98619  1.0 6.99TiB 4.95TiB 2.03TiB 70.88 0.99 101
 osd.2
 4   ssd   6.98619  1.0 6.99TiB 4.90TiB 2.09TiB 70.11 0.97 100
 osd.4
 5   ssd   6.98619  1.0 6.99TiB 4.52TiB 2.47TiB 64.67 0.90  92
 osd.5
 6   ssd   6.98619  1.0 6.99TiB 5.34TiB 1.64TiB 76.50 1.06 109
 osd.6
 7   ssd   6.98619  1.0 6.99TiB 4.56TiB 2.42TiB 65.31 0.91  93
 osd.7
 8   ssd   6.98619  1.0 6.99TiB 4.91TiB 2.08TiB 70.21 0.98 100
 osd.8
 9   ssd   6.98619  1.0 6.99TiB 4.66TiB 2.32TiB 66.76 0.93  95
 osd.9
30   ssd   6.98619  1.0 6.99TiB 5.20TiB 1.78TiB 74.49 1.04 106
 osd.30
-569.86191- 69.9TiB 50.3TiB 19.6TiB 71.93 1.00   - host
prod-pve2
10   ssd   6.98619  1.0 6.99TiB 4.47TiB 2.52TiB 63.92 0.89  91
 osd.10
11   ssd   6.98619  1.0 6.99TiB 4.86TiB 2.13TiB 69.53 0.97  99
 osd.11
12   ssd   6.98619  1.0 6.99TiB 4.46TiB 2.52TiB 63.91 0.89  91
 osd.12
13   ssd   6.98619  1.0 6.99TiB 4.71TiB 2.28TiB 67.43 0.94  96
 osd.13
14   ssd   6.98619  1.0 6.99TiB 5.50TiB 1.49TiB 78.68 1.09 112
 osd.14
15   ssd   6.98619  1.0 6.99TiB 5.20TiB 1.79TiB 74.38 1.03 106
 osd.15
16   ssd   6.98619  1.0 6.99TiB 4.66TiB 2.32TiB 66.74 0.93  95
 osd.16
17   ssd   6.98619  1.0 6.99TiB 5.51TiB 1.48TiB 78.84 1.10 112
 osd.17
18   ssd   6.98619  1.0 6.99TiB 5.40TiB 1.59TiB 77.24 1.07 110
 osd.18
19   ssd   6.98619  1.0 6.99TiB 5.50TiB 1.49TiB 78.66 1.09 112
 osd.19
-769.86191- 69.9TiB 50.2TiB 19.6TiB 71.93 1.00   - host
prod-pve3
20   ssd   6.98619  1.0 6.99TiB 4.22TiB 2.77TiB 60.40 0.84  86
 osd.20
21   ssd   6.98619  1.0 6.99TiB 4.43TiB 2.56TiB 63.35 0.88  90
 osd.21
22   ssd   6.98619  0.95001 6.99TiB 5.69TiB 1.30TiB 81.45 1.13 116
 osd.22
23   ssd   6.98619  1.0 6.99TiB 4.67TiB 2.32TiB 66.79 0.93  95
 osd.23
24   ssd   6.98619  0.95001 6.99TiB 5.74TiB 1.24TiB 82.20 1.14 117
 osd.24
25   ssd   6.98619  1.0 6.99TiB 4.51TiB 2.47TiB 64.59 0.90  92
 osd.25
26   ssd   6.98619  1.0 6.99TiB 4.90TiB 2.09TiB 70.15 0.98 100
 osd.26
27   ssd   6.98619  1.0 6.99TiB 5.39TiB 1.59TiB 77.21 1.07 110
 osd.27
28   ssd   6.98619  1.0 6.99TiB 5.69TiB 1.29TiB 81.47 1.13 116
 osd.28
29   ssd   6.98619  1.0 6.99TiB 5.00TiB 1.98TiB 71.63 1.00 102
 osd.29
  TOTAL  210TiB  151TiB 58.8TiB 71.92

MIN/MAX VAR: 0.84/1.14  STDDEV: 6.44



>
> >
> > Is it safe enough to keep tweaking this? (I believe I should run ceph osd
> > reweight-by-utilization 101 0.05 15) Is there any gotchas I need to be
> > aware of when doing this apart from the obvious load of r

Re: [PVE-User] Windows Server 2003

[Mark Adams]

AFAIK you need the serial driver for the guest agent to work correctly, so
make sure that device is up to date. If its not showing as with a missing
driver in device manager it must already be installed and might just need
updating?

https://pve.proxmox.com/wiki/Qemu-guest-agent#Windows


On Thu, 25 Apr 2019 at 11:41, David Lawley  wrote:

> something up with the guest agent, have to look further.
>
> These things have been pounded on for the past few years, something is
> lingering around.  Agent its running but no communication between guest
> and PVE
>
> On 4/25/2019 2:01 AM, Thomas Lamprecht wrote:
> > Am 4/24/19 um 8:13 PM schrieb David Lawley:
> >> I did that as part of the migration
> >>
> > and the guest agent works? i.e., things like
> > # qm guest cmd VMID get-osinfo
> >
> > also the guest config could be interesting:
> > # qm config VMID
> >
> >
> >> Serial driver?  Don't have have any odd devices showing up in the
> device list
> >>
> >>
> >>
> >> On 4/24/2019 2:02 PM, Mark Adams wrote:
> >>> Haven't tried this myself, but have you updated the qemu-agent and
> serial
> >>> driver to check it's not that?
> >>>
> >>> On Wed, 24 Apr 2019, 18:59 David Lawley,  wrote:
> >>>
> >>>> I know, its an oldie, but.. Windows Server 2003
> >>>>
> >>>> But since moving it to PVE 5.4 (from3.4) its does not reboot/restart
> on
> >>>> its own.  You can select a restart, but it does not. You have to start
> >>>> it manually.
> >>>>
> >>>> Migration from 3.4 was done via backup then restore into 5.4
> >>>>
> >>>> Poking around, but if someone has crossed this path before me I would
> >>>> appreciate a nudge in the right direction
> >>>>
> >>>> Thanks a bunch
> >
> >
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-use
> <https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user>
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Windows Server 2003

[Mark Adams]

Haven't tried this myself, but have you updated the qemu-agent and serial
driver to check it's not that?

On Wed, 24 Apr 2019, 18:59 David Lawley,  wrote:

> I know, its an oldie, but.. Windows Server 2003
>
> But since moving it to PVE 5.4 (from3.4) its does not reboot/restart on
> its own.  You can select a restart, but it does not. You have to start
> it manually.
>
> Migration from 3.4 was done via backup then restore into 5.4
>
> Poking around, but if someone has crossed this path before me I would
> appreciate a nudge in the right direction
>
> Thanks a bunch
>
>
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] ZFS Replication on different storage

[Mark Adams]

Why don't you just rename the zpool so they match?

On Fri, 15 Mar 2019, 09:10 Fabrizio Cuseo,  wrote:

> Hello Gianni.
> I wrote in my email that pve-zsync is not suitable for my need (redundancy
> with VM migration from one host to another).
>
> Fabrizio
>
> - Il 15-mar-19, alle 10:08, Gianni M.  ha
> scritto:
>
> > I think it's hardcoded to use rpool as the default target pool, but I
> might be
> > wrong.
>
> > You might want to have a look at pve-zsync instead ?
>
> > https://pve.proxmox.com/wiki/PVE-zsync
>
> > NiCK
>
> > From: pve-user  on behalf of Fabrizio
> Cuseo
> > 
> > Sent: Friday, March 15, 2019 8:48:58 AM
> > To: pve-user
> > Subject: Re: [PVE-User] ZFS Replication on different storage
> > Thank you, I have already seen this page; i know and use zfs with
> freenas and
> > other (tipically i use ceph, but for this cluster I only need the
> replication
> > feature), but pve-zsync is used more for offsite-backup than redundancy,
> and
> > the VM can't be migrated from one host to another.
> > So, is not usable :(
>
> > Thanks, Fabrizio
>
> > - Il 15-mar-19, alle 9:36, b...@todoo.biz ha scritto:
>
> > > Please check this page :
>
> >> [ https://pve.proxmox.com/wiki/PVE-zsync |
> > > https://pve.proxmox.com/wiki/PVE-zsync ]
>
>
> > > pve-zsync is a very nice tool. You need to know and understand what
> you are
> > > doing (generally speaking, it is a good advise).
>
> > > ZFS is a complex file system with major features, but It has a certain
> learning
> > > curve.
> > > If you have no notion of ZFS, use another backup strategy or learn
> some basics
> > > about ZFS.
>
> > > you should first install pve-zsync and issue a command looking similar
> to this
> > > one :
>
> > > pve-zsync create -dest 192.168.210.28:tank/proxmox1 -limit 12600
> -maxsnap 7
> > > -name kvm1.srv -source 133 -verbose
>
>
> > > Where 192.168.210.28 is the IP of your second host / backup host… and
> > > tank/proxmox1 is the dataset where you'll backup.
>
> > > First run will actually create the backup and sync It.
> > > It will also create a cron job available in /etc/cron.d/pve-zsync
>
> > > You can edit this file in order to tune the various parameters (most
> probably
> > > the frequency).
>
>
> > > Do read the doc.
>
> > >> Le 15 mars 2019 à 09:18, Fabrizio Cuseo  a
> écrit :
>
> > >> Hello Yannis.
> > >> I can't see an option to specify remote storage (or pool) name.
> > >> If you read my email, i need to replicate:
>
> > >> HostA/poolZ > HostB/poolY
>
> > >> But without specifying the remote pool name (there is no pool field
> in the gui),
> > >> it replicates from HostA/PoolZ ---> HostB/PoolZ (where i have no
> enough space)
> > >> Regards, Fabrizio
>
> > >> - Il 14-mar-19, alle 19:48, Yannis Milios <
> yannis.mil...@gmail.com> ha
> > >> scritto:
>
> > >>> Yes, it is possible...
>
>  [ [ https://pve.proxmox.com/pve-docs/chapter-pvesr.html |
> > >>> https://pve.proxmox.com/pve-docs/chapter-pvesr.html ] |
>  [ https://pve.proxmox.com/pve-docs/chapter-pvesr.html |
> > >>> https://pve.proxmox.com/pve-docs/chapter-pvesr.html ] ]
>
>  On Thu, 14 Mar 2019 at 11:19, Fabrizio Cuseo < [ [ mailto:
> f.cu...@panservice.it
> > >>> |
> > mailto:f.cu...@panservice.it ] |
> > >>> f.cu...@panservice.it ] > wrote:
>
> >  Hello.
> >  I have a customer with a small cluster, 2 servers (different
> models).
>
> >  I would like to replicate VMs from host A to host B, but from
> local-zfs (host A)
> >  to "zfs-data-2" (host B).
>
> >  On the GUI this is not possibile, what about some workaround ?
>
> >  Regards, Fabrizio
>
> >  --
> >  ---
> > Fabrizio Cuseo - mailto: [ [ mailto:f.cu...@panservice.it |
> >  mailto:f.cu...@panservice.it ] | f.cu...@panservice.it
> >  ]
> >  Direzione Generale - Panservice InterNetWorking
> >  Servizi Professionali per Internet ed il Networking
> >  Panservice e' associata AIIP - RIPE Local Registry
> >  Phone: +39 0773 410020 - Fax: +39 0773 470219
> > [ [ http://www.panservice.it/ | http://www.panservice.it/ ] | [
> >  http://www.panservice.it/ |
> > http://www.panservice.it ] ] mailto: [
> >  [ mailto:i...@panservice.it | mailto:i...@panservice.it ] |
> i...@panservice.it ]
> >  Numero verde nazionale: 800 901492
> >  ___
> >  pve-user mailing list
> > [ [ mailto:pve-user@pve.proxmox.com | mailto:
> pve-user@pve.proxmox.com ] |
> >  pve-user@pve.proxmox.com ]
> > [ [ https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user |
> >  https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user ] |
> > [ https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user |
> >  https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user ] ]
>
> > >>> --
> > >>> Sent from Gmail Mobile
>
> > >> --
> > >> ---
> > >> Fabrizio Cuseo - [ mailto:f.cu...@panservice.it | mailto:
> f.cu...@panservice.it ]
> > >> Direzione Generale - Panservice 

Re: [PVE-User] MxGPU with AMD S7150

[Mark Adams]

For posterity, I sorted this by speaking with ASRock Rack and getting an
as-yet unreleased bios that has the ARI forwarding option. Enabled this and
all working now.


On Fri, 1 Mar 2019 at 13:20, Dominik Csapak  wrote:

> On 01.03.19 14:13, Mark Adams wrote:
> > On Fri, 1 Mar 2019 at 12:52, Dominik Csapak 
> wrote:
> >
> >> On 01.03.19 13:37, Mark Adams wrote:
> >>> Hi All,
> >>>
> >>> I'm trying this out, based on the wiki post and the forum posts:
> >>>
> >>>
> >>
> https://forum.proxmox.com/threads/amd-s7150-mxgpu-with-proxmox-ve-5-x.50464/
> >>>
> >>> https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x
> >>>
> >>> However I'm having issues getting the gim driver working. Was just
> >>> wondering if the Proxmox staff member that tested this out came across
> >> this
> >>> particular issue, or if anyone else had any insights.
> >>
> >> Hi, i am the one that tested this.
> >>
> >
> > Hi Dominik, Thanks for getting back to me so quickly.
> >
>
> Hi, no problem
>
> >
> >>
> >>>
> >>> My hardware is an ASRock EPYCD8-2T motherboard (SR-IOV enabled in bios)
> >> and
> >>> an AMD S7150. Proxmox is 5.3-11.
> >>>
> >>> When running the modprobe of gim, it crashes out with the following:
> >>>
> >>> [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:299) Enable SRIOV
> >>> [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:300) Enable SRIOV vfs
> >>> count = 16
> >>> [Fri Mar  1 12:31:49 2019] pci :61:02.0: [1002:692f] type 7f class
> >>> 0xff
> >>> [Fri Mar  1 12:31:49 2019] pci :61:02.0: unknown header type 7f,
> >>> ignoring device
> >>> [Fri Mar  1 12:31:50 2019] gim error:(enable_sriov:311) Fail to enable
> >>> sriov, status = fffb
> >>> [Fri Mar  1 12:31:50 2019] gim error:(set_new_adapter:668) Failed to
> >>> properly enable SRIOV
> >>> [Fri Mar  1 12:31:50 2019] gim info:(gim_probe:91) AMD GIM probe:
> >> pf_count
> >>> = 1
> >>>
> >>
> >> mhmm i cannot really remember if that exact error message occured, but
> >> you have to enable several things in the bios
> >>
> >> AMD-Vi/VT-d
> >> SR-IOV
> >> ARI
> >> and possibly above-4g-decoding
> >>
> >> also make sure you enable the 'legacy' or non uefi oprom for
> >> that card
> >>
> >> on our supermicro board we could select the oprom for each pcie port
> >> separately
> >>
> >
> > It's the same with this ASRock Rack board. I've set the oprom to legacy
> for
> > PCIE slot 1, but it doesn't seem to make any difference. I've also tried
> > other slots but that doesn't make a difference either.
> >
> > The only thing I can't find, is any option relating to ARI. Do you recall
> > at all what the option was called? I think the supermicro and asrock
> boards
> > are pretty similar when it comes to options, but maybe this board is
> > missing ARI.
>
> With ARI i mean Alternative Routing-ID Interpretation, an PCI Extension[1]
>
> >
> > Also I have ACS enabled but that doesn't help either.
> >
> >
> >> names may be different in your bios,
> >> or some options may not exists at all
> >>
> >> at last, a different pcie port may be necessary, depending on how
> >> the mainboard is wired (with epyc all pcie ports should go to the
> >> cpu, but i do not know about your specific board)
> >>
> >> if all else fails, i would open an issue on github for the gim project
> >> and ask there if anything is known
> >>
> >
> > Thanks I will do that.
> >
>
> ok, if i remember anything else, i will answer here on the list
>
> 1:
>
> https://pcisig.com/sites/default/files/specification_documents/ECN-alt-rid-interpretation-070604.pdf
>
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] MxGPU with AMD S7150

[Mark Adams]

On Fri, 1 Mar 2019 at 12:52, Dominik Csapak  wrote:

> On 01.03.19 13:37, Mark Adams wrote:
> > Hi All,
> >
> > I'm trying this out, based on the wiki post and the forum posts:
> >
> >
> https://forum.proxmox.com/threads/amd-s7150-mxgpu-with-proxmox-ve-5-x.50464/
> >
> > https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x
> >
> > However I'm having issues getting the gim driver working. Was just
> > wondering if the Proxmox staff member that tested this out came across
> this
> > particular issue, or if anyone else had any insights.
>
> Hi, i am the one that tested this.
>

Hi Dominik, Thanks for getting back to me so quickly.


>
> >
> > My hardware is an ASRock EPYCD8-2T motherboard (SR-IOV enabled in bios)
> and
> > an AMD S7150. Proxmox is 5.3-11.
> >
> > When running the modprobe of gim, it crashes out with the following:
> >
> > [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:299) Enable SRIOV
> > [Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:300) Enable SRIOV vfs
> > count = 16
> > [Fri Mar  1 12:31:49 2019] pci :61:02.0: [1002:692f] type 7f class
> > 0xff
> > [Fri Mar  1 12:31:49 2019] pci :61:02.0: unknown header type 7f,
> > ignoring device
> > [Fri Mar  1 12:31:50 2019] gim error:(enable_sriov:311) Fail to enable
> > sriov, status = fffb
> > [Fri Mar  1 12:31:50 2019] gim error:(set_new_adapter:668) Failed to
> > properly enable SRIOV
> > [Fri Mar  1 12:31:50 2019] gim info:(gim_probe:91) AMD GIM probe:
> pf_count
> > = 1
> >
>
> mhmm i cannot really remember if that exact error message occured, but
> you have to enable several things in the bios
>
> AMD-Vi/VT-d
> SR-IOV
> ARI
> and possibly above-4g-decoding
>
> also make sure you enable the 'legacy' or non uefi oprom for
> that card
>
> on our supermicro board we could select the oprom for each pcie port
> separately
>

It's the same with this ASRock Rack board. I've set the oprom to legacy for
PCIE slot 1, but it doesn't seem to make any difference. I've also tried
other slots but that doesn't make a difference either.

The only thing I can't find, is any option relating to ARI. Do you recall
at all what the option was called? I think the supermicro and asrock boards
are pretty similar when it comes to options, but maybe this board is
missing ARI.

Also I have ACS enabled but that doesn't help either.


> names may be different in your bios,
> or some options may not exists at all
>
> at last, a different pcie port may be necessary, depending on how
> the mainboard is wired (with epyc all pcie ports should go to the
> cpu, but i do not know about your specific board)
>
> if all else fails, i would open an issue on github for the gim project
> and ask there if anything is known
>

Thanks I will do that.

Regards,
Mark


>
> hope this helps
>
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] MxGPU with AMD S7150

[Mark Adams]

Hi All,

I'm trying this out, based on the wiki post and the forum posts:

https://forum.proxmox.com/threads/amd-s7150-mxgpu-with-proxmox-ve-5-x.50464/

https://pve.proxmox.com/wiki/MxGPU_with_AMD_S7150_under_Proxmox_VE_5.x

However I'm having issues getting the gim driver working. Was just
wondering if the Proxmox staff member that tested this out came across this
particular issue, or if anyone else had any insights.

My hardware is an ASRock EPYCD8-2T motherboard (SR-IOV enabled in bios) and
an AMD S7150. Proxmox is 5.3-11.

When running the modprobe of gim, it crashes out with the following:

[Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:299) Enable SRIOV
[Fri Mar  1 12:31:49 2019] gim info:(enable_sriov:300) Enable SRIOV vfs
count = 16
[Fri Mar  1 12:31:49 2019] pci :61:02.0: [1002:692f] type 7f class
0xff
[Fri Mar  1 12:31:49 2019] pci :61:02.0: unknown header type 7f,
ignoring device
[Fri Mar  1 12:31:50 2019] gim error:(enable_sriov:311) Fail to enable
sriov, status = fffb
[Fri Mar  1 12:31:50 2019] gim error:(set_new_adapter:668) Failed to
properly enable SRIOV
[Fri Mar  1 12:31:50 2019] gim info:(gim_probe:91) AMD GIM probe: pf_count
= 1

Thanks for any advice!

Regards,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] PVE Cluster and iSCSI

[Mark Adams]

Are you are using some iscsi setup that manages the luns independently for
each VM? then take a look at this link.
https://pve.proxmox.com/wiki/Storage:_User_Mode_iSCSI

There has to be some method for creating the VM partitions - this is why
LVM is preferred as the option if you give it an entire iscsi target.

On Thu, 29 Nov 2018 at 01:06, Gilberto Nunes 
wrote:

> yes... but it's work only with LVEM Over iSCSI or can I access direct from
> all nodes?
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em qua, 28 de nov de 2018 às 22:24, Mark Adams 
> escreveu:
>
> > As long as you have access to the iSCSI storage from all nodes in the
> > cluster then why not?
> >
> > On Wed, 28 Nov 2018 at 19:20, Gilberto Nunes  >
> > wrote:
> >
> > > Hi there
> > >
> > > Is there any problem to use PVE cluster with iSCSI Direct or not ( I
> mean
> > > shared)?
> > >
> > > Thanks
> > > ---
> > > Gilberto Nunes Ferreira
> > >
> > > (47) 3025-5907
> > > (47) 99676-7530 - Whatsapp / Telegram
> > >
> > > Skype: gilberto.nunes36
> > > ___
> > > pve-user mailing list
> > > pve-user@pve.proxmox.com
> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] PVE Cluster and iSCSI

[Mark Adams]

As long as you have access to the iSCSI storage from all nodes in the
cluster then why not?

On Wed, 28 Nov 2018 at 19:20, Gilberto Nunes 
wrote:

> Hi there
>
> Is there any problem to use PVE cluster with iSCSI Direct or not ( I mean
> shared)?
>
> Thanks
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] NVMe

[Mark Adams]

Did you reinstall from the proxmox ISO after changing boot mode to legacy?

Regards.
Mark

On Tue, 30 Oct 2018 at 14:08, lord_Niedzwiedz  wrote:

>  I set legacy boot in bios.
> Use only one disk with lvm.
> And system not start with this.
>
> Any sugestion ?
> >> I have a problem.
> >> Im trying to install Proxmox on 4 NVMe drives.
> >> One on the motherboard, two on the PCIe.
> >>
> >> Proxmox see everything at the installation.
> >> I give the option zfs (RAIDZ-1).
> >>
> >> And I get a mistake error at the end.
> >> "unable to create zfs root pool"
> > GRUB is not yet working with ZFS on EFI. Try to switch to legacy boot in
> > BIOS if possible or use LVM for the installation.
> >
> >> Attached pictures (1-5) .jpg.
> >> https://help.komandor.pl/Wymiana/1.jpg
> >> https://help.komandor.pl/Wymiana/2.jpg
> >> https://help.komandor.pl/Wymiana/3.jpg
> >> https://help.komandor.pl/Wymiana/4.jpg
> >> https://help.komandor.pl/Wymiana/5.jpg
> >>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HA Failover if shared storage fails on one Node

[Mark Adams]

What interface is your cluster communication (corosync) running over? As
this is the link that needs to be unavailable to initiate a VM start on
another node AFAIK.

Basically, the other nodes in the cluster need to be seeing a problem with
the node. If its still communicating over the whichever interface you have
the cluster communication on then as far as it is concerned the node is
still up. If you just lose access to your storage, then your VM will still
be running in memory.

I don't believe there is any separate storage specific monitoring in
proxmox that could trigger a move to another node. If there is I'm sure
someone else on the list will advise.

Regards,
Mark

On Wed, 17 Oct 2018 at 12:19, Martin Holub  wrote:

> On 10/17/18 1:11 PM, Gilberto Nunes wrote:
> > Hi
> >
> > How about Node priority?
> > Look section 14.5.2 in this doc
> >
> > https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_configuration_10
> > ---
> > Gilberto Nunes Ferreira
> >
> > (47) 3025-5907
> > (47) 99676-7530 - Whatsapp / Telegram
> >
> > Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> > Em qua, 17 de out de 2018 às 08:05, Martin Holub 
> > escreveu:
> >
> >> Hi,
> >>
> >> I am currently testing the HA features on a 6 Node Cluster and a NetAPP
> >> Storage with iSCSI and multipath configured on all Nodes. I now tried
> >> what happens if, for any reason, booth Links fail (by shutting down the
> >> Interfaces on one Blade). Unfortunately, altough i had configured HA for
> >> my Test VM, Proxmox seems to not recognize the Storage outtage and
> >> therefore did not migrate the VM to a different blade or removed that
> >> Node from the Cluster (either by resetting it or fencing it somehow
> >> else). Any hints on how to get that solved?
> >>
> >> Thanks,
> >> Martin
> >>
>
> Not shure if i understood what you mean with that reference, but since
> Proxmox does not detect that the Storage is unreachable on that specific
> Cluster Node, how are HA Groups supposed to work around this?
>
> Best,
> Martin
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Proxmox per VM memory limit

[Mark Adams]

assuming the OS in the VM supports it, as much as the host hardware can
support (no limit).

On Tue, 2 Oct 2018 at 19:35, Gilberto Nunes 
wrote:

> Hi there!
>
> How many memory per VM I get in PVE?
> Is there some limit? 1 TB? 2 TB?
> Just curious
>
> Thanks
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Ceph Cluster with proxmox failure

[Mark Adams]

also, 3 out of 6 servers is not quorum. you need a majority of the total.

On Fri, 28 Sep 2018, 21:22 Mark Adams,  wrote:

> the exact same 3 servers have been down and everything has worked? do you
> run ceph mons on every server?
>
> On Fri, 28 Sep 2018, 21:19 Gilberto Nunes, 
> wrote:
>
>> Everything is working couple of hours ago!
>> Due a power failure, 3 hosts are down, but I suppose with 3 host the
>> cluster garantee the quorum and allow ceph continues to work.
>> This already happen before and ceph works perfectly with 3 servers
>>
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em sex, 28 de set de 2018 às 17:13, Woods, Ken A (DNR) <
>> ken.wo...@alaska.gov>
>> escreveu:
>>
>> > So the times are massively different.  Fix that.
>> >
>> > And corosync needs multicast , and that’s tested using omping, not ping.
>> >
>> > Go back through the initial set up documentation and read what is
>> required
>> > for the basic network configuration.
>> >
>> > If corosync and ceph are both not working, start there.
>> >
>> >
>> > > On Sep 28, 2018, at 12:08, Gilberto Nunes > >
>> > wrote:
>> > >
>> > > pve-ceph01:~# ssh pve-ceph01 date
>> > > Fri Sep 28 17:06:34 -03 2018
>> > > pve-ceph01:~# ssh pve-ceph02 date
>> > > Fri Sep 28 17:06:37 -03 2018
>> > > pve-ceph01:~# ssh pve-ceph05 date
>> > > Fri Sep 28 17:06:39 -03 2018
>> > >
>> > > pve-ceph01:~# ping -c 1 pve-ceph01
>> > > PING pve-ceph01.cepam.com.br (10.10.10.100) 56(84) bytes of data.
>> > > 64 bytes from pve-ceph01.cepam.com.br (10.10.10.100): icmp_seq=1
>> ttl=64
>> > > time=0.020 ms
>> > >
>> > > --- pve-ceph01.cepam.com.br ping statistics ---
>> > > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>> > > rtt min/avg/max/mdev = 0.020/0.020/0.020/0.000 ms
>> > > pve-ceph01:~# ping -c 1 pve-ceph02
>> > > once.
>> > > PING pve-ceph02.cepam.com.br (10.10.10.110) 56(84) bytes of data.
>> > > 64 bytes from pve-ceph02.cepam.com.br (10.10.10.110): icmp_seq=1
>> ttl=64
>> > > time=0.120 ms
>> > >
>> > > --- pve-ceph02.cepam.com.br ping statistics ---
>> > > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>> > > rtt min/avg/max/mdev = 0.120/0.120/0.120/0.000 ms
>> > > pve-ceph01:~# ping -c 1 pve-ceph05
>> > > PING pve-ceph05.cepam.com.br (10.10.10.140) 56(84) bytes of data.
>> > > 64 bytes from pve-ceph05.cepam.com.br (10.10.10.140): icmp_seq=1
>> ttl=64
>> > > time=0.078 ms
>> > >
>> > > --- pve-ceph05.cepam.com.br ping statistics ---
>> > > 1 packets transmitted, 1 received, 0% packet loss, time 0ms
>> > > rtt min/avg/max/mdev = 0.078/0.078/0.078/0.000 ms
>> > >
>> > > I can communicate with other...
>> > > ceph command stuck
>> > > all ceph service it's appears to be running...
>> > >
>> > >
>> > > ---
>> > > Gilberto Nunes Ferreira
>> > >
>> > > (47) 3025-5907
>> > > (47) 99676-7530 - Whatsapp / Telegram
>> > >
>> > > Skype: gilberto.nunes36
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Em sex, 28 de set de 2018 às 17:02, Woods, Ken A (DNR) <
>> > ken.wo...@alaska.gov>
>> > > escreveu:
>> > >
>> > >> Network issues?
>> > >> Time issues?
>> > >>
>> > >>
>> > >>> On Sep 28, 2018, at 11:50, Gilberto Nunes <
>> gilberto.nune...@gmail.com>
>> > >> wrote:
>> > >>>
>> > >>> Hi there
>> > >>> I have a 6 server Ceph Cluster maded with proxmox 5.2
>> > >>> Suddenly, after power failure, I have only 3 servers UP, but even
>> with
>> > 3
>> > >>> server, Ceph Cluster doesn't work.
>> > >>> pveceph status give me a timeout
>> > >>> pveceph status got timeout
>> > >>>
>> > >>> Any advice?
>> > >>>
>> > >>>
>> > >>> ---
>> > >>&g

Re: [PVE-User] dual host HA solution in 5.2

[Mark Adams]

If you have to stick with 2 servers, personally I would go for zfs as your
storage. Storage replication using zfs in proxmox has been made super
simple.

This is asynchronous though, unlike DRBD. You would have to manually start
your VM's should the "live" node go down and the data will be out of date
depending on how frequently you've told it to sync. IMO, this is a decent
setup if you are limited to 2 servers and is very simple.

Then you also get the great features such as high performance snapshots
(LVM sucks at this..), clones and even really simple replication to another
server (IE a disaster recovery location) with pve-zsync. Not to mention all
the other features of zfs - compression, checksumming etc (google it if you
don't know).

Regards,
Mark




On Fri, 28 Sep 2018 at 16:51, Woods, Ken A (DNR) 
wrote:

>
> > On Sep 28, 2018, at 07:12, Adam Weremczuk 
> wrote:
> > Please advise if you have better ideas
>
> Buy another server.
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Spice and Windows as VDI solution

[Mark Adams]

Sounds like you need to speak to your local Microsoft license re seller.

On Fri, 28 Sep 2018 at 13:33, Gilberto Nunes 
wrote:

> Hi there
>
> When using spice I will need the infamous Windows User Access Cal, to every
> simple user if will connect to Spice session??
> I need install 10 VM with Windows 10 Enterprise and need to know if I need
> Windows license to User Cal's or just normal Windows License for that!
>
> Thanks a lot
>
> Cheers
>
>
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Fedora 28

[Mark Adams]

If you don't have a license, you need to change the repository.

https://pve.proxmox.com/wiki/Package_Repositories#_proxmox_ve_no_subscription_repository


On 12 September 2018 at 10:01, lord_Niedzwiedz  wrote:

> How i make this upgrading  ??!!
>
> root@hayne1:~# apt upgrade
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> Calculating upgrade... Done
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> root@hayne1:~# apt update
> Ign:1 http://ftp.pl.debian.org/debian stretch InRelease
> Hit:2 http://ftp.pl.debian.org/debian stretch-updates InRelease
> Hit:3 http://ftp.pl.debian.org/debian stretch Release
> Hit:4 http://security.debian.org stretch/updates InRelease
> Ign:6 https://enterprise.proxmox.com/debian/pve stretch InRelease
> Err:7 https://enterprise.proxmox.com/debian/pve stretch Release
>   401  Unauthorized
> Reading package lists... Done
> E: The repository 'https://enterprise.proxmox.com/debian/pve stretch
> Release' does not have a Release file.
> N: Updating from such a repository can't be done securely, and is
> therefore disabled by default.
> N: See apt-secure(8) manpage for repository creation and user
> configuration details.
>
>
> W dniu 12.09.2018 o 09:27, Stoiko Ivanov pisze:
>
> Hi,
>>
>> The problem with Fedora containers was fixed with pve-container 2.0-25.
>> Could you try again after upgrading?
>>
>> Cheers,
>> stoiko
>>
>> On Wed, Sep 12, 2018 at 08:31:11AM +0200, lord_Niedzwiedz wrote:
>>
>>> root@hayne:~# pveversion -v
>>> proxmox-ve: 5.2-2 (running kernel: 4.15.17-1-pve)
>>> pve-manager: 5.2-1 (running version: 5.2-1/0fcd7879)
>>> pve-kernel-4.15: 5.2-1
>>> pve-kernel-4.15.17-1-pve: 4.15.17-9
>>> corosync: 2.4.2-pve5
>>> criu: 2.11.1-1~bpo90
>>> glusterfs-client: 3.8.8-1
>>> ksm-control-daemon: 1.2-2
>>> libjs-extjs: 6.0.1-2
>>> libpve-access-control: 5.0-8
>>> libpve-apiclient-perl: 2.0-4
>>> libpve-common-perl: 5.0-31
>>> libpve-guest-common-perl: 2.0-16
>>> libpve-http-server-perl: 2.0-8
>>> libpve-storage-perl: 5.0-23
>>> libqb0: 1.0.1-1
>>> lvm2: 2.02.168-pve6
>>> lxc-pve: 3.0.0-3
>>> lxcfs: 3.0.0-1
>>> novnc-pve: 0.6-4
>>> proxmox-widget-toolkit: 1.0-18
>>> pve-cluster: 5.0-27
>>> pve-container: 2.0-23
>>> pve-docs: 5.2-3
>>> pve-firewall: 3.0-8
>>> pve-firmware: 2.0-4
>>> pve-ha-manager: 2.0-5
>>> pve-i18n: 1.0-5
>>> pve-libspice-server1: 0.12.8-3
>>> pve-qemu-kvm: 2.11.1-5
>>> pve-xtermjs: 1.0-5
>>> qemu-server: 5.0-26
>>> smartmontools: 6.5+svn4324-1
>>> spiceterm: 3.0-5
>>> vncterm: 1.5-3
>>> zfsutils-linux: 0.7.8-pve1~bpo9
>>>
>>> W dniu 11.09.2018 o 16:31, Stoiko Ivanov pisze:
>>>
 Hi,

 cannot reproduce the problem with a similar config (mac and ip addresses
 changed, but else the same).

 which versions of our stack do you run? (please post the output of
 pveversion -v).

 Thanks!


 On Tue, Sep 11, 2018 at 03:08:35PM +0200, lord_Niedzwiedz wrote:

> Hi,
>
> I did not change anything except the two things below.
> And fedora works with network after restart.
>
> root@hayne1:/rpool# pct config 102
> arch: amd64
> cores: 2
> hostname: wanwww14
> memory: 8192
> net0: name=eth0,bridge=vmbr0,firewall=1,gw=8.8.152.1,hwaddr=F5:E4:
> 9B:64:22:84,ip=8.8.152.104/24,type=veth
> ostype: fedora
> rootfs: local-zfs:subvol-102-disk-1,size=28G
> swap: 1024
> unprivileged: 1
>
> I don't change antyhing.
> This is config from Proxmox GUI inside Fedora.
> [root@wanwww8 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
> DEVICE=eth0
> ONBOOT=yes
> BOOTPROTO=none
> IPADDR=8.8.152.104
> NETMASK=255.255.255.0
> GATEWAY=8.8.152.1
>
> Only these two changes below are required for the Fedora to start up
> with a
> working network.
> Of course, the native server should also have properly configured dns,
> etc.
>
> Gregory Bear
>
> Hi,
>>
>> could you please send the container-config (`pct config $vmid`) from
>> the
>> node and the contents of all files (redact if needed) from
>> /etc/systemd/network/* ?
>>
>> Thanks!
>>
>> On Tue, Sep 11, 2018 at 02:38:59PM +0200, lord_Niedzwiedz wrote:
>>
>>> Hi,
> I get yours offical Fedora 27.
>
 you should now be able to get the Fedora 28 template directly from
 us.
 # pveam update

 should pull the newest appliance index (gets normally done
 automatically,
 once a day) then either download it through the WebUI or with CLI:

 # pveam download STORAGE fedora-28-default_20180907_amd64.tar.xz

 cheers,
 Thomas

>>> Problem is in configuration fedora system inside (fedora-28 too).
>>>
>>> I must add this two things:
>>>
>>> chkconfig --levels 2345 network on
>>>
>>> vi /etc/sysconfig/network

Re: [PVE-User] Confusing about Bond 802.3ad

[Mark Adams]

That is it, as I understand it Josh. you basically need to turn your switch
in to X seperate switches so each nodes nic, is running on a "seperate"
network.

if you were to do the same thing physically without any config, with 3
nodes, you would need to have as many seperate switches as you wanted nics
in the balance-rr.

I understand mikrotik support balance-rr, but tbh I don't even count them
as a normal switch manufacturer. Their game is routers I don't know any
other switches which have support for balance-rr?

as for the 3Gbps limit I mentioned earlier with balance-rr (no matter how
many nics you have)... I don't know if that was just an issue of the day as
cheap 10Gbps came along and the need evaporated for me. I would love to
know if anyone has a test setup to try it though.

Cheers

On Sat, 25 Aug 2018, 00:15 Josh Knight,  wrote:

> Just guessing here, if the switch doesn't support rr on its port channels,
> then using separate VLANs instead of bundles on the switch is essentially
> wiring nodeA to nodeB. That way you don't hit the port channel hashing on
> the switch and you keep the rr as-is from A to B.
>
> I would also try using UDP mode on iperf to see if it's TCP retransmission
> that's preventing you from getting closer to 4Gbps. Another useful tool is
> maisezahn for traffic generation, though it is more complex to run.
>
> On Fri, Aug 24, 2018, 4:59 PM Gilberto Nunes 
> wrote:
>
> > I can get 3 gbps. At least 1.3 gbps.
> > Don't know why!
> > Em 24/08/2018 17:36, "mj"  escreveu:
> >
> > > Hi Mark,
> > >
> > > On 08/24/2018 06:20 PM, Mark Adams wrote:
> > >
> > >> also, balance-rr through a switch requires each nic to be on a
> seperate
> > >> vlan. You probably need to remove your lacp config also but this
> depends
> > >> on
> > >> switch model and configuration. so safest idea is remove it.
> > >>
> > >
> > > 
> > >
> > > then I belive your iperf test will return ~3Gbps... i seem to remember
> > >> performance doesnt get much better than this but I cant remember why.
> > >>
> > >> Also can't say if this is a good setup for ceph performance..
> > >>
> > >
> > > This is really interesting info, i did not know this. Someone has tried
> > > this with ceph? Any experiences to share..?
> > >
> > > Strange that performence turns out to be ~3Gbps, instead of the
> expected
> > > 4...
> > >
> > > Anyone with more informationon this subject?
> > >
> > > Have a nice weekend all!
> > >
> > > MJ
> > > ___
> > > pve-user mailing list
> > > pve-user@pve.proxmox.com
> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > >
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Confusing about Bond 802.3ad

[Mark Adams]

also, balance-rr through a switch requires each nic to be on a seperate
vlan. You probably need to remove your lacp config also but this depends on
switch model and configuration. so safest idea is remove it.

so I think you have 3 nodes

for example:

node1:
ens0 on port 1 vlan 10
ens1 on port 4 vlan 11
ens2 on port 7 vlan 12
ens3 on port 10 vlan 13

node2:
ens0 on port 2 vlan 10
ens1 on port 5 vlan 11
ens2 on port 8 vlan 12
ens3 on port 11 vlan 13

node3:
ens0 on port 3 vlan 10
ens1 on port 6 vlan 11
ens2 on port 9 vlan 12
ens3 on port 12 vlan 13

then I belive your iperf test will return ~3Gbps... i seem to remember
performance doesnt get much better than this but I cant remember why.

Also can't say if this is a good setup for ceph performance..

Cheers

On Fri, 24 Aug 2018, 18:58 Josh Knight,  wrote:

> I don't know your topology, I'm assuming you're going from   nodeA ->
> switch -> nodeB ?  Make sure that entire path is using RR.  You could
> verify this with interface counters on the various hops.  If a single hop
> is not doing it correctly, it will limit the throughput.
>
> On Fri, Aug 24, 2018 at 11:20 AM Gilberto Nunes <
> gilberto.nune...@gmail.com>
> wrote:
>
> > So I try balance-rr with LAG in the switch and still get 1 GB
> >
> > pve-ceph02:~# iperf3 -c 10.10.10.100
> > Connecting to host 10.10.10.100, port 5201
> > [  4] local 10.10.10.110 port 52674 connected to 10.10.10.100 port 5201
> > [ ID] Interval   Transfer Bandwidth   Retr  Cwnd
> > [  4]   0.00-1.00   sec   116 MBytes   974 Mbits/sec   32670
> > KBytes
> > [  4]   1.00-2.00   sec   112 MBytes   941 Mbits/sec3597
> > KBytes
> > [  4]   2.00-3.00   sec   112 MBytes   941 Mbits/sec3509
> > KBytes
> > [  4]   3.00-4.00   sec   112 MBytes   941 Mbits/sec0660
> > KBytes
> > [  4]   4.00-5.00   sec   112 MBytes   941 Mbits/sec6585
> > KBytes
> > [  4]   5.00-6.00   sec   112 MBytes   941 Mbits/sec0720
> > KBytes
> > [  4]   6.00-7.00   sec   112 MBytes   942 Mbits/sec3650
> > KBytes
> > [  4]   7.00-8.00   sec   112 MBytes   941 Mbits/sec4570
> > KBytes
> > [  4]   8.00-9.00   sec   112 MBytes   941 Mbits/sec0708
> > KBytes
> > [  4]   9.00-10.00  sec   112 MBytes   941 Mbits/sec8635
> > KBytes
> > - - - - - - - - - - - - - - - - - - - - - - - - -
> > [ ID] Interval   Transfer Bandwidth   Retr
> > [  4]   0.00-10.00  sec  1.10 GBytes   945 Mbits/sec   59
> >  sender
> > [  4]   0.00-10.00  sec  1.10 GBytes   942 Mbits/sec
> > receiver
> >
> > iperf Done.
> >
> >
> >
> > ---
> > Gilberto Nunes Ferreira
> >
> > (47) 3025-5907
> > (47) 99676-7530 - Whatsapp / Telegram
> >
> > Skype: gilberto.nunes36
> >
> >
> >
> >
> > 2018-08-24 12:02 GMT-03:00 Josh Knight :
> >
> > > Depending on your topology/configuration, you could try to use bond-rr
> > mode
> > > in Linux instead of 802.3ad.
> > >
> > > Bond-rr mode is the only mode that will put pkts for the same
> mac/ip/port
> > > tuple across multiple interfaces.  This will work well for UDP but TCP
> > may
> > > suffer performance issues because pkts can end up out of order and
> > trigger
> > > TCP retransmits.  There are some examples on this page, you may need to
> > do
> > > some testing before deploying it to ensure it does what you want.
> > >
> > >
> >
> https://wiki.linuxfoundation.org/networking/bonding#bonding-driver-options
> > >
> > > As others have stated, you can adjust the hashing, but a single flow
> > > (mac/ip/port combination) will still end up limited to 1Gbps without
> > using
> > > round robin mode.
> > >
> > >
> > > On Fri, Aug 24, 2018 at 6:52 AM mj  wrote:
> > >
> > > > Hi,
> > > >
> > > > Yes, it is our undertanding that if the hardware (switch) supports
> it,
> > > > "bond-xmit-hash-policy layer3+4" gives you best spread.
> > > >
> > > > But it will still give you 4 'lanes' of 1GB. Ceph will connect using
> > > > different ports, ip's etc, en each connection should use a different
> > > > lane, so altogether, you should see a network throughput that
> > > > (theoretically) could be as high as 4GB.
> > > >
> > > > That is how we understand it.
> > > >
> > > > You can also try something on the switch, like we did on our
> ProCurve:
> > > >
> > > > >  Procurve chassis(config)# show trunk
> > > > >
> > > > > Load Balancing Method:  L3-based (default)
> > > > >
> > > > >  Port | Name Type  | Group  Type
> > > > >   +  - + --
> 
> > > > >  D1   | Link to prn004 - 1   10GbE-T   | Trk1   LACP
> > > > >  D2   | Link to prn004 - 2   10GbE-T   | Trk1   LACP
> > > > >  D3   | Link to prn005 - 1   10GbE-T   | Trk2   LACP
> > > > >  D4   | Link to prn005 - 2   10GbE-T   | Trk2   LACP
> > > >
> > > > Namely: change the load balancing method to:
> > > >
> > > > > Procurve chassis(config)# trunk-load-balance L4
> > > >
> > > > So the load 

Re: [PVE-User] DHCP for non cloudinit VM

[Mark Adams]

What sort of OS are you using for VM's, that does not default to having
DHCP enabled? I personally can't think of one that isn't DHCP out of the
box.

As for using specific IP's based on MAC address, this would be easily set
in the dhcp server config?

On 21 August 2018 at 08:48, José Manuel Giner  wrote:

> We are talking about auto-configuring the network on the VM, and therefore
> you cannot install the qemu-guest-agent on the VM if you do not have a
> network yet.
>
>
>
> On 21/08/2018 9:40, Mark Schouten wrote:
>
>> On Tue, 2018-08-21 at 08:51 +0200, José Manuel Giner wrote:
>>
>>> I know that already :) and it doesn't change anything because the
>>> management difficulty still exists.
>>>
>>> Everything would be simpler with native integration.
>>>
>>
>> I disagree. As would many people. But, nothing stops you from writing
>> your own qemu-guest-agent script to configure the IP address on the VM,
>> I think.
>>
>>
>
> --
> José Manuel Giner
> http://ginernet.com
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] DHCP for non cloudinit VM

[Mark Adams]

Maybe lost in translation? He said "determine" not "configure".

That means installing qemu agent in the guest will allow the proxmox
interface to show you what IP it is using.

On 20 August 2018 at 23:16, Vinicius Barreto 
wrote:

> Hello please could you tell which command what qemu agent do you use to
> configure the IP settings of the VM?
>
> Thank you!
>
> Em seg, 20 de ago de 2018 às 10:22, Ian Coetzee  >
> escreveu:
>
> > Hi José,
> >
> > Using the Qemu Agent you are able to determine the IP of the VM.
> >
> > Kind regards
> >
> > On Mon, 20 Aug 2018 at 13:43, José Manuel Giner  wrote:
> >
> > > The possibility of being able to define IPs directly from the
> > > Proxmox/API interface. Just like with Cloud-init or containers.
> > >
> > >
> > >
> > > On 20/08/2018 11:14, Dominik Csapak wrote:
> > > > On 08/20/2018 09:36 AM, José Manuel Giner wrote:
> > > >> Hello,
> > > >>
> > > >> there any plan to implement DHCP for non cloudinit VMs?
> > > >>
> > > >> Thanks!
> > > >>
> > > >>
> > > >
> > > > the question does not really make sense as we did not implement dhcp
> > for
> > > > cloudinit, the config there only tells the vm how to configure its
> > > > network (same as with containers, were we also don't implement a dhcp
> > > > server)
> > > >
> > > > but where is the problem in having a host/vm in your network serving
> > > > dhcp?
> > > >
> > > > ___
> > > > pve-user mailing list
> > > > pve-user@pve.proxmox.com
> > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > >
> > >
> > > --
> > > José Manuel Giner
> > > http://ginernet.com
> > >
> > > ___
> > > pve-user mailing list
> > > pve-user@pve.proxmox.com
> > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > >
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
>
>
> --
> *Marcos Vinícius Barreto das Chagas*
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] DHCP for non cloudinit VM

[Mark Adams]

Just install your own DHCP server on proxmox if you want. I don't see this
as a feature many people would want, as in any "normal" network you always
have a dhcp server already?

On 20 August 2018 at 15:35, José Manuel Giner  wrote:

> I thought cloud-init was connecting against a DHCP server.
>
> Does Proxmox have any plan to provide a DHCP service so that VMs can
> receive IP auto-configuration, whether they use cloud-init or not?
>
>
>
>
>
> On 20/08/2018 16:28, Dominik Csapak wrote:
>
>> but that cannot happen without an agent/cloud init like program in the
>> vm, so what and why (if we have cloud-init) should we implement there?
>>
>> On 08/20/2018 03:25 PM, José Manuel Giner wrote:
>>
>>> I mean, when you install the ISO of an operating system on a VM, when
>>> configuring the network, that the user can choose the DHCP option instead
>>> of defining the values by hand.
>>>
>>>
>>>
>>> On 20/08/2018 15:21, Ian Coetzee wrote:
>>>
 Hi José,

 Using the Qemu Agent you are able to determine the IP of the VM.

 Kind regards

 On Mon, 20 Aug 2018 at 13:43, José Manuel Giner 
 wrote:

 The possibility of being able to define IPs directly from the
> Proxmox/API interface. Just like with Cloud-init or containers.
>
>
>
> On 20/08/2018 11:14, Dominik Csapak wrote:
>
>> On 08/20/2018 09:36 AM, José Manuel Giner wrote:
>>
>>> Hello,
>>>
>>> there any plan to implement DHCP for non cloudinit VMs?
>>>
>>> Thanks!
>>>
>>>
>>>
>> the question does not really make sense as we did not implement dhcp
>> for
>> cloudinit, the config there only tells the vm how to configure its
>> network (same as with containers, were we also don't implement a dhcp
>> server)
>>
>> but where is the problem in having a host/vm in your network serving
>> dhcp?
>>
>> ___
>> pve-user mailing list
>> pve-user@pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>
>
> --
> José Manuel Giner
> http://ginernet.com
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
> ___
 pve-user mailing list
 pve-user@pve.proxmox.com
 https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user


>>>
>>>
>>
>> ___
>> pve-user mailing list
>> pve-user@pve.proxmox.com
>> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>>
>
>
> --
> José Manuel Giner
> http://ginernet.com
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Poor CEPH performance? or normal?

[Mark Adams]

Hi Adam,

Thanks for your great round up there - Your points are excellent.

What I have ended up doing a few days ago (apologies have been too busy to
respond..) was set rbd cache = true under each client in the ceph.conf -
This got me from 15MB/s up to about 70MB/s. I then set the disk holding the
zfs dataset to writeback cache in proxmox (as you note below) and that has
bumped it up to about 130MB/s -- Which I am happy with for this setup.

Regards,
Mark

On 27 July 2018 at 14:46, Adam Thompson  wrote:

> On 2018-07-27 07:05, ronny+pve-u...@aasen.cx wrote:
>
>> rbd striping is a per image setting. you may need to make the rbd
>> image and migrate data.
>>
>> On 07/26/18 12:25, Mark Adams wrote:
>>
>>> Thanks for your suggestions. Do you know if it is possible to change an
>>> existing rbd pool to striping? or does this have to be done on first
>>> setup?
>>>
>>
> Please be aware that striping will not result in any increased
> performance, if you are using "safe" I/O modes, i.e. your VM waits for a
> successful flush-to-disk after every sector.  In that scenario, CEPH will
> never give you write performance equal to a local disk because you're
> limited to the bandwidth of a single remote disk [subsystem] *plus* the
> network round-trip latency, which even if measured in microseconds, still
> adds up.
>
> Based on my experience with this and other distributed storage systems, I
> believe you will likely find that you get large write-performance gains by:
>
> 1. use the largest possible block size during writes.  512B sectors are
> the worst-case scenario for any remote storage.  Try to write in chunks of
> *at least* 1 MByte, and it's not unreasonable nowadays to write in chunks
> of 64MB or larger.  The rationale here is that you're spending more time
> sending data, and less time waiting for ACKs.  The more you can tilt that
> in favor of data, the better off you are.  (There are downsides to huge
> sector/block/chunk sizes, though - this isn't a "free lunch" scenario.  See
> #5.)
>
> 2. relax your write-consistency requirements.  If you can tolerate the
> small risk with "Write Back" you should see better performance, especially
> during burst writes.  During large sequential writes, there are not many
> ways to violate the laws of physics, and CEPH automatically amplifies your
> writes by (in your case) a factor of 2x due to replication.
>
> 3. switch to storage devices with the best possible local write speed, for
> OSDs.  OSDs are limited by the performance of the underlying device or
> virtual device.  (e.g. it's totally possible to run OSDs on a hardware
> RAID6 controller)
>
> 4. Avoid CoW-on-CoW.  Write amplification means you'll lose around 50% of
> your IOPS and/or I/O bandwidth for each level of CoW nesting, depending on
> workload.  So don't put CEPH OSDs on, ssy, BTRFS or ZFS filesystems.  A
> worst-case scenario would be something like running a VM using ZFS on top
> of CEPH, where the OSDs are located on BTRFS filsystems, which are in turn
> virtual devices hosted on ZFS filesystems.  Welcome to 1980's storage
> performance, in that case!  (I did it without realizing once... seriously,
> 5 MBps sequential writes was a good day!)  FWIW, CoW filesystems are
> generally awesome - just not when stacked.  A sufficiently fast external
> NAS running ZFS with VMs stored over NFS can provide decent performance,
> *if* tuned correctly.  iX Systems, for example, spends a lot of time &
> effort making this work well, including some lovely HA NAS appliances.
>
> 5. Remember the triangle.  You can optimize a distributed storage system
> for any TWO of: a) cost, b) resiliency/reliability/HA, or c) speed.  (This
> is a specific case of the traditional good/fast/cheap:pick-any-2 adage.)
>
>
> I'm not sure I'm saying anything new here, I may have just summarized the
> discussion, but the points remain valid.
>
> Good luck with your performance problems.
> -Adam
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Poor CEPH performance? or normal?

[Mark Adams]

Hi Ronny,

Thanks for your suggestions. Do you know if it is possible to change an
existing rbd pool to striping? or does this have to be done on first setup?

Regards,
Mark

On Wed, 25 Jul 2018, 19:20 Ronny Aasen,  wrote:

> On 25. juli 2018 02:19, Mark Adams wrote:
> > Hi All,
> >
> > I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB
> GOLD
> > drives. Network is 10Gbps on X550-T2, separate network for the ceph
> cluster.
> >
> > I have 1 VM currently running on this cluster, which is debian stretch
> with
> > a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s
> > write speed. does this sound right? it seems very slow to me.
> >
> > Not only that, but when this zfs send is running - I can not do any
> > parallel sends to any other zfs datasets inside of the same VM. They just
> > seem to hang, then eventually say "dataset is busy".
> >
> > Any pointers or insights greatly appreciated!
>
> Greetings
>
> alwin gave you some good advice about filesystems and vm's, i wanted to
> say a little about ceph.
>
> with 3 nodes, and the default and reccomended size=3 pools, you can not
> tolerate any node failures. IOW, if you loose a node, or need to do
> lengthy maintainance on it, you are running degraded. I allways have a
> 4th "failure domain" node. so my cluster can selfheal (one of cephs
> killer features)  from a node failure. your cluster should be
>
> 3+[how-many-node-failures-i-want-to-be-able-to-survive-and-still-operate-sanely]
>
> spinning osd's with bluestore benefit greatly from ssd DB/WAL's if your
> osd's have ondisk DB/WAL you can gain a lot of performance by having the
> DB/WAL on a SSD or better.
>
> ceph gains performance with scale(number of osd nodes) . so while ceph's
> aggeregate performance is awesome, an individual single thread will not
> be amazing. A given set of data will exist on all 3 nodes, and you will
> hit 100% of nodes with any write.  so by using ceph with 3 nodes you
> give ceph the worst case for performance. eg
> with 4 nodes a write would hit 75%, with 6 nodes it would hit 50% of the
> cluster. you see where this is going...
>
> But a single write will only hit one disk in 3 nodes, and will not have
> a better performance then the disk it hits. you can cheat more
> performance with rbd caching. and it is important for performance to get
> a higher queue depth. afaik zfs uses a queue depth of 1, for ceph the
> worst possible. you may have some success by buffering on one or both
> ends of the transfer [1]
>
> if the vm have a RBD disk, you may (or may not) benefit from rbd fancy
> striping[2],  since operations can hit more osd's in parallel.
>
>
> good luck
> Ronny Aasen
>
>
> [1]
>
> https://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/
> [2] http://docs.ceph.com/docs/master/architecture/#data-striping
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] Poor CEPH performance? or normal?

[Mark Adams]

Hi Alwin,

On 25 July 2018 at 07:10, Alwin Antreich  wrote:

> Hi,
>
> On Wed, Jul 25, 2018, 02:20 Mark Adams  wrote:
>
> > Hi All,
> >
> > I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB
> GOLD
> > drives. Network is 10Gbps on X550-T2, separate network for the ceph
> > cluster.
> >
> Do a rados bench for testing the  cluster performance, spinners are not
> fast.
>
>
This was a typo - I'm actually on 5.2-1. I'll give rados bench a try to see
what it comes back with.


> > I have 1 VM currently running on this cluster, which is debian stretch
> with
> > a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s
> > write speed. does this sound right? it seems very slow to me.
> >
> Never ever use a CoW filesystem on top of another CoW system. This doubles
> the writes that need to be made.


>
> > Not only that, but when this zfs send is running - I can not do any
> > parallel sends to any other zfs datasets inside of the same VM. They just
> > seem to hang, then eventually say "dataset is busy".
> >
> Ceph already gives you the possibility of snapshots. You can let PVE do
> this through CLI or GUI.
>

The problem with this is the required features.. I need an HA cluster, and
zfs doesn't support this - so ceph is ideal, however I also need "restore
previous versions" usable inside a file server VM in samba, which ceph
snapshots at the VM layer is no use for Unless there is some other
smart way of doing this I don't know about!

I guess my main question is, is there any other config hints to speed this
up whether ceph or in ZFS inside the VM, and is the blocking of other IO
normal with ceph when "max" write speed is being reached? That bit doesn't
seem right to me.


> --
> Cheers,
> Alwin
>

Regards,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] Poor CEPH performance? or normal?

[Mark Adams]

Hi All,

I have a proxmox 5.1 + ceph cluster of 3 nodes, each with 12 x WD 10TB GOLD
drives. Network is 10Gbps on X550-T2, separate network for the ceph cluster.

I have 1 VM currently running on this cluster, which is debian stretch with
a zpool on it. I'm zfs sending in to it, but only getting around ~15MiB/s
write speed. does this sound right? it seems very slow to me.

Not only that, but when this zfs send is running - I can not do any
parallel sends to any other zfs datasets inside of the same VM. They just
seem to hang, then eventually say "dataset is busy".

Any pointers or insights greatly appreciated!

Thanks
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pveceph createosd after destroyed osd

[Mark Adams]

Hi Alwin, Yannis,

Yes I definitely agree with you there Yannis - Good to know how to resolve
via cli when things don't seem right. It's also good to know that things
can go wrong sometimes via the gui.

Alwin - Any improvements are good of course!

Regards,
Mark

On 5 July 2018 at 12:00, Yannis Milios  wrote:

>  > Yes I realise it is, what I'm saying is should it also be doing those
> > steps?
>
> Usually you don't have to, but as things often can go wrong you *may* have
> to do things manually sometimes.
> GUI is great and saves lots of work, however knowing how to manually solve
> problems when they arise via the CLI in my opinion is also a
> must.Especially when you deal with a complicated storage like Ceph 
>
> Y


> On Thu, Jul 5, 2018 at 11:53 AM Alwin Antreich 
> wrote:
>
> > On Thu, Jul 05, 2018 at 11:05:52AM +0100, Mark Adams wrote:
> > > On 5 July 2018 at 11:04, Alwin Antreich 
> wrote:
> > >
> > > > On Thu, Jul 05, 2018 at 10:26:34AM +0100, Mark Adams wrote:
> > > > > Hi Anwin;
> > > > >
> > > > > Thanks for that - It's all working now! Just to confirm though,
> > shouldn't
> > > > > the destroy button handle some of these actions? or is it left out
> on
> > > > > purpose?
> > > > >
> > > > > Regards,
> > > > > Mark
> > > > >
> > > > I am not sure, what you mean exactly but the destroyosd (CLI/GUI) is
> > > > doing more then those two steps.
> > > >
> > > >
> > > Yes I realise it is, what I'm saying is should it also be doing those
> > > steps?
> > Well, it is doing those too. Just with the failed creation of the OSD
> > not all entries are set and the destroy might fail on some (eg. no
> > service, no mount).
> >
> > The osd create/destroy is up for a change anyway with the move from
> > ceph-disk (deprecated in Mimic) to ceph-volume. Sure room for
> > improvement. ;)
> >
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pveceph createosd after destroyed osd

[Mark Adams]

On 5 July 2018 at 11:04, Alwin Antreich  wrote:

> On Thu, Jul 05, 2018 at 10:26:34AM +0100, Mark Adams wrote:
> > Hi Anwin;
> >
> > Thanks for that - It's all working now! Just to confirm though, shouldn't
> > the destroy button handle some of these actions? or is it left out on
> > purpose?
> >
> > Regards,
> > Mark
> >
> I am not sure, what you mean exactly but the destroyosd (CLI/GUI) is
> doing more then those two steps.
>
>
Yes I realise it is, what I'm saying is should it also be doing those
steps?
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pveceph createosd after destroyed osd

[Mark Adams]

Hi Anwin;

Thanks for that - It's all working now! Just to confirm though, shouldn't
the destroy button handle some of these actions? or is it left out on
purpose?

Regards,
Mark

On 3 July 2018 at 16:16, Alwin Antreich  wrote:

> On Tue, Jul 03, 2018 at 12:18:53PM +0100, Mark Adams wrote:
> > Hi Alwin, please see my response below.
> >
> > On 3 July 2018 at 10:07, Alwin Antreich  wrote:
> >
> > > On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote:
> > > > Currently running the newest 5.2-1 version, I had a test cluster
> which
> > > was
> > > > working fine. I since added more disks, first stopping, then setting
> out,
> > > > then destroying each osd so I could recreate it all from scratch.
> > > >
> > > > However, when adding a new osd (either via GUI or pveceph CLI) it
> seems
> > > to
> > > > show a successful create, however does not show in the gui as an osd
> > > under
> > > > the host.
> > > >
> > > > It's like the osd information is being stored by proxmox/ceph
> somewhere
> > > > else and not being correctly removed and recreated?
> > > >
> > > > I can see that the newly created disk (after it being destroyed) is
> > > > down/out.
> > > >
> > > > Is this by design? is there a way to force the disk back? shouldn't
> it
> > > show
> > > > in the gui once you create it again?
> > > >
> > > Did you zero your disks after removal? On the first ~200 MB of the
> disk,
> > > there are leftovers that need to be zeroed before use. After that the
> > > OSD should be added fine.
> > >
> > >
> > I hadn't done this, no - it has helped with the majority of disks thanks
> > and I can now re-add them (I also had to remove the folders from
> > /var/lib/ceph/osd which had other osd names - not sure if the destroy
> > process is supposed to remove them also?)
> They will not interfere. ;)
>
> >
> > However I have a strange problem on the 2nd host, where it will not make
> > osd.12 ... I get no error output from the gui or pveceph createosd
> /dev/sda
> > - it just doesn't appear as an osd.
> >
> > It successfully partitions the disk, but doesn't create a folder in
> > /var/lib/ceph/osd/ for the osd mount. I can see there is lock files in
> > /var/lib/ceph/tmp/ ... which I would think should only be there whilst
> the
> > creation is taking place?
> From the OSD creation or different lock? Shouldn't make troubles either.
>
> >
> > journalctl -xe is showing me the problem I think, "command_with_stdin:
> > Error EEXIST: entity osd.12 exists but key does not match"
> >
> > Where is this key? how should I be clearing it out so it will create?
> >
> 'ceph auth list' will show you all keys in ceph, there will be an orphan
> osd.12. Removal 'ceph auth del '.
>
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pveceph createosd after destroyed osd

[Mark Adams]

Hi Alwin, please see my response below.

On 3 July 2018 at 10:07, Alwin Antreich  wrote:

> On Tue, Jul 03, 2018 at 01:05:51AM +0100, Mark Adams wrote:
> > Currently running the newest 5.2-1 version, I had a test cluster which
> was
> > working fine. I since added more disks, first stopping, then setting out,
> > then destroying each osd so I could recreate it all from scratch.
> >
> > However, when adding a new osd (either via GUI or pveceph CLI) it seems
> to
> > show a successful create, however does not show in the gui as an osd
> under
> > the host.
> >
> > It's like the osd information is being stored by proxmox/ceph somewhere
> > else and not being correctly removed and recreated?
> >
> > I can see that the newly created disk (after it being destroyed) is
> > down/out.
> >
> > Is this by design? is there a way to force the disk back? shouldn't it
> show
> > in the gui once you create it again?
> >
> Did you zero your disks after removal? On the first ~200 MB of the disk,
> there are leftovers that need to be zeroed before use. After that the
> OSD should be added fine.
>
>
I hadn't done this, no - it has helped with the majority of disks thanks
and I can now re-add them (I also had to remove the folders from
/var/lib/ceph/osd which had other osd names - not sure if the destroy
process is supposed to remove them also?)

However I have a strange problem on the 2nd host, where it will not make
osd.12 ... I get no error output from the gui or pveceph createosd /dev/sda
- it just doesn't appear as an osd.

It successfully partitions the disk, but doesn't create a folder in
/var/lib/ceph/osd/ for the osd mount. I can see there is lock files in
/var/lib/ceph/tmp/ ... which I would think should only be there whilst the
creation is taking place?

journalctl -xe is showing me the problem I think, "command_with_stdin:
Error EEXIST: entity osd.12 exists but key does not match"

Where is this key? how should I be clearing it out so it will create?

Thanks,
Mark


--
> Cheers,
> Alwin
>
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pveceph createosd after destroyed osd

[Mark Adams]

Hi, Thanks for your response!

No, I didn't do any of that on the cli - I just did stop in the webgui,
then out, then destroy.

Note that there was no VM's or data at all on this test ceph cluster - I
had deleted it all before doing this. I was basically just removing it all
so the OSD numbers looked "nicer" for the final setup.

It's not a huge deal, I can just reinstall proxmox. But it concerns me that
it seems so fragile using the webgui to do this. I want to know where I
went wrong? Is there somewhere that a signature is being stored so when you
try to add that same drive again (even though I ticked "remove partitions")
it doesn't add back in to the ceph cluster in the next sequential order
from the last current "live" or "valid" drive?

Is it just a rule that you never actually remove drives? you just set them
stopped/out?

Regards,
Mark



On 3 July 2018 at 01:34, Woods, Ken A (DNR)  wrote:

> http://docs.ceph.com/docs/mimic/rados/operations/add-or-
> rm-osds/#removing-osds-manual
>
> Are you sure you followed the directions?
>
> ____
> From: pve-user  on behalf of Mark Adams
> 
> Sent: Monday, July 2, 2018 4:05:51 PM
> To: pve-user@pve.proxmox.com
> Subject: [PVE-User] pveceph createosd after destroyed osd
>
> Currently running the newest 5.2-1 version, I had a test cluster which was
> working fine. I since added more disks, first stopping, then setting out,
> then destroying each osd so I could recreate it all from scratch.
>
> However, when adding a new osd (either via GUI or pveceph CLI) it seems to
> show a successful create, however does not show in the gui as an osd under
> the host.
>
> It's like the osd information is being stored by proxmox/ceph somewhere
> else and not being correctly removed and recreated?
>
> I can see that the newly created disk (after it being destroyed) is
> down/out.
>
> Is this by design? is there a way to force the disk back? shouldn't it show
> in the gui once you create it again?
>
> Thanks!
>
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] pveceph createosd after destroyed osd

[Mark Adams]

Currently running the newest 5.2-1 version, I had a test cluster which was
working fine. I since added more disks, first stopping, then setting out,
then destroying each osd so I could recreate it all from scratch.

However, when adding a new osd (either via GUI or pveceph CLI) it seems to
show a successful create, however does not show in the gui as an osd under
the host.

It's like the osd information is being stored by proxmox/ceph somewhere
else and not being correctly removed and recreated?

I can see that the newly created disk (after it being destroyed) is
down/out.

Is this by design? is there a way to force the disk back? shouldn't it show
in the gui once you create it again?

Thanks!
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pve-csync version of pve-zsync?

[Mark Adams]

Hi Alexander,

Could you please elaborate more on how you have implemented ceph
replication using proxmox?

Thanks,
Mark


On Thu, 17 May 2018, 15:26 Alexandre DERUMIER, <aderum...@odiso.com> wrote:

> Hi,
>
> I'm currently a lot busy working on network code.
>
> for now, we have implemented ceph replication out of proxmox code, I'll
> try to work on it this summer.
>
>
> - Mail original -
> De: "Mark Adams" <m...@openvs.co.uk>
> À: "proxmoxve" <pve-user@pve.proxmox.com>
> Envoyé: Mardi 15 Mai 2018 00:13:03
> Objet: Re: [PVE-User] pve-csync version of pve-zsync?
>
> Hi Alexandre,
>
> Did you ever get a chance to take a look at this?
>
> Regards,
> Mark
>
> On 13 March 2018 at 18:32, Alexandre DERUMIER <aderum...@odiso.com>
> wrote:
>
> > Hi,
> >
> > I have plans to implement storage replication for rbd in proxmox,
> > like for zfs export|import. (with rbd export-diff |rbd import-diff )
> >
> > I'll try to work on it next month.
> >
> > I'm not sure that currently a plugin infrastructe in done in code,
> > and that it's able to manage storages with differents name.
> >
> > Can't tell if it'll be hard to implement, but the workflow is almost the
> > same.
> >
> > I'll try to look also at rbd mirror, but it's only work with librbd in
> > qemu, not with krbd,
> > so it can't be implemented for container.
> >
> >
> > - Mail original -
> > De: "Mark Adams" <m...@openvs.co.uk>
> > À: "proxmoxve" <pve-user@pve.proxmox.com>
> > Envoyé: Mardi 13 Mars 2018 18:52:21
> > Objet: Re: [PVE-User] pve-csync version of pve-zsync?
> >
> > Hi Alwin,
> >
> > I might have to take another look at it, but have you actually done this
> > with 2 proxmox clusters? I can't remember the exact part I got stuck on
> as
> > it was quite a while ago, but it wasn't as straight forward as you
> > suggest.
> > I think you couldn't use the same cluster name, which in turn created
> > issues trying to use the "remote" (backup/dr/whatever you wanna call it)
> > cluster with proxmox because it needed to be called ceph.
> >
> > The docs I was referring to were the ceph ones yes. Some of the options
> > listed in that doc do not work in the current proxmox version (I think
> the
> > doc hasn't been updated for newer versions...)
> >
> > Regards,
> > Mark
> >
> > On 13 March 2018 at 17:19, Alwin Antreich <a.antre...@proxmox.com>
> wrote:
> >
> > > On Mon, Mar 12, 2018 at 04:51:32PM +, Mark Adams wrote:
> > > > Hi Alwin,
> > > >
> > > > The last I looked at it, rbd mirror only worked if you had different
> > > > cluster names. Tried to get it working with proxmox but to no avail,
> > > > without really messing with how proxmox uses ceph I'm not sure it's
> > > > feasible, as proxmox assumes the default cluster name for
> > everything...
> > > That isn't mentioned anywhere in the ceph docs, they use for ease of
> > > explaining two different cluster names.
> > >
> > > If you have a config file named after the cluster, then you can
> specifiy
> > > it on the command line.
> > > http://docs.ceph.com/docs/master/rados/configuration/
> > > ceph-conf/#running-multiple-clusters
> > >
> > > >
> > > > Also the documentation was a bit poor for it IMO.
> > > Which documentation do you mean?
> > > ? -> http://docs.ceph.com/docs/master/rbd/rbd-mirroring/
> > >
> > > >
> > > > Would also be nice to choose specifically which VM's you want to be
> > > > mirroring, rather than the whole cluster.
> > > It is done either per pool or image separately. See the link above.
> > >
> > > >
> > > > I've manually done rbd export-diff and rbd import-diff between 2
> > separate
> > > > proxmox clusters over ssh, and it seems to work really well... It
> > would
> > > > just be nice to have a tool like pve-zsync so I don't have to write
> > some
> > > > script myself. Seems to me like something that would be desirable as
> > part
> > > > of proxmox as well?
> > > That would basically implement the ceph rbd mirror feature.
> > >
> > > >
> > > > Cheers,
> > > > Mark
> > > >
> > > > On 12 March 2018 at 16:37, Alwin Antreich <a.antre...@proxmox.com>
> > > wrote:
> > > >
>

Re: [PVE-User] pve-csync version of pve-zsync?

[Mark Adams]

Hi Alexandre,

Did you ever get a chance to take a look at this?

Regards,
Mark

On 13 March 2018 at 18:32, Alexandre DERUMIER <aderum...@odiso.com> wrote:

> Hi,
>
> I have plans to implement storage replication for rbd in proxmox,
> like for zfs export|import.  (with rbd export-diff |rbd import-diff )
>
> I'll try to work on it next month.
>
> I'm not sure that currently a plugin infrastructe in done in code,
> and that it's able to manage storages with differents name.
>
> Can't tell if it'll be hard to implement, but the workflow is almost the
> same.
>
> I'll try to look also at rbd mirror, but it's only work with librbd in
> qemu, not with krbd,
> so it can't be implemented for container.
>
>
> - Mail original -
> De: "Mark Adams" <m...@openvs.co.uk>
> À: "proxmoxve" <pve-user@pve.proxmox.com>
> Envoyé: Mardi 13 Mars 2018 18:52:21
> Objet: Re: [PVE-User] pve-csync version of pve-zsync?
>
> Hi Alwin,
>
> I might have to take another look at it, but have you actually done this
> with 2 proxmox clusters? I can't remember the exact part I got stuck on as
> it was quite a while ago, but it wasn't as straight forward as you
> suggest.
> I think you couldn't use the same cluster name, which in turn created
> issues trying to use the "remote" (backup/dr/whatever you wanna call it)
> cluster with proxmox because it needed to be called ceph.
>
> The docs I was referring to were the ceph ones yes. Some of the options
> listed in that doc do not work in the current proxmox version (I think the
> doc hasn't been updated for newer versions...)
>
> Regards,
> Mark
>
> On 13 March 2018 at 17:19, Alwin Antreich <a.antre...@proxmox.com> wrote:
>
> > On Mon, Mar 12, 2018 at 04:51:32PM +, Mark Adams wrote:
> > > Hi Alwin,
> > >
> > > The last I looked at it, rbd mirror only worked if you had different
> > > cluster names. Tried to get it working with proxmox but to no avail,
> > > without really messing with how proxmox uses ceph I'm not sure it's
> > > feasible, as proxmox assumes the default cluster name for
> everything...
> > That isn't mentioned anywhere in the ceph docs, they use for ease of
> > explaining two different cluster names.
> >
> > If you have a config file named after the cluster, then you can specifiy
> > it on the command line.
> > http://docs.ceph.com/docs/master/rados/configuration/
> > ceph-conf/#running-multiple-clusters
> >
> > >
> > > Also the documentation was a bit poor for it IMO.
> > Which documentation do you mean?
> > ? -> http://docs.ceph.com/docs/master/rbd/rbd-mirroring/
> >
> > >
> > > Would also be nice to choose specifically which VM's you want to be
> > > mirroring, rather than the whole cluster.
> > It is done either per pool or image separately. See the link above.
> >
> > >
> > > I've manually done rbd export-diff and rbd import-diff between 2
> separate
> > > proxmox clusters over ssh, and it seems to work really well... It
> would
> > > just be nice to have a tool like pve-zsync so I don't have to write
> some
> > > script myself. Seems to me like something that would be desirable as
> part
> > > of proxmox as well?
> > That would basically implement the ceph rbd mirror feature.
> >
> > >
> > > Cheers,
> > > Mark
> > >
> > > On 12 March 2018 at 16:37, Alwin Antreich <a.antre...@proxmox.com>
> > wrote:
> > >
> > > > Hi Mark,
> > > >
> > > > On Mon, Mar 12, 2018 at 03:49:42PM +, Mark Adams wrote:
> > > > > Hi All,
> > > > >
> > > > > Has anyone looked at or thought of making a version of pve-zsync
> for
> > > > ceph?
> > > > >
> > > > > This would be great for DR scenarios...
> > > > >
> > > > > How easy do you think this would be to do? I imagine it wouId it
> be
> > quite
> > > > > similar to pve-zsync, but using rbd export-diff and rbd
> import-diff
> > > > instead
> > > > > of zfs send and zfs receive? so could the existing script be
> > relatively
> > > > > easily modified? (I know nothing about perl)
> > > > >
> > > > > Cheers,
> > > > > Mark
> > > > > ___
> > > > > pve-user mailing list
> > > > > pve-user@pve.proxmox.com
> > > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > > Isn't ceph mirror already what you want? It can mirror a image or a
> > > > whole pool. It keeps track of changes and serves remote image
> deletes
> > > > (adjustable delay).
> > > >
> >
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> >
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pve-zsync processes

[Mark Adams]

Hi Wolfgang,

So does this mean that all those processes are sitting in a "queue" waiting
to execute? wouldn't it be more sensible for the script to terminate if a
process is already running for the same job?

Regards,
Mark

On 21 March 2018 at 12:40, Wolfgang Link  wrote:

> Hi,
>
> this indicates that the sync time is to low.
> cron fork every (default) 15 minutes a pve-zsync process.
> If the former pve-zsync process is not finished, it will wait until the
> former process is done.
>
> You should rise your sync interval this can be done in the
> /etc/cron.d/pve-zsync.
>
> Best Regards,
>
> Wolfgang Link
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] pve-zsync processes

[Mark Adams]

Hi All,

I've been using pve-zsync for a few months - it seems to work pretty well.

However, I have just noticed it doesn't seem to be terminating itself
correctly. at present I have around 800 pve-zsync processes (sleeping)
which all seems to be duplicates. (I would expect 1 per VMID?)

Has anyone noticed this behaviour? any idea why or how to stop it?

Best Regards,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pve-csync version of pve-zsync?

[Mark Adams]

Hi Alwin,

I might have to take another look at it, but have you actually done this
with 2 proxmox clusters? I can't remember the exact part I got stuck on as
it was quite a while ago, but it wasn't as straight forward as you suggest.
I think you couldn't use the same cluster name, which in turn created
issues trying to use the "remote" (backup/dr/whatever you wanna call it)
cluster with proxmox because it needed to be called ceph.

The docs I was referring to were the ceph ones yes. Some of the options
listed in that doc do not work in the current proxmox version (I think the
doc hasn't been updated for newer versions...)

Regards,
Mark

On 13 March 2018 at 17:19, Alwin Antreich <a.antre...@proxmox.com> wrote:

> On Mon, Mar 12, 2018 at 04:51:32PM +, Mark Adams wrote:
> > Hi Alwin,
> >
> > The last I looked at it, rbd mirror only worked if you had different
> > cluster names. Tried to get it working with proxmox but to no avail,
> > without really messing with how proxmox uses ceph I'm not sure it's
> > feasible, as proxmox assumes the default cluster name for everything...
> That isn't mentioned anywhere in the ceph docs, they use for ease of
> explaining two different cluster names.
>
> If you have a config file named after the cluster, then you can specifiy
> it on the command line.
> http://docs.ceph.com/docs/master/rados/configuration/
> ceph-conf/#running-multiple-clusters
>
> >
> > Also the documentation was a bit poor for it IMO.
> Which documentation do you mean?
> ? -> http://docs.ceph.com/docs/master/rbd/rbd-mirroring/
>
> >
> > Would also be nice to choose specifically which VM's you want to be
> > mirroring, rather than the whole cluster.
> It is done either per pool or image separately. See the link above.
>
> >
> > I've manually done rbd export-diff and rbd import-diff between 2 separate
> > proxmox clusters over ssh, and it seems to work really well... It would
> > just be nice to have a tool like pve-zsync so I don't have to write some
> > script myself. Seems to me like something that would be desirable as part
> > of proxmox as well?
> That would basically implement the ceph rbd mirror feature.
>
> >
> > Cheers,
> > Mark
> >
> > On 12 March 2018 at 16:37, Alwin Antreich <a.antre...@proxmox.com>
> wrote:
> >
> > > Hi Mark,
> > >
> > > On Mon, Mar 12, 2018 at 03:49:42PM +, Mark Adams wrote:
> > > > Hi All,
> > > >
> > > > Has anyone looked at or thought of making a version of pve-zsync for
> > > ceph?
> > > >
> > > > This would be great for DR scenarios...
> > > >
> > > > How easy do you think this would be to do? I imagine it wouId it be
> quite
> > > > similar to pve-zsync, but using rbd export-diff and rbd import-diff
> > > instead
> > > > of zfs send and zfs receive? so could the existing script be
> relatively
> > > > easily modified? (I know nothing about perl)
> > > >
> > > > Cheers,
> > > > Mark
> > > > ___
> > > > pve-user mailing list
> > > > pve-user@pve.proxmox.com
> > > > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> > > Isn't ceph mirror already what you want? It can mirror a image or a
> > > whole pool. It keeps track of changes and serves remote image deletes
> > > (adjustable delay).
> > >
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] pve-csync version of pve-zsync?

[Mark Adams]

Hi Alwin,

The last I looked at it, rbd mirror only worked if you had different
cluster names. Tried to get it working with proxmox but to no avail,
without really messing with how proxmox uses ceph I'm not sure it's
feasible, as proxmox assumes the default cluster name for everything...

Also the documentation was a bit poor for it IMO.

Would also be nice to choose specifically which VM's you want to be
mirroring, rather than the whole cluster.

I've manually done rbd export-diff and rbd import-diff between 2 separate
proxmox clusters over ssh, and it seems to work really well... It would
just be nice to have a tool like pve-zsync so I don't have to write some
script myself. Seems to me like something that would be desirable as part
of proxmox as well?

Cheers,
Mark

On 12 March 2018 at 16:37, Alwin Antreich <a.antre...@proxmox.com> wrote:

> Hi Mark,
>
> On Mon, Mar 12, 2018 at 03:49:42PM +0000, Mark Adams wrote:
> > Hi All,
> >
> > Has anyone looked at or thought of making a version of pve-zsync for
> ceph?
> >
> > This would be great for DR scenarios...
> >
> > How easy do you think this would be to do? I imagine it wouId it be quite
> > similar to pve-zsync, but using rbd export-diff and rbd import-diff
> instead
> > of zfs send and zfs receive? so could the existing script be relatively
> > easily modified? (I know nothing about perl)
> >
> > Cheers,
> > Mark
> > ___
> > pve-user mailing list
> > pve-user@pve.proxmox.com
> > https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
> Isn't ceph mirror already what you want? It can mirror a image or a
> whole pool. It keeps track of changes and serves remote image deletes
> (adjustable delay).
>
>
> --
> Cheers,
> Alwin
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] pve-csync version of pve-zsync?

[Mark Adams]

Hi All,

Has anyone looked at or thought of making a version of pve-zsync for ceph?

This would be great for DR scenarios...

How easy do you think this would be to do? I imagine it wouId it be quite
similar to pve-zsync, but using rbd export-diff and rbd import-diff instead
of zfs send and zfs receive? so could the existing script be relatively
easily modified? (I know nothing about perl)

Cheers,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] ZFS Replication

[Mark Adams]

Im just trying out the zfs replication in proxmox, nice work!

Just a few questions..

- Is it possible to change the network that does the replication? (IE be
good to use a direct connected with balance-rr for throughput)

- Is it possible to replicate between machines that are not in the same
cluster?

Both can be easily done via zfs send/recv in cli of course, but wonder if
this is possible through the web interface?

And lastly, what is the correct procedure for using a replicated VM, should
it be needed?


Thanks,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HA Fencing

[Mark Adams]

On 5 December 2017 at 08:52, Thomas Lamprecht <t.lampre...@proxmox.com>
wrote:

> Hi,
>
> On 12/04/2017 07:51 PM, Mark Adams wrote:
> > On 17 November 2017 at 10:55, Thomas Lamprecht <t.lampre...@proxmox.com>
> wrote:
> >> On 11/16/2017 07:20 PM, Mark Adams wrote:
> >>> Hi all,
> >>>
> >>> It looks like in newer versions of proxmox, the only fencing type
> advised
> >>> is watchdog. Is that the case?
> >>>
> >>
> >> Yes, since PVE 4.0 watchdog fencing is the norm.
> >> There is a patch set of mine which implements the use of external fence
> >> device,
> >> but it has seen no review. I should probably dust it up, look over it
> and
> >> re send
> >> it again, it's about time we finally get this feature.
> >>
> >
> > I think you should definitely get this feature in - I would even say it
> is
> > necessary for an enterprise HA setup?
> >
>
> Not really a necessary. Watchdog based fencing is no less secure than
> traditional
> fence devices. In fact, as there's much less to configure, and much less
> protocols
> between them I'd say its the opposite. I.e., you do not must fire up a
> command
> over TCP/IP to fence a node to a device. Here are multiple problem points,
> Link problems, high load problems delaying fencing, fence devices whit a
> setup not
> well tested, at least not under failure conditions, ...
> A watchdog, which triggers as soon as the node did not pulled it up,
> independent
> of link failures, cluster load is here the safer bet. They are often the
> norm in
> highly-secure critical embedded systems to, not without reason.
> It's the difference between a emergency shutdown button and a
> dead-man-switch.
>

AFAIK It's the only way to know for sure, that your server has actually
been fenced when it is not contactable by other means, For instance some
network issue on the host.

Yes the Watchdog on the machine that goes offline should fence itself, but
still the only way to know for sure that the machine is dead is to power it
off right?


> Maybe you didn't even meant the reliability stand point but that a better
> best-case SLA could be possible with fence devices?
>

This does make a difference too, it could fail over in seconds with faster
fencing.


>
> But nonetheless agreeing that we should really get it in. I'll try to
> pickup the
> series before this month ends, after the Cluster over API stuff got in.
>

Thanks it would be great to see it in.


>
> cheers,
> Thomas
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] HA Fencing

[Mark Adams]

Hi,

On 17 November 2017 at 10:55, Thomas Lamprecht <t.lampre...@proxmox.com>
wrote:

> Hi,
>
> On 11/16/2017 07:20 PM, Mark Adams wrote:
> > Hi all,
> >
> > It looks like in newer versions of proxmox, the only fencing type advised
> > is watchdog. Is that the case?
> >
>
> Yes, since PVE 4.0 watchdog fencing is the norm.
> There is a patch set of mine which implements the use of external fence
> device,
> but it has seen no review. I should probably dust it up, look over it and
> re send
> it again, it's about time we finally get this feature.
>

I think you should definitely get this feature in - I would even say it is
necessary for an enterprise HA setup?


> > Is it still possible to do PDU fencing as well? This should enable us to
> be
> > able to fail over faster as the fence will not fail if the machine has no
> > power right?
> >
>
> No, at the moment external fence devices are not integrated.
> You can expect an faster recovery with external fence devices, at least in
> simple setups (i.e., not multiple fence device hierachy)
>
> cheers,
> Thomas
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] HA Fencing

[Mark Adams]

Hi all,

It looks like in newer versions of proxmox, the only fencing type advised
is watchdog. Is that the case?

Is it still possible to do PDU fencing as well? This should enable us to be
able to fail over faster as the fence will not fail if the machine has no
power right?

Thanks
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] snapshot=off

[Mark Adams]

Hi All,

On proxmox 5.1, with ceph as storage, I'm trying to disable the
snapshotting of a specific disk on a VM.

This is not an option in the gui, but I've added the option to the disk in
the conf file

scsi1: ssd_ceph_vm:vm-100-disk-2,discard=on,size=32G,snapshot=off

However, this seems to be ignored and the snapshot is still created.

Is this just not supported?

Thanks in advance!
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] ZFS over iSCSI in Linux box versus OmniOS box

[Mark Adams]

Try create more than 10 vdisks (doesn't have to be all in the same VM...)

It never worked after 10 for me.

On 28 October 2017 at 14:11, Gilberto Nunes 
wrote:

> Just a note: with Ubuntu I am using IET ( iSCSI Enterprise Target) and it's
> seems faster than ever!
>
> ---
> Gilberto Ferreira
>
> Consultor TI Linux | IaaS Proxmox, CloudStack, KVM | Zentyal Server |
> Zimbra Mail Server
>
> (47) 3025-5907
> (47) 99676-7530
>
> Skype: gilberto.nunes36
>
>
> konnectati.com.br 
>
>
> https://www.youtube.com/watch?v=2rkgOxuyuu8
>  com%2Fwatch%3Fv%3D2rkgOxuyuu8=ATP9BJPCTVrxQ0HlXLOw0kw0XCzPJ4
> U4ykUa7tFWFuBvgTD0wmC801aq_95_stzEFRQVw7Kr0HZutpDHKLccDjjj0t
> TUxU4JsKeQ3WPxS07szD2CKzq-xOsR7zZYZwv7JP_D9tuk1dBIa2yZWUYlXdd83_
> PYwmBhJ00TxUioCq5_XnrZnanIvpOhKXGwiULeTXsdPqjKmB8Wh_
> 6fLjaFYpAmrSW3eghz4vtEgJcoCPm82hLIgJuS4tLlFXX8y2RJFNiZU8Ke2M
> _tbEOUgk95FP2jqhPlvpkYHLnmr0Q>
>
>
> 2017-10-28 10:52 GMT-02:00 Gilberto Nunes :
>
> > Hi guys
> >
> > I swear I just cannot explain that, but a Ubuntu Server box seems to be
> > faster than OmniOS in the same hardware.
> > Before, I have deployed a OmniOS lastest version, I it's very slow... I
> > cannot even make movedisk to two VM's at sametime...
> > Now, I deploy a Ubuntu Server 16.04 and the storage are really really
> > faster
> > And, which is better, I can now make movedisk for more than one VM at
> > sametime...
> > Amazing
> > I don't know it's something about OmniOS or what!
> > But I am very happy now...
> >
> > Cheers
> >
> > ---
> > Gilberto Ferreira
> >
> >
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] PVE ZFS Over ISCSI

[Mark Adams]

Imo, don't even bother trying to do this with tgt. I got it working some
time ago but it was flakey, and started having dataset naming issues after
about 10 disks.

The only working (stable) zfs over iscsi with proxmox afaik is using
comstar (which to be fair to proxmox devs, is what they say in the docs...)

If you want to get this working I suggest you use omnios for the zfs
storage end. The free version of napp-it is easy to get going for a proof
of concept.

On 11 Aug 2017 19:23, "Mikhail"  wrote:

Hello,

Even if you're able to insert and create the disk, this does not mean
you will be able to start/run VM with that disk. Apart from creating
disk on ZFS storage you also need to export it via iSCSI.

Unfortunately, ZFS-over-iSCSI is not documented well. I came across same
problem as you 3 weeks ago (posted here on the list), but my questions
left unanswered.

Mikhail.

On 08/11/2017 07:14 PM, Gilberto Nunes wrote:

> If I use the command:
>
> qm set 100 -scsi1 ZFS-OVER:vm-100-disk-1
> update VM 100: -scsi1 ZFS-OVER:vm-100-disk-1
>
> I can insert the disk with sucess
>
> But inside web interface I am no able to!

___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
___
pve-user mailing list
pve-user@pve.proxmox.com
https://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

Re: [PVE-User] ZFS on iSCSI + Pacemaker/corosync/DRBD

[Mark Adams]

Hi,

Thanks for the response.

I was planning on using active/backup bonding on 10Gbe for my network fault
tolerance so no multipath support shouldn't be an issue.

I've come across some strange behaviour with the iet provider though, in
that after 9 LUNS it starts changing the existing luns rather than adding
additional.

Hard disk config in proxmox for VM:

Hard Disk (virtio0) ZFSOVERISCSI:vm-112-disk-1,size=10G
Hard Disk (virtio1) ZFSOVERISCSI:vm-112-disk-2,size=10G
Hard Disk (virtio2) ZFSOVERISCSI:vm-112-disk-3,size=10G
Hard Disk (virtio3) ZFSOVERISCSI:vm-112-disk-4,size=10G
Hard Disk (virtio4) ZFSOVERISCSI:vm-112-disk-5,size=10G
Hard Disk (virtio5) ZFSOVERISCSI:vm-112-disk-6,size=10G
Hard Disk (virtio6) ZFSOVERISCSI:vm-112-disk-7,size=10G
Hard Disk (virtio7) ZFSOVERISCSI:vm-112-disk-8,size=10G
Hard Disk (virtio8) ZFSOVERISCSI:vm-112-disk-9,size=10G
Hard Disk (virtio9) ZFSOVERISCSI:vm-112-disk-10,size=10G

ietd.conf file on zfs/iscsi storage host:

Lun 0 Path=/dev/VMSTORE/vm-112-disk-1,Type=blockio
Lun 1 Path=/dev/VMSTORE/vm-112-disk-2,Type=blockio
Lun 2 Path=/dev/VMSTORE/vm-112-disk-3,Type=blockio
Lun 3 Path=/dev/VMSTORE/vm-112-disk-4,Type=blockio
Lun 4 Path=/dev/VMSTORE/vm-112-disk-6,Type=blockio
Lun 5 Path=/dev/VMSTORE/vm-112-disk-7,Type=blockio
Lun 6 Path=/dev/VMSTORE/vm-112-disk-8,Type=blockio
Lun 7 Path=/dev/VMSTORE/vm-112-disk-9,Type=blockio
Lun 8 Path=/dev/VMSTORE/vm-112-disk-10,Type=blockio

as you can see, "disk-5" is missing since I added "disk-10"

Is anyone using zfs over iscsi with iet? have you seen this behaviour?

Thanks,
Mark

On 23 November 2016 at 20:40, Michael Rasmussen <m...@miras.org> wrote:

> On Wed, 23 Nov 2016 09:40:55 +
> Mark Adams <m...@openvs.co.uk> wrote:
>
> >
> > Has anyone else tried to get this or a similar setup working? Any views
> > greatly received.
> >
> What you are trying to achieve is not a good idea with
> corosync/pacemaker since iSCSI is a block device. To create a cluster
> over a LUN will require a cluster aware filesystem like NFS, CIFS etc.
> The proper way of doing this with iSCSI would be using multipath to a
> SAN since iSCSI LUNs cannot be shared. Unfortunately the current
> implementation of ZFS over iSCSI does not support multipath (a
> limitation in libiscsi). Also may I remind you that Iet development has
> stopped in favor of LIO targets (http://linux-iscsi.org/wiki/LIO). I am
> currently working on making an implementation of LIO for proxmox which
> will use a different architecture than the current ZFS over iSCSI
> implementation. The new implementation will support multipath. As this
> is developed in my spare time progress is not a high as it could be.
>
> Alternatively you could look at this:
> http://www.napp-it.org/doc/downloads/z-raid.pdf
>
> --
> Hilsen/Regards
> Michael Rasmussen
>
> Get my public GnuPG keys:
> michael  rasmussen  cc
> http://pgp.mit.edu:11371/pks/lookup?op=get=0xD3C9A00E
> mir  datanom  net
> http://pgp.mit.edu:11371/pks/lookup?op=get=0xE501F51C
> mir  miras  org
> http://pgp.mit.edu:11371/pks/lookup?op=get=0xE3E80917
> --
> /usr/games/fortune -es says:
> The computer should be doing the hard work.  That's what it's paid to
> do, after all.
> -- Larry Wall in <199709012312.qaa08...@wall.org>
>
> ___
> pve-user mailing list
> pve-user@pve.proxmox.com
> http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
>
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user

[PVE-User] ZFS on iSCSI + Pacemaker/corosync/DRBD

[Mark Adams]

Hi All,

I'm testing out proxmox and trying to get a working ZFS on iSCSI HA setup
going.

Because ZFS on iSCSI logs on to the iscsi server via ssh and creates a zfs
dataset then adds iscsi config to /etc/ietd.conf it works fine when you've
got a single iscsi host, but I haven't figured out a way to use it with
pacemaker/corosync resources.

I believe the correct configuration would be for the ZFS on iSCSI script to
create the pacemaker iSCSILogicalUnit resource using pcs, after creating
the zfs dataset, but this musn't be something that is supported as yet.

Has anyone else tried to get this or a similar setup working? Any views
greatly received.

Thanks,
Mark
___
pve-user mailing list
pve-user@pve.proxmox.com
http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user