Re: [PVE-User] network setup problems in 4.2
hi, On Fri, May 13, 2016 at 10:49:08AM +0200, Dietmar Maurer wrote: > > > i now tried to recreate it, but it new tells me that "tag" must have > > > a minimum value of 2, and if i try to edit the config by hand i get > > > and error on starting the vm: > > > unable to parse value of 'net1' - unable to parse network options > > Yes, we use tag 1 internally on the bridge, so you cannot use > that for your network. hm..that's unfortunate, it means that i have to completly redo my network setup (including the cluster networking...) is that documented somewhere? i did not see it in the release notes and it worked perfectly until 4.1...i did not expect such a change in i minor release. (and for the record, as long as i don't reboot the vms, it is still working) regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] network setup problems in 4.2
On Fri, May 13, 2016 at 09:37:58AM +0200, Wolfgang Link wrote: > Sorry forget what I wrote this is not relevant to your problem. but yes for the record, the setups uses openvswitch (i forgot to add it to the original mail) regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] network setup problems in 4.2
On Fri, May 13, 2016 at 09:14:55AM +0200, Wolfgang Link wrote: > Hi Albert, > > Do you have already installed openv-switch 2.5? > > If yes make a downgrade to 2.4. do i need to add additional repositories for newer openvswitch versions? i currently have openvswitch-switch: 2.3.2-3 from the enterprise repository. thx regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] network setup problems in 4.2
hi, since i'm having problems migrating already running VMs (see the other thread) i stopped 2 VMs and moved them to a new host with offline migration i ran into another problem: it seems the upgrade dropped the second network interface on all my VMs (the currently running ones have it until they are stopped and it works until then, but it's already missing in the config). i now tried to recreate it, but it new tells me that "tag" must have a minimum value of 2, and if i try to edit the config by hand i get and error on starting the vm: unable to parse value of 'net1' - unable to parse network options this unfortunatly breaks my network setup...does anybody know a workaround for it? thx regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online migration problems with pve 4.2
On Thu, May 12, 2016 at 01:00:58PM +0200, Eneko Lacunza wrote: > Hi, > > El 12/05/16 a las 12:38, Albert Dengg escribió: > > hi, > > On Thu, May 12, 2016 at 08:45:30AM +0200, Eneko Lacunza wrote: > > > Hi Albert, > > > > > > I just updated yesterday afternoon our cluster to PVE 4.2 non-subscrition > > > and migration works OK. Did so using your procedure. > > > > > > I found migration problems about a month ago, you can search in the > > > mailing > > > list the good advice I got from the developers. > > i think i know wich thread you mean, and they advised you to upgrade > > to > > qemu-server 4.0-59 > > pve-qemu-kvm 2.5-7 > > > > however, i'm already running > > qemu-server: 4.0-72 > > pve-qemu-kvm: 2.5-14 > Yes, notice I had to upgrade also on migration-source machines: qemu-server > 4.0-59 . What version do you have in the to-be-upgraded machines? i saw that (i stumbled upon the thread when i head the initial problem. as i wrote in my initial mail, i then also upgraded the other two noeds (carefully checking that i had disabled HA on all VMs as to not run into problems fencing a node during the upgrade by accident). here is the complete output for the two other noeds: [dengg@pve2:~]> pveversion -v proxmox-ve: 4.2-48 (running kernel: 4.2.8-1-pve) pve-manager: 4.2-2 (running version: 4.2-2/725d76f0) pve-kernel-2.6.32-37-pve: 2.6.32-150 pve-kernel-4.4.6-1-pve: 4.4.6-48 pve-kernel-4.2.6-1-pve: 4.2.6-36 pve-kernel-2.6.32-32-pve: 2.6.32-136 pve-kernel-2.6.32-43-pve: 2.6.32-166 pve-kernel-4.2.8-1-pve: 4.2.8-41 pve-kernel-2.6.32-39-pve: 2.6.32-157 pve-kernel-3.10.0-11-pve: 3.10.0-36 pve-kernel-2.6.32-34-pve: 2.6.32-140 pve-kernel-4.2.2-1-pve: 4.2.2-16 pve-kernel-2.6.32-40-pve: 2.6.32-160 pve-kernel-3.10.0-13-pve: 3.10.0-38 pve-kernel-2.6.32-41-pve: 2.6.32-164 pve-kernel-4.2.3-2-pve: 4.2.3-22 lvm2: 2.02.116-pve2 corosync-pve: 2.3.5-2 libqb0: 1.0-1 pve-cluster: 4.0-39 qemu-server: 4.0-72 pve-firmware: 1.1-8 libpve-common-perl: 4.0-59 libpve-access-control: 4.0-16 libpve-storage-perl: 4.0-50 pve-libspice-server1: 0.12.5-2 vncterm: 1.2-1 pve-qemu-kvm: 2.5-14 pve-container: 1.0-62 pve-firewall: 2.0-25 pve-ha-manager: 1.0-28 ksm-control-daemon: 1.2-1 glusterfs-client: 3.5.2-2+deb8u1 lxc-pve: 1.1.5-7 lxcfs: 2.0.0-pve2 cgmanager: 0.39-pve1 criu: 1.6.0-1 fence-agents-pve: 4.0.20-1 openvswitch-switch: 2.3.2-3 [dengg@pve3:~]> pveversion -v proxmox-ve: 4.2-48 (running kernel: 4.2.8-1-pve) pve-manager: 4.2-2 (running version: 4.2-2/725d76f0) pve-kernel-4.4.6-1-pve: 4.4.6-48 pve-kernel-4.2.6-1-pve: 4.2.6-36 pve-kernel-2.6.32-32-pve: 2.6.32-136 pve-kernel-2.6.32-43-pve: 2.6.32-166 pve-kernel-4.2.8-1-pve: 4.2.8-41 pve-kernel-4.2.2-1-pve: 4.2.2-16 pve-kernel-3.10.0-13-pve: 3.10.0-38 pve-kernel-4.2.3-2-pve: 4.2.3-22 lvm2: 2.02.116-pve2 corosync-pve: 2.3.5-2 libqb0: 1.0-1 pve-cluster: 4.0-39 qemu-server: 4.0-72 pve-firmware: 1.1-8 libpve-common-perl: 4.0-59 libpve-access-control: 4.0-16 libpve-storage-perl: 4.0-50 pve-libspice-server1: 0.12.5-2 vncterm: 1.2-1 pve-qemu-kvm: 2.5-14 pve-container: 1.0-62 pve-firewall: 2.0-25 pve-ha-manager: 1.0-28 ksm-control-daemon: 1.2-1 glusterfs-client: 3.5.2-2+deb8u1 lxc-pve: 1.1.5-7 lxcfs: 2.0.0-pve2 cgmanager: 0.39-pve1 criu: 1.6.0-1 fence-agents-pve: 4.0.20-1 openvswitch-switch: 2.3.2-3 please note that this was the upgrade from pve 4.1 to 4.2 yesterday, as i pushed it back a bit because of the reports about problems with virtio-net on this list... > > > as for storage: > > we are using ceph storage for the VMs > We too. (i did not yet install the ceph upgrad the seems to have been pushed to the mirror today, as i wanted to have the cluster in a working order before doing additional changes...) lg albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online migration problems with pve 4.2
hi, On Thu, May 12, 2016 at 08:45:30AM +0200, Eneko Lacunza wrote: > Hi Albert, > > I just updated yesterday afternoon our cluster to PVE 4.2 non-subscrition > and migration works OK. Did so using your procedure. > > I found migration problems about a month ago, you can search in the mailing > list the good advice I got from the developers. i think i know wich thread you mean, and they advised you to upgrade to qemu-server 4.0-59 pve-qemu-kvm 2.5-7 however, i'm already running qemu-server: 4.0-72 pve-qemu-kvm: 2.5-14 > > Can you attach the full migration task log? What storage are you using? here is the log from the latest try: task started by HA resource agent May 12 12:10:26 starting migration of VM 105 to node 'pve1' (172.23.22.1) May 12 12:10:26 copying disk images May 12 12:10:26 starting VM 105 on remote node 'pve1' May 12 12:10:28 starting ssh migration tunnel May 12 12:10:29 starting online/live migration on localhost:6 May 12 12:10:29 migrate_set_speed: 8589934592 May 12 12:10:29 migrate_set_downtime: 0.1 May 12 12:10:31 ERROR: online migrate failure - aborting May 12 12:10:31 aborting phase 2 - cleanup resources May 12 12:10:31 migrate_cancel May 12 12:10:33 ERROR: migration finished with problems (duration 00:00:07) TASK ERROR: migration problems i also saw the task starting the vm on the new node and starting the migration tunnel, which succeeded. as for storage: we are using ceph storage for the VMs just for the record: i reran some test just now: * offline migration works (i have a test vm that i could use for that * even though i got an error yesterday, it worked just now for a freshly booted VM i still would like to find a way that does not involve shutting down all VMs to get migration working again...the next possible time for that would be at some point end of next week most likly... > > Not really related to this, but you should remove 2.6.32 and 3.10 kernels, > they're not able to boot PVE 4/Debian jessie as far as I've tested. yes i know, i just saw it...however i really don't think non-running kernels are the problem here... > > Versions on our cluster: > # pveversion -v > proxmox-ve: 4.2-49 (running kernel: 4.4.8-1-pve) > pve-manager: 4.2-4 (running version: 4.2-4/2660193c) hm...slightly newer, however i think that's becaus you are running non-subscription.. thanks for all the pointers sofar... regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Online migration problems with pve 4.2
On Wed, May 11, 2016 at 01:13:10PM -0500, Gerald Brandt wrote: > Hi, > > Try from the command line: > > qm migrate --online unfortunatly this produces the same result as ha-manager migrate: it returns, but the migration does not happen. in the webinterface i then can see the failed job and the error message mentioned in my mail. regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] Online migration problems with pve 4.2
hi, i just upgrade a pve cluster to pve 4.2 (enterprise repo), but i have the problem that i cannot do any online migrations since the upgrade. pve versions (this node has alerady rebooted after the upgrade): [dengg@pve1:~]> pveversion -v proxmox-ve: 4.2-48 (running kernel: 4.4.6-1-pve) pve-manager: 4.2-2 (running version: 4.2-2/725d76f0) pve-kernel-2.6.32-37-pve: 2.6.32-150 pve-kernel-4.4.6-1-pve: 4.4.6-48 pve-kernel-4.2.6-1-pve: 4.2.6-36 pve-kernel-2.6.32-32-pve: 2.6.32-136 pve-kernel-2.6.32-43-pve: 2.6.32-166 pve-kernel-4.2.8-1-pve: 4.2.8-41 pve-kernel-2.6.32-39-pve: 2.6.32-157 pve-kernel-3.10.0-11-pve: 3.10.0-36 pve-kernel-2.6.32-34-pve: 2.6.32-140 pve-kernel-4.2.2-1-pve: 4.2.2-16 pve-kernel-2.6.32-40-pve: 2.6.32-160 pve-kernel-3.10.0-13-pve: 3.10.0-38 pve-kernel-2.6.32-41-pve: 2.6.32-164 pve-kernel-4.2.3-2-pve: 4.2.3-22 lvm2: 2.02.116-pve2 corosync-pve: 2.3.5-2 libqb0: 1.0-1 pve-cluster: 4.0-39 qemu-server: 4.0-72 pve-firmware: 1.1-8 libpve-common-perl: 4.0-59 libpve-access-control: 4.0-16 libpve-storage-perl: 4.0-50 pve-libspice-server1: 0.12.5-2 vncterm: 1.2-1 pve-qemu-kvm: 2.5-14 pve-container: 1.0-62 pve-firewall: 2.0-25 pve-ha-manager: 1.0-28 ksm-control-daemon: 1.2-1 glusterfs-client: 3.5.2-2+deb8u1 lxc-pve: 1.1.5-7 lxcfs: 2.0.0-pve2 cgmanager: 0.39-pve1 criu: 1.6.0-1 fence-agents-pve: 4.0.20-1 openvswitch-switch: 2.3.2-3 (yes, i should cleanup old kernel versions...) steps i took sofar: * migrated all vm's of one node * upgraded this node * rebooted it * tried to migrate vms back to this host to upgrade another one since there was a thread on the mailling list about migration problems that where fixed with newer qemu-server/pve-qemu-kvm version, i then upgraded the other two nodes (disabeling HA first). i still cannot online migrate, even a vm that has been booted on the fully upgraded node. unfortunatly the error message i get is not really informative: ERROR: online migration failure - aborting am i running into some known problem here or is this a new issue? thx regards, albert ps: rebooting the remaining nodes without migrating the VMs off them would be really inconvinient... signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] ceph.conf permissions
On Thu, Mar 10, 2016 at 10:49:06AM +, James Bailey wrote: ... > > You could use extended ACLs to allow the ceph user read access to that file > only. > > https://wiki.debian.org/Permissions#Access_Control_Lists_in_Linux have you tried it in this case?b i'm not shure if the filesystem used for /etc/pve (pmxcfs) supports posix acls... as with putting the user in the www-data group: at least on my machines the keyrings used for ceph are only readable by root. this means that most likely, you are going to run into the next problem there. regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] pve 4 cluster status problem
hi, i have a problem with one of my clusters the problem is the following: even though i migrated all HA managed VMs of the node, for some reason one of the nodes got fenced upon applying updates. after the node cam back up i applied all updates and rebooted again. now i have the problem that this node is not working correctly, here is the output of "ha-manager status": root@pve3:~# ha-manager status quorum OK master pve1 (old timestamp - dead?, Thu Mar 3 16:47:03 2016) lrm pve1 (active, Thu Mar 3 17:20:54 2016) lrm pve2 (active, Thu Mar 3 17:20:48 2016) lrm pve3 (wait_for_agent_lock, Thu Mar 3 17:20:53 2016) service vm:100 (pve2, started) service vm:101 (pve1, started) service vm:102 (pve1, started) service vm:103 (pve3, fence) service vm:104 (pve1, started) service vm:105 (pve3, fence) service vm:109 (pve3, fence) (vms 103, 105 & 109 are running fine on pve2 atm and did not have any downtime/restart) cluster status also looks ok: root@pve3:~# pvecm status Quorum information -- Date: Thu Mar 3 17:22:40 2016 Quorum provider: corosync_votequorum Nodes:3 Node ID: 0x0002 Ring ID: 2164 Quorate: Yes Votequorum information -- Expected votes: 3 Highest expected: 3 Total votes: 3 Quorum: 2 Flags:Quorate Membership information -- Nodeid Votes Name 0x0001 1 172.23.22.1 0x0003 1 172.23.22.2 0x0002 1 172.23.22.3 (local) the HA managment daemons are also running fine: root@pve3:~# systemctl |grep ha- pve-ha-crm.service loaded active running PVE Cluster Ressource Manager Daemon pve-ha-lrm.service loaded active running PVE Local HA Ressource Manager Daemon can anybody help me to find out whats going wrong here and how to best solve it? thanks in adavance regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Proxmox VE 4.1 released!
hi, On Fri, Dec 11, 2015 at 08:16:44PM +0100, Dietmar Maurer wrote: > > does somebody have an idea whats going wrong here or to be more > > exact, where to search next? > > Do you run a VM on that node? ah only one without HA having an HA managed VM on it made the difference. thanks for the quick reply, i must have missed that it does not fence if there are currently no HA VMs on it... regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] Proxmox VE 4.1 released!
hi, On Fri, Dec 11, 2015 at 12:24:16PM +0100, Martin Maurer wrote: > Hi all, > > we are proud to announce the final release of our Proxmox VE 4.1! :) i have a slight problem however and i'm wondering if anybody else has this problem: while i had fencing working fine using the iTCO watchdog in 4.0, in 4.1 i cannot get pve to fence a node. my setup is the following: 3x DELL R620 Servers, with full iDRAC running current pve from the enterprise repository when i try to force a node to fence itself by setting all links to down using this simple command for i in $(seq 0 5); do ip l s eth$i down; done to set all links to down, the node will not reboot itself even though pvecm status correctly displays that quorum has been lost i have tried: iTCO_wdt ipmi_watchdog both with and without nmi_watchdog=0 the only differnece is that if i use ipmi_watchdog, i get a message in the kernel ring buffer every 5 secondds like that: Dec 11 18:59:29 pve2 kernel: [ 361.067004] ipmi_si ipmi_si.0: Could not set the global enables: 0xcc. fencing was working before (in fact it seems all nodes where fencing themselves on the upgrade from 4.0 to 4.1...) does somebody have an idea whats going wrong here or to be more exact, where to search next? thx regards, albert signature.asc Description: PGP signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] timezone problem
hi, i have a small problem concering timezones: we have a windows (server 2008r2) VM running, that needs to be running with UTC as its current timezone. since windows insits on the RTC running on local time, i changed Use local time for RTC to no and RTC start date to now, however the machine still gets the wrong time (the clock is set to CEST, so there is an error of 2hrs). the question is now, have i missed something? thanks regards, albert dengg signature.asc Description: Digital signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] windows performance problem
hi, we have 2 windows VMs (windows server 2008r2), that show some performance problems. we traced it down to calls to QueryPerformanceCounter() to get timing information. we tried enaling hv_relaxed,hv_spinlocks, hv_apic and hv_time in an test, however that did not really help we still have significant slowdown for every call to the performance counters (for some artifical tests we got to up 12 times the runtime compared to relativly similar physical hardware). does anybody here now about any solution to this problem? regards, albert dengg signature.asc Description: Digital signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] kernel sources
hi, it's probably a dumb question, but: where do i find the soruces for the pve-kernel-2.6.32-39-pve package? if i try to just add deb-src https://enterprise.proxmox.com/debian wheezy pve-enterprise i get a file not found error... my current reson for that: we are currently chasing performance issues and i wanted to compile the perf utility for the kernel, since there seems to be no prebuilt package for the pve kernel version... thx regards, albert signature.asc Description: Digital signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
Re: [PVE-User] kernel sources
hi, On Mon, Jun 29, 2015 at 06:48:11PM +0300, Lex Rivera wrote: https://git.proxmox.com/ ? ah thanks for the pointer... i guess i'm to used to work with standard debian...so looking for debian source packages was my first idea :) regards, albert signature.asc Description: Digital signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
[PVE-User] openvswitch and proxmox with sflow
hi, i'm currently in the process of building a 3 node cluster for production use and have selected openvswitch for the networking (setup more or less as in [0]) since i'm still in the process of testing fine tuning i tried to enable sending sflow data from the openvswitch to an vm running ntop to get a few statistics and that lead to the situation that i get frequent kernel panics and the message about fatal exception in interrupt on the console. is this a known problem (configuration mistake on my side) or is it a bug (i did not find a bugreport sofar)? thx regards, albert dengg [0] http://pve.proxmox.com/wiki/Open_vSwitch signature.asc Description: Digital signature ___ pve-user mailing list pve-user@pve.proxmox.com http://pve.proxmox.com/cgi-bin/mailman/listinfo/pve-user
