Re: [ovirt-users] Unable to reinstall hosts after network removal.
Ok I found the conflict. The ip rule list does not have any record on IB0 interface, all other networks are updated correctly but not IB one. Probably this is not a good idea to have an ISCSI storage devices with the same interface as VM network. the network update operation trying to put interface up and down in that point all iscsi devices going to IO/failure this stops to update IB interface data. Maybe one need to put whole cluster to maintenance mode in order to change Storage dependent network setup? this is the main trouble which prevents to fix/change any network setup on IB0 interface. tail -f /var/log/vdsm/vdsm.log Traceback (most recent call last): File /usr/share/vdsm/storage/hsm.py, line 2430, in connectStorageServer conObj.connect() File /usr/share/vdsm/storage/storageServer.py, line 367, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File /usr/share/vdsm/storage/iscsi.py, line 164, in addIscsiNode iscsiadm.node_login(iface.name, portalStr, target.iqn) File /usr/share/vdsm/storage/iscsiadm.py, line 295, in node_login raise IscsiNodeError(rc, out, err) IscsiNodeError: (8, ['Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.data01.x8664:sn.f5c9b017e103, portal: 10.10.10.31,3260] (multiple)'], ['iscsiadm: Could not login to [iface: default, target: iqn.2003-01.org.linux-iscsi.data01.x8664:sn.f5c9b017e103, portal: 10.10.10.31,3260].', 'iscsiadm: initiator reported error (8 - connection timed out)', 'iscsiadm: Could not log into all portals']) PS save network configuration is always checked. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** On Tue, Dec 30, 2014 at 7:05 AM, Moti Asayag masa...@redhat.com wrote: - Original Message - From: Arman Khalatyan arm2...@gmail.com To: Moti Asayag masa...@redhat.com Cc: users users@ovirt.org Sent: Monday, December 29, 2014 3:59:46 PM Subject: Re: [ovirt-users] Unable to reinstall hosts after network removal. My setup has 3 networks: 1xIB,1x10Gbit+1Gbit for ovirt-management. The ovirt network does not have any trouble, it is always there. I was trying to rename or remove my IB network which was used for VM migrations. I was using web-GUI, which was removing IB0 network w/o problem. After removal the hosts where ok. Have you checked the option of save network configuration in the setup networks dialog (iirc should have been checked by default) ? Could you also attach the /var/log/ovirt-engine/engine.log from the engine server and /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log from the node so we can see which request was sent to vdsm and its result ? Then I put them to maintenance mode. To refresh iptables rules I did reinstall. then reinstall was failing with message that IB0 not attached to any interface. But IB0 interface is not possible to attach it is already deleted and not visible in any network dialog. After creating interface with the same name everything is online now. My current interface list is following: virsh -r net-list Name State Autostart Persistent -- ;vdsmdummy; active nono vdsm-cls10G active yes yes vdsm-IB0 active yes yes vdsm-ovirtmgmt active yes yes On this host the IB0 is not atached to any interface, I wondered, if it should show up in the net list? I think the GUI does not rename/remove the interface from the DB. Some constrain keeps still IB0 in DB. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** On Sun, Dec 28, 2014 at 10:31 AM, Moti Asayag masa...@redhat.com wrote: - Original Message - From: Arman Khalatyan arm2...@gmail.com To: users users@ovirt.org Sent: Wednesday, December 24, 2014 1:22:43 PM Subject: [ovirt-users] Unable to reinstall hosts after network removal. Hello, I have a little trouble with ovirt 3.5 on CentOS6.6: I was removing all networks from all hosts. Did you use the setup networks dialog from the UI in order to remove those networks ? Or have you removed those networks from the host directly (where you should used the: 1. virsh net-destroy 'the-network-name' 2. virsh net-undefine 'the-network-name' ) can you report the output of 'virsh -r net-list' ? Then after removing network from data center the hosts went to unusable. What was the host's status
Re: [ovirt-users] Unable to reinstall hosts after network removal.
- Original Message - From: Arman Khalatyan arm2...@gmail.com To: Moti Asayag masa...@redhat.com Cc: users users@ovirt.org Sent: Tuesday, December 30, 2014 2:21:26 PM Subject: Re: [ovirt-users] Unable to reinstall hosts after network removal. Ok I found the conflict. The ip rule list does not have any record on IB0 interface, all other networks are updated correctly but not IB one. Probably this is not a good idea to have an ISCSI storage devices with the same interface as VM network. the network update operation trying to put interface up and down in that point all iscsi devices going to IO/failure this stops to update IB interface data. Maybe one need to put whole cluster to maintenance mode in order to change Storage dependent network setup? Adding Dan and Allon. this is the main trouble which prevents to fix/change any network setup on IB0 interface. tail -f /var/log/vdsm/vdsm.log Traceback (most recent call last): File /usr/share/vdsm/storage/hsm.py, line 2430, in connectStorageServer conObj.connect() File /usr/share/vdsm/storage/storageServer.py, line 367, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File /usr/share/vdsm/storage/iscsi.py, line 164, in addIscsiNode iscsiadm.node_login(iface.name, portalStr, target.iqn) File /usr/share/vdsm/storage/iscsiadm.py, line 295, in node_login raise IscsiNodeError(rc, out, err) IscsiNodeError: (8, ['Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.data01.x8664:sn.f5c9b017e103, portal: 10.10.10.31,3260] (multiple)'], ['iscsiadm: Could not login to [iface: default, target: iqn.2003-01.org.linux-iscsi.data01.x8664:sn.f5c9b017e103, portal: 10.10.10.31,3260].', 'iscsiadm: initiator reported error (8 - connection timed out)', 'iscsiadm: Could not log into all portals']) PS save network configuration is always checked. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** On Tue, Dec 30, 2014 at 7:05 AM, Moti Asayag masa...@redhat.com wrote: - Original Message - From: Arman Khalatyan arm2...@gmail.com To: Moti Asayag masa...@redhat.com Cc: users users@ovirt.org Sent: Monday, December 29, 2014 3:59:46 PM Subject: Re: [ovirt-users] Unable to reinstall hosts after network removal. My setup has 3 networks: 1xIB,1x10Gbit+1Gbit for ovirt-management. The ovirt network does not have any trouble, it is always there. I was trying to rename or remove my IB network which was used for VM migrations. I was using web-GUI, which was removing IB0 network w/o problem. After removal the hosts where ok. Have you checked the option of save network configuration in the setup networks dialog (iirc should have been checked by default) ? Could you also attach the /var/log/ovirt-engine/engine.log from the engine server and /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log from the node so we can see which request was sent to vdsm and its result ? Then I put them to maintenance mode. To refresh iptables rules I did reinstall. then reinstall was failing with message that IB0 not attached to any interface. But IB0 interface is not possible to attach it is already deleted and not visible in any network dialog. After creating interface with the same name everything is online now. My current interface list is following: virsh -r net-list Name State Autostart Persistent -- ;vdsmdummy; active nono vdsm-cls10G active yes yes vdsm-IB0 active yes yes vdsm-ovirtmgmt active yes yes On this host the IB0 is not atached to any interface, I wondered, if it should show up in the net list? I think the GUI does not rename/remove the interface from the DB. Some constrain keeps still IB0 in DB. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** On Sun, Dec 28, 2014 at 10:31 AM, Moti Asayag masa...@redhat.com wrote: - Original Message - From: Arman Khalatyan arm2...@gmail.com To: users users@ovirt.org Sent: Wednesday, December 24, 2014 1:22:43 PM Subject: [ovirt-users] Unable to reinstall hosts after network removal. Hello, I have a little trouble with ovirt 3.5 on CentOS6.6: I was removing all networks from all hosts. Did you use the setup networks dialog from
Re: [ovirt-users] Unable to reinstall hosts after network removal.
My setup has 3 networks: 1xIB,1x10Gbit+1Gbit for ovirt-management. The ovirt network does not have any trouble, it is always there. I was trying to rename or remove my IB network which was used for VM migrations. I was using web-GUI, which was removing IB0 network w/o problem. After removal the hosts where ok. Then I put them to maintenance mode. To refresh iptables rules I did reinstall. then reinstall was failing with message that IB0 not attached to any interface. But IB0 interface is not possible to attach it is already deleted and not visible in any network dialog. After creating interface with the same name everything is online now. My current interface list is following: virsh -r net-list Name State Autostart Persistent -- ;vdsmdummy; active nono vdsm-cls10G active yes yes vdsm-IB0 active yes yes vdsm-ovirtmgmt active yes yes On this host the IB0 is not atached to any interface, I wondered, if it should show up in the net list? I think the GUI does not rename/remove the interface from the DB. Some constrain keeps still IB0 in DB. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** On Sun, Dec 28, 2014 at 10:31 AM, Moti Asayag masa...@redhat.com wrote: - Original Message - From: Arman Khalatyan arm2...@gmail.com To: users users@ovirt.org Sent: Wednesday, December 24, 2014 1:22:43 PM Subject: [ovirt-users] Unable to reinstall hosts after network removal. Hello, I have a little trouble with ovirt 3.5 on CentOS6.6: I was removing all networks from all hosts. Did you use the setup networks dialog from the UI in order to remove those networks ? Or have you removed those networks from the host directly (where you should used the: 1. virsh net-destroy 'the-network-name' 2. virsh net-undefine 'the-network-name' ) can you report the output of 'virsh -r net-list' ? Then after removing network from data center the hosts went to unusable. What was the host's status prior to removing its networks ? Was it up ? Every time after reinstall the host claims that the network is not configured, but it s already removed from network tab in DC. What is the missing network name ? Is it 'ovirtmgmt' ? Where from it gets the old configuration? the old interfaces also restored every time on the reinstalled hosts. The hosts via vdsm reports their network configuration via the 'getCapabilities' verb of vdsm. You can try running it on the host: vdsClient -s 0 getVdsCaps and examine the nics / neworks / bridges / vlans / bonds elements. Which DB table is in charge of dc-networks? The retrieved information from vdsm is reported to 'vds_interace' table. The dc networks are stored in 'networks' table and networks attached to clusters are stored in network_cluster table. I wouldn't recommend on deleting entries from the tables directly. There are certain constraints which shouldn't be violated, i.e. the management network 'ovirtmgnt' is blocked for removal from the engine. Thanks, Arman. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Unable to reinstall hosts after network removal.
I think pressing the Refresh Capabilities button at the hosts main tab might help after you remove the IB interface - this forces the engine to update its DB state to what's reported by the hypervisor. Then I would expect things to work better... On 29/12/14 15:59, Arman Khalatyan wrote: My setup has 3 networks: 1xIB,1x10Gbit+1Gbit for ovirt-management. The ovirt network does not have any trouble, it is always there. I was trying to rename or remove my IB network which was used for VM migrations. I was using web-GUI, which was removing IB0 network w/o problem. After removal the hosts where ok. Then I put them to maintenance mode. To refresh iptables rules I did reinstall. then reinstall was failing with message that IB0 not attached to any interface. But IB0 interface is not possible to attach it is already deleted and not visible in any network dialog. After creating interface with the same name everything is online now. My current interface list is following: virsh -r net-list Name State Autostart Persistent -- ;vdsmdummy; active nono vdsm-cls10G active yes yes vdsm-IB0 active yes yes vdsm-ovirtmgmt active yes yes On this host the IB0 is not atached to any interface, I wondered, if it should show up in the net list? I think the GUI does not rename/remove the interface from the DB. Some constrain keeps still IB0 in DB. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** On Sun, Dec 28, 2014 at 10:31 AM, Moti Asayag masa...@redhat.com mailto:masa...@redhat.com wrote: - Original Message - From: Arman Khalatyan arm2...@gmail.com mailto:arm2...@gmail.com To: users users@ovirt.org mailto:users@ovirt.org Sent: Wednesday, December 24, 2014 1:22:43 PM Subject: [ovirt-users] Unable to reinstall hosts after network removal. Hello, I have a little trouble with ovirt 3.5 on CentOS6.6: I was removing all networks from all hosts. Did you use the setup networks dialog from the UI in order to remove those networks ? Or have you removed those networks from the host directly (where you should used the: 1. virsh net-destroy 'the-network-name' 2. virsh net-undefine 'the-network-name' ) can you report the output of 'virsh -r net-list' ? Then after removing network from data center the hosts went to unusable. What was the host's status prior to removing its networks ? Was it up ? Every time after reinstall the host claims that the network is not configured, but it s already removed from network tab in DC. What is the missing network name ? Is it 'ovirtmgmt' ? Where from it gets the old configuration? the old interfaces also restored every time on the reinstalled hosts. The hosts via vdsm reports their network configuration via the 'getCapabilities' verb of vdsm. You can try running it on the host: vdsClient -s 0 getVdsCaps and examine the nics / neworks / bridges / vlans / bonds elements. Which DB table is in charge of dc-networks? The retrieved information from vdsm is reported to 'vds_interace' table. The dc networks are stored in 'networks' table and networks attached to clusters are stored in network_cluster table. I wouldn't recommend on deleting entries from the tables directly. There are certain constraints which shouldn't be violated, i.e. the management network 'ovirtmgnt' is blocked for removal from the engine. Thanks, Arman. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** ___ Users mailing list Users@ovirt.org mailto:Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Unable to reinstall hosts after network removal.
- Original Message - From: Arman Khalatyan arm2...@gmail.com To: Moti Asayag masa...@redhat.com Cc: users users@ovirt.org Sent: Monday, December 29, 2014 3:59:46 PM Subject: Re: [ovirt-users] Unable to reinstall hosts after network removal. My setup has 3 networks: 1xIB,1x10Gbit+1Gbit for ovirt-management. The ovirt network does not have any trouble, it is always there. I was trying to rename or remove my IB network which was used for VM migrations. I was using web-GUI, which was removing IB0 network w/o problem. After removal the hosts where ok. Have you checked the option of save network configuration in the setup networks dialog (iirc should have been checked by default) ? Could you also attach the /var/log/ovirt-engine/engine.log from the engine server and /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log from the node so we can see which request was sent to vdsm and its result ? Then I put them to maintenance mode. To refresh iptables rules I did reinstall. then reinstall was failing with message that IB0 not attached to any interface. But IB0 interface is not possible to attach it is already deleted and not visible in any network dialog. After creating interface with the same name everything is online now. My current interface list is following: virsh -r net-list Name State Autostart Persistent -- ;vdsmdummy; active nono vdsm-cls10G active yes yes vdsm-IB0 active yes yes vdsm-ovirtmgmt active yes yes On this host the IB0 is not atached to any interface, I wondered, if it should show up in the net list? I think the GUI does not rename/remove the interface from the DB. Some constrain keeps still IB0 in DB. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** On Sun, Dec 28, 2014 at 10:31 AM, Moti Asayag masa...@redhat.com wrote: - Original Message - From: Arman Khalatyan arm2...@gmail.com To: users users@ovirt.org Sent: Wednesday, December 24, 2014 1:22:43 PM Subject: [ovirt-users] Unable to reinstall hosts after network removal. Hello, I have a little trouble with ovirt 3.5 on CentOS6.6: I was removing all networks from all hosts. Did you use the setup networks dialog from the UI in order to remove those networks ? Or have you removed those networks from the host directly (where you should used the: 1. virsh net-destroy 'the-network-name' 2. virsh net-undefine 'the-network-name' ) can you report the output of 'virsh -r net-list' ? Then after removing network from data center the hosts went to unusable. What was the host's status prior to removing its networks ? Was it up ? Every time after reinstall the host claims that the network is not configured, but it s already removed from network tab in DC. What is the missing network name ? Is it 'ovirtmgmt' ? Where from it gets the old configuration? the old interfaces also restored every time on the reinstalled hosts. The hosts via vdsm reports their network configuration via the 'getCapabilities' verb of vdsm. You can try running it on the host: vdsClient -s 0 getVdsCaps and examine the nics / neworks / bridges / vlans / bonds elements. Which DB table is in charge of dc-networks? The retrieved information from vdsm is reported to 'vds_interace' table. The dc networks are stored in 'networks' table and networks attached to clusters are stored in network_cluster table. I wouldn't recommend on deleting entries from the tables directly. There are certain constraints which shouldn't be violated, i.e. the management network 'ovirtmgnt' is blocked for removal from the engine. Thanks, Arman. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Unable to reinstall hosts after network removal.
- Original Message - From: Arman Khalatyan arm2...@gmail.com To: users users@ovirt.org Sent: Wednesday, December 24, 2014 1:22:43 PM Subject: [ovirt-users] Unable to reinstall hosts after network removal. Hello, I have a little trouble with ovirt 3.5 on CentOS6.6: I was removing all networks from all hosts. Did you use the setup networks dialog from the UI in order to remove those networks ? Or have you removed those networks from the host directly (where you should used the: 1. virsh net-destroy 'the-network-name' 2. virsh net-undefine 'the-network-name' ) can you report the output of 'virsh -r net-list' ? Then after removing network from data center the hosts went to unusable. What was the host's status prior to removing its networks ? Was it up ? Every time after reinstall the host claims that the network is not configured, but it s already removed from network tab in DC. What is the missing network name ? Is it 'ovirtmgmt' ? Where from it gets the old configuration? the old interfaces also restored every time on the reinstalled hosts. The hosts via vdsm reports their network configuration via the 'getCapabilities' verb of vdsm. You can try running it on the host: vdsClient -s 0 getVdsCaps and examine the nics / neworks / bridges / vlans / bonds elements. Which DB table is in charge of dc-networks? The retrieved information from vdsm is reported to 'vds_interace' table. The dc networks are stored in 'networks' table and networks attached to clusters are stored in network_cluster table. I wouldn't recommend on deleting entries from the tables directly. There are certain constraints which shouldn't be violated, i.e. the management network 'ovirtmgnt' is blocked for removal from the engine. Thanks, Arman. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Unable to reinstall hosts after network removal.
Hello, I have a little trouble with ovirt 3.5 on CentOS6.6: I was removing all networks from all hosts. Then after removing network from data center the hosts went to unusable. Every time after reinstall the host claims that the network is not configured, but it s already removed from network tab in DC. Where from it gets the old configuration? the old interfaces also restored every time on the reinstalled hosts. Which DB table is in charge of dc-networks? Thanks, Arman. *** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *** ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
