Hi guys,
I'm experiencing a big issue with networking.
First of all, we're running CS 4.11.2 on XS 6.5 with Advanced Networking.
Our XS Pool network conf was:
- eth0 Management
- eth1 empty
- eth2 Public
- eth3 empty
- eth4 + eth5: bond LACP for GuestVM
During our last maintenance last week, we decided to create a new bond
active-passive for Management (eth0 + eth1) and a new bond for Public
(eth2 + eth3)
In addition, we would change GuestVM bond from LACP to active-active.
So, we created from pool master a new bond and put eth0 + eth1 interface in.
MGT_NET_UUID=$(xe network-create name-label=Management)
PMI_PIF_UUID=$(xe pif-list host-uuid=xxx management=true params=uuid | awk
'{ print $5 }')
MGT_PIF0_UUID=$(xe pif-list host-uuid=xxx device=eth0 params=uuid | awk '{
print $5 }')
MGT_PIF1_UUID=$(xe pif-list host-uuid=xxx device=eth1 params=uuid | awk '{
print $5 }')
xe bond-create network-uuid=$MGT_NET_UUID
pif-uuids=$PMI_PIF_UUID,$MGT_PIF1_UUID mode=active-backup
I used the same method to create new bond for Public network (obviously
changing nics).
To change bond mode for GuestVM network I've used:
xe pif-list device=bond0 VLAN=-1
xe pif-param-set uuid=<Bond0 UUID> other-config:bond-mode=balance-slb
I've repeated Public and GuestVM commands on each of the three hosts in the
pool, for Management I've done it only on Pool Master.
After that I've restarted toolstack and (after the issue I'll explain) also
reboot every host.
However, this is the result:
- Existing VMs runs fine and I can stop, start, migrate
- New VM that require new network will fail
- restart network with clean will fail and makes network and instance
unavailable
This is the CS log:
2019-06-25 00:14:35,751 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Created VM
dbb045ef-5072-96ef-fbb0-1d7e3af0a0ea for r-899-VM
2019-06-25 00:14:35,756 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) PV args are
%template=domP%name=r-899-VM%eth2ip=31.44.38.53%eth2mask=255.255.255.240%gateway=31.44.38.49%eth0ip=10.122.12.1%eth0mask=255.255.255.0%domain=
tet.com
%cidrsize=24%dhcprange=10.122.12.1%eth1ip=169.254.3.241%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=8.8.8.8%dns2=8.8.4.4
2019-06-25 00:14:35,757 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) HVM args are template=domP
name=r-899-VM eth2ip=31.44.38.53 eth2mask=255.255.255.240
gateway=31.44.38.49 eth0ip=10.122.12.1 eth0mask=255.255.255.0 domain=tet.com
cidrsize=24 dhcprange=10.122.12.1 eth1ip=169.254.3.241 eth1mask=255.255.0.0
type=router disable_rp_filter=true dns1=8.8.8.8 dns2=8.8.4.4
2019-06-25 00:14:35,790 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) VBD
ec5e1e54-5902-cbcb-a6a8-abec0479d27c created for
com.cloud.agent.api.to.DiskTO@36117847
2019-06-25 00:14:35,790 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Creating VIF for r-899-VM on
nic [Nic:Public-31.44.38.53-vlan://untagged]
2019-06-25 00:14:35,792 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Looking for network named
Public
2019-06-25 00:14:35,793 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Found more than one network
with the name Public
2019-06-25 00:14:35,802 DEBUG [c.c.h.x.r.XsLocalNetwork]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Found a network called
Public on host=192.168.200.39;
Network=9fa48b75-d68e-feaf-2eb4-8a7340f8c89b;
pif=ca4c1679-fa36-bc93-37de-28a74ddc4f2c
2019-06-25 00:14:35,807 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Created a vif
dfaab3d7-7921-e4d5-ba27-537e8d549a5c on 2
2019-06-25 00:14:35,807 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Creating VIF for r-899-VM on
nic [Nic:Guest-10.122.12.1-vlan://384]
2019-06-25 00:14:35,809 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Looking for network named
GuestVM
2019-06-25 00:14:35,825 DEBUG [c.c.h.x.r.XsLocalNetwork]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Found a network called
GuestVM on host=192.168.200.39;
Network=300e55f0-88ff-a460-e498-e75424bc292a;
pif=b67841c5-6361-0dbf-a63d-a3e9c1b9f2fc
2019-06-25 00:14:35,826 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Creating VLAN 384 on host
192.168.200.39 on device bond0
2019-06-25 00:14:36,467 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) VLAN is created for 384.
The uuid is e34dc684-8a87-7ef6-5a49-8214011f8c3c
2019-06-25 00:14:36,480 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Created a vif
94558211-6bc6-ce64-9535-6d424b2b072c on 0
2019-06-25 00:14:36,481 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Creating VIF for r-899-VM on
nic [Nic:Control-169.254.3.241-null]
2019-06-25 00:14:36,531 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) already have a vif on dom0
for link local network
2019-06-25 00:14:36,675 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Created a vif
1ee58631-8524-ffa8-7ef7-e0acde48449f on 1
2019-06-25 00:14:37,688 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-35:ctx-c5073156) (logid:0d9e7907) Task failed! Task record:
uuid: 3177dcfc-ade9-3079-00f3-191efa1f90b2
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Tue Jun 25 00:14:42 CEST 2019
finished: Tue Jun 25 00:14:42 CEST 2019
status: failure
residentOn: com.xensource.xenapi.Host@85c62ee8
progress: 1.0
type: <none/>
result:
errorInfo: [HOST_CANNOT_ATTACH_NETWORK,
OpaqueRef:a323ff49-04bb-3e0f-50e3-b1bc7ef31630,
OpaqueRef:f9e9b47a-28c3-5031-a8fb-5e103c970a8a]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
And this is the xensource.log
INET 0.0.0.0:80|VBD.create R:47cd5359e955|audit] VBD.create: VM =
'dc9b4f1a-960c-8f02-a34c-32ad20e053f6 (r-899-VM)'; VDI =
'f80241ff-2a00-4565-bfa4-a980f1462f3e'
INET 0.0.0.0:80|VBD.create R:47cd5359e955|xapi] Checking whether there's a
migrate in progress...
INET 0.0.0.0:80|VBD.create R:47cd5359e955|xapi] VBD.create (device = 0;
uuid = f2b9e26f-66d2-d939-e2a1-04af3ae41ac9; ref =
OpaqueRef:48deb7cb-2d3b-8370-107a-68ba25213c3e)
INET 0.0.0.0:80|VIF.create R:ac85e8d39cd8|audit] VIF.create: VM =
'dc9b4f1a-960c-8f02-a34c-32ad20e053f6 (r-899-VM)'; network =
'9fa48b75-d68e-feaf-2eb4-8a7340f8c89b'
INET 0.0.0.0:80|VIF.create R:ac85e8d39cd8|xapi] VIF.create running
INET 0.0.0.0:80|VIF.create R:ac85e8d39cd8|xapi] Found mac_seed on VM:
supplied MAC parameter = '1e:00:19:00:00:4e'
INET 0.0.0.0:80|VIF.create R:ac85e8d39cd8|xapi] VIF
ref='OpaqueRef:96cfcfaa-cf0f-e8a2-6ff5-4029815999b1' created (VM =
'OpaqueRef:d3de28ef-ef26-02f7-46aa-ec01fe5c035e'; MAC address =
'1e:00:19:00:00:4e')
INET 0.0.0.0:80|VLAN.create R:6046e28d22a8|audit] VLAN.create: network =
'2ce5f897-376f-4562-46c5-6f161106584b'; VLAN tag = 363
INET 0.0.0.0:80|VLAN.create R:6046e28d22a8|xapi] Session.create
trackid=70355f146f1dd9d4873b9fdde8d1b8eb pool=true uname= originator=
is_local_superuser=true auth_user_sid=
parent=trackid=ab7ab58a3d7585f75b89abdab8725787
INET 0.0.0.0:80|VLAN.create R:6046e28d22a8|mscgen] xapi=>xapi
[label="(XML)"];
UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid
D:9eca0ccc2d01 created by task R:6046e28d22a8
INET 0.0.0.0:80|VLAN.create R:6046e28d22a8|mscgen] xapi=>dst_xapi
[label="(XML)"];
INET 0.0.0.0:80|VLAN.create R:6046e28d22a8|xmlrpc_client] stunnel pid:
10191 (cached = true) connected to 192.168.200.36:443
INET 0.0.0.0:80|VLAN.create R:6046e28d22a8|xmlrpc_client]
with_recorded_stunnelpid
task_opt=OpaqueRef:6046e28d-22a8-72de-9c3e-b08b87bb1ca6 s_pid=10191
INET 0.0.0.0:80|VLAN.create R:6046e28d22a8|xmlrpc_client] stunnel pid:
10191 (cached = true) returned stunnel to cache
INET 0.0.0.0:80|local logout in message forwarder D:457fa1a34193|xapi]
Session.destroy trackid=70355f146f1dd9d4873b9fdde8d1b8eb
INET 0.0.0.0:80|VLAN.create R:6046e28d22a8|taskhelper] the status of
R:6046e28d22a8 is: success; cannot set it to `success
INET 0.0.0.0:80|VIF.create R:320e6896912a|audit] VIF.create: VM =
'dc9b4f1a-960c-8f02-a34c-32ad20e053f6 (r-899-VM)'; network =
'2ce5f897-376f-4562-46c5-6f161106584b'
INET 0.0.0.0:80|VIF.create R:320e6896912a|xapi] VIF.create running
INET 0.0.0.0:80|VIF.create R:320e6896912a|xapi] Found mac_seed on VM:
supplied MAC parameter = '02:00:19:9e:00:02'
INET 0.0.0.0:80|VIF.create R:320e6896912a|xapi] VIF
ref='OpaqueRef:d4db98cf-6c5c-bb9f-999a-291c33e5c0cd' created (VM =
'OpaqueRef:d3de28ef-ef26-02f7-46aa-ec01fe5c035e'; MAC address =
'02:00:19:9e:00:02')
INET 0.0.0.0:80|host.call_plugin R:620d4a6a391b|audit] Host.call_plugin
host = '18d712b3-f64a-4178-9404-144f4f8fce2f (LIONARCH)'; plugin = 'vmops';
fn = 'setLinkLocalIP'; args = [ brName: xapi4 ]
INET 0.0.0.0:80|host.call_plugin R:620d4a6a391b|xapi] Session.create
trackid=f3e9f44a74e7dad26498e75c6de5eeba pool=true uname= originator=
is_local_superuser=true auth_user_sid=
parent=trackid=ab7ab58a3d7585f75b89abdab8725787
INET 0.0.0.0:80|host.call_plugin R:620d4a6a391b|mscgen] xapi=>xapi
[label="(XML)"];
UNIX /var/xapi/xapi||dummytaskhelper] task dispatch:session.get_uuid
D:30eddea1c3fd created by task R:620d4a6a391b
INET 0.0.0.0:80|host.call_plugin R:620d4a6a391b|mscgen] xapi=>dst_xapi
[label="(XML)"];
INET 0.0.0.0:80|host.call_plugin R:620d4a6a391b|xmlrpc_client] stunnel
pid: 10224 (cached = true) connected to 192.168.200.36:443
INET 0.0.0.0:80|host.call_plugin R:620d4a6a391b|xmlrpc_client]
with_recorded_stunnelpid
task_opt=OpaqueRef:620d4a6a-391b-e414-e4f3-40ea69e89d7a s_pid=10224
INET 0.0.0.0:80|host.call_plugin R:620d4a6a391b|xmlrpc_client] stunnel
pid: 10224 (cached = true) returned stunnel to cache
INET 0.0.0.0:80|local logout in message forwarder D:f89f813a5123|xapi]
Session.destroy trackid=f3e9f44a74e7dad26498e75c6de5eeba
INET 0.0.0.0:80|host.call_plugin R:620d4a6a391b|taskhelper] the status of
R:620d4a6a391b is: success; cannot set it to `success
INET 0.0.0.0:80|VIF.create R:1685ff9e765e|audit] VIF.create: VM =
'dc9b4f1a-960c-8f02-a34c-32ad20e053f6 (r-899-VM)'; network =
'e667320f-0e48-4cc6-6329-d723846cf8be'
INET 0.0.0.0:80|VIF.create R:1685ff9e765e|xapi] VIF.create running
INET 0.0.0.0:80|VIF.create R:1685ff9e765e|xapi] Found mac_seed on VM:
supplied MAC parameter = '0e:00:a9:fe:01:73'
INET 0.0.0.0:80|VIF.create R:1685ff9e765e|xapi] VIF
ref='OpaqueRef:36a3163e-2a8e-fda1-7e13-0d6802b8ba12' created (VM =
'OpaqueRef:d3de28ef-ef26-02f7-46aa-ec01fe5c035e'; MAC address =
'0e:00:a9:fe:01:73')
|Async.VM.start_on R:96fa354a6a0d|dispatcher] spawning a new thread to
handle the current task (trackid=ab7ab58a3d7585f75b89abdab8725787)
|Async.VM.start_on R:96fa354a6a0d|audit] VM.start_on: VM =
'dc9b4f1a-960c-8f02-a34c-32ad20e053f6 (r-899-VM)'; host
'18d712b3-f64a-4178-9404-144f4f8fce2f (LIONARCH)'
|Async.VM.start_on R:96fa354a6a0d|xapi] No operations are valid because
current-operations = [ OpaqueRef:96fa354a-6a0d-da55-f1d4-15eaa51d6640 ->
attach ]
es
|Async.VM.start_on R:96fa354a6a0d|xapi] The VM's BIOS strings were not yet
filled in. The VM is now made BIOS-generic.
|Async.VM.start_on R:96fa354a6a0d|xapi] Checking whether VM
OpaqueRef:d3de28ef-ef26-02f7-46aa-ec01fe5c035e can run on host
OpaqueRef:f0e56fe4-004c-31af-4767-531e22622c1e
|Async.VM.start_on R:96fa354a6a0d|backtrace] Raised at
xapi_network_attach_helpers.ml:50.8-90 -> list.ml:69.12-15 ->
xapi_vm_helpers.ml:379.4-111
|Async.VM.start_on R:96fa354a6a0d|xapi] Caught exception while checking if
network OpaqueRef:f9e9b47a-28c3-5031-a8fb-5e103c970a8a could be attached on
host OpaqueRef:f0e56fe4-004c-31af-4767-531e22622c1e:CANNOT_PLUG_BOND_SLAVE:
[ OpaqueRef:b9238dea-802c-d8da-a7b7-ee5bc642a615 ]
|Async.VM.start_on R:96fa354a6a0d|xapi] Raised at xapi_vm_helpers.ml:390.10-134
-> list.ml:69.12-15 -> xapi_vm_helpers.ml:507.1-47 ->
message_forwarding.ml:932.5-85 -> threadext.ml:20.20-24 ->
threadext.ml:20.62-65
-> message_forwarding.ml:40.25-57 -> message_forwarding.ml:1262.9-276 ->
pervasiveext.ml:22.2-9
|Async.VM.start_on R:96fa354a6a0d|xapi] Raised at pervasiveext.ml:26.22-25
-> pervasiveext.ml:22.2-9
|Async.VM.start_on R:96fa354a6a0d|xapi] Raised at pervasiveext.ml:26.22-25
-> pervasiveext.ml:22.2-9
|Async.VM.start_on R:96fa354a6a0d|backtrace] Raised at pervasiveext.ml:26.22-25
-> message_forwarding.ml:1248.3-1023 -> rbac.ml:229.16-23
|Async.VM.start_on R:96fa354a6a0d|backtrace] Raised at rbac.ml:238.10-15 ->
server_helpers.ml:79.11-41
|Async.VM.start_on R:96fa354a6a0d|dispatcher] Server_helpers.exec
exception_handler: Got exception HOST_CANNOT_ATTACH_NETWORK: [
OpaqueRef:f0e56fe4-004c-31af-4767-531e22622c1e;
OpaqueRef:f9e9b47a-28c3-5031-a8fb-5e103c970a8a ]
Please, PLEASE, let me know that someone knows how to fix it!
