Hi, Anonymous fellow ;)
Let’s see if I can help you a little bit. I am checking the ACS 4.9 source
code.
The error is thrown at line 896 of class
“com.cloud.hypervisor.kvm.resource.LibvirtComputingResource”.
The condition that causes the error is “_pifs.get("private") == null”.
“_pifs” if a map. The key “private” is added to the map at line 1124, if
condition “_guestBridgeName != null && bridge.equals(_guestBridgeName)” is
met.
The variable “_guestBridgeName” is a String that can receive the value of
“guest.network.device” parameter or “_privBridgeName” variable. This
process happens at lines 752-755. The process of assigning a value to
“_privBridgeName” happens at lines 747-750. The default value for
“_privBridgeName” is “cloudbr1”. The default can be overridden by
“private.network.device” parameter.
Having detailed the parameter. Let's see how ACS gets the “bridge” value.
It gets that value from code at line 1113 “cmdout.split("%")”. The variable
“cmdout” contains the output of the following OS command: “ovs-vsctl
list-br | sed '{:q;N;s/\\n/%/g;t q}'”.
Can you run the command and check its output?
On Thu, Oct 20, 2016 at 8:49 AM, Cloud List <cloud-l...@sg.or.id> wrote:
> Hi,
>
> We are using ACS version 4.2 / 4.9 on our test environment. We are using
> Ubuntu 12.04 as the operating system and KVM as the hypervisor.
>
> We are trying to simulate an upgrade from ACS 4.2 to 4.9 and roll-back from
> 4.9 to 4.2 on our test environment. The upgrade went smooth, and the
> roll-back went well as well except when we need to start the agent after
> downgrading the agent.
>
> After uninstalling cloudstack-agent version 4.9 and installing back
> cloudstack-agent version 4.2, I am not able to start the agent with below
> error messages:
>
> ====
> 2016-10-20 17:32:28,187 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> cloud0/bridge
> 2016-10-20 17:32:28,187 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) Found bridge cloud0
> 2016-10-20 17:32:28,188 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/lo/bridge
> 2016-10-20 17:32:28,188 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file
> /sys/devices/virtual/net/cloudbr1/bridge
> 2016-10-20 17:32:28,188 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) Found bridge cloudbr1
> 2016-10-20 17:32:28,188 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet4/bridge
> 2016-10-20 17:32:28,188 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet5/bridge
> 2016-10-20 17:32:28,188 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet6/bridge
> 2016-10-20 17:32:28,188 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet7/bridge
> 2016-10-20 17:32:28,188 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet8/bridge
> 2016-10-20 17:32:28,189 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet9/bridge
> 2016-10-20 17:32:28,189 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet0/bridge
> 2016-10-20 17:32:28,189 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet1/bridge
> 2016-10-20 17:32:28,189 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet2/bridge
> 2016-10-20 17:32:28,189 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet3/bridge
> 2016-10-20 17:32:28,189 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> virbr0/bridge
> 2016-10-20 17:32:28,189 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) Found bridge virbr0
> 2016-10-20 17:32:28,189 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet11/bridge
> 2016-10-20 17:32:28,190 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet12/bridge
> 2016-10-20 17:32:28,190 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet13/bridge
> 2016-10-20 17:32:28,190 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet14/bridge
> 2016-10-20 17:32:28,190 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet10/bridge
> 2016-10-20 17:32:28,190 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking for pif for bridge cloud0
> 2016-10-20 17:32:28,190 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) matchPifFileInDirectory: file name 'vnet1'
> 2016-10-20 17:32:28,191 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) matchPifFileInDirectory: file name 'vnet2'
> 2016-10-20 17:32:28,191 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) matchPifFileInDirectory: file name 'vnet5'
> 2016-10-20 17:32:28,191 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) failing to get physical interface from bridge cloud0,
> did not find an eth*, bond*, vlan*, em*, or p*p* in
> /sys/devices/virtual/net/cloud0/brif
> 2016-10-20 17:32:28,191 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking for pif for bridge cloudbr1
> 2016-10-20 17:32:28,191 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) matchPifFileInDirectory: file name 'eth1'
> 2016-10-20 17:32:28,192 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking for pif for bridge virbr0
> 2016-10-20 17:32:28,192 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) failing to get physical interface from bridge virbr0,
> did not find an eth*, bond*, vlan*, em*, or p*p* in
> /sys/devices/virtual/net/virbr0/brif
> 2016-10-20 17:32:28,192 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) done looking for pifs, no more bridges
> 2016-10-20 17:32:28,192 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) Failed to get private nic name
> 2016-10-20 17:32:28,192 ERROR [cloud.agent.AgentShell] (main:null) (logid:)
> Unable to start agent: Failed to get private nic name
> ====
>
> Below is the result of brctl show and the content of
> /etc/network/interfaces:
>
> ====
> root@test-kvm-03:/var/log/cloudstack/agent# brctl show
> bridge name bridge id STP enabled interfaces
> cloud0 8000.fe00a9fe00f8 no vnet1
> vnet2
> vnet5
> cloudbr1 8000.d067e5ec82c0 no eth1
> vnet0
> vnet10
> vnet11
> vnet12
> vnet13
> vnet14
> vnet3
> vnet4
> vnet6
> vnet7
> vnet8
> vnet9
> virbr0 8000.000000000000 yes
> ====
>
> /etc/network/interfaces:
>
> ====
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> auto eth1
> #iface eth1 inet static
> iface eth1 inet manual
>
> auto cloudbr1
> iface cloudbr1 inet static
> bridge_ports eth1
> address 192.168.0.201
> netmask 255.255.255.0
> network 192.168.0.0
> broadcast 192.168.0.255
> gateway 192.168.0.1
> dns-nameservers 8.8.8.8 8.8.4.4
> dns-search xxxxx.com
>
> auto cloudbr1:0
> iface cloudbr1:0 inet static
> address 192.168.3.201
> netmask 255.255.255.0
> network 192.168.3.0
> broadcast 192.168.3.255
> ====
>
> It seems that the error messages are complaining about no physical
> interface being added into cloud0 and virbr0 bridges. I tried to add eth1,
> cloudbr1 into the bridges but it didn't work. Deleting the cloud0 and
> virbr0 bridges doesn't help either, agent is still complaining about cannot
> find "pifs":
>
> ====
> 2016-10-20 18:00:48,331 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/lo/bridge
> 2016-10-20 18:00:48,332 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file
> /sys/devices/virtual/net/cloudbr1/bridge
> 2016-10-20 18:00:48,332 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) Found bridge cloudbr1
> 2016-10-20 18:00:48,332 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet4/bridge
> 2016-10-20 18:00:48,332 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet5/bridge
> 2016-10-20 18:00:48,332 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet6/bridge
> 2016-10-20 18:00:48,332 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet7/bridge
> 2016-10-20 18:00:48,332 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet8/bridge
> 2016-10-20 18:00:48,332 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet9/bridge
> 2016-10-20 18:00:48,333 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet0/bridge
> 2016-10-20 18:00:48,333 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet1/bridge
> 2016-10-20 18:00:48,333 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet2/bridge
> 2016-10-20 18:00:48,333 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/vnet3/bridge
> 2016-10-20 18:00:48,333 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet11/bridge
> 2016-10-20 18:00:48,333 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet12/bridge
> 2016-10-20 18:00:48,333 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet13/bridge
> 2016-10-20 18:00:48,333 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet14/bridge
> 2016-10-20 18:00:48,334 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking in file /sys/devices/virtual/net/
> vnet10/bridge
> 2016-10-20 18:00:48,334 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) looking for pif for bridge cloudbr1
> 2016-10-20 18:00:48,334 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) matchPifFileInDirectory: file name 'eth1'
> 2016-10-20 18:00:48,334 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) done looking for pifs, no more bridges
> 2016-10-20 18:00:48,334 DEBUG [kvm.resource.LibvirtComputingResource]
> (main:null) (logid:) Failed to get private nic name
> 2016-10-20 18:00:48,334 ERROR [cloud.agent.AgentShell] (main:null) (logid:)
> Unable to start agent: Failed to get private nic name
> ====
>
> I understand that the required bridge information is supposed to be added
> by CloudStack during the time when the host is added. Is there a way how I
> can add the bridge information again manually without having to delete and
> re-add the host into CloudStack? The reason is because we want to keep the
> VMs running during the downgrade, deleting and re-adding the host into
> CloudStack will shutdown the VMs.
>
> Any advice is greatly appreciated.
>
> Thank you.
>
> -ip-
>
--
Rafael Weingärtner
