The error when running manually:
]# /usr/share/cloudstack-common/scripts/vm/network/security_group.py
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
line 26, in <module>
import libvirt
Package libvirt-4.5.0-33.el7_8.1.x86_64 already installed and latest version
so I guess, cannot runjust like that ?
On Tue, Sep 29, 2020 at 3:08 PM Hean Seng <heans...@gmail.com> wrote:
> Hi
>
> I am running on CentOS 7.8.2003
>
> Cloudstack Agent , and KVM hypervisor:
>
> cloudstack-agent-4.14.0.0-1.el7.x86_64
>
> iptables and ebtables aldy install, iptables -L and ebtables -L show
> default rules .
>
> This is dumpxml, seem the bridge is there, the VLAN for Guest is VLAN401,
> which the setup seem right :
>
> <interface type='bridge'>
>
> <mac address='02:00:21:d9:00:01'/>
>
> <source bridge='brem1-401'/>
>
> <bandwidth>
>
> <inbound average='1280' peak='1280'/>
>
> <outbound average='1280' peak='1280'/>
>
> </bandwidth>
>
> <target dev='vnet9'/>
>
> <model type='virtio'/>
>
> <link state='up'/>
>
> <alias name='net0'/>
>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>
>
>
> # ifconfig brem1-401
>
> brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>
> inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid 0x20<link>
>
> ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
>
> RX packets 7191638 bytes 333185781 (317.7 MiB)
>
> RX errors 0 dropped 0 overruns 0 frame 0
>
> TX packets 8 bytes 656 (656.0 B)
>
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> brctl show
>
> bridge name bridge id STP enabled interfaces
>
> brem1-401 8000.047d7b600966 no em1.401
>
> vnet10
>
> vnet11
>
> vnet12
>
> vnet14
>
> vnet4
>
> vnet8
>
> vnet9
>
> cloud0 8000.fe00a9fe1cea no vnet13
>
> vnet15
>
> vnet2
>
> vnet6
>
> cloudbr0 8000.047d7b600966 yes em1
>
> vnet3
>
> vnet5
>
> vnet7
>
> cloudbr1 8000.000000000000 yes
>
>
>
> I always have /var/log/cloudstack/agent/ this path , just inside
>
> ls /var/log/cloudstack/agent/
>
> agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
> agent.log.2020-09-28.gz resizevolume.log setup.log
>
> it doesnt have securitygroup folder/ log, all the log ip pasted is at
> agent.log
>
>
>
>
> On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <pearl.dsi...@shapeblue.com>
> wrote:
>
>> Hi Hean,
>>
>> I was able to deploy a VM in a shared network in an advanced zone with
>> security groups enabled and could see the iptables rules getting added for
>> the VM on the host, so it is probably not a bug that we are seeing. Seems
>> like a setup issue.
>> Applying the rules can fail because:
>>
>> * The hypervisor host doesn't have iptables / ebtables command -
>> which is probably unlikely
>> * The VM deployed doesn't have an interface - can be verified by
>> dumping its configuration - virsh dumpxml <domain_name> | grep interface
>> * or application of the default network rules for the VM itself
>> failed -If you are up for some debugging, can you try running the script
>> manually in the hypervisor host:
>>
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip <vm_ip>
>> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
>> --nicsecips 0; --isFirstNic --check
>>
>> when you do a virsh dumpxml <vm_domain_name>, look out for an interface
>> section that looks something like:
>>
>> <interface type='bridge'>
>> <mac address='1e:00:86:00:00:ba'/>
>> <source bridge='breth1-11'/>
>> <bandwidth>
>> <inbound average='25600' peak='25600'/>
>> <outbound average='25600' peak='25600'/>
>> </bandwidth>
>> <target dev='vnet0'/>
>> <model type='virtio'/>
>> <link state='up'/>
>> <alias name='net0'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> function='0x0'/>
>> </interface>
>>
>>
>> the inputs to the script can be found as follows:
>> domain_name - internal name of your vm (i-x-y-VM); can also be obtained
>> by "virsh list"
>> vm_id - the 'y' part of the internal name : i-x-y-VM
>> vm_ip - Now that debug logs have been enabled, you can search for
>> "StartCommand" in your logs get the ip from the 'nics' section or you could
>> choose any ip from the shared network range
>> vm_mac - from the above bridge section, you will be able to get the MAC
>> address
>> vif / device name - taget dev name - in the above case - vnet0
>> bridge_name - again can be obtained from the above snippet (eg, breth1-11)
>> if there are no secondary ips, then pass 0 for nicsecips otherwise
>> specify the seconday ips separated by a semicolon (;)
>>
>> Basically, at the end of it, it should look something like:
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
>> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0; --isFirstNic
>>
>> Hopefully, now you should be able to see a security_group.log file in the
>> /var/log/cloudstack/agent/ path.
>>
>> Thanks,
>> Pearl
>>
>>
>>
>>
>>
>> ________________________________
>> From: Hean Seng <heans...@gmail.com>
>> Sent: Tuesday, September 29, 2020 10:45 AM
>> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
>> Subject: Re: Cloudstack Advance with Security Group
>>
>> Hi Pearl,
>>
>> I had change to debug, following is the error captured:
>>
>>
>> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
>> com.cloud.agent.api.SecurityGroupRulesCmd
>>
>> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
>> connection at: qemu:///system
>>
>> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
>> rules for vm i-23-77-VM
>>
>> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
>> network rule for nic cloudbr0 for VM i-23-77-VM
>>
>> 2020-09-29 01:13:30,094 WARN
>> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
>> network rules for vm i-23-77-VM
>>
>> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
>> 11-6057904448766738456: {
>> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>>
>> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
>> default network rules failed","wait":0}}] }
>>
>>
>>
>>
>>
>> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <
>> pearl.dsi...@shapeblue.com>
>> wrote:
>>
>> > Hi Hean,
>> >
>> >
>> > Could you try enabling debug logging on your hypervisor hosts, by
>> running
>> > the following:
>> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>> >
>> > And then restart the cloudstack-agent service on the hosts. Post this,
>> try
>> > deploying a VM in the shared network. This may give a better insight
>> into
>> > where exactly the issue lies.
>> >
>> > Thanks,
>> > Pearl
>> >
>> >
>> > Software Engineer
>> > pearl.dsi...@shapeblue.com
>> > www.shapeblue.com<http://www.shapeblue.com>
>> >
>> >
>> >
>> >
>> >
>> >
>> > ------------------------------
>> > *From:* Ivan Kudryavtsev <i...@bw-sw.com>
>> > *Sent:* Monday, September 28, 2020 2:17 PM
>> > *To:* users <users@cloudstack.apache.org>
>> > *Subject:* Re: Cloudstack Advance with Security Group
>> >
>> > This is it.
>> >
>>
>> pearl.dsi...@shapeblue.com
>> www.shapeblue.com
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <heans...@gmail.com> wrote:
>> >
>> > > In the log, I cannot see anything much, except a few lines showing
>> above
>> > > . Not sure if this is a bug on 4.14.
>> > >
>> > >
>> > >
>> > > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
>> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
>> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> > >
>> > > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
>> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
>> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> > >
>> > > 2020-09-28 03:53:36,858 WARN
>> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
>> default
>> > > network rules for vm i-2-81-VM
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <i...@bw-sw.com>
>> wrote:
>> > >
>> > > > Hi,
>> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
>> > > though,
>> > > > but for any KVM hypervisor Linux distribution, the logic is the
>> same.
>> > > >
>> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <heans...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi
>> > > > >
>> > > > > Are you running on CentOS7 ?
>> > > > >
>> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log
>> at
>> > of
>> > > > > security_group.log
>> > > > >
>> > > > > # ls /var/log/cloudstack/agent/
>> > > > >
>> > > > > agent.log resizevolume.log setup.log
>> > > > >
>> > > > >
>> > > > > I recheck back the Intall guide, seems no missing anything.
>> > > > >
>> > > > >
>> > > > > Older intallation guide, 4.11 mentioned need , allow
>> > > > > /usr/lib/sysctl.d/00-system.conf
>> > > > >
>> > > > > # Enable netfilter on bridges.
>> net.bridge.bridge-nf-call-ip6tables =
>> > 1
>> > > > > net.bridge.bridge-nf-call-iptables = 1
>> > > > net.bridge.bridge-nf-call-arptables
>> > > > > = 1
>> > > > >
>> > > > > And it has been done too.
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <i...@bw-sw.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > No, this is not the issue.
>> > > > > > It's a normal state of the system, as KVM hooks are a new and
>> > > optional
>> > > > > > feature of 4.14.
>> > > > > >
>> > > > > > You should find some sort of messages regarding security_groups
>> at
>> > > > > > /var/log/cloudstack/agent/security_group.log
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <heans...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
>> > have
>> > > > this
>> > > > > > > error too, do you think is this contribute to the error as
>> well:
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:52,762 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
>> > > > > > >
>> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
>> > is
>> > > > not
>> > > > > > > available. Transformations will not be applied.
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:52,762 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>> scripting
>> > > > engine
>> > > > > is
>> > > > > > > not initialized. Data transformation skipped.
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:53,083 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
>> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy'
>> is
>> > not
>> > > > > > > available. Transformations will not be applied.
>> > > > > > >
>> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
>> i...@bw-sw.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > This just means you installed it in the wrong way. Ebtables
>> and
>> > > > > > Iptables
>> > > > > > > > must be filled with rules like
>> > > > > > > >
>> > > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
>> > > ACCEPT
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
>> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> > > > > --physdev-is-bridged
>> > > > > > > -m
>> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
>> -m
>> > > udp
>> > > > > > > --dport
>> > > > > > > > 53 -j RETURN
>> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
>> -m
>> > > tcp
>> > > > > > > --dport
>> > > > > > > > 53 -j RETURN
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> > > > > --physdev-is-bridged
>> > > > > > > -m
>> > > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
>> > > > > > --physdev-is-bridged
>> > > > > > > -j
>> > > > > > > > i-6242-10304-vm
>> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
>> > --state
>> > > > NEW
>> > > > > > -j
>> > > > > > > > ACCEPT
>> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
>> > --state
>> > > > NEW
>> > > > > > -j
>> > > > > > > > ACCEPT
>> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
>> > > > > > > > -A i-6242-10304-vm -j DROP
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
>> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
>> > > > > > > > -p ARP --arp-op Request -j ACCEPT
>> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
>> > > > > > > > -p ARP -j DROP
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
>> heans...@gmail.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I checked the hypervisor , it seems iptables is nothing
>> > inside
>> > > ,
>> > > > > > this
>> > > > > > > is
>> > > > > > > > > centos7 , initially i turnoff firewalld , but even i
>> turn
>> > on
>> > > it
>> > > > > now
>> > > > > > > and
>> > > > > > > > > try to update the security group rules, it seems empty
>> > iptable
>> > > > > rules
>> > > > > > :
>> > > > > > > > >
>> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
>> > > > > > > > >
>> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
>> > > > > > > > pearl.dsi...@shapeblue.com
>> > > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Hean,
>> > > > > > > > > >
>> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
>> > default,
>> > > > > > egress
>> > > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
>> > > > denied.
>> > > > > > > Hence,
>> > > > > > > > > as
>> > > > > > > > > > you rightly mentioned, security group rules are added
>> > > > > accordingly.
>> > > > > > > > These
>> > > > > > > > > > rules get added on the hypervisor host, and you can
>> verify
>> > > > them,
>> > > > > by
>> > > > > > > > going
>> > > > > > > > > > into the host and searching for iptables rules
>> > corresponding
>> > > to
>> > > > > the
>> > > > > > > VM
>> > > > > > > > > > (internal name - i-x-y-VM).
>> > > > > > > > > > This blog maybe helpful in providing further details:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Pearl
>> > > > > > > > > > ________________________________
>> > > > > > > > > > From: Hean Seng <heans...@gmail.com>
>> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
>> > > > > > > > > > To: users@cloudstack.apache.org <
>> > users@cloudstack.apache.org
>> > > >
>> > > > > > > > > > Subject: Cloudstack Advance with Security Group
>> > > > > > > > > >
>> > > > > > > > > > Hi
>> > > > > > > > > >
>> > > > > > > > > > I created advance zone with security group, all working
>> > fine.
>> > > > > > > > > >
>> > > > > > > > > > But VMcreated ， seems the default security group that
>> > > assigned
>> > > > to
>> > > > > > the
>> > > > > > > > VM.
>> > > > > > > > > > all accept policy , i understand is Default Deny, and
>> once
>> > > add
>> > > > > in
>> > > > > > > the
>> > > > > > > > > port
>> > > > > > > > > > in Security Group Ingress and Egress, only is allowed
>> > > > > > > > > >
>> > > > > > > > > > Also, is this rules created at VirtualRouter of the
>> > > > > SharedNetwork,
>> > > > > > or
>> > > > > > > > at
>> > > > > > > > > > the Hypervisor?
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Regards,
>> > > > > > > > > > Hean Seng
>> > > > > > > > > >
>> > > > > > > > > > pearl.dsi...@shapeblue.com
>> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
>> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building,
>> London
>> > > SE1
>> > > > > > 9SGUK
>> > > > > > > > > > @shapeblue
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Regards,
>> > > > > > > > > Hean Seng
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Regards,
>> > > > > > > Hean Seng
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Regards,
>> > > > > Hean Seng
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Hean Seng
>> > >
>> >
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
