Hi,
Yes, the script wouldn't do anything, if it's just run without args, however,
it should prompt you with the args to be provided. Based on the error, it seems
that you are missing a python library, hence the script terminates and that's
why the log file isn't generated in the first place. You can install the
package using:
yum install libvirt-python
Thanks,
Pearl
________________________________
From: Hean Seng <heans...@gmail.com>
Sent: Tuesday, September 29, 2020 1:10 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: Cloudstack Advance with Security Group
The error when running manually:
]# /usr/share/cloudstack-common/scripts/vm/network/security_group.py
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
line 26, in <module>
import libvirt
Package libvirt-4.5.0-33.el7_8.1.x86_64 already installed and latest version
so I guess, cannot runjust like that ?
pearl.dsi...@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
On Tue, Sep 29, 2020 at 3:08 PM Hean Seng <heans...@gmail.com> wrote:
> Hi
>
> I am running on CentOS 7.8.2003
>
> Cloudstack Agent , and KVM hypervisor:
>
> cloudstack-agent-4.14.0.0-1.el7.x86_64
>
> iptables and ebtables aldy install, iptables -L and ebtables -L show
> default rules .
>
> This is dumpxml, seem the bridge is there, the VLAN for Guest is VLAN401,
> which the setup seem right :
>
> <interface type='bridge'>
>
> <mac address='02:00:21:d9:00:01'/>
>
> <source bridge='brem1-401'/>
>
> <bandwidth>
>
> <inbound average='1280' peak='1280'/>
>
> <outbound average='1280' peak='1280'/>
>
> </bandwidth>
>
> <target dev='vnet9'/>
>
> <model type='virtio'/>
>
> <link state='up'/>
>
> <alias name='net0'/>
>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
>
>
>
> # ifconfig brem1-401
>
> brem1-401: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
>
> inet6 fe80::3824:b2ff:fe09:873f prefixlen 64 scopeid 0x20<link>
>
> ether 04:7d:7b:60:09:66 txqueuelen 1000 (Ethernet)
>
> RX packets 7191638 bytes 333185781 (317.7 MiB)
>
> RX errors 0 dropped 0 overruns 0 frame 0
>
> TX packets 8 bytes 656 (656.0 B)
>
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> brctl show
>
> bridge name bridge id STP enabled interfaces
>
> brem1-401 8000.047d7b600966 no em1.401
>
> vnet10
>
> vnet11
>
> vnet12
>
> vnet14
>
> vnet4
>
> vnet8
>
> vnet9
>
> cloud0 8000.fe00a9fe1cea no vnet13
>
> vnet15
>
> vnet2
>
> vnet6
>
> cloudbr0 8000.047d7b600966 yes em1
>
> vnet3
>
> vnet5
>
> vnet7
>
> cloudbr1 8000.000000000000 yes
>
>
>
> I always have /var/log/cloudstack/agent/ this path , just inside
>
> ls /var/log/cloudstack/agent/
>
> agent.log agent.log.2020-09-26.gz agent.log.2020-09-27.gz
> agent.log.2020-09-28.gz resizevolume.log setup.log
>
> it doesnt have securitygroup folder/ log, all the log ip pasted is at
> agent.log
>
>
>
>
> On Tue, Sep 29, 2020 at 2:44 PM Pearl d'Silva <pearl.dsi...@shapeblue.com>
> wrote:
>
>> Hi Hean,
>>
>> I was able to deploy a VM in a shared network in an advanced zone with
>> security groups enabled and could see the iptables rules getting added for
>> the VM on the host, so it is probably not a bug that we are seeing. Seems
>> like a setup issue.
>> Applying the rules can fail because:
>>
>> * The hypervisor host doesn't have iptables / ebtables command -
>> which is probably unlikely
>> * The VM deployed doesn't have an interface - can be verified by
>> dumping its configuration - virsh dumpxml <domain_name> | grep interface
>> * or application of the default network rules for the VM itself
>> failed -If you are up for some debugging, can you try running the script
>> manually in the hypervisor host:
>>
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> default_network_rules --vmname <domain_name> --vmid <vm_id> --vmip <vm_ip>
>> --vmmac <vm_mac> --vif <device/networkname> --brname <bridge_name>
>> --nicsecips 0; --isFirstNic --check
>>
>> when you do a virsh dumpxml <vm_domain_name>, look out for an interface
>> section that looks something like:
>>
>> <interface type='bridge'>
>> <mac address='1e:00:86:00:00:ba'/>
>> <source bridge='breth1-11'/>
>> <bandwidth>
>> <inbound average='25600' peak='25600'/>
>> <outbound average='25600' peak='25600'/>
>> </bandwidth>
>> <target dev='vnet0'/>
>> <model type='virtio'/>
>> <link state='up'/>
>> <alias name='net0'/>
>> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
>> function='0x0'/>
>> </interface>
>>
>>
>> the inputs to the script can be found as follows:
>> domain_name - internal name of your vm (i-x-y-VM); can also be obtained
>> by "virsh list"
>> vm_id - the 'y' part of the internal name : i-x-y-VM
>> vm_ip - Now that debug logs have been enabled, you can search for
>> "StartCommand" in your logs get the ip from the 'nics' section or you could
>> choose any ip from the shared network range
>> vm_mac - from the above bridge section, you will be able to get the MAC
>> address
>> vif / device name - taget dev name - in the above case - vnet0
>> bridge_name - again can be obtained from the above snippet (eg, breth1-11)
>> if there are no secondary ips, then pass 0 for nicsecips otherwise
>> specify the seconday ips separated by a semicolon (;)
>>
>> Basically, at the end of it, it should look something like:
>> /usr/share/cloudstack-common/scripts/vm/network/security_group.py
>> default_network_rules --vmname i-2-3-VM --vmid 3 --vmip <ip> --vmmac
>> 1e:00:86:00:00:ba --vif vnet0 --brname breth1-11 --nicsecips 0; --isFirstNic
>>
>> Hopefully, now you should be able to see a security_group.log file in the
>> /var/log/cloudstack/agent/ path.
>>
>> Thanks,
>> Pearl
>>
>>
>>
>>
>>
>> ________________________________
>> From: Hean Seng <heans...@gmail.com>
>> Sent: Tuesday, September 29, 2020 10:45 AM
>> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
>> Subject: Re: Cloudstack Advance with Security Group
>>
>> Hi Pearl,
>>
>> I had change to debug, following is the error captured:
>>
>>
>> 2020-09-29 01:13:30,090 DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Processing command:
>> com.cloud.agent.api.SecurityGroupRulesCmd
>>
>> 2020-09-29 01:13:30,090 DEBUG [kvm.resource.LibvirtConnection]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Looking for libvirtd
>> connection at: qemu:///system
>>
>> 2020-09-29 01:13:30,094 DEBUG [kvm.resource.LibvirtComputingResource]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Checking default network
>> rules for vm i-23-77-VM
>>
>> 2020-09-29 01:13:30,094 ERROR [kvm.resource.LibvirtComputingResource]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Unable to apply default
>> network rule for nic cloudbr0 for VM i-23-77-VM
>>
>> 2020-09-29 01:13:30,094 WARN
>> [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Failed to program default
>> network rules for vm i-23-77-VM
>>
>> 2020-09-29 01:13:30,095 DEBUG [cloud.agent.Agent]
>> (agentRequest-Handler-2:null) (logid:388b92e0) Seq
>> 11-6057904448766738456: {
>> Ans: , MgmtId: 72752492851638, via: 11, Ver: v1, Flags: 110,
>>
>> [{"com.cloud.agent.api.SecurityGroupRuleAnswer":{"logSequenceNumber":15,"vmId":77,"reason":"PROGRAMMING_FAILED","result":false,"details":"programming
>> default network rules failed","wait":0}}] }
>>
>>
>>
>>
>>
>> On Tue, Sep 29, 2020 at 12:38 PM Pearl d'Silva <
>> pearl.dsi...@shapeblue.com>
>> wrote:
>>
>> > Hi Hean,
>> >
>> >
>> > Could you try enabling debug logging on your hypervisor hosts, by
>> running
>> > the following:
>> > *sed -i 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml*
>> >
>> > And then restart the cloudstack-agent service on the hosts. Post this,
>> try
>> > deploying a VM in the shared network. This may give a better insight
>> into
>> > where exactly the issue lies.
>> >
>> > Thanks,
>> > Pearl
>> >
>> >
>> > Software Engineer
>> > pearl.dsi...@shapeblue.com
>> > www.shapeblue.com<http://www.shapeblue.com>
>> >
>> >
>> >
>> >
>> >
>> >
>> > ------------------------------
>> > *From:* Ivan Kudryavtsev <i...@bw-sw.com>
>> > *Sent:* Monday, September 28, 2020 2:17 PM
>> > *To:* users <users@cloudstack.apache.org>
>> > *Subject:* Re: Cloudstack Advance with Security Group
>> >
>> > This is it.
>> >
>>
>> pearl.dsi...@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>> > On Mon, Sep 28, 2020 at 3:46 PM Hean Seng <heans...@gmail.com> wrote:
>> >
>> > > In the log, I cannot see anything much, except a few lines showing
>> above
>> > > . Not sure if this is a bug on 4.14.
>> > >
>> > >
>> > >
>> > > 2020-09-28 03:53:36,585 ERROR [kvm.resource.LibvirtComputingResource]
>> > > (agentRequest-Handler-4:null) (logid:b6a4c077) Unable to apply default
>> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> > >
>> > > 2020-09-28 03:53:36,858 ERROR [kvm.resource.LibvirtComputingResource]
>> > > (agentRequest-Handler-3:null) (logid:74058678) Unable to apply default
>> > > network rule for nic cloudbr0 for VM i-2-81-VM
>> > >
>> > > 2020-09-28 03:53:36,858 WARN
>> > > [resource.wrapper.LibvirtSecurityGroupRulesCommandWrapper]
>> > > (agentRequest-Handler-3:null) (logid:74058678) Failed to program
>> default
>> > > network rules for vm i-2-81-VM
>> > >
>> > >
>> > >
>> > >
>> > > On Mon, Sep 28, 2020 at 4:34 PM Ivan Kudryavtsev <i...@bw-sw.com>
>> wrote:
>> > >
>> > > > Hi,
>> > > > no I'm on 4.11, so can not help with exact 4.14, and I'm on Ubuntu,
>> > > though,
>> > > > but for any KVM hypervisor Linux distribution, the logic is the
>> same.
>> > > >
>> > > > On Mon, Sep 28, 2020 at 3:31 PM Hean Seng <heans...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi
>> > > > >
>> > > > > Are you running on CentOS7 ?
>> > > > >
>> > > > > I am running on CentOS7 , ACS 4.14 , and seem there is no log
>> at
>> > of
>> > > > > security_group.log
>> > > > >
>> > > > > # ls /var/log/cloudstack/agent/
>> > > > >
>> > > > > agent.log resizevolume.log setup.log
>> > > > >
>> > > > >
>> > > > > I recheck back the Intall guide, seems no missing anything.
>> > > > >
>> > > > >
>> > > > > Older intallation guide, 4.11 mentioned need , allow
>> > > > > /usr/lib/sysctl.d/00-system.conf
>> > > > >
>> > > > > # Enable netfilter on bridges.
>> net.bridge.bridge-nf-call-ip6tables =
>> > 1
>> > > > > net.bridge.bridge-nf-call-iptables = 1
>> > > > net.bridge.bridge-nf-call-arptables
>> > > > > = 1
>> > > > >
>> > > > > And it has been done too.
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Sep 28, 2020 at 4:05 PM Ivan Kudryavtsev <i...@bw-sw.com>
>> > > wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > No, this is not the issue.
>> > > > > > It's a normal state of the system, as KVM hooks are a new and
>> > > optional
>> > > > > > feature of 4.14.
>> > > > > >
>> > > > > > You should find some sort of messages regarding security_groups
>> at
>> > > > > > /var/log/cloudstack/agent/security_group.log
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Sep 28, 2020 at 2:10 PM Hean Seng <heans...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > > I not sure where goes wrong, are you running on CentOS 7 ? I
>> > have
>> > > > this
>> > > > > > > error too, do you think is this contribute to the error as
>> well:
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:52,762 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
>> > > > > > >
>> '/etc/cloudstack/agent/hooks/libvirt-vm-xml-transformer.groovy'
>> > is
>> > > > not
>> > > > > > > available. Transformations will not be applied.
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:52,762 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy
>> scripting
>> > > > engine
>> > > > > is
>> > > > > > > not initialized. Data transformation skipped.
>> > > > > > >
>> > > > > > > 2020-09-28 03:04:53,083 WARN
>> [kvm.resource.LibvirtKvmAgentHook]
>> > > > > > > (agentRequest-Handler-5:null) (logid:4f23845b) Groovy script
>> > > > > > > '/etc/cloudstack/agent/hooks/libvirt-vm-state-change.groovy'
>> is
>> > not
>> > > > > > > available. Transformations will not be applied.
>> > > > > > >
>> > > > > > > On Mon, Sep 28, 2020 at 2:27 PM Ivan Kudryavtsev <
>> i...@bw-sw.com
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > This just means you installed it in the wrong way. Ebtables
>> and
>> > > > > > Iptables
>> > > > > > > > must be filled with rules like
>> > > > > > > >
>> > > > > > > > -A i-6242-10304-def -m state --state RELATED,ESTABLISHED -j
>> > > ACCEPT
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m udp --sport 68 --dport 67 -j ACCEPT
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-out vnet18
>> > > > > > > > --physdev-is-bridged -m udp --sport 67 --dport 68 -j ACCEPT
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> > > > > --physdev-is-bridged
>> > > > > > > -m
>> > > > > > > > set ! --match-set i-6242-10304-vm src -j DROP
>> > > > > > > > -A i-6242-10304-def -p udp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
>> -m
>> > > udp
>> > > > > > > --dport
>> > > > > > > > 53 -j RETURN
>> > > > > > > > -A i-6242-10304-def -p tcp -m physdev --physdev-in vnet18
>> > > > > > > > --physdev-is-bridged -m set --match-set i-6242-10304-vm src
>> -m
>> > > tcp
>> > > > > > > --dport
>> > > > > > > > 53 -j RETURN
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-in vnet18
>> > > > > --physdev-is-bridged
>> > > > > > > -m
>> > > > > > > > set --match-set i-6242-10304-vm src -j i-6242-10304-vm-eg
>> > > > > > > > -A i-6242-10304-def -m physdev --physdev-out vnet18
>> > > > > > --physdev-is-bridged
>> > > > > > > -j
>> > > > > > > > i-6242-10304-vm
>> > > > > > > > -A i-6242-10304-vm -p udp -m udp --dport 1:65535 -m state
>> > --state
>> > > > NEW
>> > > > > > -j
>> > > > > > > > ACCEPT
>> > > > > > > > -A i-6242-10304-vm -p tcp -m tcp --dport 1:65535 -m state
>> > --state
>> > > > NEW
>> > > > > > -j
>> > > > > > > > ACCEPT
>> > > > > > > > -A i-6242-10304-vm -p icmp -m icmp --icmp-type any -j ACCEPT
>> > > > > > > > -A i-6242-10304-vm -j DROP
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > Bridge chain: i-4435-8929-vm-in, entries: 7, policy: ACCEPT
>> > > > > > > > -s ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP -s ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP --arp-mac-src ! 1e:0:32:0:2:2 -j DROP
>> > > > > > > > -p ARP -j i-4435-8929-vm-in-ips
>> > > > > > > > -p ARP --arp-op Request -j ACCEPT
>> > > > > > > > -p ARP --arp-op Reply -j ACCEPT
>> > > > > > > > -p ARP -j DROP
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, Sep 28, 2020 at 1:10 PM Hean Seng <
>> heans...@gmail.com>
>> > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I checked the hypervisor , it seems iptables is nothing
>> > inside
>> > > ,
>> > > > > > this
>> > > > > > > is
>> > > > > > > > > centos7 , initially i turnoff firewalld , but even i
>> turn
>> > on
>> > > it
>> > > > > now
>> > > > > > > and
>> > > > > > > > > try to update the security group rules, it seems empty
>> > iptable
>> > > > > rules
>> > > > > > :
>> > > > > > > > >
>> > > > > > > > > [root@kvm03 ~]# iptables -L -v -n
>> > > > > > > > >
>> > > > > > > > > Chain INPUT (policy ACCEPT 82903 packets, 1170M bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Chain OUTPUT (policy ACCEPT 80505 packets, 25M bytes)
>> > > > > > > > >
>> > > > > > > > > pkts bytes target prot opt in out source
>> > > > > > > > > destination
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Mon, Sep 28, 2020 at 12:05 PM Pearl d'Silva <
>> > > > > > > > pearl.dsi...@shapeblue.com
>> > > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Hean,
>> > > > > > > > > >
>> > > > > > > > > > In an Advanced Zone with Security Groups enabled, by
>> > default,
>> > > > > > egress
>> > > > > > > > > > traffic from the VM is allowed, while Ingress traffic is
>> > > > denied.
>> > > > > > > Hence,
>> > > > > > > > > as
>> > > > > > > > > > you rightly mentioned, security group rules are added
>> > > > > accordingly.
>> > > > > > > > These
>> > > > > > > > > > rules get added on the hypervisor host, and you can
>> verify
>> > > > them,
>> > > > > by
>> > > > > > > > going
>> > > > > > > > > > into the host and searching for iptables rules
>> > corresponding
>> > > to
>> > > > > the
>> > > > > > > VM
>> > > > > > > > > > (internal name - i-x-y-VM).
>> > > > > > > > > > This blog maybe helpful in providing further details:
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://shankerbalan.net/blog/cloudstack-advanced-zone-with-security-groups/
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > > Pearl
>> > > > > > > > > > ________________________________
>> > > > > > > > > > From: Hean Seng <heans...@gmail.com>
>> > > > > > > > > > Sent: Sunday, September 27, 2020 2:48 PM
>> > > > > > > > > > To: users@cloudstack.apache.org <
>> > users@cloudstack.apache.org
>> > > >
>> > > > > > > > > > Subject: Cloudstack Advance with Security Group
>> > > > > > > > > >
>> > > > > > > > > > Hi
>> > > > > > > > > >
>> > > > > > > > > > I created advance zone with security group, all working
>> > fine.
>> > > > > > > > > >
>> > > > > > > > > > But VMcreated ， seems the default security group that
>> > > assigned
>> > > > to
>> > > > > > the
>> > > > > > > > VM.
>> > > > > > > > > > all accept policy , i understand is Default Deny, and
>> once
>> > > add
>> > > > > in
>> > > > > > > the
>> > > > > > > > > port
>> > > > > > > > > > in Security Group Ingress and Egress, only is allowed
>> > > > > > > > > >
>> > > > > > > > > > Also, is this rules created at VirtualRouter of the
>> > > > > SharedNetwork,
>> > > > > > or
>> > > > > > > > at
>> > > > > > > > > > the Hypervisor?
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Regards,
>> > > > > > > > > > Hean Seng
>> > > > > > > > > >
>> > > > > > > > > > pearl.dsi...@shapeblue.com
>> > > > > > > > > > www.shapeblue.com<http://www.shapeblue.com>
>> > > > > > > > > > 3 London Bridge Street, 3rd floor, News Building,
>> London
>> > > SE1
>> > > > > > 9SGUK
>> > > > > > > > > > @shapeblue
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Regards,
>> > > > > > > > > Hean Seng
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Regards,
>> > > > > > > Hean Seng
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Regards,
>> > > > > Hean Seng
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Hean Seng
>> > >
>> >
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
