Hi Dave,
As basic zone with SG isn’t widely used or popular, it’s possible you’re
hitting a regression or bug.
Regards.
________________________________
From: Dave Lapointe <da...@island.net.INVALID>
Sent: Friday, March 21, 2025 8:23:31 PM
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Re: Looking for suggestion on network configuration - followup
Hi Rohit,
Thanks for the suggestions.
Before I saw your reply here to use Advanced Zones, I tried Basic Zone because
my CloudStack configuration requirements are so simple (only needing one
server). And it turns out that a Basic Zone without SG works as expected...
the VM's can communicate with each other. However, for the sake of being
consistent with typical deployments, I'll try Advanced Zones now.
Unfortunately, I won't have physical access to the server for at least another
week so I can't install Ubuntu.
I confirmed that physdev is available and bridge-nf-call-iptables is 1.
I think that the Basic Zone with SG isn't working as it should, since the
default firewall rules created by CloudStack prevent all communication across
cloudbr0, and the SG rules appear to be ignored. But maybe it's not so
important since, as you said, most people use Advanced Zones and what I've been
trying is a bit of an outlier case.
Thanks,
Dave
On 03-20-25 10:07 p.m., Rohit Yadav wrote:
> Hi Dave,
>
> This shouldn't normally happen, most people using security groups with KVM
> have been using Ubuntu and Adv zone with security groups. In fact, most
> people should use Adv zone without SG groups now, as one can create shared
> network with SG in such zones now. I would suggest that you at least
> re-deploy zone with SG and try again, my notes are for Adv zone without
> security groups.
>
> Can you also try this:
>
> 1.
> Check if physdev is available:
> lsmod | grep xt_physdev
>
>
> 1.
> Enable it when missing:
> modprobe xt_physdev
>
> Also ensure bridge-based firewalling is enabled:
>
> modprobe br_netfilter
> echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
>
>
> Regards.
>
>
>
>
> ________________________________
> From: Dave Lapointe <da...@island.net>
> Sent: Tuesday, March 18, 2025 05:57
> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
> Subject: Re: Looking for suggestion on network configuration - followup
>
> Hi Rohit,
>
> Thanks for the link. I'm running ACS 4.20 on Rocky 8. I did a "basic" setup
> when first connecting to the CloudStack UI. Hardware is Dell PowerEdge 450
> with 64GB RAM. I've tried Windows 10 and Debian Linux VM's.
>
> I did some more checking and found that the firewall rules for the bridge are
> being set in
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py:default_network_rules
> where it blocks all traffic between VM's and then fails to define VM
> specific rules because a table doesn't exist. Here is the log from agent.log:
>
> 2025-03-17 07:26:58,267 - Executing command: default_network_rules
> 2025-03-17 07:26:58,267 - modprobe br_netfilter
> 2025-03-17 07:26:58,270 - sysctl -w net.bridge.bridge-nf-call-arptables=1
> 2025-03-17 07:26:58,273 - sysctl -w net.bridge.bridge-nf-call-iptables=1
> 2025-03-17 07:26:58,276 - sysctl -w net.bridge.bridge-nf-call-ip6tables=1
> 2025-03-17 07:26:58,278 - iptables-save |grep physdev-is-bridged |grep
> FORWARD |grep BF |grep '\-o' | grep -w cloudbr0|awk '{print $9}' | head -1
> 2025-03-17 07:26:58,283 - iptables -L BF-cloudbr0
> 2025-03-17 07:26:58,286 - iptables -L BF-cloudbr0-OUT
> 2025-03-17 07:26:58,289 - iptables -L BF-cloudbr0-IN
> 2025-03-17 07:26:58,292 - ip6tables -L BF-cloudbr0
> 2025-03-17 07:26:58,295 - ip6tables -L BF-cloudbr0-OUT
> 2025-03-17 07:26:58,298 - ip6tables -L BF-cloudbr0-IN
> 2025-03-17 07:26:58,300 - bridge -o link show | awk '/master cloudbr0 / &&
> !/^[0-9]+: vnet/ {print $2}' | head -1 | cut -d ':' -f1 | cut -d '@' -f1
> 2025-03-17 07:26:58,304 - iptables -n -L BF-cloudbr0 | awk
> '/BF-cloudbr0(.*)references/ {gsub(/\(/, "") ;print $3}'
> 2025-03-17 07:26:58,307 - iptables -n -L BF-cloudbr0-IN | awk
> '/BF-cloudbr0-IN(.*)references/ {gsub(/\(/, "") ;print $3}'
> 2025-03-17 07:26:58,311 - iptables -n -L BF-cloudbr0-OUT | awk
> '/BF-cloudbr0-OUT(.*)references/ {gsub(/\(/, "") ;print $3}'
> 2025-03-17 07:26:58,314 - ip6tables -n -L BF-cloudbr0 | awk
> '/BF-cloudbr0(.*)references/ {gsub(/\(/, "") ;print $3}'
> 2025-03-17 07:26:58,318 - iptables -I FORWARD -i cloudbr0 -j DROP
> 2025-03-17 07:26:58,320 - iptables -I FORWARD -o cloudbr0 -j DROP
> 2025-03-17 07:26:58,323 - iptables -I FORWARD -i cloudbr0 -m physdev
> --physdev-is-bridged -j BF-cloudbr0
> 2025-03-17 07:26:58,326 - iptables -I FORWARD -o cloudbr0 -m physdev
> --physdev-is-bridged -j BF-cloudbr0
> 2025-03-17 07:26:58,329 - iptables -A BF-cloudbr0 -m state --state
> RELATED,ESTABLISHED -j ACCEPT
> 2025-03-17 07:26:58,332 - iptables -A BF-cloudbr0 -m physdev
> --physdev-is-bridged --physdev-is-in -j BF-cloudbr0-IN
> 2025-03-17 07:26:58,335 - iptables -A BF-cloudbr0 -m physdev
> --physdev-is-bridged --physdev-is-out -j BF-cloudbr0-OUT
> 2025-03-17 07:26:58,337 - iptables -A BF-cloudbr0 -m physdev
> --physdev-is-bridged --physdev-out -j ACCEPT
> 2025-03-17 07:26:58,340 - Command exited non-zero: iptables -A BF-cloudbr0 -m
> physdev --physdev-is-bridged --physdev-out -j ACCEPT
> Traceback (most recent call last):
> File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
> line 53, in execute
> return check_output(cmd, shell=True).decode()
> File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
> **kwargs).stdout
> File "/usr/lib64/python3.6/subprocess.py", line 438, in run
> output=stdout, stderr=stderr)
> subprocess.CalledProcessError: Command 'iptables -A BF-cloudbr0 -m physdev
> --physdev-is-bridged --physdev-out -j ACCEPT' returned non-zero exit status
> 2.
> 2025-03-17 07:26:58,341 - iptables -F BF-cloudbr0
> 2025-03-17 07:26:58,344 - ip6tables -F BF-cloudbr0
>
> The exception that's thrown is caused by the error:
> Bad argument `ACCEPT'
>
> Just to confirm that the rules set by default_network_rules were causing the
> issue after a VM was started I tried adding:
>
> iptables -I FORWARD -i cloudbr0 -j ACCEPT
> iptables -I FORWARD -i cloudbr0 -j ACCEPT
>
> and communications between VM's starting working, including DHCP. I also
> tried setting rules in Security Group but it appears that they're not added
> to iptables because the VM specific table doesn't exist.
>
> I was hoping that the file at
> /usr/share/cloudstack-common/scripts/vm/network/security_group.py was
> actually the one that was being invoked but it's not... I guess it's a
> resource in the agent package. If it was then I thought I could kind of
> "bludgeon" a solution by adding:
>
> iptables -I FORWARD -i cloudbr0 -j ACCEPT
> iptables -I FORWARD -i cloudbr0 -j ACCEPT
>
> in default_network_rules.
>
> In your notes on installing CloudStack the suggestion of setting:
>
> net.bridge.bridge-nf-call-arptables = 0
> net.bridge.bridge-nf-call-iptables = 0
>
> in sysctl.conf doesn't work because default_network_rules always re-enables
> them.
>
> So I think something isn't working right with CloudStack. It appears to be
> ignoring security group rules, and at least on of the iptables rules appear
> to have invalid syntax. If it was possible to modify security_group.py then
> I could at least make it work. But default_network_rules is called quite
> often, like whenever a VM is started so I can't think of anyway to add
> firewall rules that won't be replaced by the rules from default_network_rules.
>
> Thanks,
> Dave
>
>
> On 03-16-25 12:35 a.m., Rohit Yadav wrote:
>> Hi Dave,
>>
>> Can you share your OS and other env details? For ACS 4.20 you don't have to
>> seed the systemvmtemplate manually.
>>
>> My notes on installing CloudStack are here -
>> https://rohityadav.cloud/blog/cloudstack-kvm/ if you're trying this on
>> Ubuntu.
>>
>> Regards.
>>
>>
>>
>> ________________________________
>> From: Dave Lapointe <da...@island.net.INVALID>
>> Sent: Sunday, March 16, 2025 06:12
>> To: users@cloudstack.apache.org <users@cloudstack.apache.org>
>> Subject: Looking for suggestion on network configuration - followup
>>
>> Hi,
>>
>> I decided to go back to the basic CloudStack installation as described in
>> the Quick Installation Guide, and followed every instruction there (there is
>> an error with
>> /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
>> -m /export/secondary -u
>> http://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.0.0-x86_64-kvm.qcow2.bz2
>> -h kvm -F. The SystemVMtemplate should be
>> systemvmtemplate-4.20.0-x86_64-kvm.qcow2.bz2).
>>
>> After completing the install and observing that there were no errors in
>> management-server.log and agent.log I assumed everything was setup
>> correctly. Once the two system VM's started up I ssh'ed into them, (they
>> are bridged through cloudbr0 and cloud0) and found that I couldn't ping on
>> the subnet for cloudbr0 but could ping on the cloud0 subnet. I deleted all
>> of the firewall rules and set the default policies to ACCEPT but it made no
>> difference. The routing tables appeared to be okay, with traffic for the
>> cloudbr0 subnet being routed through the appropriate NIC.
>>
>> On the host I tried tcpdump -i cloudbr0 -w cloudbr0.cap and Wireshark showed
>> that the ICMP packets are arriving on cloudbr0 but there is no response. I
>> tried tcpdump on one of the system VM's and confirmed that there is a
>> response to pinging the cloudbr0 ip address, and that there was no response
>> from the ping target.
>>
>> I've done lots of bridging in different environments, and haven't
>> encountered anything like this before, so I'm at a loss to know how to
>> continue investigating this.
>>
>> I've spent a few days on this now and have made no progress. Please, any
>> suggestions? I'm close to the point now of abandoning CloudStack because I
>> can't get it to work as documented.
>>
>> Thanks,
>> Dave
>>
>
>
