Hi Rohit,
Thanks for the link. I'm running ACS 4.20 on Rocky 8. I did a "basic" setup
when first connecting to the CloudStack UI. Hardware is Dell PowerEdge 450 with 64GB
RAM. I've tried Windows 10 and Debian Linux VM's.
I did some more checking and found that the firewall rules for the bridge are
being set in
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:default_network_rules
where it blocks all traffic between VM's and then fails to define VM specific
rules because a table doesn't exist. Here is the log from agent.log:
2025-03-17 07:26:58,267 - Executing command: default_network_rules
2025-03-17 07:26:58,267 - modprobe br_netfilter
2025-03-17 07:26:58,270 - sysctl -w net.bridge.bridge-nf-call-arptables=1
2025-03-17 07:26:58,273 - sysctl -w net.bridge.bridge-nf-call-iptables=1
2025-03-17 07:26:58,276 - sysctl -w net.bridge.bridge-nf-call-ip6tables=1
2025-03-17 07:26:58,278 - iptables-save |grep physdev-is-bridged |grep FORWARD
|grep BF |grep '\-o' | grep -w cloudbr0|awk '{print $9}' | head -1
2025-03-17 07:26:58,283 - iptables -L BF-cloudbr0
2025-03-17 07:26:58,286 - iptables -L BF-cloudbr0-OUT
2025-03-17 07:26:58,289 - iptables -L BF-cloudbr0-IN
2025-03-17 07:26:58,292 - ip6tables -L BF-cloudbr0
2025-03-17 07:26:58,295 - ip6tables -L BF-cloudbr0-OUT
2025-03-17 07:26:58,298 - ip6tables -L BF-cloudbr0-IN
2025-03-17 07:26:58,300 - bridge -o link show | awk '/master cloudbr0 / &&
!/^[0-9]+: vnet/ {print $2}' | head -1 | cut -d ':' -f1 | cut -d '@' -f1
2025-03-17 07:26:58,304 - iptables -n -L BF-cloudbr0 | awk '/BF-cloudbr0(.*)references/
{gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,307 - iptables -n -L BF-cloudbr0-IN | awk
'/BF-cloudbr0-IN(.*)references/ {gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,311 - iptables -n -L BF-cloudbr0-OUT | awk
'/BF-cloudbr0-OUT(.*)references/ {gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,314 - ip6tables -n -L BF-cloudbr0 | awk '/BF-cloudbr0(.*)references/
{gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,318 - iptables -I FORWARD -i cloudbr0 -j DROP
2025-03-17 07:26:58,320 - iptables -I FORWARD -o cloudbr0 -j DROP
2025-03-17 07:26:58,323 - iptables -I FORWARD -i cloudbr0 -m physdev
--physdev-is-bridged -j BF-cloudbr0
2025-03-17 07:26:58,326 - iptables -I FORWARD -o cloudbr0 -m physdev
--physdev-is-bridged -j BF-cloudbr0
2025-03-17 07:26:58,329 - iptables -A BF-cloudbr0 -m state --state
RELATED,ESTABLISHED -j ACCEPT
2025-03-17 07:26:58,332 - iptables -A BF-cloudbr0 -m physdev
--physdev-is-bridged --physdev-is-in -j BF-cloudbr0-IN
2025-03-17 07:26:58,335 - iptables -A BF-cloudbr0 -m physdev
--physdev-is-bridged --physdev-is-out -j BF-cloudbr0-OUT
2025-03-17 07:26:58,337 - iptables -A BF-cloudbr0 -m physdev
--physdev-is-bridged --physdev-out -j ACCEPT
2025-03-17 07:26:58,340 - Command exited non-zero: iptables -A BF-cloudbr0 -m
physdev --physdev-is-bridged --physdev-out -j ACCEPT
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py",
line 53, in execute
return check_output(cmd, shell=True).decode()
File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/lib64/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'iptables -A BF-cloudbr0 -m physdev
--physdev-is-bridged --physdev-out -j ACCEPT' returned non-zero exit status 2.
2025-03-17 07:26:58,341 - iptables -F BF-cloudbr0
2025-03-17 07:26:58,344 - ip6tables -F BF-cloudbr0
The exception that's thrown is caused by the error:
Bad argument `ACCEPT'
Just to confirm that the rules set by default_network_rules were causing the
issue after a VM was started I tried adding:
iptables -I FORWARD -i cloudbr0 -j ACCEPT
iptables -I FORWARD -i cloudbr0 -j ACCEPT
and communications between VM's starting working, including DHCP. I also tried
setting rules in Security Group but it appears that they're not added to
iptables because the VM specific table doesn't exist.
I was hoping that the file at
/usr/share/cloudstack-common/scripts/vm/network/security_group.py was actually the one
that was being invoked but it's not... I guess it's a resource in the agent package. If
it was then I thought I could kind of "bludgeon" a solution by adding:
iptables -I FORWARD -i cloudbr0 -j ACCEPT
iptables -I FORWARD -i cloudbr0 -j ACCEPT
in default_network_rules.
In your notes on installing CloudStack the suggestion of setting:
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0
in sysctl.conf doesn't work because default_network_rules always re-enables
them.
So I think something isn't working right with CloudStack. It appears to be
ignoring security group rules, and at least on of the iptables rules appear to
have invalid syntax. If it was possible to modify security_group.py then I
could at least make it work. But default_network_rules is called quite often,
like whenever a VM is started so I can't think of anyway to add firewall rules
that won't be replaced by the rules from default_network_rules.
Thanks,
Dave
On 03-16-25 12:35 a.m., Rohit Yadav wrote:
Hi Dave,
Can you share your OS and other env details? For ACS 4.20 you don't have to
seed the systemvmtemplate manually.
My notes on installing CloudStack are here -
https://rohityadav.cloud/blog/cloudstack-kvm/ if you're trying this on Ubuntu.
Regards.
________________________________
From: Dave Lapointe <da...@island.net.INVALID>
Sent: Sunday, March 16, 2025 06:12
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Looking for suggestion on network configuration - followup
Hi,
I decided to go back to the basic CloudStack installation as described in the
Quick Installation Guide, and followed every instruction there (there is an
error with
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
-m /export/secondary -u
http://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.0.0-x86_64-kvm.qcow2.bz2
-h kvm -F. The SystemVMtemplate should be
systemvmtemplate-4.20.0-x86_64-kvm.qcow2.bz2).
After completing the install and observing that there were no errors in
management-server.log and agent.log I assumed everything was setup correctly.
Once the two system VM's started up I ssh'ed into them, (they are bridged
through cloudbr0 and cloud0) and found that I couldn't ping on the subnet for
cloudbr0 but could ping on the cloud0 subnet. I deleted all of the firewall
rules and set the default policies to ACCEPT but it made no difference. The
routing tables appeared to be okay, with traffic for the cloudbr0 subnet being
routed through the appropriate NIC.
On the host I tried tcpdump -i cloudbr0 -w cloudbr0.cap and Wireshark showed
that the ICMP packets are arriving on cloudbr0 but there is no response. I
tried tcpdump on one of the system VM's and confirmed that there is a response
to pinging the cloudbr0 ip address, and that there was no response from the
ping target.
I've done lots of bridging in different environments, and haven't encountered
anything like this before, so I'm at a loss to know how to continue
investigating this.
I've spent a few days on this now and have made no progress. Please, any
suggestions? I'm close to the point now of abandoning CloudStack because I
can't get it to work as documented.
Thanks,
Dave