Hi Rohit,

Thanks for the link.  I'm running ACS 4.20 on Rocky 8.  I did a "basic" setup 
when first connecting to the CloudStack UI.  Hardware is Dell PowerEdge 450 with 64GB 
RAM.  I've tried Windows 10 and Debian Linux VM's.

I did some more checking and found that the firewall rules for the bridge are 
being set in 
/usr/share/cloudstack-common/scripts/vm/network/security_group.py:default_network_rules
 where it blocks all traffic between VM's and then fails to define VM specific 
rules because a table doesn't exist.  Here is the log from agent.log:

2025-03-17 07:26:58,267 - Executing command: default_network_rules
2025-03-17 07:26:58,267 - modprobe br_netfilter
2025-03-17 07:26:58,270 - sysctl -w net.bridge.bridge-nf-call-arptables=1
2025-03-17 07:26:58,273 - sysctl -w net.bridge.bridge-nf-call-iptables=1
2025-03-17 07:26:58,276 - sysctl -w net.bridge.bridge-nf-call-ip6tables=1
2025-03-17 07:26:58,278 - iptables-save |grep physdev-is-bridged |grep FORWARD 
|grep BF |grep '\-o' | grep -w cloudbr0|awk '{print $9}' | head -1
2025-03-17 07:26:58,283 - iptables -L BF-cloudbr0
2025-03-17 07:26:58,286 - iptables -L BF-cloudbr0-OUT
2025-03-17 07:26:58,289 - iptables -L BF-cloudbr0-IN
2025-03-17 07:26:58,292 - ip6tables -L BF-cloudbr0
2025-03-17 07:26:58,295 - ip6tables -L BF-cloudbr0-OUT
2025-03-17 07:26:58,298 - ip6tables -L BF-cloudbr0-IN
2025-03-17 07:26:58,300 - bridge -o link show | awk '/master cloudbr0 / && 
!/^[0-9]+: vnet/ {print $2}' | head -1 | cut -d ':' -f1 | cut -d '@' -f1
2025-03-17 07:26:58,304 - iptables -n -L BF-cloudbr0 | awk '/BF-cloudbr0(.*)references/ 
{gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,307 - iptables -n -L BF-cloudbr0-IN | awk 
'/BF-cloudbr0-IN(.*)references/ {gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,311 - iptables -n -L BF-cloudbr0-OUT | awk 
'/BF-cloudbr0-OUT(.*)references/ {gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,314 - ip6tables -n -L BF-cloudbr0 | awk '/BF-cloudbr0(.*)references/ 
{gsub(/\(/, "") ;print $3}'
2025-03-17 07:26:58,318 - iptables -I FORWARD -i cloudbr0 -j DROP
2025-03-17 07:26:58,320 - iptables -I FORWARD -o cloudbr0 -j DROP
2025-03-17 07:26:58,323 - iptables -I FORWARD -i cloudbr0 -m physdev 
--physdev-is-bridged -j BF-cloudbr0
2025-03-17 07:26:58,326 - iptables -I FORWARD -o cloudbr0 -m physdev 
--physdev-is-bridged -j BF-cloudbr0
2025-03-17 07:26:58,329 - iptables -A BF-cloudbr0 -m state --state 
RELATED,ESTABLISHED -j ACCEPT
2025-03-17 07:26:58,332 - iptables -A BF-cloudbr0 -m physdev 
--physdev-is-bridged --physdev-is-in -j BF-cloudbr0-IN
2025-03-17 07:26:58,335 - iptables -A BF-cloudbr0 -m physdev 
--physdev-is-bridged --physdev-is-out -j BF-cloudbr0-OUT
2025-03-17 07:26:58,337 - iptables -A BF-cloudbr0 -m physdev 
--physdev-is-bridged --physdev-out  -j ACCEPT
2025-03-17 07:26:58,340 - Command exited non-zero: iptables -A BF-cloudbr0 -m 
physdev --physdev-is-bridged --physdev-out  -j ACCEPT
Traceback (most recent call last):
  File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", 
line 53, in execute
    return check_output(cmd, shell=True).decode()
  File "/usr/lib64/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib64/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'iptables -A BF-cloudbr0 -m physdev 
--physdev-is-bridged --physdev-out  -j ACCEPT' returned non-zero exit status 2.
2025-03-17 07:26:58,341 - iptables -F BF-cloudbr0
2025-03-17 07:26:58,344 - ip6tables -F BF-cloudbr0

The exception that's thrown is caused by the error:
Bad argument `ACCEPT'

Just to confirm that the rules set by default_network_rules were causing the 
issue after a VM was started I tried adding:

iptables -I FORWARD -i cloudbr0 -j ACCEPT
iptables -I FORWARD -i cloudbr0 -j ACCEPT

and communications between VM's starting working, including DHCP.  I also tried 
setting rules in  Security Group but it appears that they're not added to 
iptables because the VM specific table doesn't exist.

I was hoping that the file at 
/usr/share/cloudstack-common/scripts/vm/network/security_group.py was actually the one 
that was being invoked but it's not... I guess it's a resource in the agent package.  If 
it was then I thought I could kind of "bludgeon" a solution by adding:

iptables -I FORWARD -i cloudbr0 -j ACCEPT
iptables -I FORWARD -i cloudbr0 -j ACCEPT

in default_network_rules.

In your notes on installing CloudStack the suggestion of setting:

net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-iptables = 0

in sysctl.conf doesn't work because default_network_rules always re-enables 
them.

So I think something isn't working right with CloudStack.  It appears to be 
ignoring security group rules, and at least on of the iptables rules appear to 
have invalid syntax.  If it was possible to modify security_group.py then I 
could at least make it work.  But default_network_rules is called quite often, 
like whenever a VM is started so I can't think of anyway to add firewall rules 
that won't be replaced by the rules from default_network_rules.

Thanks,
Dave


On 03-16-25 12:35 a.m., Rohit Yadav wrote:
Hi Dave,

Can you share your OS and other env details? For ACS 4.20 you don't have to 
seed the systemvmtemplate manually.

My notes on installing CloudStack are here - 
https://rohityadav.cloud/blog/cloudstack-kvm/ if you're trying this on Ubuntu.

Regards.

________________________________
From: Dave Lapointe <da...@island.net.INVALID>
Sent: Sunday, March 16, 2025 06:12
To: users@cloudstack.apache.org <users@cloudstack.apache.org>
Subject: Looking for suggestion on network configuration - followup

Hi,

I decided to go back to the basic CloudStack installation as described in the 
Quick Installation Guide, and followed every instruction there (there is an 
error with 
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt 
-m /export/secondary -u 
http://download.cloudstack.org/systemvm/4.20/systemvmtemplate-4.20.0.0-x86_64-kvm.qcow2.bz2
 -h kvm -F.  The SystemVMtemplate should be 
systemvmtemplate-4.20.0-x86_64-kvm.qcow2.bz2).

After completing the install and observing that there were no errors in 
management-server.log and agent.log I assumed everything was setup correctly.  
Once the two system VM's started up I ssh'ed into them, (they are bridged 
through cloudbr0 and cloud0) and found that I couldn't ping on the subnet for 
cloudbr0 but could ping on the cloud0 subnet.  I deleted all of the firewall 
rules and set the default policies to ACCEPT but it made no difference.  The 
routing tables appeared to be okay, with traffic for the cloudbr0 subnet being 
routed through the appropriate NIC.

On the host I tried tcpdump -i cloudbr0 -w cloudbr0.cap and Wireshark showed 
that the ICMP packets are arriving on cloudbr0 but there is no response.  I 
tried tcpdump on one of the system VM's and confirmed that there is a response 
to pinging the cloudbr0 ip address, and that there was no response from the 
ping target.

I've done lots of bridging in different environments, and haven't encountered 
anything like this before, so I'm at a loss to know how to continue 
investigating this.

I've spent a few days on this now and have made no progress.  Please, any 
suggestions?  I'm close to the point now of abandoning CloudStack because I 
can't get it to work as documented.

Thanks,
Dave


Reply via email to