kiranchavala opened a new issue, #14077: URL: https://github.com/apache/cloudstack/issues/14077
### problem Network Extension feature doesn't persist the vlan information ### versions ACS 4.23, KVM ### The steps to reproduce the bug Follow the steps mentioned here to register network extension https://github.com/apache/cloudstack-extensions/blob/network-namespace/Network-Namespace/README.md 1. Create a network offering with vlan enabled 2. Create a network based on the above network offering 3. Launch a vm in the network > Vm deployed successfully 4. Destroy the vm After some time the network state changes from implemented to allocated since there are no vm's 5. Check the networks table, the brodcast_uri changes to NULL 6. The subsequent vm launch will result in a failure logs ``` root@Cloudstack-423:/var/log/cloudstack/management# cat /var/log/cloudstack/management/management-server.log.2026-09-07 | grep -i "logid:63df11b4" | grep -iE "Script output|failed with exit code|Cannot find|Ensuring network device|Implementing network extension|applyFWRules|Applying|dhcp|dns" 2026-09-07 11:14:23,762 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-49:[ctx-564cbc63, job-1216, ctx-099a3943, FirstFitRoutingAllocator]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:24,714 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:24,716 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:24,717 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:24,756 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Implementing network extension for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (VLAN null) 2026-09-07 11:14:24,756 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={}) 2026-09-07 11:14:25,306 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:25,316 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={"host":"192.168.55.179","namespace":"cs-net-227"}) 2026-09-07 11:14:25,628 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:27,212 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Script output: Cannot find device "vn--227" 2026-09-07 11:14:27,212 ERROR [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Network extension script implement-network failed with exit code 3: Cannot find device "vn--227" 2026-09-07 11:14:27,261 DEBUG [c.c.n.f.FirewallManagerImpl] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) applying default firewall egress rules 2026-09-07 11:14:27,263 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) applyFWRules: network=Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} activeRules=0 defaultEgressAllow=true 2026-09-07 11:14:27,263 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={"host":"192.168.55.179","namespace":"cs-net-227"}) 2026-09-07 11:14:27,565 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:29,149 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Script output: apply-fw-rules: built 0 ingress rule(s) across 0 public IP(s), 0 egress rule(s), default_egress=ALLOW, default-DROP chains for 0 static-NAT IP(s) with no rules 2026-09-07 11:14:29,149 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Ignoring non-JSON script output: apply-fw-rules: built 0 ingress rule(s) across 0 public IP(s), 0 egress rule(s), default_egress=ALLOW, default-DROP chains for 0 static-NAT IP(s) with no rules 2026-09-07 11:14:29,155 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Applying 1 IPs for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} 2026-09-07 11:14:29,156 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={"host":"192.168.55.179","namespace":"cs-net-227"}) 2026-09-07 11:14:29,463 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:31,012 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={"host":"192.168.55.179","namespace":"cs-net-227"}) 2026-09-07 11:14:31,319 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:33,117 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:33,117 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-63:[ctx-806840fb, job-1216/job-1217, ctx-f951d8a4]) (logid:63df11b4) Hosts's actual total RAM: (6.57 GB) 7053959168 and RAM after applying overprovisioning: (6.57 GB) 7053959168 2026-09-07 11:14:34,682 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32, FirstFitRoutingAllocator]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:34,709 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:34,711 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:34,711 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:34,751 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Implementing network extension for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (VLAN null) 2026-09-07 11:14:34,751 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={}) 2026-09-07 11:14:35,058 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:35,084 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={"host":"192.168.55.179","namespace":"cs-net-227"}) 2026-09-07 11:14:35,391 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:36,929 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Script output: Cannot find device "vn--227" 2026-09-07 11:14:36,929 ERROR [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Network extension script implement-network failed with exit code 3: Cannot find device "vn--227" 2026-09-07 11:14:36,978 DEBUG [c.c.n.f.FirewallManagerImpl] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) applying default firewall egress rules 2026-09-07 11:14:36,982 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) applyFWRules: network=Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} activeRules=0 defaultEgressAllow=true 2026-09-07 11:14:36,982 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={"host":"192.168.55.179","namespace":"cs-net-227"}) 2026-09-07 11:14:37,288 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:38,864 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Script output: apply-fw-rules: built 0 ingress rule(s) across 0 public IP(s), 0 egress rule(s), default_egress=ALLOW, default-DROP chains for 0 static-NAT IP(s) with no rules 2026-09-07 11:14:38,864 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Ignoring non-JSON script output: apply-fw-rules: built 0 ingress rule(s) across 0 public IP(s), 0 egress rule(s), default_egress=ALLOW, default-DROP chains for 0 static-NAT IP(s) with no rules 2026-09-07 11:14:38,868 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Applying 1 IPs for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} 2026-09-07 11:14:38,868 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={"host":"192.168.55.179","namespace":"cs-net-227"}) 2026-09-07 11:14:39,170 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:40,738 INFO [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Ensuring network device for network Network {"id": 227, "name": "test6", "uuid": "2e690dff-8d4e-4603-8454-7e06672bdc56", "networkofferingid": 40} (current={"host":"192.168.55.179","namespace":"cs-net-227"}) 2026-09-07 11:14:41,046 DEBUG [o.a.c.f.e.n.NetworkExtensionElement] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Script output: {"host":"192.168.55.179","namespace":"cs-net-227"} 2026-09-07 11:14:42,881 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Hosts's actual total CPU: 12748 and CPU after applying overprovisioning: 12748 2026-09-07 11:14:42,881 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-64:[ctx-69cc0292, job-1216/job-1218, ctx-c4e49f32]) (logid:63df11b4) Hosts's actual total RAM: (6.57 GB) 7053959168 and RAM after applying overprovisioning: (6.57 GB) 7053959168 ``` ``` *************************** 28. row *************************** id: 227 name: test6 uuid: 2e690dff-8d4e-4603-8454-7e06672bdc56 display_text: test6 traffic_type: Guest broadcast_domain_type: Vlan broadcast_uri: NULL gateway: 10.1.1.1 cidr: 10.1.1.0/24 mode: Dhcp network_offering_id: 40 physical_network_id: 200 data_center_id: 1 guru_name: NetworkExtensionGuestNetworkGuru state: Allocated related: 227 domain_id: 1 account_id: 2 dns1: 8.8.8.8 dns2: NULL ip6dns1: NULL ip6dns2: NULL guru_data: NULL set_fields: 0 acl_type: Account network_domain: cs2cloud.internal reservation_id: f9fe4b5c-831e-4903-9258-7ab5338e27bd guest_type: Isolated restart_required: 0 created: 2026-09-07 10:31:15 removed: NULL specify_ip_ranges: 0 vpc_id: NULL ip6_gateway: NULL ip6_cidr: NULL network_cidr: NULL display_network: 1 network_acl_id: NULL streched_l2: 0 redundant: 0 external_id: NULL public_mtu: 1500 private_mtu: 1500 keep_mac_address_on_public_nic: 1 ``` ### What to do about it? The vlan information of the guest network should be persisted in the database so that the vlan Root cause found ``` NetworkExtensionGuestNetworkGuru.implement() (server/src/main/java/com/cloud/network/guru/NetworkExtensionGuestNetworkGuru.java:104-120): public Network implement(final Network network, final NetworkOffering offering, final DeployDestination dest, final ReservationContext context) throws InsufficientVirtualNetworkCapacityException { ... // The broadcast_domain_type and broadcast_uri are set by the extension script output final NetworkVO implemented = getNetworkVO(network, offering, physicalNetworkId); return implemented; } private static NetworkVO getNetworkVO(Network network, NetworkOffering offering, Long physicalNetworkId) { final NetworkVO implemented = new NetworkVO(...); ... if (network.getBroadcastUri() != null) { implemented.setBroadcastUri(network.getBroadcastUri()); // only copies through — never allocates } return implemented; } ``` Compare to the parent it extends, GuestNetworkGuru.implement() (server/src/main/java/com/cloud/network/guru/GuestNetworkGuru.java:398), which calls allocateVnet(): ``` protected void allocateVnet(final Network network, final NetworkVO implemented, final long dcId, final long physicalNetworkId, final String reservationId) throws InsufficientVirtualNetworkCapacityException { if (network.getBroadcastUri() == null) { final String vnet = _dcDao.allocateVnet(dcId, physicalNetworkId, network.getAccountId(), reservationId, ...); implemented.setBroadcastUri(BUri(vnet)); // auto-allocates a fresh VLAN } else { implemented.setBroadcastUri(n } } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
