GitHub user muthukrishnang1100 created a discussion: CloudStack 4.20.3.0 - Inter-VPC Routing via Private Gateway Does Not Support Multiple Next-Hops in Hub-and-Spoke Architecture
Environment CloudStack Version: 4.20.3.0 VPC Offering: Redundant VPC Transit VLAN: 1931 Transit Network: 10.31.252.0/22 Private Gateways used for Inter-VPC communication Architecture Common-Hub VPC CIDR: 10.30.64.0/23 Private Gateway IP: 10.31.252.5 Gateway: 10.31.252.1 Development VPC CIDR: 10.30.70.0/24 Private Gateway IP: 10.31.252.6 Gateway: 10.31.252.5 Cloud-Test VPC CIDR: 10.30.74.0/24 Private Gateway IP: 10.31.252.7 Gateway: 10.31.252.5 The design is Hub-and-Spoke. Common-Hub VPC provides shared services such as: DNS LDAP Monitoring Backup Shared Infrastructure Services Development and Cloud-Test VPCs should communicate with Common-Hub through the transit network. Expected Behaviour Common-Hub VPC should be able to reach: Development VPC (10.30.70.0/24) via next-hop 10.31.252.6 Cloud-Test VPC (10.30.74.0/24) via next-hop 10.31.252.7 Expected routing on Common-Hub VR: 10.30.70.0/24 via 10.31.252.6 10.30.74.0/24 via 10.31.252.7 This would allow bidirectional communication between Hub and Spoke VPCs. Actual Behaviour When static routes are created on the Common-Hub Private Gateway through the CloudStack UI, only the Destination CIDR field is available. There is no option to specify a route-specific next-hop IP. Static Routes configured: 10.30.70.0/24 10.30.74.0/24 CloudStack automatically generates the following routes on the VPC Router: 10.30.70.0/24 via 10.31.252.1 dev eth4 10.30.74.0/24 via 10.31.252.1 dev eth4 The next-hop is automatically taken from the Private Gateway "Gateway" field (10.31.252.1). However, the correct next-hops should be: 10.30.70.0/24 via 10.31.252.6 10.30.74.0/24 via 10.31.252.7 As a result: Development VPC -> Common-Hub VPC = SUCCESS Cloud-Test VPC -> Common-Hub VPC = SUCCESS Common-Hub VPC -> Development VPC = FAILED Common-Hub VPC -> Cloud-Test VPC = FAILED Proof 1 - Generated Route on Common-Hub VR Output from active Common-Hub VPC Router: ip route Output: 10.30.70.0/24 via 10.31.252.1 dev eth4 10.30.74.0/24 via 10.31.252.1 dev eth4 These routes are generated automatically by CloudStack. Proof 2 - Transit Network Connectivity Exists >From Common-Hub VR: ping 10.31.252.6 Result: 64 bytes from 10.31.252.6 0% packet loss >From Common-Hub VR: ping 10.31.252.7 Result: 64 bytes from 10.31.252.7 0% packet loss This proves: Transit VLAN is working Layer 2 connectivity exists Layer 3 connectivity exists Private Gateways can reach each other Proof 3 - Manual Route Fix Works When manually adding routes on the active Common-Hub VPC Router: ip route add 10.30.70.0/24 via 10.31.252.6 ip route add 10.30.74.0/24 via 10.31.252.7 Communication immediately starts working. Common-Hub -> Development = SUCCESS Common-Hub -> Cloud-Test = SUCCESS Additional Problem Manual routes are not persistent. The manually added routes disappear after: VPC Router restart VPC Router failover Management Server restart Router reconfigure Router stop/start CloudStack regenerates the routes and restores: 10.30.70.0/24 via 10.31.252.1 10.30.74.0/24 via 10.31.252.1 which breaks communication again. Database Observation Table: static_routes Columns: id uuid vpc_gateway_id cidr state vpc_id account_id domain_id created There does not appear to be any column that stores a route-specific next-hop IP. This suggests CloudStack derives the next-hop from the Private Gateway gateway field rather than storing a next-hop per route. Question Is this expected behaviour in CloudStack 4.20.3.0? If yes, how should a Hub-and-Spoke Inter-VPC architecture be implemented when a Hub VPC must route to multiple spoke VPCs using different Private Gateway IP addresses? If not, is this a bug? Feature Request Please consider supporting route-specific next-hop selection for VPC Private Gateway Static Routes through: CloudStack UI API Database model This would allow implementation of standard Hub-and-Spoke Inter-VPC routing designs using Private Gateways without requiring manual route injection on VPC Routers. GitHub link: https://github.com/apache/cloudstack/discussions/13362 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
