MitchDrage commented on issue #14687:
URL: https://github.com/apache/cloudstack/issues/14687#issuecomment-7007831127

   This will be quite a large feature, so I went through the process of 
developing my own list of requirements based on my experience, reviewed how 
that compared to the hyperscale cloud providers' approach, and then 
independently had Claude Ultracode develop a spec and run its council over the 
compiled list with voting personas as:
   
   1. Cloud operator / product owner.
   2. Customer network engineer.
   3. Security and multi-tenant isolation reviewer.
   4. Apache CloudStack maintainer.
   5. VR / systemvm implementation engineer.
   
   Design diagrams, including three UI mock-ups, and an API spec are attached 
for reference as well.
   
   I've developed a series of decisions to share for this feature. Included 
below are the main ones, and the full list with the rationale and alternatives 
for each is in the attached file. I'm happy to discuss/change any of them.
   
   As you can tell there's some heavy Claude usage here to develop this spec - 
no one is going to write 1100 lines of spec by hand. It has streamlined the 
design approach for this and helped me to develop some very detailed 
collateral, along with pre-testing a significant amount of these decisions - 
ruling out lots of options and saving me a significant amount of time in the 
process.
   I have hand reviewed and worked on these decisions in detail over many days 
and I'm comfortable that they represent fairly sound logic, however of course I 
would I'd appreciate a detailed review and feedback from maintainers prior to 
jumping into any code and testing.
   
   The path to implement this will take some time. Some bugs I've already 
reproduced go first as small fixes, and then I'd deliver this through 
manageably sized PRs so they can be thoroughly reviewed and tested.
   
   Delivery:
   Small bug fixes first to 4.20/4.22 where they reproduce without BGP. These 
are listed below. The feature then goes to main for the release after 24.0, as 
four PRs linked from this issue:
     1. frr-reload, the template package and FRR logs in router diagnostics. 
Operator peers benefit from this one on its own.
     2. Core: API, schema, management server, VR renderer, firewall and BFD.
     3. Status, the route table and the UI.
     4. Usage, reviewed with the usage maintainers.
   
     Each PR can merge on its own, and they merge in order. The offering flag 
that turns the feature on lands in the last PR, so a release that holds only 
part of the series exposes nothing to operators or tenants.
   
   Redundant ROUTED VPCs also depend on #14446.
   
   Decision summary:
   - API: a BGP flag on the private gateway, named apart from the offering's 
routingmode, with the BGP settings stored against the gateway and the peer 
address defaulting to the gateway's own gateway field, with multihop as an 
option. Both admin VLAN gateways and gateways on a tenant's own L2 network can 
use BGP. Dedicated commands update and list them, so operators can control 
access through roles.
   - Enablement: a new VPC offering capability that is off by default. It is 
independent of routingmode=Dynamic and works on NATTED and ROUTED VPCs.
   - Scope: IPv4 and one VPC first. BGP runs between a VPC and on-prem routers 
only, and routes learned from one peer are never sent to another. IPv6 needs 
these first:
     - IPv6 addressing on private gateways. createPrivateGateway only accepts 
an IPv4 gateway, netmask and IP address.
     - IPv6 static routes. createStaticRoute only accepts IPv4 CIDRs.
     - IPv6 firewall rules on the gateway NIC so the VR accepts BGP.
     - #12811 settling NATTED IPv6.
   - Isolation: tenant and operator peers sit in separate peer groups in the 
existing router bgp. Every neighbour has mandatory prefix-lists and route-maps, 
and no routes pass between the two groups.
   - Inbound: the VR accepts the routes the customer announces, as AWS and 
Azure do. It always rejects link-local, its own connected subnets, the VPC's 
own ranges, the default route and very short prefixes. A max-prefix limit 
counts accepted routes and trips as soon as it is passed. Its default is 1,000, 
which a load test on a small VR handled with no session drops. A tripped 
session restarts on its own after an interval, 5 minutes by default. An 
optional import filter per gateway is a future capability.
   - Local AS: every VPC presents one operator-set AS to customer routers, as 
Azure and AWS do. The VPC owner can override it with a private ASN when it 
clashes with their network. No public ASN is needed. Dynamic ROUTED VPCs keep 
their allocated AS for operator peers.
   - Peer AS: any valid ASN except the VPC's own AS, reserved values and an 
operator deny list of ASNs and ranges.
   - Outbound: only the tiers the tenant selects for that peer. A follow-up 
release adds the remote-access VPN client range and the tiers of peered VPCs to 
the same list. VPCs connect to each other with static routes over private 
gateways. Site-to-site VPN stays blocked.
   - Failover: a priority per gateway from 1 to 4, where equal priorities share 
load, tenant timers within operator bounds (keepalive 3 s and hold 9 s at 
minimum), and BFD down to 300 ms x 3. ECMP needs the same peer ASN on both 
links.
   - Redundant VRs: sessions follow the primary. A keepalived switchover lab 
test passed in NATTED mode and in ROUTED mode with #14446, which redundant 
ROUTED VPCs need.
   - Visibility: a Routing tab on the VPC shows the effective route table, and 
each gateway shows its accepted and advertised routes, why the session last 
went down, and what was negotiated.
   - Gateway ACL: the gateway's ACL filters all traffic on a BGP link. A 
gateway on `default_deny` brings its session up and installs routes but passes 
no traffic until the tenant attaches an ACL that allows it. The status shows 
this.
   - Operator controls: an admin can hold a session down, and the operator can 
disable tenant BGP for an account. Each hold stays until the party that set it 
clears it, and the status lists the holds that apply. Operator-wide limits are 
root-admin only.
   - Events: customers get session up and down events through webhooks. 
Operators get one alert when more than half of a zone's sessions drop in one 
check interval, since that points at the platform.
   - Hypervisors: KVM, VMware, XenServer and Hyper-V, within each hypervisor's 
existing NIC limit.
   - Usage: a usage record per BGP gateway, in the same way as VPN users.
   
   Full list of Design Decisions:
   
[bgp-private-gateway-decisions.md](https://github.com/user-attachments/files/35082816/bgp-private-gateway-decisions.md)
   
   API Spec:
   
[bgp-private-gateway-api-spec.md](https://github.com/user-attachments/files/35082809/bgp-private-gateway-api-spec.md)
   
   Design diagrams:
   1. Physical/Logical topology - In this case the example is using VXLAN/EVPN.
   <img width="2480" height="1320" alt="01-topology" 
src="https://github.com/user-attachments/assets/eddbd903-75d6-4ac3-ac7d-b88a01c4de8a";
 />
   
   
   2. Supported gateway types and deployment
   <img width="2420" height="1400" alt="02-gateway-types" 
src="https://github.com/user-attachments/assets/7454467d-0458-4504-a271-77610652952d";
 />
   
   3. Peer group arrangement to separate operator upstream routing and private 
routing
   <img width="2440" height="1400" alt="03-bgp-process" 
src="https://github.com/user-attachments/assets/1137409b-4a3b-42e9-ad72-c6fb578015b7";
 />
   
   4. Route filtering approach
   <img width="2600" height="1668" alt="04-route-filtering" 
src="https://github.com/user-attachments/assets/2d69df94-69dc-4658-92c7-d2473e7f889f";
 />
   
   5. Failover process/timers
   <img width="2440" height="1300" alt="05-failover" 
src="https://github.com/user-attachments/assets/06963edc-f04d-47a4-9599-6b50a4f31fca";
 />
   
   6. VR config flow
   <img width="2520" height="2332" alt="06-config-flow" 
src="https://github.com/user-attachments/assets/4708d784-e7fc-4ef2-88b0-8b5a858eed96";
 />
   
   7. VR Firewall
   <img width="2440" height="1560" alt="07-vr-firewall" 
src="https://github.com/user-attachments/assets/e2eac3dc-4782-4474-8ca2-49c73d9957e3";
 />
   
   8. NATted vs Routed VPCs
   <img width="2440" height="1520" alt="08-natted-routed" 
src="https://github.com/user-attachments/assets/647eaa29-f96e-4e3e-b727-fc3ade534795";
 />
   
   9. Redundant VR switchover
   <img width="2440" height="1480" alt="09-redundant-vr" 
src="https://github.com/user-attachments/assets/e9d54895-0910-46af-b2de-0637a91cbca6";
 />
   
   10. API and Data model
   <img width="2440" height="1520" alt="10-data-model" 
src="https://github.com/user-attachments/assets/0c760712-69f9-4a74-8aaa-4fcf5995c8b7";
 />
   
   UI mockups:
   1. VPC Routing tab
   <img width="2560" height="1852" alt="11a-ui-vpc-routing" 
src="https://github.com/user-attachments/assets/88d55156-9576-48f8-8622-5a0527517572";
 />
   
   2. Private Gateway BGP - Received Routes tab
   <img width="2560" height="1852" alt="11b-ui-gateway-bgp" 
src="https://github.com/user-attachments/assets/0e13a392-8114-44df-b443-ecbd8d17be13";
 />
   
   3. Private Gateway BGP - Advertised Routes tab
   <img width="2560" height="1852" alt="11c-ui-gateway-advertised" 
src="https://github.com/user-attachments/assets/83df8975-a52a-4897-8228-8724cb923131";
 />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to