Hi, Op 10-05-2022 om 11:42 schreef Abhishek Kumar:
Yes. When a public IPv6 is assigned or released, CloudStack will publish event with type NET.IP6ASSIGN, NET.IP6RELEASE. These event notifications can be tracked. And with improvements in events framework, these event messages will have network uuid as entityuuid and Network as entitytype. Using this network can be queried using to list IPv6 routes that need to be added.Sample event message, {"eventDateTime":"2022-05-10 09:32:12 +0000","entityuuid":"14658b39-9d20-4783-a1bc-12fb58bcbd98","Network":"14658b39-9d20-4783-a1bc-12fb58bcbd98","description":"Assigned public IPv6 address: fd23:313a:2f53:3000:1c00:baff:fe00:4 for network ID: 14658b39-9d20-4783-a1bc-12fb58bcbd98","event":"NET.IP6ASSIGN","user":"bde866ba-c600-11ec-af19-1e00320001f3","account":"bde712c9-c600-11ec-af19-1e00320001f3","entity":"Network","status":"Completed"} ?Sample API call,list networks id=14658b39-9d20-4783-a1bc-12fb58bcbd98 filter=id,name,ip6routes{ "count": 1, "network": [ { "id": "14658b39-9d20-4783-a1bc-12fb58bcbd98", "ip6routes": [ { "gateway": "fd23:313a:2f53:3000:1c00:baff:fe00:4", "subnet": "fd23:313a:2f53:3cbf::/64" }
Looking at this JSON, does this mean that fd23:313a:2f53:3000:1c00:baff:fe00:4 is the WAB address of the VR?
And that I would need to (statically) route fd23:313a:2f53:3cbf::/64 to that IP?
The event message does not include the subnet, that makes it a bit more difficult as you would then also need to do a API-call to gather that information.
WidoP.S.: Who controls the DNS of qa.cloudstack.cloud? It lacks an AAAA-record for IPv6!
], "name": "routing_test" } ] } ________________________________ From: Wido den Hollander <w...@widodh.nl> Sent: 10 May 2022 13:59 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org>; Abhishek Kumar <abhishek.ku...@shapeblue.com> Subject: Re: IPV6 in Isolated/VPC networks Op 10-05-2022 om 10:19 schreef Abhishek Kumar:Hi all, IPv6 Support in Isolated Network and VPC with Static Routing based on the design doc [1] has been implemented and is available in 4.17.0 RC2. I hope while testing 4.17.0 RC2 you will also try to test it ? Documentation for it is available at http://qa.cloudstack.cloud/docs/WIP-PROOFING/pr/262/plugins/ipv6.html#isolated-network-and-vpc-tier (will be available in the official docs once 4.17.0 version of docs is built).Great work! I see only static routing is supported. But do we publish something on the message bus once a new VR/VPC is created? This way you could pick up these messages and have the network create a (static) route based on those. ExaBGP for example could be used to inject such routes. Wido[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing Regards, Abhishek ________________________________ From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 13 September 2021 14:30 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Thanks Alex, Wei. I've updated the docs here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in+Isolated+Network+and+VPC+with+Static+Routing I'll leave the thread open for futher discussion/ideas/feedback. I think we've completed the phase1 design doc including all feedback comments for adding IPv6 support in CloudStack and some initial poc/work can be started. My colleagues and I will keep everyone posted on this thread and/or on a Github PR as and when we're able to start our work on the same (after 4.16, potentially towards 4.17). Regards. ________________________________ From: Wei ZHOU <ustcweiz...@gmail.com> Sent: Friday, September 10, 2021 20:22 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Agree with Alex. We only need to know how many /64 are allocated. We do not care how many ipv6 addresses are used by VMs. -Wei On Fri, 10 Sept 2021 at 16:36, Alex Mattioli <alex.matti...@shapeblue.com> wrote:Hi Rohit, I'd go for option 2, don't see a point tracking anything smaller than a /64 tbh. Cheers Alex -----Original Message----- From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 09 September 2021 12:44 To: dev@cloudstack.apache.org Subject: Re: IPV6 in Isolated/VPC networks Thanks Alex, Kristaps. I've updated the design doc to reflect two agreements: * Allocate /64 for both isolated network and VPC tiers, no large allocation of prefixes to VPC (cons: more static routing rules for upstream router/admins) * All systemvms (incl. ssvm, cpvm, VRs) get IPv6 address if zone has a dedicated /64 prefix/block for systemvms The only outstanding question now is: * How do we manage IPv6 usage? Can anyone advise how we do IPv6 usage for shared network (design, implementation and use-cases?) Option1: We don't do it, all user VMs nics have ipv4 address which whose usage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address. Option2: Implement a separate, first-class IPv6 address or /64 prefix tracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records) Option3: other thoughts? Regards. ________________________________ From: Alex Mattioli <alex.matti...@shapeblue.com> Sent: Wednesday, September 8, 2021 23:24 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: RE: IPV6 in Isolated/VPC networks Hi Rohit, Kristaps, I'd say option 1 as well, it does create a bit more overhead with static routes but if that's automated for a VPC it can also be easily automated for several tiers of a VPC. We also don't constrain the amount of tiers in a VPC. It has the added advantage to be closer to the desired behaviour with dynamic routing in the future, where a VPC VR can announce several subnets upstream. Cheers Alex -----Original Message----- From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: 08 September 2021 19:04 To: dev@cloudstack.apache.org Subject: Re: IPV6 in Isolated/VPC networks Hi Kristaps, Thanks for sharing, I suppose that means individual tiers should be allocated /64 instead of larger ipv6 blocks to the whole VPC which could cause wastage. Any objection from anybody? Regards. ________________________________ From: Kristaps Cudars <kristaps.cud...@gmail.com> Sent: Wednesday, September 8, 2021 9:24:01 PM To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Hello, I asked networking team to comment on "How should the IPv6 block/allocation work in VPCs?" Option1: They haven't seen lately devices with limits on how many static routes can be created. Option2: With /60 and /62 assignments and big quantity of routers IPv6 assignment from RIPE NNC can be drained fast. /48 contains 64000 /64 /60 contains 16 /64 64000 / 16 = 4000 routers On 2021/09/07 11:59:09, Rohit Yadav <rohit.ya...@shapeblue.com> wrote:All, After another iteration with Alex, I've updated the design doc. Kindlyreview:https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in +Isolated+Network+and+VPC+with+Static+Routing ... and advise on some outstanding questions: * How should the IPv6 block/allocation work in VPCs? Option1: Should this be simply /64 allocation on any new tier, the cons of this option is one static route/rule per VPC tier. (many upstream routers may have limit on no. of static routes?) Option2: Let user ask/specify tier size, say /60 (for 16 tiers) or /62(4 tiers) for the VPC, this can be filtered based on the vpc.max.networks global setting (3 is default). The pros of this option are less no. of static route/rule and easy programming, but potential wastage of multiple /64 prefix blocks for unused/uncreated tiers.* How do we manage IPv6 usage? Can anyone advise how we do IPv6usage for shared network (design, implementation and use-cases?)Option1: We don't do it, all user VMs nics have ipv4 address which whoseusage we don't track. For public VR/nics/networks, we can simply add the IPv6 details for a related IPv4 address.Option2: Implement a separate, first-class IPv6 address or /64 prefixtracking/management and usage for all VMs and systemvms nic (this means account/domain level limits and new billing/records)* Enable IPv6 on systemvms (specifically SSVM and CPVM) by defaultif zone has a IPv6 address block allocated/assigned for use for systemvms (this was mainly thought for VRs, but why no ssvm and cpvms too - any cons of this?)* Regards. ________________________________ From: Rohit Yadav <rohit.ya...@shapeblue.com> Sent: Thursday, August 19, 2021 15:45 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Hi all, I've taken feedback from this thread and wrote this design doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+Support+in +Isolated+Network+and+VPC+with+Static+Routing Kindly review and advise if I missed anything or anything that needs tobe changed/updated. You may comment on the wiki directly too.Kindly suggest your views on the following (also in the design docabove):Outstanding Questions: * Should admin or user be able to specify how VPC super CIDRs arecreated/needed; for example a user can ask for VPC with /60 super CIDR? Or should CloudStack automatically find/allocate a /64 for a new VPC tier from the root-admin configured /64-/48 block?* Should we explore FRR and iBGP or other strategies to do dynamicrouting which may not require advance/complex configuration in the VR or for the users/admin?* With SLAAC and no dhcpv6, is there a way to support secondary IPv6addresses (or floating IPv6 addresses for VR/public traffic) for guest VM's nics?* Any thoughts on UI/UX for firewall/routing management? * Any other feature/support for isolated network or VPC feature thatmust be explored or supported such as PF, VPN, LB, vpc static routes, vpc gateway etc.* For usage - should we have any consideration, or should we assumethat IPv4 and IPv6 address will go together for every nic; so IPv6 usage for a nic is in tandem with Ipv4 address for a nic, i.e. no explicit/new biling/usage needed?* For smoketests, local dev-test should we explore ULA? Unique LocalAddress - in the range fc00::/7. Typically only within the 'local' half fd00::/8. ULA for IPv6 is analogous to IPv4 private network addressing. This prefix can be randomly generated at first install by CloudStack in a zone using zoneid etc?* Should we expand support for VR diagnostic feature to includesupport for ipv6, traceroute6?* Discuss - expand support/ability to allocate a /60, or /56 etcprefix to an individual VM in shared network (Wido's suggestion)Regards. ________________________________ From: Wei ZHOU <ustcweiz...@gmail.com> Sent: Tuesday, August 17, 2021 21:16 To: dev@cloudstack.apache.org <dev@cloudstack.apache.org> Subject: Re: IPV6 in Isolated/VPC networks Thanks Kristaps, Wido, Rohit and Alex for your replies. I am fine with not having stateful dhcpv6 and privacy extension/temporary address in phase 1. If community decides not to do eventually , it is also ok to me. We could explore how to better use secondary ipv6 addresses as Wido advised. It would be great if anyone share some user experience. -Wei On Tuesday, 17 August 2021, Kristaps Cudars <kristaps.cud...@gmail.com> wrote:Hi Wei, Published this month's RFC 9099 and explains in different words/perspective what has been written by Alex, Rohit and Wido. https://www.rfc-editor.org/rfc/rfc9099.html On 2021/08/17 09:20:21, Wei ZHOU <ustcweiz...@gmail.com> wrote:Hi Wido, (cc to Rohit and Alex) It is a good suggestion to use FRR for ipv6. The configuration is quite simple and the VMs can get SLAAC, routes, etc. Privacy extension looks not the same as what you mentioned. see https://datatracker.ietf.org/doc/html/rfc4941 You are right. To use static routing, the admins need to configure the routes in the upstream router, and add some ipv6 ranges (eg /56 for VPCs and /64 for isolated networks) and their next-hop (which will be configured in VRs) in CloudStack. CloudStack will pick up an IPv6 rangeandassign it to an isolated network or vpc. @Rohit, correct me if I'mwrong.I have a question, it looks stateless dhcpv6 (SLAAC from router/VR, router/dns etc via RA messages) will be the only option for now (relatedtoyour pr https://github.com/apache/cloudstack/pull/3077) . Would it begoodto provide stateful dhcpv6 (which can be implemented by dnsmasq) as an option in cloudstack ? The advantages are (1) support other ipv6 cidr sizes than /64. (2) we can assign a specified Ipv6 address to a vm. vm Ipv6 addresses can be changed (4) an Ipv6 addresses can be re-used by multiple vms. The problem is, stateful dhcpv6 does not support routers,nameservers,etc.we need to figure it out (probably use radvd/frr and dnsmasq both). -WeiOn Fri, 13 Aug 2021 at 12:19, Wido den Hollander <w...@widodh.nl> wrote:Hi, See my inline responses: Op 11-08-2021 om 14:26 schreef Rohit Yadav:Hi all, Thanks for your feedback and ideas, I've gone ahead with discussingthemwith Alex and came up with a PoC/design which can be implemented in the following phases:* Phase1: implement ipv6 support in isolated networks andVPCwithstatic routing* Phase2: discuss and implement support for dynamic routing(TBD)For Phase1 here's the high-level proposal: * IPv6 address management: * At the zone level root-admin specifies a /64 publicrangethatwill be used for VRs, then they can add a /48, or /56 IPv6 range forguestnetworks (to be used by isolated networks and VPC tiers)* On creation of any IPv6 enabled isolated network or VPCtier,from the /48 or /56 block a /64 network is allocated/used* We assume SLAAC and autoconfiguration, no DHCPv6 in thezone(discuss: is privacy a concern, can privacy extensions rfc4941 ofslaac beexplored?) Privacy Extensions are only a concern for client devices which roam between different IPv6 networks. If you IPv6 address of a client keeps the same suffix (MAC based) and switches network then only the prefix (/64) will change. This way a network like Google, Facebook, etc could track your device moving from network to network if they only look at the last 64-bits of the IPv6 address. For servers this is not a problem as you already know in which network they are.* Network offerings: root-admin can create new networkofferings(with VPC too) that specifies a network stack option:* ipv4 only (default, for backward compatibility allnetworks/offerings post-upgrade migrate to this option)* ipv4-and-ipv6 * ipv6-only (this can be phase 1.b) * A new routing option: static (phase1), dynamic (phase2,withmultiple sub-options such as ospf/bgp etc...) This means that the network admin will need to statically route theIPv6subnet to the VR's outside IPv6 address, for example, on a JunOSrouter:set routing-options rib inet6.0 static route 2001:db8:500::/48 next-hop 2001:db8:100::50 I'm assuming that 2001:db8:100::50 is the address of the VR on the outside (/64) network. In reality this will probably be a longer address, but this is for just the example.* VR changes: * VR gets its guest and public nics set to inet6 auto * For each /64 allocated to guest network and VPC tiers,radvdis configured to do RA radvd is fine, but looking at phase 2 with dynamic routing you might already want to look into FRRouting. FRR can also advertise RAs while not doing any routing. interface ens4 no ipv6 nd suppress-ra ipv6 nd prefix 2001:db8:500::/64 ipv6 nd rdnss 2001:db8:400::53 2001:db8:200::53 See: http://docs.frrouting.org/en/latest/ipv6.html* Firewall: a new ipv6 zone/chain is created for ipv6whereipv6firewall rules (ACLs, ingress, egress) are implemented; ACLs betweenVPCtiers are managed/implemented by ipv6 firewall on VR Please take a look at the existing security_group.py script which implements RFC4890 https://datatracker.ietf.org/doc/html/rfc4890 ICMPv6 is a vital part of IPv6 and certain packets should always be allowed.* It is assumed that static routes are created on thecore/mainrouter by the admin or automated using some scripts/tools; for this CloudStack will announce events with details of /64 networks and VR's public IPv6 address that can be consumed by a rabbitmq/message busclient(for example), or a custom cron job or script as part oforchestration.(this wouldn't be necessary for dynamic routing bgp with phase2)\\ You would only need to announce the /48 or /56 allocated to the VR, that's all. You don't need to inform the upstream router about the /64 subnets created within that larger subnet.* Guest Networking: With SLAAC, it's easy for CloudStack tocalculate allocate and use a /64 and determine the IPv6 address of VRnicsand guest VM nics* A user create an isolated network/VPC with an offeringthat isipv6 enabled* A user can manage firewall for the IPv6 address/guestnics;there'll be no port forward and LB feature though for IPv6* A users can run workloads in the guest VMs that listenonpublically routable ipv6 addresses* Usage/billing etc continue to work, no change needed Network layout: [core/ISP router] -> [VR] -> [guest netwokr or VPC tier on a VLAN] ->[guest VMs/nics]*core/ISP router needs static routes to be added (manually orautomated), assumes a /48 or /56 configured for the zoneThoughts, feedback?Looks doable! Side-note: It would be very cool if you could use parts of this implementation to also route /48, /56, or /60 subnets to individual VMs in Shared networks. Why? This allows for running Docker containers with native IPv6 inside the VM or running a (Open)VPN server inside a VM which then assigns public IPv6 addresses to clients connected. Instead of routing the subnet to a VR we route the subnet to a single instance in a shared network. If we could then also move these subnets between Instances easily one can quickly migrate to a different instance while keeping the same IPv6 subnet. WidoProof-of-concept commentary: here's what I did to test the idea: * Created an isolated network and deployed a VM in my homelabThe VR running on KVM has following nics eth0 - guest network eth1 - link local eth2 - public network * I setup a custom openwrt router on a RPi4 to serve as atoy-corerouter where I create a wan6 IPv6 tunnel using tunnel broker and I gota/48 allocated. My configuration looks like:/48 - 2001:470:ed36::/48 (allocated by tunnel broker) /64 - 2001:470:36:3e2::/64 (default allocated by) I create a LAN ipv6 (public network for CloudStack VR): atsubnet/prefix0:LAN IPv6 address: 2001:470:ed36:0::1/64 Address mode: SLAAC+stateless DHCP (no dhcpv6) * * In the isolated VR, I enabled ipv6 as: net.ipv6.conf.all.disable_ipv6 = 0 net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.all.accept_ra = 1 net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.autoconf = 1 Set up a IPv6 nameserver/dns in /etc/resolve.conf And configured the nics: echo iface eth0 inet6 auto >> /etc/network/interfaces echo iface eth2 inet6 auto >> /etc/network/interfaces /etc/init.d/networking restart Next, restart ACS isolated network without cleanup to have itreconfigure IPv4 nics, firewall, NAT etc* Next, I created a /64 network for the isolated guest network on eth0ofVR using radvd:# cat /etc/radvd.conf interface eth0 { AdvSendAdvert on; MinRtrAdvInterval 5; MaxRtrAdvInterval 15; prefix 2001:470:ed36:1::/64 { AdvOnLink on; AdvAutonomous on; }; }; systemctl restart radvd All guest VMs nics and VR's eth0 gets IPv6 address (SLAAC) in this...:1::/64 network* Finally I added a static route in toy core-router for thenew/64IPv6 range in the isolated network2001:470:ed36:1::/64 via <public IPv6 address of the VR on eth2> dev<local lan nic>* ... and I enabled firewall rules to allow any traffic to pass for thenew /64 networkAnd voila all done! I create a domain AAAA record that points to myguest VM IPv6 address a test webserver onhttp://ipv6-isolated-ntwk-demo.yadav.cloud/ (Note: I'll get rid of the tunnel and request a new /48 block after afew days, sharing this solely for testing purposes)Regards.
