Hi Wido, I assume that flouting ip will not work grate with ingress/egress acl on VR.
>From regular ACS user perspective: I have Instance with dualstack its running web app on 443. I want to swap instances for whatever reason. In case of IPv4 change d-nat rule. In case of IPv6 if flouting IP was not created upfront he will need to change dns entry that usually has 24h ttl. Inconvenience degradation in experience. >From ACS admin perspective: I don’t want to have these tickets in helpdesk. You needed to create another flouting IP that it would be seamless- will not work as answer. On 2021/07/19 09:05:54, Wido den Hollander <w...@widodh.nl> wrote: > > > Op 16-07-2021 om 21:46 schreef Kristaps Cudars: > > Hi Wido, > > > > Your proposal is to sacrifice ability to reassign IPv6 to instance, have > > internal domain prefix, and list/db in ACS what IPv6 has been assigned to > > what instance and go with RA and SLAAC. For route signaling to switch use > > BGP/OSPFv3 or manual pre-creation. > > > > You can still list the IPs which have been assigned. You'll know exactly > what IPv6 address a VM has because of the prefix + MAC. Privacy > Extensions need to be disabled in the VM. > > This already works in CloudStack in Shared Networks in this way. > > Using secondary IPs you can always have 'floating' IPv6 addressess. > > Wido > > > Option with RA and managed flag that DHCPv6 is in use to support preset > > information and ability to create route information from ACS is not an > > option as DHCPv6 its failing? > > > > > > On 2021/07/16 15:17:42, Wido den Hollander <w...@widodh.nl> wrote: > >> > >> > >> Op 16-07-2021 om 16:42 schreef Hean Seng: > >>> Hi Wido, > >>> > >>> In current setup, each Cloudstack have own VR, so in this new IPv6 > >>> subnet > >>> allocation , each VR (which have Frr) will need to have peering with ISP > >>> router (and either BGP or Static Route) , and there is 1000 Acocunts, it > >>> will 1000 BGP session with ISP router , Am I right for this ? or I > >>> understand wrong . > >>> > >> > >> Yes, that is correct. A /56 would also be sufficient or a /60 which is > >> enough to allocate a few /64 subnets. > >> > >> 1000 BGP connections isn't really a problem for a proper router at the > >> ISP. OSPF(v3) would be better, but as I said that's poorly supported. > >> > >> The ISP could also install 1000 static routes, but that means that the > >> ISP's router needs to have those configured. > >> > >> http://docs.frrouting.org/en/latest/ospf6d.html > >> (While looking up this URL I see that Frr recently put in a lot of work > >> in OSPFv3, seems better now) > >> > >>> I understand IPv6 is different then IPv4, and in IPv6 it suppose each > >>> devices have own IP. It just how to realize in easy way. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <w...@widodh.nl> wrote: > >>> > >>>> > >>>> > >>>> Op 16-07-2021 om 05:54 schreef Hean Seng: > >>>>> Hi Wido, > >>>>> > >>>>> My initial thought is not like this, it is the /48 at ISP router, and > >>>> /64 > >>>>> subnet assign to AdvanceZoneVR, AdvanceZoneVR responsible is > >>>>> distribule IPv6 ip (from the assigned /64 sunet) to VM, and not routing > >>>>> the traffic, in the VM that get the IPv6 IP will default route to ISP > >>>>> router as gw. It can may be a bridge over via Advancezone-VR. > >>>>> > >>>> > >>>> How would you bridge this? That sounds like NAT? > >>>> > >>>> IPv6 is meant to be routed. Not to be translated or bridged in any way. > >>>> > >>>> The way a made the drawing is exactly how IPv6 should work in a VPC > >>>> environment. > >>>> > >>>> Traffic flows through the VR where it can do firewalling of the traffic. > >>>> > >>>>> However, If do as the way described in the drawing, then i suppose will > >>>> be > >>>>> another kind of virtual router going to introduce , to get hold the /48 > >>>> in > >>>>> this virtual router right ? > >>>>> > >>>> > >>>> It can be the same VR. But keep in mind that IPv6 != IPv4. > >>>> > >>>> The VR will get Frr as a new daemon which can talk BGP with the upper > >>>> network to route traffic. > >>>> > >>>>> After this, The Advance Zone, NAT's VR will peer with this new IPv6 VR > >>>>> for getting the IPv6 /64 prefix ? > >>>>> > >>>> > >>>> IPv4 will be behind NAT, but IPv6 will not be behind NAT. > >>>> > >>>>> If do in this way, then I guess you just only need Static route, with > >>>>> peering ip both end as one /48 can have a lot of /64 on it. And > >>>> hardware > >>>>> budgeting for new IPv6-VR will become very important, as all traffic > >>>>> will > >>>>> need to pass over it . > >>>>> > >>>> > >>>> Routing or NAT is the same for the VR. You don't need a very beefy VR > >>>> for this. > >>>> > >>>>> It will be like > >>>>> > >>>>> ISP Router ------ > (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM > >>>>> > >>>>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering on > >>>>> OSPF instead of BGP , otherwise few thousand of AdvanceZone-VR wil have > >>>>> few thousand of BGP session. on new-IPv6-VR > >>>>> > >>>>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct, , > >>>>> otherwise ISP router will be full of /64 prefix route either on BGP( > >>>>> Many > >>>>> BGP Session) , or Many Static route . If few thousand account, ti > >>>>> will > >>>>> be few thousand of BGP session with ISP router or few thousand static > >>>> route > >>>>> which is not possible . > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <w...@widodh.nl> > >>>> wrote: > >>>>> > >>>>>> But you still need routing. See the attached PNG (and draw.io XML). > >>>>>> > >>>>>> You need to route the /48 subnet TO the VR which can then route it to > >>>>>> the Virtual Networks behind the VR. > >>>>>> > >>>>>> There is no other way then routing with either BGP or a Static route. > >>>>>> > >>>>>> Wido > >>>>>> > >>>>>> Op 15-07-2021 om 12:39 schreef Hean Seng: > >>>>>>> Or explain like this : > >>>>>>> > >>>>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin > >>>>>>> assigned to Cloudstack > >>>>>>> 2) Cloudsack allocated the subnet (that generated from step1) to > >>>> Virtual > >>>>>>> Router, one Virtual Router have one subniet /64 > >>>>>>> 3) Virtual Router allocate single IPv6 (within the range of /64 > >>>>>>> allocated to VR) to VM > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heans...@gmail.com > >>>>>>> <mailto:heans...@gmail.com>> wrote: > >>>>>>> > >>>>>>> Hi Wido, > >>>>>>> > >>>>>>> I think the /48 is at physical router as gateway , and subnet > >>>>>>> of > >>>> /64 > >>>>>>> at VR of Cloudstack. Cloudstack only keep which /48 prefix > >>>>>>> and > >>>>>>> vlan information of this /48 to be later split the /64. to VR. > >>>>>>> > >>>>>>> And the instances is getting singe IPv6 of /64 IP. The VR is > >>>>>>> getting /64. The default gateway shall goes to /48 of physical > >>>>>>> router ip . In this case ,does not need any BGP router . > >>>>>>> > >>>>>>> > >>>>>>> Similar concept as IPv4 : > >>>>>>> > >>>>>>> /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 > >>>> that > >>>>>>> created in Network. > >>>>>>> and /64 of IPv6 is equivalent to single IP of IPv4 assign to > >>>>>>> VM. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander < > >>>> w...@widodh.nl > >>>>>>> <mailto:w...@widodh.nl>> wrote: > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Op 14-07-2021 om 16:44 schreef Hean Seng: > >>>>>>> > Hi > >>>>>>> > > >>>>>>> > I replied in another thread, i think do not need > >>>>>>> implement > >>>>>>> BGP or OSPF, > >>>>>>> > that would be complicated . > >>>>>>> > > >>>>>>> > We only need assign IPv6 's /64 prefix to Virtual > >>>>>>> Router > >>>>>>> (VR) in NAT > >>>>>>> > zone, and the VR responsible to deliver single IPv6 to > >>>>>>> VM > >>>> via > >>>>>>> DHCP6. > >>>>>>> > > >>>>>>> > In VR, you need to have Default IPv6 route to Physical > >>>>>>> Router's /48. IP > >>>>>>> > as IPv6 Gateway. Thens should be done . > >>>>>>> > > >>>>>>> > Example : > >>>>>>> > Physical Router Interface > >>>>>>> > IPv6 IP : 2000:aaaa::1/48 > >>>>>>> > > >>>>>>> > Cloudstack virtual router : 2000:aaaa:200:201::1/64 > >>>>>>> with > >>>>>>> default ipv6 > >>>>>>> > route to router ip 2000:aaaa::1 > >>>>>>> > and Clodustack Virtual router dhcp allocate IP to VM , > >>>>>>> and > >>>>>>> VM will have > >>>>>>> > default route to VR. IPv6 2000:aaaa:200:201::1 > >>>>>>> > > >>>>>>> > So in cloudstack need to allow user to enter , IPv6 > >>>>>>> gwateway , and > >>>>>>> > the /48 Ipv6 prefix , then it will self allocate the > >>>>>>> /64 > >>>> ip > >>>>>>> to the VR , > >>>>>>> > and maintain make sure not ovelap allocation > >>>>>>> > > >>>>>>> > > >>>>>>> > >>>>>>> But NAT is truly not the solution with IPv6. IPv6 is > >>>>>>> supposed > >>>> to > >>>>>> be > >>>>>>> routable. In addition you should avoid DHCPv6 as much as > >>>>>>> possible as > >>>>>>> that's not really the intended use-case for address > >>>>>>> allocation > >>>>>>> with IPv6. > >>>>>>> > >>>>>>> In order to route an /48 IPv6 subnet to the VR you have a > >>>>>>> few > >>>>>>> possibilities: > >>>>>>> > >>>>>>> - Static route from the upperlying routers which are > >>>>>>> outside > >>>> of > >>>>>>> CloudStack > >>>>>>> - BGP > >>>>>>> - OSPFv3 (broken in most cases!) > >>>>>>> - DHCPv6 Prefix Delegation > >>>>>>> > >>>>>>> BGP and/or Static routes are still the best bet here. > >>>>>>> > >>>>>>> So what you do is that you tell CloudStack that you will > >>>>>>> route > >>>>>>> 2001:db8::/48 to the VR, the VR can then use that to split > >>>>>>> it > >>>> up > >>>>>>> into > >>>>>>> multiple /64 subnets going towards the instances: > >>>>>>> > >>>>>>> - 2001:db8::/64 > >>>>>>> - 2001:db8:1::/64 > >>>>>>> - 2001:db8:2::/64 > >>>>>>> ... > >>>>>>> - 2001:db8:f::/64 > >>>>>>> > >>>>>>> And go on. > >>>>>>> > >>>>>>> In case of BGP you indeed have to tell the VR a few things: > >>>>>>> > >>>>>>> - It's own AS number > >>>>>>> - The peer's address(es) > >>>>>>> > >>>>>>> With FRR you can simply say: > >>>>>>> > >>>>>>> neighbor 2001:db8:4fa::179 remote-as external > >>>>>>> > >>>>>>> The /48 you need to have at the VR anyway in case of > >>>>>>> either a > >>>>>>> static > >>>>>>> route or BGP. > >>>>>>> > >>>>>>> We just need to add a NullRoute on the VR for that /48 so > >>>>>>> that > >>>>>>> traffic > >>>>>>> will not be routed to the upper gateway in case of the VR > >>>> can't > >>>>>>> find a > >>>>>>> route. > >>>>>>> > >>>>>>> Wido > >>>>>>> > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli > >>>>>>> > <alex.matti...@shapeblue.com > >>>>>>> <mailto:alex.matti...@shapeblue.com> > >>>>>>> <mailto:alex.matti...@shapeblue.com > >>>>>>> <mailto:alex.matti...@shapeblue.com>>> wrote: > >>>>>>> > > >>>>>>> > Hi Wido, > >>>>>>> > That's pretty much in line with our thoughts, thanks > >>>> for > >>>>>>> the input. > >>>>>>> > I believe we agree on the following points then: > >>>>>>> > > >>>>>>> > - FRR with BGP (no OSPF) > >>>>>>> > - Route /48 (or/56) down to the VR > >>>>>>> > - /64 per network > >>>>>>> > - SLACC for IP addressing > >>>>>>> > > >>>>>>> > I believe the next big question is then "on which > >>>>>>> level > >>>>>>> of ACS do we > >>>>>>> > manage AS numbers?". I see two options: > >>>>>>> > 1) Private AS number on a per-zone basis > >>>>>>> > 2) Root Admin assigned AS number on a domain/account > >>>> basis > >>>>>>> > 3) End-user driven AS number on a per network basis > >>>> (for > >>>>>>> bring your > >>>>>>> > own AS and IP scenario) > >>>>>>> > > >>>>>>> > Thoughts? > >>>>>>> > > >>>>>>> > Cheers > >>>>>>> > Alex > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > -----Original Message----- > >>>>>>> > From: Wido den Hollander <w...@widodh.nl > >>>>>>> <mailto:w...@widodh.nl> <mailto:w...@widodh.nl > >>>>>>> <mailto:w...@widodh.nl>>> > >>>>>>> > Sent: 13 July 2021 15:08 > >>>>>>> > To: dev@cloudstack.apache.org > >>>>>>> <mailto:dev@cloudstack.apache.org> > >>>>>>> <mailto:dev@cloudstack.apache.org > >>>>>>> <mailto:dev@cloudstack.apache.org>>; > >>>>>>> > Alex Mattioli <alex.matti...@shapeblue.com > >>>>>>> <mailto:alex.matti...@shapeblue.com> > >>>>>>> > <mailto:alex.matti...@shapeblue.com > >>>>>>> <mailto:alex.matti...@shapeblue.com>>> > >>>>>>> > Cc: Wei Zhou <wei.z...@shapeblue.com > >>>>>>> <mailto:wei.z...@shapeblue.com> > >>>>>>> > <mailto:wei.z...@shapeblue.com > >>>>>>> <mailto:wei.z...@shapeblue.com>>>; Rohit Yadav > >>>>>>> > <rohit.ya...@shapeblue.com > >>>>>>> <mailto:rohit.ya...@shapeblue.com> > >>>>>>> <mailto:rohit.ya...@shapeblue.com > >>>>>>> <mailto:rohit.ya...@shapeblue.com>>>; > >>>>>>> > Gabriel Beims Bräscher <gabr...@pcextreme.nl > >>>>>>> <mailto:gabr...@pcextreme.nl> > >>>>>>> > <mailto:gabr...@pcextreme.nl <mailto: > >>>> gabr...@pcextreme.nl > >>>>>>>>> > >>>>>>> > Subject: Re: IPV6 in Isolated/VPC networks > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > On 7/7/21 1:16 PM, Alex Mattioli wrote: > >>>>>>> > > Hi all, > >>>>>>> > > @Wei Zhou<mailto:wei.z...@shapeblue.com > >>>>>>> <mailto:wei.z...@shapeblue.com> > >>>>>>> > <mailto:wei.z...@shapeblue.com > >>>>>>> <mailto:wei.z...@shapeblue.com>>> @Rohit > >>>>>>> > Yadav<mailto:rohit.ya...@shapeblue.com > >>>>>>> <mailto:rohit.ya...@shapeblue.com> > >>>>>>> > <mailto:rohit.ya...@shapeblue.com > >>>>>>> <mailto:rohit.ya...@shapeblue.com>>> and myself are > >>>>>>> investigating how > >>>>>>> > to enable IPV6 support on Isolated and VPC networks > >>>>>>> and > >>>>>>> would like > >>>>>>> > your input on it. > >>>>>>> > > At the moment we are looking at implementing FRR > >>>> with > >>>>>>> BGP (and > >>>>>>> > possibly OSPF) on the ACS VR. > >>>>>>> > > > >>>>>>> > > We are looking for requirements, recommendations, > >>>>>>> ideas, rants, > >>>>>>> > etc...etc... > >>>>>>> > > > >>>>>>> > > >>>>>>> > Ok! Here we go. > >>>>>>> > > >>>>>>> > I think that you mean that the VR will actually > >>>>>>> route > >>>> the > >>>>>>> IPv6 > >>>>>>> > traffic and for that you need to have a way of > >>>>>>> getting > >>>> a > >>>>>>> subnet > >>>>>>> > routed to the VR. > >>>>>>> > > >>>>>>> > BGP is probably you best bet here. Although OSPFv3 > >>>>>>> technically > >>>>>>> > supports this it is very badly implemented in Frr > >>>>>>> for > >>>>>>> example. > >>>>>>> > > >>>>>>> > Now FRR is a very good router and one of the fancy > >>>>>>> features it > >>>>>>> > supports is BGP Unnumered. This allows for auto > >>>>>>> configuration of BGP > >>>>>>> > over a L2 network when both sides are sending Router > >>>>>>> Advertisements. > >>>>>>> > This is very easy for flexible BGP configurations > >>>>>>> where > >>>>>>> both sides > >>>>>>> > have dynamic IPs. > >>>>>>> > > >>>>>>> > What you want to do is that you get a /56, /48 or > >>>>>>> something which is > >>>>>>> > >/64 bits routed to the VR. > >>>>>>> > > >>>>>>> > Now you can sub-segment this into separate /64 > >>>>>>> subnets. > >>>>>>> You don't > >>>>>>> > want to go smaller then a /64 is that prevents you > >>>>>>> from > >>>>>>> using SLAAC > >>>>>>> > for IPv6 address configuration. This is how it works > >>>> for > >>>>>>> Shared > >>>>>>> > Networks now in Basic and Advanced Zones. > >>>>>>> > > >>>>>>> > FRR can now also send out the Router Advertisements > >>>>>>> on > >>>>>>> the downlinks > >>>>>>> > sending out: > >>>>>>> > > >>>>>>> > - DNS servers > >>>>>>> > - DNS domain > >>>>>>> > - Prefix (/64) to be used > >>>>>>> > > >>>>>>> > There is no need for DHCPv6. You can calculate the > >>>>>>> IPv6 > >>>>>>> address the > >>>>>>> > VM will obtain by using the MAC and the prefix. > >>>>>>> > > >>>>>>> > So in short: > >>>>>>> > > >>>>>>> > - Using BGP you routed a /48 to the VR > >>>>>>> > - Now you split this into /64 subnets towards the > >>>>>>> isolated networks > >>>>>>> > > >>>>>>> > Wido > >>>>>>> > > >>>>>>> > > Alex Mattioli > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Regards, > >>>>>>> > Hean Seng > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Regards, > >>>>>>> Hean Seng > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Regards, > >>>>>>> Hean Seng > >>>>> > >>>>> > >>>>> > >>>> > >>> > >>> > >> >
