I think if doing this way , since you were to implement on peering ip between vr and phsical router , then would need keep /56 or 48 at Clodustack ? We can only add /64 subnet to Cloudstack only (instead of keep the /56 or 48 there).
I saw other software provider do is adding /64 subnet to their system, and after that allocate subnet to the VM (from the previous added list). May be considering the OSPF if really on this. It really a nightmare for maintaining 1000 or few thousand of BGP session. You can imagine your Cisco Router list of few thousand BGP session there. On Fri, Jul 16, 2021 at 11:17 PM Wido den Hollander <w...@widodh.nl> wrote: > > > Op 16-07-2021 om 16:42 schreef Hean Seng: > > Hi Wido, > > > > In current setup, each Cloudstack have own VR, so in this new IPv6 > subnet > > allocation , each VR (which have Frr) will need to have peering with ISP > > router (and either BGP or Static Route) , and there is 1000 Acocunts, it > > will 1000 BGP session with ISP router , Am I right for this ? or I > > understand wrong . > > > > Yes, that is correct. A /56 would also be sufficient or a /60 which is > enough to allocate a few /64 subnets. > > 1000 BGP connections isn't really a problem for a proper router at the > ISP. OSPF(v3) would be better, but as I said that's poorly supported. > > The ISP could also install 1000 static routes, but that means that the > ISP's router needs to have those configured. > > http://docs.frrouting.org/en/latest/ospf6d.html > (While looking up this URL I see that Frr recently put in a lot of work > in OSPFv3, seems better now) > > > I understand IPv6 is different then IPv4, and in IPv6 it suppose each > > devices have own IP. It just how to realize in easy way. > > > > > > > > > > > > > > > > > > > > On Fri, Jul 16, 2021 at 8:17 PM Wido den Hollander <w...@widodh.nl> > wrote: > > > >> > >> > >> Op 16-07-2021 om 05:54 schreef Hean Seng: > >>> Hi Wido, > >>> > >>> My initial thought is not like this, it is the /48 at ISP router, and > >> /64 > >>> subnet assign to AdvanceZoneVR, AdvanceZoneVR responsible is > >>> distribule IPv6 ip (from the assigned /64 sunet) to VM, and not > routing > >>> the traffic, in the VM that get the IPv6 IP will default route to ISP > >>> router as gw. It can may be a bridge over via Advancezone-VR. > >>> > >> > >> How would you bridge this? That sounds like NAT? > >> > >> IPv6 is meant to be routed. Not to be translated or bridged in any way. > >> > >> The way a made the drawing is exactly how IPv6 should work in a VPC > >> environment. > >> > >> Traffic flows through the VR where it can do firewalling of the traffic. > >> > >>> However, If do as the way described in the drawing, then i suppose will > >> be > >>> another kind of virtual router going to introduce , to get hold the /48 > >> in > >>> this virtual router right ? > >>> > >> > >> It can be the same VR. But keep in mind that IPv6 != IPv4. > >> > >> The VR will get Frr as a new daemon which can talk BGP with the upper > >> network to route traffic. > >> > >>> After this, The Advance Zone, NAT's VR will peer with this new IPv6 > VR > >>> for getting the IPv6 /64 prefix ? > >>> > >> > >> IPv4 will be behind NAT, but IPv6 will not be behind NAT. > >> > >>> If do in this way, then I guess you just only need Static route, with > >>> peering ip both end as one /48 can have a lot of /64 on it. And > >> hardware > >>> budgeting for new IPv6-VR will become very important, as all traffic > will > >>> need to pass over it . > >>> > >> > >> Routing or NAT is the same for the VR. You don't need a very beefy VR > >> for this. > >> > >>> It will be like > >>> > >>> ISP Router ------ > (new IPV6-VR ) ---- > AdvanceZone-VR ----> VM > >>> > >>> Relationship of (new IPv6 VR) and AdvanceZone-VR , may be considering > on > >>> OSPF instead of BGP , otherwise few thousand of AdvanceZone-VR wil > have > >>> few thousand of BGP session. on new-IPv6-VR > >>> > >>> Also, I suppose we cannot do ISP router. -->. Advancezone VR direct, > , > >>> otherwise ISP router will be full of /64 prefix route either on BGP( > Many > >>> BGP Session) , or Many Static route . If few thousand account, ti > will > >>> be few thousand of BGP session with ISP router or few thousand static > >> route > >>> which is not possible . > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Thu, Jul 15, 2021 at 10:47 PM Wido den Hollander <w...@widodh.nl> > >> wrote: > >>> > >>>> But you still need routing. See the attached PNG (and draw.io XML). > >>>> > >>>> You need to route the /48 subnet TO the VR which can then route it to > >>>> the Virtual Networks behind the VR. > >>>> > >>>> There is no other way then routing with either BGP or a Static route. > >>>> > >>>> Wido > >>>> > >>>> Op 15-07-2021 om 12:39 schreef Hean Seng: > >>>>> Or explain like this : > >>>>> > >>>>> 1) Cloudstack generate list of /64 subnet from /48 that Network admin > >>>>> assigned to Cloudstack > >>>>> 2) Cloudsack allocated the subnet (that generated from step1) to > >> Virtual > >>>>> Router, one Virtual Router have one subniet /64 > >>>>> 3) Virtual Router allocate single IPv6 (within the range of /64 > >>>>> allocated to VR) to VM > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Jul 15, 2021 at 6:25 PM Hean Seng <heans...@gmail.com > >>>>> <mailto:heans...@gmail.com>> wrote: > >>>>> > >>>>> Hi Wido, > >>>>> > >>>>> I think the /48 is at physical router as gateway , and subnet > of > >> /64 > >>>>> at VR of Cloudstack. Cloudstack only keep which /48 prefix > and > >>>>> vlan information of this /48 to be later split the /64. to VR. > >>>>> > >>>>> And the instances is getting singe IPv6 of /64 IP. The VR is > >>>>> getting /64. The default gateway shall goes to /48 of physical > >>>>> router ip . In this case ,does not need any BGP router . > >>>>> > >>>>> > >>>>> Similar concept as IPv4 : > >>>>> > >>>>> /48 subnet of IPv6 is equivalent to current /24 subnet of IPv4 > >> that > >>>>> created in Network. > >>>>> and /64 of IPv6 is equivalent to single IP of IPv4 assign to > VM. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Thu, Jul 15, 2021 at 5:31 PM Wido den Hollander < > >> w...@widodh.nl > >>>>> <mailto:w...@widodh.nl>> wrote: > >>>>> > >>>>> > >>>>> > >>>>> Op 14-07-2021 om 16:44 schreef Hean Seng: > >>>>> > Hi > >>>>> > > >>>>> > I replied in another thread, i think do not need > implement > >>>>> BGP or OSPF, > >>>>> > that would be complicated . > >>>>> > > >>>>> > We only need assign IPv6 's /64 prefix to Virtual > Router > >>>>> (VR) in NAT > >>>>> > zone, and the VR responsible to deliver single IPv6 to > VM > >> via > >>>>> DHCP6. > >>>>> > > >>>>> > In VR, you need to have Default IPv6 route to Physical > >>>>> Router's /48. IP > >>>>> > as IPv6 Gateway. Thens should be done . > >>>>> > > >>>>> > Example : > >>>>> > Physical Router Interface > >>>>> > IPv6 IP : 2000:aaaa::1/48 > >>>>> > > >>>>> > Cloudstack virtual router : 2000:aaaa:200:201::1/64 > with > >>>>> default ipv6 > >>>>> > route to router ip 2000:aaaa::1 > >>>>> > and Clodustack Virtual router dhcp allocate IP to VM , > and > >>>>> VM will have > >>>>> > default route to VR. IPv6 2000:aaaa:200:201::1 > >>>>> > > >>>>> > So in cloudstack need to allow user to enter , IPv6 > >>>>> gwateway , and > >>>>> > the /48 Ipv6 prefix , then it will self allocate the > /64 > >> ip > >>>>> to the VR , > >>>>> > and maintain make sure not ovelap allocation > >>>>> > > >>>>> > > >>>>> > >>>>> But NAT is truly not the solution with IPv6. IPv6 is > supposed > >> to > >>>> be > >>>>> routable. In addition you should avoid DHCPv6 as much as > >>>>> possible as > >>>>> that's not really the intended use-case for address > allocation > >>>>> with IPv6. > >>>>> > >>>>> In order to route an /48 IPv6 subnet to the VR you have a > few > >>>>> possibilities: > >>>>> > >>>>> - Static route from the upperlying routers which are > outside > >> of > >>>>> CloudStack > >>>>> - BGP > >>>>> - OSPFv3 (broken in most cases!) > >>>>> - DHCPv6 Prefix Delegation > >>>>> > >>>>> BGP and/or Static routes are still the best bet here. > >>>>> > >>>>> So what you do is that you tell CloudStack that you will > route > >>>>> 2001:db8::/48 to the VR, the VR can then use that to split > it > >> up > >>>>> into > >>>>> multiple /64 subnets going towards the instances: > >>>>> > >>>>> - 2001:db8::/64 > >>>>> - 2001:db8:1::/64 > >>>>> - 2001:db8:2::/64 > >>>>> ... > >>>>> - 2001:db8:f::/64 > >>>>> > >>>>> And go on. > >>>>> > >>>>> In case of BGP you indeed have to tell the VR a few things: > >>>>> > >>>>> - It's own AS number > >>>>> - The peer's address(es) > >>>>> > >>>>> With FRR you can simply say: > >>>>> > >>>>> neighbor 2001:db8:4fa::179 remote-as external > >>>>> > >>>>> The /48 you need to have at the VR anyway in case of > either a > >>>>> static > >>>>> route or BGP. > >>>>> > >>>>> We just need to add a NullRoute on the VR for that /48 so > that > >>>>> traffic > >>>>> will not be routed to the upper gateway in case of the VR > >> can't > >>>>> find a > >>>>> route. > >>>>> > >>>>> Wido > >>>>> > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > On Wed, Jul 14, 2021 at 8:55 PM Alex Mattioli > >>>>> > <alex.matti...@shapeblue.com > >>>>> <mailto:alex.matti...@shapeblue.com> > >>>>> <mailto:alex.matti...@shapeblue.com > >>>>> <mailto:alex.matti...@shapeblue.com>>> wrote: > >>>>> > > >>>>> > Hi Wido, > >>>>> > That's pretty much in line with our thoughts, thanks > >> for > >>>>> the input. > >>>>> > I believe we agree on the following points then: > >>>>> > > >>>>> > - FRR with BGP (no OSPF) > >>>>> > - Route /48 (or/56) down to the VR > >>>>> > - /64 per network > >>>>> > - SLACC for IP addressing > >>>>> > > >>>>> > I believe the next big question is then "on which > level > >>>>> of ACS do we > >>>>> > manage AS numbers?". I see two options: > >>>>> > 1) Private AS number on a per-zone basis > >>>>> > 2) Root Admin assigned AS number on a domain/account > >> basis > >>>>> > 3) End-user driven AS number on a per network basis > >> (for > >>>>> bring your > >>>>> > own AS and IP scenario) > >>>>> > > >>>>> > Thoughts? > >>>>> > > >>>>> > Cheers > >>>>> > Alex > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > -----Original Message----- > >>>>> > From: Wido den Hollander <w...@widodh.nl > >>>>> <mailto:w...@widodh.nl> <mailto:w...@widodh.nl > >>>>> <mailto:w...@widodh.nl>>> > >>>>> > Sent: 13 July 2021 15:08 > >>>>> > To: dev@cloudstack.apache.org > >>>>> <mailto:dev@cloudstack.apache.org> > >>>>> <mailto:dev@cloudstack.apache.org > >>>>> <mailto:dev@cloudstack.apache.org>>; > >>>>> > Alex Mattioli <alex.matti...@shapeblue.com > >>>>> <mailto:alex.matti...@shapeblue.com> > >>>>> > <mailto:alex.matti...@shapeblue.com > >>>>> <mailto:alex.matti...@shapeblue.com>>> > >>>>> > Cc: Wei Zhou <wei.z...@shapeblue.com > >>>>> <mailto:wei.z...@shapeblue.com> > >>>>> > <mailto:wei.z...@shapeblue.com > >>>>> <mailto:wei.z...@shapeblue.com>>>; Rohit Yadav > >>>>> > <rohit.ya...@shapeblue.com > >>>>> <mailto:rohit.ya...@shapeblue.com> > >>>>> <mailto:rohit.ya...@shapeblue.com > >>>>> <mailto:rohit.ya...@shapeblue.com>>>; > >>>>> > Gabriel Beims Bräscher <gabr...@pcextreme.nl > >>>>> <mailto:gabr...@pcextreme.nl> > >>>>> > <mailto:gabr...@pcextreme.nl <mailto: > >> gabr...@pcextreme.nl > >>>>>>> > >>>>> > Subject: Re: IPV6 in Isolated/VPC networks > >>>>> > > >>>>> > > >>>>> > > >>>>> > On 7/7/21 1:16 PM, Alex Mattioli wrote: > >>>>> > > Hi all, > >>>>> > > @Wei Zhou<mailto:wei.z...@shapeblue.com > >>>>> <mailto:wei.z...@shapeblue.com> > >>>>> > <mailto:wei.z...@shapeblue.com > >>>>> <mailto:wei.z...@shapeblue.com>>> @Rohit > >>>>> > Yadav<mailto:rohit.ya...@shapeblue.com > >>>>> <mailto:rohit.ya...@shapeblue.com> > >>>>> > <mailto:rohit.ya...@shapeblue.com > >>>>> <mailto:rohit.ya...@shapeblue.com>>> and myself are > >>>>> investigating how > >>>>> > to enable IPV6 support on Isolated and VPC networks > and > >>>>> would like > >>>>> > your input on it. > >>>>> > > At the moment we are looking at implementing FRR > >> with > >>>>> BGP (and > >>>>> > possibly OSPF) on the ACS VR. > >>>>> > > > >>>>> > > We are looking for requirements, recommendations, > >>>>> ideas, rants, > >>>>> > etc...etc... > >>>>> > > > >>>>> > > >>>>> > Ok! Here we go. > >>>>> > > >>>>> > I think that you mean that the VR will actually > route > >> the > >>>>> IPv6 > >>>>> > traffic and for that you need to have a way of > getting > >> a > >>>>> subnet > >>>>> > routed to the VR. > >>>>> > > >>>>> > BGP is probably you best bet here. Although OSPFv3 > >>>>> technically > >>>>> > supports this it is very badly implemented in Frr > for > >>>>> example. > >>>>> > > >>>>> > Now FRR is a very good router and one of the fancy > >>>>> features it > >>>>> > supports is BGP Unnumered. This allows for auto > >>>>> configuration of BGP > >>>>> > over a L2 network when both sides are sending Router > >>>>> Advertisements. > >>>>> > This is very easy for flexible BGP configurations > where > >>>>> both sides > >>>>> > have dynamic IPs. > >>>>> > > >>>>> > What you want to do is that you get a /56, /48 or > >>>>> something which is > >>>>> > >/64 bits routed to the VR. > >>>>> > > >>>>> > Now you can sub-segment this into separate /64 > subnets. > >>>>> You don't > >>>>> > want to go smaller then a /64 is that prevents you > from > >>>>> using SLAAC > >>>>> > for IPv6 address configuration. This is how it works > >> for > >>>>> Shared > >>>>> > Networks now in Basic and Advanced Zones. > >>>>> > > >>>>> > FRR can now also send out the Router Advertisements > on > >>>>> the downlinks > >>>>> > sending out: > >>>>> > > >>>>> > - DNS servers > >>>>> > - DNS domain > >>>>> > - Prefix (/64) to be used > >>>>> > > >>>>> > There is no need for DHCPv6. You can calculate the > IPv6 > >>>>> address the > >>>>> > VM will obtain by using the MAC and the prefix. > >>>>> > > >>>>> > So in short: > >>>>> > > >>>>> > - Using BGP you routed a /48 to the VR > >>>>> > - Now you split this into /64 subnets towards the > >>>>> isolated networks > >>>>> > > >>>>> > Wido > >>>>> > > >>>>> > > Alex Mattioli > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > >>>>> > > >>>>> > > >>>>> > -- > >>>>> > Regards, > >>>>> > Hean Seng > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Regards, > >>>>> Hean Seng > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Regards, > >>>>> Hean Seng > >>> > >>> > >>> > >> > > > > > -- Regards, Hean Seng
