RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

Ipv6 can be shorter than ipv4.

Here is the proof:

ping6 ::1 

is shorter than 

ping 127.1

ipv6 addresses can be very small when done properly.

Jean

-Original Message-
From: NANOG  On Behalf Of Mark Tinka
Sent: November 28, 2021 5:39 AM
To: nanog@nanog.org
Subject: Re: IPv6 and CDN's



On 11/28/21 05:37, Masataka Ohta wrote:

> Try to type in raw IPv6 addresses.

There is DNS for that.

Mark.

Re: SentryPeer: A distributed peer to peer list of bad IP addresses and phone numbers collected via a SIP Honeypot

[Gavin Henry]

This should help
https://github.com/SentryPeer/SentryPeer/blob/aea3b3762c7df9e4d19901fa2dd82fe93a38f4cf/CHANGELOG.md#unreleased

Re: IPv6 and CDN's

[Mark Tinka]

On 11/28/21 06:43, Masataka Ohta wrote:



Here in nanog, we are talking about network operations, considerable
part of which can not rely on DNS.


And yet Facebook were unable to access their kit to fix their recent 
outage because of it (or, lack of it).


There was a time when knowing the IP(v4) address of every interface of 
every router in your network was cool. I have never had to care about 
that in close to 15 years. Right up there with losing interest in making 
software modems work in Linux, when it was a thing :-).


If you want to remain stuck in the past, we don't have to join you.

Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 11/28/21 14:58, Masataka Ohta wrote:


Exactly.

That facebook poorly managed their DNS to cause the recent disaster
is an important evidence to support my point that DNS, so often, may
not be helpful for network operations against disastrous failures,
including, but not limited to, DNS failures.


Yes, but that does not mean that DNS is not valuable, or cannot be 
hardened.


Everything can break, even an IPv4 interface on a router port. Good 
practice in network operations is what keeps these kinds of problems at 
bay. I mean, why else do we have lists like these?


I am certain Facebook have hardened their DNS infrastructure, and that 
particular failure scenario should not recur, given all the clever 
people there, and around them.





There was a time when knowing the IP(v4) address of every interface 
of every router in your network was cool.


I surely acknowledge your point that it is impossible to do so with
MAC address based IPv6 addresses, which is why IPv6 opex is so high.

But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.


That's just satisfying one's mental (or emotional) nits.

Routers (and customers) don't care about how anally we assign address 
space. As long as it is compliant, does not conflict, and is correctly 
routed.


That we cannot transpose our IPv4 mental/emotional habits on to IPv6 
does not make IPv6 more complicated. It just makes us more stuck in our 
ways.


After all, DHCPv6 still does not offer a default gateway.



So, you are saying you haven't faced real operational problems
to loss DNS information for these 15 years.

Congratulations for your luck!


I am sure I have had a DNS issue of some sort or other in the past 15 
years. The fact that I can't remember what it was is telling.




Surely, the recent disaster of facebook happened in the recent
past.

So what?


And they have learned from it, and I dare say, fixed it.

Facebook will neither be disposing of DNS any time soon, nor will they 
be dropping IPv6.


Mark.

Re: IPv6 and CDN's

[Masataka Ohta]

Mark Tinka wrote:


That facebook poorly managed their DNS to cause the recent disaster
is an important evidence to support my point that DNS, so often, may
not be helpful for network operations against disastrous failures,
including, but not limited to, DNS failures.


Yes, but that does not mean that DNS is not valuable, or cannot be 
hardened.


As a person who proposed anycast DNS servers, against which facebook
operated their DNS, I'm so sure you are right.

I am certain Facebook have hardened their DNS infrastructure, and that 
particular failure scenario should not recur, given all the clever 
people there, and around them.


All I can see is that there were a lot of stupid people in facebook.

I really hope less of them still remain there.

Masataka Ohta

Re: IPv6 and CDN's

[Mark Tinka]

On 11/28/21 05:37, Masataka Ohta wrote:


Try to type in raw IPv6 addresses.


There is DNS for that.

Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 11/28/21 15:33, Masataka Ohta wrote:


As a person who proposed anycast DNS servers, against which facebook
operated their DNS, I'm so sure you are right.


Facebook's mistake on this is an easily fixable one.

We've all been there. Nothing groundbreaking.



All I can see is that there were a lot of stupid people in facebook.

I really hope less of them still remain there.


Can't help you there...

Mark.

RE: IPv6 and CDN's

[Jean St-Laurent via NANOG]

I like to put some servers behind that scheme.

 

2601::443: for https servers

2601::25: for MTA servers.

2601::993: for IMAP

 

It gives a quick note of what is that ip even though it’s ipv6 and usually 
non-human readable. 

 

Not sure what kind of scheme is use by medium/big ISP. 

 

Do you go by zip code of the area covered or some kind of logical to help 
people know what is behind that ipv6 network?

 

Jean

 

From: NANOG  On Behalf Of Baldur 
Norddahl
Sent: November 28, 2021 8:22 AM
To: NANOG 
Subject: Re: IPv6 and CDN's

 

 

søn. 28. nov. 2021 13.59 skrev Masataka Ohta mailto:mo...@necom830.hpcl.titech.ac.jp> >:


But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.

 

99% if not 100% of our subnets have either only routers or only hosts + a 
gateway. So that would be a strange rule to follow. Also very expensive if we 
are talking public addressing.

 

I find that 10.x.y.z is not much if you want to have a system in your subnet 
numbering. With ipv6 there is much more space to enable systematic numbering 
schemes. 

Re: IPv6 and CDN's

[Mark Tinka]

On 11/28/21 16:13, Masataka Ohta wrote:



Certainly, but, merely because it is an easily avoided one.


None of the us came out the womb knowing anything. We learned as we went 
along. And we keep learning, right until our death.


To expect experience before it is experienced has always been unreasonable.



People who really understand DNS, including but not limited to
anycast one, have never been there.


Anycast was the result of things that broke. And even with Anycast, lots 
of things break, still.


Continuity of service != never having been there.

Mark.

Re: IPv6 and CDN's

[Baldur Norddahl]

søn. 28. nov. 2021 13.59 skrev Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp>:

>
> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.
>

99% if not 100% of our subnets have either only routers or only hosts + a
gateway. So that would be a strange rule to follow. Also very expensive if
we are talking public addressing.

I find that 10.x.y.z is not much if you want to have a system in your
subnet numbering. With ipv6 there is much more space to enable systematic
numbering schemes.

>

Re: IPv6 and CDN's

[Masataka Ohta]

Mark Tinka wrote:


As a person who proposed anycast DNS servers, against which facebook
operated their DNS, I'm so sure you are right.


Facebook's mistake on this is an easily fixable one.


Certainly, but, merely because it is an easily avoided one.


We've all been there.


People who really understand DNS, including but not limited to
anycast one, have never been there.

Masataka Ohta

Re: IPv6 and CDN's

[sronan]

It certainly sounds like you’ve never operated a network at scale if you think 
knowing the IP address of something reduces Operational expense. The only way 
to truly reduce Opex at scale is automation.

Shane

> On Nov 28, 2021, at 9:13 AM, Masataka Ohta  
> wrote:
> 
> Mark Tinka wrote:
> 
>>> As a person who proposed anycast DNS servers, against which facebook
>>> operated their DNS, I'm so sure you are right.
>> Facebook's mistake on this is an easily fixable one.
> 
> Certainly, but, merely because it is an easily avoided one.
> 
>> We've all been there.
> 
> People who really understand DNS, including but not limited to
> anycast one, have never been there.
> 
>Masataka Ohta

Re: IPv6 and CDN's

[Mark Tinka]

On 11/28/21 14:09, Jean St-Laurent wrote:


Ipv6 can be shorter than ipv4.

Here is the proof:

ping6 ::1

is shorter than

ping 127.1

ipv6 addresses can be very small when done properly.


The good news is the point of an IP address is not for its own sake.

Mark.

Re: IPv6 and CDN's

[Masataka Ohta]

Baldur Norddahl wrote:


But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.



99% if not 100% of our subnets have either only routers or only hosts + a
gateway.


It merely means you should not use MAC address based IP addresses for,
at least, routers, which is partly why opex of IPv4 is low.


So that would be a strange rule to follow. Also very expensive if we
are talking public addressing.

I find that 10.x.y.z is not much if you want to have a system in
your subnet numbering.


In most cases, it is enough that addresses are unique within
certain domain, which is why many ISPs are assigning addresses,
not guaranteed to be globally unique, to there internal routers.


With ipv6 there is much more space to enable systematic numbering
schemes.


More space, only to encourage stupid idea of MAC address based
addresses of IPv6/ND, is not required for systematic numbering.

Masataka Ohta

Re: IPv6 and CDN's

[Masataka Ohta]

Mark Tinka wrote:


Here in nanog, we are talking about network operations, considerable
part of which can not rely on DNS.


And yet Facebook were unable to access their kit to fix their recent 
outage because of it (or, lack of it).


Exactly.

That facebook poorly managed their DNS to cause the recent disaster
is an important evidence to support my point that DNS, so often, may
not be helpful for network operations against disastrous failures,
including, but not limited to, DNS failures.

There was a time when knowing the IP(v4) address of every interface of 
every router in your network was cool.


I surely acknowledge your point that it is impossible to do so with
MAC address based IPv6 addresses, which is why IPv6 opex is so high.

But, with manually configured IP addresses, it is trivially easy
to have a rule to assign lower part of IP addresses within a subnet
for hosts and upper part for routers, which is enough to troubleshoot
most network failures.

I have never had to care about 
that in close to 15 years. Right up there with losing interest in making 
software modems work in Linux, when it was a thing :-).


So, you are saying you haven't faced real operational problems
to loss DNS information for these 15 years.

Congratulations for your luck!


If you want to remain stuck in the past, we don't have to join you.


Surely, the recent disaster of facebook happened in the recent
past.

So what?

Masataka Ohta

Re: IPv6 and CDN's

[Mark Tinka]

On 11/28/21 15:59, Masataka Ohta wrote:


It merely means you should not use MAC address based IP addresses for,
at least, routers, which is partly why opex of IPv4 is low.


I often wonder what Internet you use :-)...



More space, only to encourage stupid idea of MAC address based
addresses of IPv6/ND, is not required for systematic numbering.


I'm sure a router won't stop working because we did not assign it an IP 
address "systematically". Nor will the spinning of the earth.


Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 11/28/21 16:20, Jean St-Laurent via NANOG wrote:


I like to put some servers behind that scheme.

2601::443: for https servers

2601::25: for MTA servers.

2601::993: for IMAP

It gives a quick note of what is that ip even though it’s ipv6 and 
usually non-human readable.


Not sure what kind of scheme is use by medium/big ISP.

Do you go by zip code of the area covered or some kind of logical to 
help people know what is behind that ipv6 network?




We really aren't clever with IPv6 address design and assignment. The 
most we do is assign:


    - 1x /48 to Loopbacks for all routers.
    - /48's per PoP for infrastructure links.
    - /48's per city for /56 assignments to customers.
    - /48's per city for assignments to customers.

We don't try to co-ordinate them in a "meaningful" way that would offer 
visual identification. We feel that is just too much work, and that 
getting IPv4/IPv6 parity for day-to-day operations is a challenge in and 
of itself, without trying to be fancy.


Mark.

Re: Latency/Packet Loss on ASR1006

[Colin Legendre]

Thanks, will look into this.

---
Colin Legendre
President and CTO

Coextro - Unlimited. Fast. Reliable.
w: www.coextro.com
e: clegen...@coextro.com

p: 647-693-7686 ext.101
m: 416-560-8502
f: 647-812-4132


On Sat, Nov 27, 2021 at 7:42 AM Tassos  wrote:

> In the past we had packet loss issues due SIP's PLIM buffer.
>
> The following docs may provide some guidance:
>
>
> https://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregation-services-routers/200674-Throughput-issues-on-ASR1000-Series-rout.html
>
> https://www.cisco.com/c/en/us/td/docs/interfaces_modules/shared_port_adapters/configuration/ASR1000/asr1000-sip-spa-book/asr-spa-pkt-class.html
>
> --
> Tassos
>
> On 27/11/21 02:11, Fiona Weber via NANOG wrote:
>
> Hi
>
> we see similar problems on ASR1006-X with ESP100 and MIP100. At about ~45
> Gbit/s of traffic (on ~30k PPPoE Sessions and ~700k CGN sessions) the QFP
> utilization skyrockets from ~45 % straight to ~95 % :(
> I don't know if it's the CGN sessions or the traffic/packets causing the
> load increase, the datasheet says it supports something like 10M
> sessions but maybe not if you really intend to push packets through it?
> We have not seen such spikes with way higher pps, but lower CGN session
> count, when we had DDoS Attacks against end customers.
>
> Fiona
> On 11/26/21 20:09, Colin Legendre wrote:
>
> Hi,
>
> We have ...
>
> ASR1006  that has following cards...
> 1 x ESP40
> 1 x SIP40
> 4 x SPA-1x10GE-L-V2
> 1 x 6TGE
> 1 x RP2
>
> We've been having latency and packet loss during peak periods...
>
> We notice all is good until we reach 50% utilization on output of...
>
> 'show platform hardware qfp active datapath utilization summary'
>
> Literally ... 47% good... 48% good... 49% latency to next hop goes from
> 1ms to 15-20ms... 50% we see 1-2% packet-loss and 30-40ms latency... 53% we
> see 60-70ms latency and 8-10% packet loss.
>
> Is this expected... the ESP40 can only really push 20G and then starts to
> have performance issues?
>
>
>
> ---
> Colin Legendre
>
>
>

Re: IPv6 and CDN's

[Owen DeLong via NANOG]

> On Nov 27, 2021, at 19:37 , Masataka Ohta  
> wrote:
> 
> Mark Tinka wrote:
> 
>> On 11/27/21 17:07, Masataka Ohta wrote:
>>> Because lengthy IPv6 addresses mean a lot more opex than IPv4.
>> I disagree
> 
> Try to type in raw IPv6 addresses.

Rarely necessary in the modern age, but really not significantly more difficult 
than IPv4 once
you become accustomed to it.

Owen

Re: AWS and IPv6

[Karl Auer]

On Sun, 2021-11-28 at 12:53 -0800, Michael Thomas wrote:
> I was reading their howto yesterday and it seems they are only 
> allocating a /64? Why?

That's a /64 *per subnet*...

But the size of a VPC's IPv6 CIDR block does seem to be fixed at /56.
Would have been nice to see /48 instead.

Regards, K.

-- 
~~~
Karl Auer (ka...@biplane.com.au)
http://www.biplane.com.au/kauer

GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58
Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170

Re: AWS and IPv6

[Michael Thomas]

On 11/28/21 1:17 PM, Karl Auer wrote:

On Sun, 2021-11-28 at 12:53 -0800, Michael Thomas wrote:

I was reading their howto yesterday and it seems they are only
allocating a /64? Why?

That's a /64 *per subnet*...

But the size of a VPC's IPv6 CIDR block does seem to be fixed at /56.
Would have been nice to see /48 instead.


Ah ok, I must have missed that.

Mike

Re: IPv6 and CDN's

[Mark Andrews]

> On 29 Nov 2021, at 09:41, scott  wrote:
> 
> 
> On 11/28/2021 9:47 AM, Owen DeLong via NANOG wrote:
>> Why not properly assign /48s to customers and /40s to cities?
>> --
> 
> Side note: I recently tried to get /48 per customer with ARIN on repeated 
> emails and they refused.  We were already given an IPv6 block a while back.  
> I told them I wanted to expand it so I could give out a /48 per customer and 
> that we had more than 65535 customers, which is the block we got; 65535 /48s. 
>  I didn't even account for our needs.
> 
> Without arguing the reasons, we will have to hand out /56s, rather than /48s 
> because of this.  So, it's not all /48-unicorns, puppies and rainbows.
> 
> scott

Looks like a policy omission.  You should be able to grow the per customer 
allocation up to /48 per customer.
One shouldn’t be stuck with /56 because one made a bad choice of prefix size 
initially.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: AWS and IPv6

[William Herrin]

On Sun, Nov 28, 2021 at 1:18 PM Karl Auer  wrote:
> On Sun, 2021-11-28 at 12:53 -0800, Michael Thomas wrote:
> > I was reading their howto yesterday and it seems they are only
> > allocating a /64? Why?
>
> That's a /64 *per subnet*...
>
> But the size of a VPC's IPv6 CIDR block does seem to be fixed at /56.
> Would have been nice to see /48 instead.

Hi Karl,

To what purpose? You can't alter the VPC routing of any of the IP
addresses (v4 or v6) assigned to an AWS VPC. If you try, for example,
to assign a /64 to an instance you get a funky error: "Route
destination doesn't match any subnet CIDR blocks." You can only assign
the block's IP addresses to subnets or not and then assign addresses
from the subnet to the instances. You can't have more than 256 subnets
in a VPC so why would you need more than a /56 of IPv6 addresses?

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: IPv6 and CDN's

[Owen DeLong via NANOG]

> On Nov 28, 2021, at 04:58 , Masataka Ohta  
> wrote:
> 
> Mark Tinka wrote:
> 
>>> Here in nanog, we are talking about network operations, considerable
>>> part of which can not rely on DNS.
>> And yet Facebook were unable to access their kit to fix their recent outage 
>> because of it (or, lack of it).
> 
> Exactly.
> 
> That facebook poorly managed their DNS to cause the recent disaster
> is an important evidence to support my point that DNS, so often, may
> not be helpful for network operations against disastrous failures,
> including, but not limited to, DNS failures.

I’d argue that the true failure was failure to document the system adequately 
to provide for prompt
resolution of the DNS problems in the absence of DNS and failure to properly 
distribute that knowledge
to those that would need it in such a circumstance.


>> There was a time when knowing the IP(v4) address of every interface of every 
>> router in your network was cool.
> 
> I surely acknowledge your point that it is impossible to do so with
> MAC address based IPv6 addresses, which is why IPv6 opex is so high.

Hardly anyone uses MAC address based IPv6 addresses for anything, so your claim 
here is specious.

Statically or manually configured IPv6 addresses are no more difficult than the 
equivalent in IPv4, just
slightly longer.

Consider, for example, 192.159.10.7 vs. 2620:0:930::400:7

The 400 could have been omitted leaving 2620:0:930::7, but I chose to have 
additional flexibility that
is available from IPv6 in classifying addresses for particular purposes and 
decided to use the next
order quartet for that purpose.

> But, with manually configured IP addresses, it is trivially easy
> to have a rule to assign lower part of IP addresses within a subnet
> for hosts and upper part for routers, which is enough to troubleshoot
> most network failures.

It’s equally easy to do that in IPv6 as well (while I prefer to use the reverse 
strategy, using low order
addresses for routers and higher numbers for non-forwarding hosts).

>> I have never had to care about that in close to 15 years. Right up there 
>> with losing interest in making software modems work in Linux, when it was a 
>> thing :-).
> 
> So, you are saying you haven't faced real operational problems
> to loss DNS information for these 15 years.
> 
> Congratulations for your luck!

In reality, DNS properly implemented is extraordinarily reliable these days. 
It’s not completely failure-proof,
as Facebook recently demonstrated so thoroughly, but consider how extraordinary 
that failure was in order
to garner so much attention. Gone are the days when DNS failures were a part of 
daily operational life
that we simply accepted.

>> If you want to remain stuck in the past, we don't have to join you.
> 
> Surely, the recent disaster of facebook happened in the recent
> past.

Yes, but if you really look at it, that was a failure of preparedness much more 
than a failure of DNS.

> So what?

So, the world has moved on and modern networks are built using tools you appear 
to prefer not to
depend on… I remember when an RS-232 port was the gold standard of being able 
to recover your
router. Today, we accept at least USB and in most cases even Ethernet as a 
viable alternative.

Things get better, but that comes with higher-level dependencies on underlying 
infrastructure. Fortunately,
the underlying infrastructure gets better as well, making that possible and 
even reasonable. The fact
that you prefer not to recognize this is a sign that you are failing to adapt 
to the present in favor of
living in the past as Mark stated.

Owen

Re: AWS and IPv6

[Michael Thomas]

On 11/27/21 2:44 PM, Fletcher Kittredge wrote:


The Register  says: AWS claims 
'monumental step forward' with optional IPv6-only networks 




I was reading their howto yesterday and it seems they are only 
allocating a /64? Why?


I guess I just don't get the point of the VPC in the first place. I get 
the firewall aspect but it seem to be more than that.


Mike

Re: AWS and IPv6

[Matt Palmer]

On Sun, Nov 28, 2021 at 02:10:40PM -0800, William Herrin wrote:
> On Sun, Nov 28, 2021 at 1:18 PM Karl Auer  wrote:
> > On Sun, 2021-11-28 at 12:53 -0800, Michael Thomas wrote:
> > > I was reading their howto yesterday and it seems they are only
> > > allocating a /64? Why?
> >
> > That's a /64 *per subnet*...
> >
> > But the size of a VPC's IPv6 CIDR block does seem to be fixed at /56.
> > Would have been nice to see /48 instead.
> 
> To what purpose? You can't alter the VPC routing of any of the IP
> addresses (v4 or v6) assigned to an AWS VPC.

Which is, fundamentally, half the problem with IPv6 in AWS.  I'd have much
preferred that they'd added the ability to do actually-useful IPv6 routing
rather than IPv6-only subnets, which strikes me as more of a toy than
something *actually* useful.

- Matt

Re: IPv6 and CDN's

[Owen DeLong via NANOG]

> On Nov 28, 2021, at 08:55 , Mark Tinka  wrote:
> 
> 
> 
> On 11/28/21 16:20, Jean St-Laurent via NANOG wrote:
> 
>> I like to put some servers behind that scheme.
>>  
>> 2601::443: for https servers
>> 2601::25: for MTA servers.
>> 2601::993: for IMAP
>>  
>> It gives a quick note of what is that ip even though it’s ipv6 and usually 
>> non-human readable. 
>>  
>> Not sure what kind of scheme is use by medium/big ISP. 
>>  
>> Do you go by zip code of the area covered or some kind of logical to help 
>> people know what is behind that ipv6 network?
> 
> We really aren't clever with IPv6 address design and assignment. The most we 
> do is assign:
> 
> - 1x /48 to Loopbacks for all routers.
> - /48's per PoP for infrastructure links.
> - /48's per city for /56 assignments to customers.
> - /48's per city for assignments to customers. 

Why are you so stingy with customer assignments?

Why not properly assign /48s to customers and /40s to cities?

Owen

Re: AWS and IPv6

[Dave Bell]

It's a /56 per VPC, and a /64 per subnet.

Seems reasonable to me.

https://docs.aws.amazon.com/vpc/latest/userguide/get-started-ipv6.html

Dave

On Sun, 28 Nov 2021 at 20:54, Michael Thomas  wrote:

>
> On 11/27/21 2:44 PM, Fletcher Kittredge wrote:
>
>
> The Register  says: AWS claims 'monumental
> step forward' with optional IPv6-only networks
> 
>
>
> I was reading their howto yesterday and it seems they are only allocating
> a /64? Why?
>
> I guess I just don't get the point of the VPC in the first place. I get
> the firewall aspect but it seem to be more than that.
>
> Mike
>

Re: IPv6 and CDN's

[Dave Bell]

On Sun, 28 Nov 2021 at 13:00, Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> That facebook poorly managed their DNS to cause the recent disaster
> is an important evidence to support my point that DNS, so often, may
> not be helpful for network operations against disastrous failures,
> including, but not limited to, DNS failures.
>

 I don't want to wade into the middle of this argument, but has there been
more information about the recent facebook outage released that I missed?
All I've read seems to say that the loss of connectivity to their DNS
servers was a symptom, rather than the cause of the outage.

Dave

Re: AWS and IPv6

[Oliver O'Boyle]

On Sun., Nov. 28, 2021, 17:13 William Herrin,  wrote:

> On Sun, Nov 28, 2021 at 1:18 PM Karl Auer  wrote:
> > On Sun, 2021-11-28 at 12:53 -0800, Michael Thomas wrote:
> > > I was reading their howto yesterday and it seems they are only
> > > allocating a /64? Why?
> >
> > That's a /64 *per subnet*...
> >
> > But the size of a VPC's IPv6 CIDR block does seem to be fixed at /56.
> > Would have been nice to see /48 instead.
>
> Hi Karl,
>
> To what purpose? You can't alter the VPC routing of any of the IP
> addresses (v4 or v6) assigned to an AWS VPC. If you try, for example,
> to assign a /64 to an instance you get a funky error: "Route
> destination doesn't match any subnet CIDR blocks." You can only assign
> the block's IP addresses to subnets or not and then assign addresses
> from the subnet to the instances. You can't have more than 256 subnets
> in a VPC so why would you need more than a /56 of IPv6 addresses?
>

Agreed, those limits align and are reasonable. If you BYO, then you can
bring up to 5 /48's per account, but only use one per region. The limit of
a /56 per VPC remains, but you can create multiple VPCs per region and most
companies use multiple accounts. There are some other limitations but some
of these may change over time:


   -

   The most specific IPv6 address range that you can bring is /48 for CIDRs
   that are publicly advertised, and /56 for CIDRs that are not publicly
   advertised
   

   .
   -

   You can bring each address range to one Region at a time.
   -

   You can bring a total of five IPv4 and IPv6 address ranges per Region to
   your AWS account.
   -

   You cannot share your IP address range with other accounts using AWS
   Resource Access Manager (AWS RAM).


Regards,
> Bill Herrin
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/



>

Re: IPv6 and CDN's

[Dave Taht]

On Sat, Nov 27, 2021 at 12:18 PM William Herrin  wrote:
>
> On Fri, Nov 26, 2021 at 3:07 PM Michael Thomas  wrote:
> >> On 11/26/21 1:44 PM, Jean St-Laurent via NANOG wrote:
> >> Here are some maths and 1 argument kicking ass pitch for CFO’s that use 
> >> iphones.
> >> Apple tells app devs to use IPv6 as it's 1.4 times faster than IPv4
>
> > This really hits my bs meter big time.
>
> If I had to guess, this is an example of correlation is not causation.
> Folks with IPv6 tend to be on savvier service providers who have
> better performance for both IPv4 and IPv6.  To find out for sure,
> you'd have to do an experiment where same-user-same-server connections
> are split between IPv4 and IPv6 and then measure the performance
> difference. I don't know if anyone has done that but these particular
> articles look like someone is just looking at the high-level metrics.
> Those won't hold any statistical validity because they're not actually
> random samples.

We wrote 110+ tests for flent.org to test services under load, you can
use -4 or -6, and all the plots against all different sorts of test
conditions, can be compared against each other easily.

Example of use:

flent --socket-stats --step-size=.05 -l 300 -H
fremont.starlink.taht.net -4 -t ipv4 rrul
flent  --socket-stats --step-size=.05 -l 300 -H
fremont.starlink.taht.net -6-t ipv6 rrul
flent-gui *.flent.gz # and add-other-datafiles

There are also several tests in the flent suite (like rrul46) that try
ipv4 and 6 at exactly the same time. You typically run into (ISP)
bottleneck or (DC) host bandwidth limits first, unless you also
distribute the generated load across multiple machines (I use pdsh for
this, I'd like to know of tools in modern clouds to fire off a load
generator like this, or of other load generators). We have also been
using the irtt tool at a very high resolution (3ms) to map networks'
jitter and latency at "idle" with rather interesting results for
3/4/5g, wifi and starlink, but haven't been breaking down that data
between ipv4 and ipv6 as yet.

Other useful statistics to perhaps gather at scale would be TCP_INFO
rtt, loss, marking, retransmit stats from customer-facing apache or
other proxy servers, and break those down between ipv4 and ipv6 and by
AS.

> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: AWS and IPv6

[William Herrin]

On Sun, Nov 28, 2021 at 3:52 PM Matt Palmer  wrote:
> Which is, fundamentally, half the problem with IPv6 in AWS.  I'd have much
> preferred that they'd added the ability to do actually-useful IPv6 routing
> rather than IPv6-only subnets, which strikes me as more of a toy than
> something *actually* useful.

Yeah, they don't even have a practical way to implement a firewall
instance for IPv6. Unless you want to mirror 1:many NAT for IPv6 like
you do IPv4. You just can't route an IPv6 block to an instance. And
with 1:many NAT you wouldn't want public IP addresses inside but AWS
doesn't let you assign ULA addresses inside the subnet, only global
addresses.

Regards,
Bill Herrin


-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: IPv6 and CDN's

[Owen DeLong via NANOG]

> On Nov 28, 2021, at 02:42 , Mark Tinka  wrote:
> 
> 
> 
> On 11/28/21 06:43, Masataka Ohta wrote:
> 
>> 
>> Here in nanog, we are talking about network operations, considerable
>> part of which can not rely on DNS.
> 
> And yet Facebook were unable to access their kit to fix their recent outage 
> because of it (or, lack of it).

I’d argue that failing to put the correct documentation in place for coping 
with a DNS outage was the bigger
issue than the DNS failure in the Facebook outage… So would a number of the 
engineers I know at Facebook.

> There was a time when knowing the IP(v4) address of every interface of every 
> router in your network was cool. I have never had to care about that in close 
> to 15 years. Right up there with losing interest in making software modems 
> work in Linux, when it was a thing :-).

There was a time when every router in a moderately large network was less than 
50. Those days are gone.

Today, it’s impossible to build large scale networks without depending on 
certain tools. That means that the
failure of those tools can be catastrophic if one is not properly prepared. 
This is simply the modern reality.
Proper preparation is harder than it used to be, but for any such network, 
there should be online and off-line
copies of sufficient documentation (which is adequately maintained) to cope 
restore service of any such
underlying facility quickly in the event of a failure.

Owen

Re: IPv6 and CDN's

[scott]

On 11/28/2021 9:47 AM, Owen DeLong via NANOG wrote:

Why not properly assign /48s to customers and /40s to cities?
--


Side note: I recently tried to get /48 per customer with ARIN on 
repeated emails and they refused.  We were already given an IPv6 block a 
while back.  I told them I wanted to expand it so I could give out a /48 
per customer and that we had more than 65535 customers, which is the 
block we got; 65535 /48s.  I didn't even account for our needs.


Without arguing the reasons, we will have to hand out /56s, rather than 
/48s because of this.  So, it's not all /48-unicorns, puppies and rainbows.


scott

Re: IPv6 and CDN's

[Masataka Ohta]

sro...@ronan-online.com wrote:


It certainly sounds like you’ve never operated a network at scale if
you think knowing the IP address of something reduces Operational
expense.


It's Mark, not me, who said:

: There was a time when knowing the IP(v4) address of every interface
: of every router in your network was cool.


The only way to truly reduce Opex at scale is automation.


Automation by what? DNS?

Masataka Ohta

Re: IPv6 and CDN's

[William Herrin]

On Sun, Nov 28, 2021 at 1:28 PM Dave Bell  wrote:
> On Sun, 28 Nov 2021 at 13:00, Masataka Ohta 
>  wrote:
>> That facebook poorly managed their DNS to cause the recent disaster
>  I don't want to wade into the middle of this argument, but has
> there been more information about the recent facebook outage
> released that I missed? All I've read seems to say that the loss of
> connectivity to their DNS servers was a symptom, rather than the
> cause of the outage.

Hi Dave,

You haven't missed anything. Facebook broke its backbone by mistake.
Before they could restore it, the DNS records went stale and the
now-stale servers withdrew themselves from the network as Facebook
designed them to do when the records go stale. Unfortunately for
Facebook, the internal servers did the same thing as the external
ones. This broke the authentication system which in turn broke
everything else, complicating their efforts to access the various
systems including the ones they could have copied and pasted IP
addresses from.

But, to hear Masataka tell it, copy and paste hasn't been invented yet
so we all type IP addresses by hand on our vt100 CRT terminals.

Regards,
Bill Herrin




--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: AWS and IPv6

[Michael Thomas]

On 11/28/21 3:50 PM, Matt Palmer wrote:

On Sun, Nov 28, 2021 at 02:10:40PM -0800, William Herrin wrote:

On Sun, Nov 28, 2021 at 1:18 PM Karl Auer  wrote:

On Sun, 2021-11-28 at 12:53 -0800, Michael Thomas wrote:

I was reading their howto yesterday and it seems they are only
allocating a /64? Why?

That's a /64 *per subnet*...

But the size of a VPC's IPv6 CIDR block does seem to be fixed at /56.
Would have been nice to see /48 instead.

To what purpose? You can't alter the VPC routing of any of the IP
addresses (v4 or v6) assigned to an AWS VPC.

Which is, fundamentally, half the problem with IPv6 in AWS.  I'd have much
preferred that they'd added the ability to do actually-useful IPv6 routing
rather than IPv6-only subnets, which strikes me as more of a toy than
something *actually* useful.

Maybe they're future proofing themselves until they can figure out how 
to put a meter on it for more $$$?


Mike

Re: IPv6 and CDN's

[Masataka Ohta]

Dave Bell wrote:


That facebook poorly managed their DNS to cause the recent disaster
is an important evidence to support my point that DNS, so often, may
not be helpful for network operations against disastrous failures,
including, but not limited to, DNS failures.



I don't want to wade into the middle of this argument, but has there been
more information about the recent facebook outage released that I missed?


You should have missed:

https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
The end result was that our DNS servers became unreachable even though 
they were still operational. This made it impossible for the rest of the 
internet to find our servers.



All I've read seems to say that the loss of connectivity to their DNS
servers was a symptom, rather than the cause of the outage.


See above.

Another part of the release should also be interesting:

   and second, the total loss of DNS broke many of the internal
   tools we'd normally use to investigate and resolve outages
   like this.

That is an evidence for my statement above "that DNS, so often,
may not be helpful for network operations against disastrous
failures".

You can't rely on automatic tools over DNS, when DNS is failing.

Masataka Ohta

Re: AWS and IPv6

[William Herrin]

On Sun, Nov 28, 2021 at 4:13 PM William Herrin  wrote:
> Yeah, they don't even have a practical way to implement a firewall
> instance for IPv6. Unless you want to mirror 1:many NAT for IPv6 like
> you do IPv4. You just can't route an IPv6 block to an instance. And
> with 1:many NAT you wouldn't want public IP addresses inside but AWS
> doesn't let you assign ULA addresses inside the subnet, only global
> addresses.

I stand corrected on this.

https://aws.amazon.com/blogs/aws/inspect-subnet-to-subnet-traffic-with-amazon-vpc-more-specific-routing/

https://aws.amazon.com/blogs/aws/new-vpc-ingress-routing-simplifying-integration-of-third-party-appliances/

This technique does in fact work for IPv6, allowing you to insert a
firewall at the edge. Interestingly though, it won't receive IPv6
packets for an address that isn't attached to a running instance in
the interior subnet.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: IPv6 and CDN's

[Masataka Ohta]

William Herrin wrote:


But, to hear Masataka tell it, copy and paste hasn't been invented yet
so we all type IP addresses by hand on our vt100 CRT terminals.


You should be using so advanced technologies to input ASCII
text with touch and swipe, which is very slow, even slower
than cut and paste.

But, you should remember that using ASCII keyboard (of vt100
or whatever) is the fastest way to input IPv4 addresses.

Or, maybe, you can't touch type.

Masataka Ohta

Re: IPv6 and CDN's

[Mark Tinka]

On 11/29/21 00:41, scott wrote:

Side note: I recently tried to get /48 per customer with ARIN on 
repeated emails and they refused.  We were already given an IPv6 block 
a while back.  I told them I wanted to expand it so I could give out a 
/48 per customer and that we had more than 65535 customers, which is 
the block we got; 65535 /48s.  I didn't even account for our needs.


Without arguing the reasons, we will have to hand out /56s, rather 
than /48s because of this.  So, it's not all /48-unicorns, puppies and 
rainbows.


We have two types of customers - that that get assigned a /48, and those 
that get assigned a /56.


Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 11/29/21 03:33, Masataka Ohta wrote:

The end result was that our DNS servers became unreachable even though 
they were still operational. This made it impossible for the rest of 
the internet to find our servers.


So your suggestion to map machine addresses to human-readable names 
is... what?


Or should we all just get bigger brains and remember machine addresses 
by heart :-)?




That is an evidence for my statement above "that DNS, so often,
may not be helpful for network operations against disastrous
failures".


Don't be drawn into Facebook's size as being what all operators on the 
Internet do.


If DNS, for all operators, died in the same way it did for Facebook, I'm 
certain we'd all be too busy to answer each other on this thread.


Operations significantly smaller than Facebook have had well-architected 
DNS deployment for yonks. Don't let Facebook's scale leave you with the 
assumption that if they cocked it up, the rest of us don't have a chance 
in hell.


Mark.

Re: IPv6 and CDN's

[Masataka Ohta]

Mark Tinka wrote:


It's Mark, not me, who said:

: There was a time when knowing the IP(v4) address of every interface
: of every router in your network was cool.


In case you missed the nuance, I haven't had to do this in over 20 years.


Say it to Shane, not me. That you two can not communicate well
is not my problem.

Masataka Ohta

Re: IPv6 and CDN's

[Mark Tinka]

On 11/29/21 03:11, Masataka Ohta wrote:



It's Mark, not me, who said:

: There was a time when knowing the IP(v4) address of every interface
: of every router in your network was cool.


In case you missed the nuance, I haven't had to do this in over 20 years.

Mark.