Re: Short-circuited traceroutes on FIOS

[Christopher Morrow]

On Tue, Dec 10, 2019 at 11:44 PM Lee  wrote:
> It's protocol specific.  Windows tracert uses icmp instead of udp.
> On a linux box try
>   ping -t 2 205.132.109.90
>
> You should get a time to live exceeded but the Verizon router gives
> you an echo reply instead.

that's hilariously bad :( I think this is the OLT really that's doing this...
$ ping -t 3 205.132.109.90
PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data.
>From 130.81.32.236 icmp_seq=1 Time to live exceeded

$ ping -t 1 205.132.109.90
PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data.
>From 192.168.100.1 icmp_seq=1 Time to live exceeded

$ ping -t 2 205.132.109.90
PING 205.132.109.90 (205.132.109.90) 56(84) bytes of data.
64 bytes from 205.132.109.90: icmp_seq=1 ttl=254 time=3.38 ms

An outbound traceroute has:
 1  _gateway (192.168.100.1)  2.537 ms  2.587 ms  2.703 ms
 2  * * *
 3  B3320.WASHDC-LCR-21.verizon-gni.net (130.81.32.236)  6.638 ms
B3320.WASHDC-LCR-22.verizon-gni.net (130.81.32.238)  6.223 ms  6.414
ms
...

and inbound that hop 2 is:
 6  HundredGigE2-4-0-3.WASHDC-LCR-22.verizon-gni.NET (140.222.238.55)
5.504 ms HundredGigE2-6-0-3.WASHDC-LCR-21.verizon-gni.NET
(140.222.234.53)  9.261 ms  9.266 ms
 7  ae203-0.WASHDC-VFTTP-320.verizon-gni.net ()  7.955 ms  3.026 ms
ae204-0.WASHDC-VFTTP-320.verizon-gni.net (130.81.32.239)  2.347 ms

oh well, just wonky gpon again?

Re: Short-circuited traceroutes on FIOS

[Lee]

On 12/10/19, Christopher Morrow  wrote:
> On Tue, Dec 10, 2019 at 5:36 PM Nimrod Levy  wrote:
>>
>> Is that unique to the FiOS gateway device? I don't use their router and my
>> traces go right out.
>>
>
> I also don't use their device  and:
> $ traceroute 205.132.109.90
> traceroute to 205.132.109.90 (205.132.109.90), 30 hops max, 60 byte packets
>  1  _gateway (192.168.100.1)  3.085 ms  2.990 ms  2.795 ms
> 
> 15  * 12.250.138.90 (12.250.138.90)  65.970 ms *^C
>
> -chris
> (perhaps this is location specific? I'm in the ashburn-ish-area-ish)

It's protocol specific.  Windows tracert uses icmp instead of udp.
On a linux box try
  ping -t 2 205.132.109.90

You should get a time to live exceeded but the Verizon router gives
you an echo reply instead.

Regards,
Lee


>> On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon  wrote:
>>>
>>> Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes
>>> right on their gateways.
>>>
>>> More internet breakage. Thanks for the information to all who responded.
>>>
>>> Random control test.
>>>
>>> C:\Users\Home>tracert -d 1.4.5.6
>>>
>>> Tracing route to 1.4.5.6 over a maximum of 30 hops
>>>
>>>115 ms 5 ms<1 ms  172.18.24.1
>>>2 3 ms23 ms24 ms  192.168.2.33
>>>3 3 ms 6 ms 3 ms  1.4.5.6
>>>
>>> Trace complete.
>>>
>>>
>>> Joe
>>>
>>> Joe Maimon wrote:
>>> > Anyone have an idea why there are some destinations that on
>>> > residential verizon fios here in NY area terminate right on first
>>> > external hop?
>>> >
>>> > There seems to be a CDN common denominator here. On other networks
>>> > with more typical BGP paths and traceroutes, users are reporting
>>> > issues accessing these sites.
>>> >
>>> > C:\Users\Home>tracert www.usfoods.com
>>> >
>>> > Tracing route to statics.usfoods.com [205.132.109.90]
>>> > over a maximum of 30 hops:
>>> >
>>> >   1 3 ms<1 ms<1 ms  172.18.24.1
>>> >   2 4 ms 3 ms 3 ms  192.168.2.33
>>> >   317 ms 6 ms 3 ms  statics.usfoods.com [205.132.109.90]
>>> >
>>> > Trace complete.
>>> >
>>> > C:\Users\Home>tracert atworkhp.americanexpress.com
>>> >
>>> > Tracing route to atworkhp.americanexpress.com.akadns.net
>>> > [139.71.19.87]
>>> > over a maximum of 30 hops:
>>> >
>>> >   1 2 ms<1 ms<1 ms  172.18.24.1
>>> >   2 3 ms 4 ms23 ms  192.168.2.33
>>> >   321 ms11 ms 5 ms atworkhomepage2.americanexpress.com
>>> > [139.71.19.87]
>>> >
>>> > Trace complete.
>>> >
>>> > C:\Users\Home>tracert portal.discover.com
>>> >
>>> > Tracing route to e14577.x.akamaiedge.net [23.51.172.254]
>>> > over a maximum of 30 hops:
>>> >
>>> >   1 3 ms 1 ms18 ms  172.18.24.1
>>> >   221 ms 7 ms 6 ms  192.168.2.33
>>> >   3 4 ms 2 ms 2 ms
>>> > a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
>>> >
>>> > Trace complete.
>>> >
>>> >
>>> >
>>>
>

Re: Short-circuited traceroutes on FIOS

[Christopher Morrow]

On Tue, Dec 10, 2019 at 5:36 PM Nimrod Levy  wrote:
>
> Is that unique to the FiOS gateway device? I don't use their router and my 
> traces go right out.
>

I also don't use their device  and:
$ traceroute 205.132.109.90
traceroute to 205.132.109.90 (205.132.109.90), 30 hops max, 60 byte packets
 1  _gateway (192.168.100.1)  3.085 ms  2.990 ms  2.795 ms

15  * 12.250.138.90 (12.250.138.90)  65.970 ms *^C

-chris
(perhaps this is location specific? I'm in the ashburn-ish-area-ish)

>
> On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon  wrote:
>>
>> Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes
>> right on their gateways.
>>
>> More internet breakage. Thanks for the information to all who responded.
>>
>> Random control test.
>>
>> C:\Users\Home>tracert -d 1.4.5.6
>>
>> Tracing route to 1.4.5.6 over a maximum of 30 hops
>>
>>115 ms 5 ms<1 ms  172.18.24.1
>>2 3 ms23 ms24 ms  192.168.2.33
>>3 3 ms 6 ms 3 ms  1.4.5.6
>>
>> Trace complete.
>>
>>
>> Joe
>>
>> Joe Maimon wrote:
>> > Anyone have an idea why there are some destinations that on
>> > residential verizon fios here in NY area terminate right on first
>> > external hop?
>> >
>> > There seems to be a CDN common denominator here. On other networks
>> > with more typical BGP paths and traceroutes, users are reporting
>> > issues accessing these sites.
>> >
>> > C:\Users\Home>tracert www.usfoods.com
>> >
>> > Tracing route to statics.usfoods.com [205.132.109.90]
>> > over a maximum of 30 hops:
>> >
>> >   1 3 ms<1 ms<1 ms  172.18.24.1
>> >   2 4 ms 3 ms 3 ms  192.168.2.33
>> >   317 ms 6 ms 3 ms  statics.usfoods.com [205.132.109.90]
>> >
>> > Trace complete.
>> >
>> > C:\Users\Home>tracert atworkhp.americanexpress.com
>> >
>> > Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87]
>> > over a maximum of 30 hops:
>> >
>> >   1 2 ms<1 ms<1 ms  172.18.24.1
>> >   2 3 ms 4 ms23 ms  192.168.2.33
>> >   321 ms11 ms 5 ms atworkhomepage2.americanexpress.com
>> > [139.71.19.87]
>> >
>> > Trace complete.
>> >
>> > C:\Users\Home>tracert portal.discover.com
>> >
>> > Tracing route to e14577.x.akamaiedge.net [23.51.172.254]
>> > over a maximum of 30 hops:
>> >
>> >   1 3 ms 1 ms18 ms  172.18.24.1
>> >   221 ms 7 ms 6 ms  192.168.2.33
>> >   3 4 ms 2 ms 2 ms
>> > a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
>> >
>> > Trace complete.
>> >
>> >
>> >
>>

Re: Starting to Drop Invalids for Customers

[Christopher Morrow]

On Tue, Dec 10, 2019 at 7:32 PM Rubens Kuhl  wrote:
>
>
>>
>> RPKI ROAs (compared to IRR objects) carry different meaning: the existence 
>> of a ROA (both by definition and common implementation) supersedes other 
>> data sources (IRR, LOAs, or comments in whois records, etc), and as such can 
>> be used on any type of EBGP session for validation of the received Internet 
>> routing information.
>>
>
> Which brings me to my favorite possible RPKI-IRR integration: a ROA that says 
> that IRR objects on IRR source x with maintainer Y are authoritative for a 
> given number resource. Kinda like SPF for BGP.
>

Is this required? or a crutch for use until a network can publish all
of their routing data in the RPKI?

-chris

Re: Short-circuited traceroutes on FIOS

[Javier J]

mtr -u 4.2.2.2 --report-wide
Start: 2019-12-10T21:26:20-0500
HOST: fedora-lenovo   Loss%   Snt   Last   Avg  Best
 Wrst StDev
  1.|-- _gateway 0.0%101.3   1.4   1.1
  2.3   0.3
  2.|-- ??? 100.0100.0   0.0   0.0
  0.0   0.0
  3.|-- b3346.nwrknj-lcr-21.verizon-gni.net  0.0%106.7   4.8   2.2
  8.2   1.9
  4.|-- ??? 100.0100.0   0.0   0.0
  0.0   0.0
  5.|-- 0.ae1.br1.ewr6.alter.net 0.0%10   18.9   6.2   3.6
 18.9   4.6
  6.|-- lag-12.ear3.newark1.level3.net  20.0%103.9   3.3   2.2
  4.4   1.0
  7.|-- ae-2-3602.ear2.newyork1.level3.net  90.0%105.6   5.6   5.6
  5.6   0.0
  8.|-- ??? 100.0100.0   0.0   0.0
  0.0   0.0

Verizon FIOS
They are blocking ICMP and returning false responses right at their gateway
at the CO, not the CPE (I'm using my own router)

You have to do it using UDP to get real results of a traceroute.

- Javier



On Tue, Dec 10, 2019 at 3:56 PM Joe Maimon  wrote:

> This is not from a verizon CPE. Its happening on their CO internet
> gateway customer facing routers.
>
> tcptraceroute looks more legit
>
> Joe
>
> Nimrod Levy wrote:
> > Is that unique to the FiOS gateway device? I don't use their router
> > and my traces go right out.
> >
> >
> > On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon  > > wrote:
> >
> > Apparently Verizon FIOS is a red herring, terminating ICMP
> > traceroutes
> > right on their gateways.
> >
> > More internet breakage. Thanks for the information to all who
> > responded.
> >
> > Random control test.
> >
> > C:\Users\Home>tracert -d 1.4.5.6
> >
> > Tracing route to 1.4.5.6 over a maximum of 30 hops
> >
> >115 ms 5 ms<1 ms  172.18.24.1
> >2 3 ms23 ms24 ms  192.168.2.33
> >3 3 ms 6 ms 3 ms  1.4.5.6
> >
> > Trace complete.
> >
> >
> > Joe
> >
> > Joe Maimon wrote:
> > > Anyone have an idea why there are some destinations that on
> > > residential verizon fios here in NY area terminate right on first
> > > external hop?
> > >
> > > There seems to be a CDN common denominator here. On other networks
> > > with more typical BGP paths and traceroutes, users are reporting
> > > issues accessing these sites.
> > >
> > > C:\Users\Home>tracert www.usfoods.com 
> > >
> > > Tracing route to statics.usfoods.com
> >  [205.132.109.90]
> > > over a maximum of 30 hops:
> > >
> > >   1 3 ms<1 ms<1 ms  172.18.24.1
> > >   2 4 ms 3 ms 3 ms  192.168.2.33
> > >   317 ms 6 ms 3 ms statics.usfoods.com
> >  [205.132.109.90]
> > >
> > > Trace complete.
> > >
> > > C:\Users\Home>tracert atworkhp.americanexpress.com
> > 
> > >
> > > Tracing route to atworkhp.americanexpress.com.akadns.net
> >  [139.71.19.87]
> > > over a maximum of 30 hops:
> > >
> > >   1 2 ms<1 ms<1 ms  172.18.24.1
> > >   2 3 ms 4 ms23 ms  192.168.2.33
> > >   321 ms11 ms 5 ms
> > atworkhomepage2.americanexpress.com
> > 
> > > [139.71.19.87]
> > >
> > > Trace complete.
> > >
> > > C:\Users\Home>tracert portal.discover.com
> > 
> > >
> > > Tracing route to e14577.x.akamaiedge.net
> >  [23.51.172.254]
> > > over a maximum of 30 hops:
> > >
> > >   1 3 ms 1 ms18 ms  172.18.24.1
> > >   221 ms 7 ms 6 ms  192.168.2.33
> > >   3 4 ms 2 ms 2 ms
> > > a23-51-172-254.deploy.static.akamaitechnologies.com
> > 
> > [23.51.172.254]
> > >
> > > Trace complete.
> > >
> > >
> > >
> >
>
>

Re: Starting to Drop Invalids for Customers

[Rubens Kuhl]

>
> RPKI ROAs (compared to IRR objects) carry different meaning: the existence
> of a ROA (both by definition and common implementation) supersedes other
> data sources (IRR, LOAs, or comments in whois records, etc), and as such
> can be used on any type of EBGP session for validation of the received
> Internet routing information.
>
>
Which brings me to my favorite possible RPKI-IRR integration: a ROA that
says that IRR objects on IRR source x with maintainer Y are authoritative
for a given number resource. Kinda like SPF for BGP.



Rubens

Re: DDoS attack

[Töma Gavrichenkov]

Peace,

On Mon, Dec 9, 2019 at 11:35 PM Florian Brandstetter via NANOG
 wrote:
> if that was to be amplification, the source addresses
> would not be within Google or CloudFlare ranges
> (especially not CloudFlare, as they are not running
> a vulnerable recursor

Well, vulnerable — arguably of course, amplifying — yes, a few, around
twenty.  Not sure if they have any kind of rate limiting there (also
not sure if it's legal for me to check it), expecially given that the
queries could come from spoofed sources.  Anyway, in theory, their
sources *could* be present in a DDoS (though not likely).

12:11:23.726699 IP (tos 0x0, ttl 64, id 9173, offset 0, flags [none],
proto UDP (17), length 60)
$IP.60801 > 172.65.253.110.53: 45631+ [1au] ANY? com. (32)
12:11:23.733976 IP (tos 0x0, ttl 60, id 30234, offset 0, flags [+],
proto UDP (17), length 1500)
172.65.253.110.53 > $IP.60801: 45631$ 22/0/1 com. SOA
a.gtld-servers.net. nstld.verisign-grs.com. 1576020207 1800 900 604800
86400, com. RRSIG, com. NS a.gtld-servers.net., com. NS
b.gtld-servers.net., com. NS c.gtld-servers.net., com. NS
e.gtld-servers.net., com. NS i.gtld-servers.net., com. NS
j.gtld-servers.net., com. NS g.gtld-servers.net., com. NS
f.gtld-servers.net., com. NS l.gtld-servers.net., com. NS
d.gtld-servers.net., com. NS k.gtld-servers.net., com. NS
h.gtld-servers.net., com. NS m.gtld-servers.net., com. RRSIG, com.
DNSKEY, com. DNSKEY, com. DNSKEY, com. RRSIG[|domain]

--
Töma

RE: [EXTERNAL] RE: DDoS attack

[Paul Amaral via NANOG]

Rarely will sourced ips be the same every time a victim gets DDOS'd. Good 
telemetry is key but every time the attack happens it needs to be looked at.  I 
find bogon prefixes are not as used much, especially amplification attacks.  
Gathering good intel and blocking bogons will help,  but there is no one 
strategy that works. You also will always risk blocking some good traffic. 
Again, there's a reason why you can only mitigate and not stop a DDOS 
completely. 

Paul  

-Original Message-
From: Nikos Leontsinis  
Sent: Tuesday, December 10, 2019 5:19 PM
To: Aaron Gould ; 'Paul Amaral' ; 
ahmed.dala...@hrins.net; Nanog@nanog.org
Subject: RE: [EXTERNAL] RE: DDoS attack 

You can get the bogon prefixes from Cymru and defend your network using them in 
combination with rpf The key with the attacks dos or ddos is to have proper 
telemetry (streaming telemetry not polling telemetry) and baselines without 
this information you run the danger of blocking good traffic.

Based on the thread below I don't see any evidence of an attack only 
speculations.

nikos

-Original Message-
From: NANOG  On Behalf Of Aaron Gould
Sent: Tuesday, December 10, 2019 5:05 PM
To: 'Paul Amaral' ; ahmed.dala...@hrins.net; Nanog@nanog.org
Subject: [EXTERNAL] RE: DDoS attack

Years ago, we looked at netflow data and precursors to attacks, and found that 
UDP 3074 Xbox Live was showing up just prior to the attacks...and through other 
research we concluded that gamers are a big cause of large ddos attacks 
apparently they go after each other in retaliation

I've crafted a series of things for dealing with the results of volumetric ddos 
attacks... I've had attacks in upwards of 50 or 60 gig as I recall across 
all of my (3) internet connections at times

- deny acl's ... for ports/protocols that I know are absolutely not needed
- policers of various well known port attack vectors (gleaned from netflow data)
- policers of well-known *good* ports/protocols (like ntp, dns, etc) to some 
realistic level
- a repeat-victims list of ip's with policing udp for this group (note1)
- rtbh (note2)

Note 1 - Also, I've learned that if a customer has been attack once, the 
chances of them being the target of an attack again is highso by crafting 
the repeat victims list, you can catch next-day attacks of differing vectors.
Note 2 - for sustained attacks lasting a long time (30 mins, an hour, etc), we 
trigger a bgp/community route that goes out to the inet cloud and stops attack 
further into the upstream providers network... I know I "complete" the attack, 
but, I save my network ;) ...I use an old cisco 2600 as my trigger router and 
wrote a job aid that I shared with the NOC for triggering rtbh when needed, 
couple commands.
...I would like to automate my rtbh using what I understand is a possibly use 
case for FastNetMon, but haven't got around to it

I also wonder if team cymru's utrs project and other things like that would 
benefit my security posture.


-Aaron


This email is from Equinix (EMEA) B.V. or one of its associated companies in 
the territory from where this email has been sent. This email, and any files 
transmitted with it, contains information which is confidential, is solely for 
the use of the intended recipient and may be legally privileged. If you have 
received this email in error, please notify the sender and delete this email 
immediately. Equinix (EMEA) B.V.. Registered Office: Amstelplein 1, 1096 HA 
Amsterdam, The Netherlands. Registered in The Netherlands No. 57577889.

Re: Short-circuited traceroutes on FIOS

[Nimrod Levy]

Is that unique to the FiOS gateway device? I don't use their router and my
traces go right out.


On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon  wrote:

> Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes
> right on their gateways.
>
> More internet breakage. Thanks for the information to all who responded.
>
> Random control test.
>
> C:\Users\Home>tracert -d 1.4.5.6
>
> Tracing route to 1.4.5.6 over a maximum of 30 hops
>
>115 ms 5 ms<1 ms  172.18.24.1
>2 3 ms23 ms24 ms  192.168.2.33
>3 3 ms 6 ms 3 ms  1.4.5.6
>
> Trace complete.
>
>
> Joe
>
> Joe Maimon wrote:
> > Anyone have an idea why there are some destinations that on
> > residential verizon fios here in NY area terminate right on first
> > external hop?
> >
> > There seems to be a CDN common denominator here. On other networks
> > with more typical BGP paths and traceroutes, users are reporting
> > issues accessing these sites.
> >
> > C:\Users\Home>tracert www.usfoods.com
> >
> > Tracing route to statics.usfoods.com [205.132.109.90]
> > over a maximum of 30 hops:
> >
> >   1 3 ms<1 ms<1 ms  172.18.24.1
> >   2 4 ms 3 ms 3 ms  192.168.2.33
> >   317 ms 6 ms 3 ms  statics.usfoods.com [205.132.109.90]
> >
> > Trace complete.
> >
> > C:\Users\Home>tracert atworkhp.americanexpress.com
> >
> > Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87]
> > over a maximum of 30 hops:
> >
> >   1 2 ms<1 ms<1 ms  172.18.24.1
> >   2 3 ms 4 ms23 ms  192.168.2.33
> >   321 ms11 ms 5 ms atworkhomepage2.americanexpress.com
> > [139.71.19.87]
> >
> > Trace complete.
> >
> > C:\Users\Home>tracert portal.discover.com
> >
> > Tracing route to e14577.x.akamaiedge.net [23.51.172.254]
> > over a maximum of 30 hops:
> >
> >   1 3 ms 1 ms18 ms  172.18.24.1
> >   221 ms 7 ms 6 ms  192.168.2.33
> >   3 4 ms 2 ms 2 ms
> > a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
> >
> > Trace complete.
> >
> >
> >
>
>

RE: [EXTERNAL] RE: DDoS attack

[Nikos Leontsinis]

You can get the bogon prefixes from Cymru and defend your network using them in 
combination with rpf
The key with the attacks dos or ddos is to have proper telemetry (streaming 
telemetry not polling telemetry)
and baselines without this information you run the danger of blocking good 
traffic.

Based on the thread below I don't see any evidence of an attack only 
speculations.

nikos

-Original Message-
From: NANOG  On Behalf Of Aaron Gould
Sent: Tuesday, December 10, 2019 5:05 PM
To: 'Paul Amaral' ; ahmed.dala...@hrins.net; Nanog@nanog.org
Subject: [EXTERNAL] RE: DDoS attack

Years ago, we looked at netflow data and precursors to attacks, and found that 
UDP 3074 Xbox Live was showing up just prior to the attacks...and through other 
research we concluded that gamers are a big cause of large ddos attacks 
apparently they go after each other in retaliation

I've crafted a series of things for dealing with the results of volumetric ddos 
attacks... I've had attacks in upwards of 50 or 60 gig as I recall across 
all of my (3) internet connections at times

- deny acl's ... for ports/protocols that I know are absolutely not needed
- policers of various well known port attack vectors (gleaned from netflow data)
- policers of well-known *good* ports/protocols (like ntp, dns, etc) to some 
realistic level
- a repeat-victims list of ip's with policing udp for this group (note1)
- rtbh (note2)

Note 1 - Also, I've learned that if a customer has been attack once, the 
chances of them being the target of an attack again is highso by crafting 
the repeat victims list, you can catch next-day attacks of differing vectors.
Note 2 - for sustained attacks lasting a long time (30 mins, an hour, etc), we 
trigger a bgp/community route that goes out to the inet cloud and stops attack 
further into the upstream providers network... I know I "complete" the attack, 
but, I save my network ;) ...I use an old cisco 2600 as my trigger router and 
wrote a job aid that I shared with the NOC for triggering rtbh when needed, 
couple commands.
...I would like to automate my rtbh using what I understand is a possibly use 
case for FastNetMon, but haven't got around to it

I also wonder if team cymru's utrs project and other things like that would 
benefit my security posture.


-Aaron


This email is from Equinix (EMEA) B.V. or one of its associated companies in 
the territory from where this email has been sent. This email, and any files 
transmitted with it, contains information which is confidential, is solely for 
the use of the intended recipient and may be legally privileged. If you have 
received this email in error, please notify the sender and delete this email 
immediately. Equinix (EMEA) B.V.. Registered Office: Amstelplein 1, 1096 HA 
Amsterdam, The Netherlands. Registered in The Netherlands No. 57577889.

Re: Starting to Drop Invalids for Customers

[Job Snijders]

Dear Arturo, group,

On Tue, Dec 10, 2019 at 20:51 Arturo Servin  wrote:

>
> Invalid according to RPKI or IRR? Or both?
>

In this context the use of the word “invalid” refers to the result of
validation procedure described in RFC 6811 - which is to match received BGP
updates to the RPKI and attach either of “valid”, “invalid”, or “not-found”.

In IRR, the challenge has always been that “route:” objects describe a
state of the network that may exist, but the semantics of “route:” objects
don’t allow extrapolation towards what should definitely *not* exist in the
BGP Default-Free Zone.

RPKI ROAs (compared to IRR objects) carry different meaning: the existence
of a ROA (both by definition and common implementation) supersedes other
data sources (IRR, LOAs, or comments in whois records, etc), and as such
can be used on any type of EBGP session for validation of the received
Internet routing information.

Kind regards,

Job

>

Re: Short-circuited traceroutes on FIOS

[Joe Maimon]

This is not from a verizon CPE. Its happening on their CO internet 
gateway customer facing routers.


tcptraceroute looks more legit

Joe

Nimrod Levy wrote:
Is that unique to the FiOS gateway device? I don't use their router 
and my traces go right out.



On Tue, Dec 10, 2019 at 3:08 PM Joe Maimon > wrote:


Apparently Verizon FIOS is a red herring, terminating ICMP
traceroutes
right on their gateways.

More internet breakage. Thanks for the information to all who
responded.

Random control test.

C:\Users\Home>tracert -d 1.4.5.6

Tracing route to 1.4.5.6 over a maximum of 30 hops

   115 ms 5 ms<1 ms  172.18.24.1
   2 3 ms23 ms24 ms  192.168.2.33
   3 3 ms 6 ms 3 ms  1.4.5.6

Trace complete.


Joe

Joe Maimon wrote:
> Anyone have an idea why there are some destinations that on
> residential verizon fios here in NY area terminate right on first
> external hop?
>
> There seems to be a CDN common denominator here. On other networks
> with more typical BGP paths and traceroutes, users are reporting
> issues accessing these sites.
>
> C:\Users\Home>tracert www.usfoods.com 
>
> Tracing route to statics.usfoods.com
 [205.132.109.90]
> over a maximum of 30 hops:
>
>   1 3 ms<1 ms<1 ms  172.18.24.1
>   2 4 ms 3 ms 3 ms  192.168.2.33
>   317 ms 6 ms 3 ms statics.usfoods.com
 [205.132.109.90]
>
> Trace complete.
>
> C:\Users\Home>tracert atworkhp.americanexpress.com

>
> Tracing route to atworkhp.americanexpress.com.akadns.net
 [139.71.19.87]
> over a maximum of 30 hops:
>
>   1 2 ms<1 ms<1 ms  172.18.24.1
>   2 3 ms 4 ms23 ms  192.168.2.33
>   321 ms11 ms 5 ms
atworkhomepage2.americanexpress.com

> [139.71.19.87]
>
> Trace complete.
>
> C:\Users\Home>tracert portal.discover.com

>
> Tracing route to e14577.x.akamaiedge.net
 [23.51.172.254]
> over a maximum of 30 hops:
>
>   1 3 ms 1 ms18 ms  172.18.24.1
>   221 ms 7 ms 6 ms  192.168.2.33
>   3 4 ms 2 ms 2 ms
> a23-51-172-254.deploy.static.akamaitechnologies.com

[23.51.172.254]
>
> Trace complete.
>
>
>

Re: Short-circuited traceroutes on FIOS

[Filip Hruska]

I had this issue while looking at Ripe Atlas measurements.

Turns out these Verizon boxes spoof ICMP with TTL = 3 (or 2, I don't 
recall). Try doing a UDP or TCP based traceroute instead.


Maybe you're seeing the same problem.

Kind Regards,
Filip

On 12/10/19 8:47 PM, Joe Maimon wrote:
Anyone have an idea why there are some destinations that on 
residential verizon fios here in NY area terminate right on first 
external hop?


There seems to be a CDN common denominator here. On other networks 
with more typical BGP paths and traceroutes, users are reporting 
issues accessing these sites.


C:\Users\Home>tracert www.usfoods.com

Tracing route to statics.usfoods.com [205.132.109.90]
over a maximum of 30 hops:

  1 3 ms    <1 ms    <1 ms  172.18.24.1
  2 4 ms 3 ms 3 ms  192.168.2.33
  3    17 ms 6 ms 3 ms  statics.usfoods.com [205.132.109.90]

Trace complete.

C:\Users\Home>tracert atworkhp.americanexpress.com

Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87]
over a maximum of 30 hops:

  1 2 ms    <1 ms    <1 ms  172.18.24.1
  2 3 ms 4 ms    23 ms  192.168.2.33
  3    21 ms    11 ms 5 ms atworkhomepage2.americanexpress.com 
[139.71.19.87]


Trace complete.

C:\Users\Home>tracert portal.discover.com

Tracing route to e14577.x.akamaiedge.net [23.51.172.254]
over a maximum of 30 hops:

  1 3 ms 1 ms    18 ms  172.18.24.1
  2    21 ms 7 ms 6 ms  192.168.2.33
  3 4 ms 2 ms 2 ms 
a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]


Trace complete.



--
Filip Hruska
Linux System Administrator

Re: Short-circuited traceroutes on FIOS

[Christopher Morrow]

wasn't vz pursuing some 'get the a cdn in the central office' for a
time? :) perhaps this is the manifestation of that? :)
or perhaps jared arranged to get links back from each CO to his
network gear in akamai-land?

I love conspiracies!

On Tue, Dec 10, 2019 at 2:48 PM Joe Maimon  wrote:
>
> Anyone have an idea why there are some destinations that on residential
> verizon fios here in NY area terminate right on first external hop?
>
> There seems to be a CDN common denominator here. On other networks with
> more typical BGP paths and traceroutes, users are reporting issues
> accessing these sites.
>
> C:\Users\Home>tracert www.usfoods.com
>
> Tracing route to statics.usfoods.com [205.132.109.90]
> over a maximum of 30 hops:
>
>1 3 ms<1 ms<1 ms  172.18.24.1
>2 4 ms 3 ms 3 ms  192.168.2.33
>317 ms 6 ms 3 ms  statics.usfoods.com [205.132.109.90]
>
> Trace complete.
>
> C:\Users\Home>tracert atworkhp.americanexpress.com
>
> Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87]
> over a maximum of 30 hops:
>
>1 2 ms<1 ms<1 ms  172.18.24.1
>2 3 ms 4 ms23 ms  192.168.2.33
>321 ms11 ms 5 ms  atworkhomepage2.americanexpress.com
> [139.71.19.87]
>
> Trace complete.
>
> C:\Users\Home>tracert portal.discover.com
>
> Tracing route to e14577.x.akamaiedge.net [23.51.172.254]
> over a maximum of 30 hops:
>
>1 3 ms 1 ms18 ms  172.18.24.1
>221 ms 7 ms 6 ms  192.168.2.33
>3 4 ms 2 ms 2 ms
> a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]
>
> Trace complete.
>
>

Re: Short-circuited traceroutes on FIOS

[Joe Maimon]

Apparently Verizon FIOS is a red herring, terminating ICMP traceroutes 
right on their gateways.


More internet breakage. Thanks for the information to all who responded.

Random control test.

C:\Users\Home>tracert -d 1.4.5.6

Tracing route to 1.4.5.6 over a maximum of 30 hops

  115 ms 5 ms<1 ms  172.18.24.1
  2 3 ms23 ms24 ms  192.168.2.33
  3 3 ms 6 ms 3 ms  1.4.5.6

Trace complete.


Joe

Joe Maimon wrote:
Anyone have an idea why there are some destinations that on 
residential verizon fios here in NY area terminate right on first 
external hop?


There seems to be a CDN common denominator here. On other networks 
with more typical BGP paths and traceroutes, users are reporting 
issues accessing these sites.


C:\Users\Home>tracert www.usfoods.com

Tracing route to statics.usfoods.com [205.132.109.90]
over a maximum of 30 hops:

  1 3 ms<1 ms<1 ms  172.18.24.1
  2 4 ms 3 ms 3 ms  192.168.2.33
  317 ms 6 ms 3 ms  statics.usfoods.com [205.132.109.90]

Trace complete.

C:\Users\Home>tracert atworkhp.americanexpress.com

Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87]
over a maximum of 30 hops:

  1 2 ms<1 ms<1 ms  172.18.24.1
  2 3 ms 4 ms23 ms  192.168.2.33
  321 ms11 ms 5 ms atworkhomepage2.americanexpress.com 
[139.71.19.87]


Trace complete.

C:\Users\Home>tracert portal.discover.com

Tracing route to e14577.x.akamaiedge.net [23.51.172.254]
over a maximum of 30 hops:

  1 3 ms 1 ms18 ms  172.18.24.1
  221 ms 7 ms 6 ms  192.168.2.33
  3 4 ms 2 ms 2 ms 
a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]


Trace complete.

Re: Starting to Drop Invalids for Customers

[Arturo Servin]

Mark

Invalid according to RPKI or IRR? Or both?

Regards
as

On Tue, 10 Dec 2019, 18:22 Randy Bush,  wrote:

> mark,
>
> > Just to let this group know that we've started the process of
> > activating the dropping of Invalids for all our eBGP customers.
>
> cool.  any stats and lessons appreciated.
>
> randy
>

Short-circuited traceroutes on FIOS

[Joe Maimon]

Anyone have an idea why there are some destinations that on residential 
verizon fios here in NY area terminate right on first external hop?


There seems to be a CDN common denominator here. On other networks with 
more typical BGP paths and traceroutes, users are reporting issues 
accessing these sites.


C:\Users\Home>tracert www.usfoods.com

Tracing route to statics.usfoods.com [205.132.109.90]
over a maximum of 30 hops:

  1 3 ms<1 ms<1 ms  172.18.24.1
  2 4 ms 3 ms 3 ms  192.168.2.33
  317 ms 6 ms 3 ms  statics.usfoods.com [205.132.109.90]

Trace complete.

C:\Users\Home>tracert atworkhp.americanexpress.com

Tracing route to atworkhp.americanexpress.com.akadns.net [139.71.19.87]
over a maximum of 30 hops:

  1 2 ms<1 ms<1 ms  172.18.24.1
  2 3 ms 4 ms23 ms  192.168.2.33
  321 ms11 ms 5 ms  atworkhomepage2.americanexpress.com 
[139.71.19.87]


Trace complete.

C:\Users\Home>tracert portal.discover.com

Tracing route to e14577.x.akamaiedge.net [23.51.172.254]
over a maximum of 30 hops:

  1 3 ms 1 ms18 ms  172.18.24.1
  221 ms 7 ms 6 ms  192.168.2.33
  3 4 ms 2 ms 2 ms 
a23-51-172-254.deploy.static.akamaitechnologies.com [23.51.172.254]


Trace complete.

NFL Sunday Ticket - Online Streaming service

[Justin Krejci]

I am looking for a contact in the network group (may be called National 
Escalation team or NatEsc team internally) within AT/DirecTV pertaining to 
the NFL Sunday Ticket online streaming service. I have been attempting to work 
through their normal support process for quite some time, they are extremely 
isolated and handicapped in the support center and I am so frustrated at the 
impossibility to get any traction through their call center to deal with a 
network related issue. I have had 4 cases go to their escalation team and they 
all mysteriously close with no real or valid resolution. I've spoken with about 
4 or 5 supervisors in their Tulsa call center and they are unable to do much. 
It has been a nightmare and I am hoping someone has a contact they can get me 
in touch with.

Re: restricted hotel block

[Matthew Petach]

Which hotel was that?  I might want to go, just to take advantage of the
discount...  ^_^

Matt



On Tue, Dec 10, 2019, 09:36 Randy Bush  wrote:

> is anyone aware of any conference other than nanog which does
>
> Online Reservations: (Open exclusively to NANOG Members only from
> December 2 - December 16)
>
> randy
>
>

Re: restricted hotel block

[Josh Luthman]

Online reservations?  Yes

Exclusively only reservations?  Yes

Restricted to a 2 week window?  No - I'd guess this was to keep it from
being so open ended and increase the cost of running the show.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Tue, Dec 10, 2019 at 12:35 PM Randy Bush  wrote:

> is anyone aware of any conference other than nanog which does
>
> Online Reservations: (Open exclusively to NANOG Members only from
> December 2 - December 16)
>
> randy
>

restricted hotel block

[Randy Bush]

is anyone aware of any conference other than nanog which does

Online Reservations: (Open exclusively to NANOG Members only from
December 2 - December 16)

randy

Re: DDoS attack

[Saku Ytti]

On Tue, 10 Dec 2019 at 19:08, Aaron Gould  wrote:

> - policers of well-known *good* ports/protocols (like ntp, dns, etc) to some 
> realistic level

You might want to downpref these to a scavanger class, instead of
police. Since ultimately policing makes it just easier to ddos the
service, which is actually needed.

-- 
  ++ytti

Re: Starting to Drop Invalids for Customers

[Randy Bush]

mark,

> Just to let this group know that we've started the process of
> activating the dropping of Invalids for all our eBGP customers.

cool.  any stats and lessons appreciated.

randy

RE: DDoS attack

[Aaron Gould]

Years ago, we looked at netflow data and precursors to attacks, and found that 
UDP 3074 Xbox Live was showing up just prior to the attacks...and through other 
research we concluded that gamers are a big cause of large ddos attacks 
apparently they go after each other in retaliation

I've crafted a series of things for dealing with the results of volumetric ddos 
attacks... I've had attacks in upwards of 50 or 60 gig as I recall across 
all of my (3) internet connections at times

- deny acl's ... for ports/protocols that I know are absolutely not needed
- policers of various well known port attack vectors (gleaned from netflow data)
- policers of well-known *good* ports/protocols (like ntp, dns, etc) to some 
realistic level
- a repeat-victims list of ip's with policing udp for this group (note1)
- rtbh (note2)

Note 1 - Also, I've learned that if a customer has been attack once, the 
chances of them being the target of an attack again is highso by crafting 
the repeat victims list, you can catch next-day attacks of differing vectors.
Note 2 - for sustained attacks lasting a long time (30 mins, an hour, etc), we 
trigger a bgp/community route that goes out to the inet cloud and stops attack 
further into the upstream providers network... I know I "complete" the attack, 
but, I save my network ;)
...I use an old cisco 2600 as my trigger router and wrote a job aid that I 
shared with the NOC for triggering rtbh when needed, couple commands.
...I would like to automate my rtbh using what I understand is a possibly use 
case for FastNetMon, but haven't got around to it

I also wonder if team cymru's utrs project and other things like that would 
benefit my security posture.


-Aaron

Re: Is anyone able to contact GTT?

[Mat Perkins]

i...@gtt.net

On Tue, Dec 10, 2019 at 7:52 AM Bottiger  wrote:

> I sent an email to noc at gtt.net from 2 different emails and both got a
> reply saying:
>
> 5.1.0 - Unknown address error 550-'5.4.1 Recipient address rejected:
> Access denied [HE1EUR01FT058.eop-EUR01.prod.protection.outlook.com]'
>
> Not sure if this means if they are blocking my email or if their email is
> broken.
>

RE: Is anyone able to contact GTT?

[Rob Wcislo]

I’d like to assist here.

Do you have access to Ethervision.  The customer portal is the most efficient 
way to initiate and track NOC tickets.

OR Try calling: USA Toll Free: +1 877-385-5252, +1 800-583-1388.

If you still have trouble, please reach me directly and I’ll get you to the 
right folks…


Rob Wcislo
VP, Sales
office: +1(908)516-4211
www.gtt.net

[GTT_Logo_RGB_Blue]

Video Overview:
https://vimeo.com/257739650


From: NANOG  On Behalf Of Matt Harris
Sent: Tuesday, December 10, 2019 9:58 AM
To: Bottiger 
Cc: North American Network Operators' Group 
Subject: Re: Is anyone able to contact GTT?

On Tue, Dec 10, 2019 at 8:51 AM Bottiger 
mailto:bottige...@gmail.com>> wrote:
I sent an email to noc at gtt.net from 2 different emails and 
both got a reply saying:

5.1.0 - Unknown address error 550-'5.4.1 Recipient address rejected: Access 
denied 
[HE1EUR01FT058.eop-EUR01.prod.protection.outlook.com]'

Not sure if this means if they are blocking my email or if their email is 
broken.

Are you a GTT customer? If so, then I suspect the email you're looking for is 
inoc@ not noc@ ? The response indicates that the recipient address was 
rejected, not the sender address and I've always used inoc to contact them 
about my transit circuits.

Re: Is anyone able to contact GTT?

[Matt Harris]

On Tue, Dec 10, 2019 at 8:51 AM Bottiger  wrote:

> I sent an email to noc at gtt.net from 2 different emails and both got a
> reply saying:
>
> 5.1.0 - Unknown address error 550-'5.4.1 Recipient address rejected:
> Access denied [HE1EUR01FT058.eop-EUR01.prod.protection.outlook.com]'
>
> Not sure if this means if they are blocking my email or if their email is
> broken.
>

Are you a GTT customer? If so, then I suspect the email you're looking for
is inoc@ not noc@ ? The response indicates that the recipient address was
rejected, not the sender address and I've always used inoc to contact them
about my transit circuits.

Re: Software Defined Networks

[Asif Shaikat]

The term " Software Defined Networks " is open to interpretation. But
chapter 1 & 2  of bellow course give a concise idea about general concept
around Software Defined Networks.

https://courses.edx.org/courses/course-v1:LinuxFoundationX+LFS165x+2T2018/course/


Regards

Asif





On Wed, Dec 4, 2019 at 11:57 PM Rod Beck 
wrote:

> Can someone explain what is all the fuss? SDN is like the latest telecom
> craze but the articles do a poor job of explaining the advantages. I seek
> concrete examples.
>
> Regards,
>
> Roderick.
>
>
> Roderick Beck
> VP of Business Development
>
> United Cable Company
>
> www.unitedcablecompany.com
>
> New York City & Budapest
>
> rod.b...@unitedcablecompany.com
>
> 36-70-605-5144
>
>
> [image: 1467221477350_image005.png]
>

Is anyone able to contact GTT?

[Bottiger]

I sent an email to noc at gtt.net from 2 different emails and both got a
reply saying:

5.1.0 - Unknown address error 550-'5.4.1 Recipient address rejected: Access
denied [HE1EUR01FT058.eop-EUR01.prod.protection.outlook.com]'

Not sure if this means if they are blocking my email or if their email is
broken.

RE: DDoS attack

[Paul Amaral via NANOG]

Normally these attacks are spoofed IPs, usually amplification attacks based on 
UDP using DNS/LDAP etc. This is something that is common and usually is towards 
schools, financial institutions. This an easy attack to orchestrate by anyone, 
most of these attacks can be launch via stresser services online. 800mbs to 
most smaller ISPs is a lot of traffic and can deeply impact not only the victim 
prefix but other non-targeted customers, as traffic consumed by the attack will 
cause problems for all users on that circuit.

There's a few things you can do, ask your upstream provider to rate limit UDP 
packets towards you. Rate limit them to what you think a normal UDP rate should 
be. I don’t recommend blocking UDP as you will block legit UDP packets from 
reaching any of your customer when the attack is not ongoing. Note most larger 
providers will not help or care to help, I know Comcast probably will not help 
you, their support techs will have no idea what you are taking about neither 
will most entry level engineers. However, it's worth taking a shot and asking 
you upstream provider. 

Another way you can minimize this is if you are multi-hommed with BGP. In this 
case take the targeted prefix and advertise to be preferred through one of your 
upstreams and move all over prefixes to the other link. This will ensure that 
most of your customers will not be impacted during the DDOS. Once you have the 
victim prefix preferred on that specific BGP link then you can rate limit on 
your edge, or the provider can do this for you. You will still have the full 
force of the attack at the edge unless you can get one of your providers to 
help you out. With DDOS you can only mitigate it and not necessarily stop it.  
Someone will always get that DDOS traffic. rather is your, your provider or 
your customers. The problem is figuring out where you want the traffic to be 
rate-limited, stopped etc and that who's expense. 

BTW those stresser services are usually free for a set about 0-15 min than you 
must pay thus why its not ongoing. 


Good luck, 

Paul 



-Original Message-
From: NANOG  On Behalf Of ahmed.dala...@hrins.net
Sent: Monday, December 09, 2019 3:08 PM
To: nanog@nanog.org
Subject: DDoS attack 

Dear All, 

My network is being flooded with UDP packets, Denial of Service attack, soucing 
from Cloud flare and Google IP Addresses, with 200-300 mbps minimum traffic, 
the destination in my network are IP prefixes that is currnetly not used but 
still getting traffic with high volume.
The traffic is being generated with high intervals between 10-30 Minutes for 
each time, maxing to 800 mbps When reached out cloudflare support, they 
mentioned that there services are running on Nat so they can’t pin out which 
server is attacking based on ip address alone, as a single IP has more than 
5000 server behind it, providing 1 source IP and UDP source port, didn’t help 
either Any suggestions?

Regards,
Ahmed Dala Ali 

Re: DDoS attack

[Alain Hebert]

     BCP38 

    After all this time and knowledge why people still think ip> are legit evidence in DDoS instances...


-
Alain Hebertaheb...@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 2019-12-09 15:15, Tim Požár wrote:

This is lame.  They should be able to view NAT translation tables or
better yet have some method of watching flows.

Tim

On 12/9/19 12:11 PM, Christopher Morrow wrote:

I'd note that: "what prefixes?" isn't answered here... like: "what is
the thing on your network which is being attacked?"

On Mon, Dec 9, 2019 at 3:08 PM ahmed.dala...@hrins.net
 wrote:

Dear All,

My network is being flooded with UDP packets, Denial of Service attack, soucing 
from Cloud flare and Google IP Addresses, with 200-300 mbps minimum traffic, 
the destination in my network are IP prefixes that is currnetly not used but 
still getting traffic with high volume.
The traffic is being generated with high intervals between 10-30 Minutes for 
each time, maxing to 800 mbps
When reached out cloudflare support, they mentioned that there services are 
running on Nat so they can’t pin out which server is attacking based on ip 
address alone, as a single IP has more than 5000 server behind it, providing 1 
source IP and UDP source port, didn’t help either
Any suggestions?

Regards,
Ahmed Dala Ali

Starting to Drop Invalids for Customers

[Mark Tinka]

Hi all.

Just to let this group know that we've started the process of activating
the dropping of Invalids for all our eBGP customers.

We're starting off with our Juniper edge routers. Once those are done,
we'll move on to our Cisco ASR1006 routers, finishing off with our Cisco
ASR920 routers.

I'll let you know if anything catches fire :-).

Mark.