Cloudflare contact?

[John Von Essen]

I work with DuckDuckGo, and earlier today our macOS browser (which is currently 
available via the App store now) started getting caught by Cloudflare’s 
bot/fraud system. We did a fair amount of debugging, it appears to be some kind 
of browser/UA fingerprinting. This is happening for pretty much anyone using 
our browser, anywhere in the world, when browsing cloudflare powered sites. My 
hunch is this is accidental, but since we have no direct contacts at 
Cloudflare, we’re having a hard time escalating this. 

Thanks
John

AS3352

[John Von Essen]

If anyone from AS3352 (Telefonica Espana) is on list, please contact me 
off-list. We’ve detected a problem with geolocation (possibly your resolvers) 
sending a lot of traffic to the west coast US when it should be going to Europe.

Thanks
John

Re: Congrats to AS701

[John Von Essen]

Philly suburbs here, v6 is live for me. At home I use an Orbi router, just 
enabled v6 with autoconfig and got a native v6 WAN. So far looks good. Had to 
manually configure v6 DNS though.

The only downside is the geolocation of my v6 IP is pretty bad.

John

Sent from my iPhone
> On Jun 16, 2022, at 9:45 AM, Jamie Bowden via NANOG  wrote:
> 
> 
> I had to log in to my FiOS provided CPE (Verizon Quantum Gateway) and enable 
> IPv6.  It’s off by default. 
>  
> This is what I see in Reston, VA:
>  
> Ethernet adapter Ethernet:
>  
>Connection-specific DNS Suffix  . : fios-router.home
>Description . . . . . . . . . . . : Intel(R) Ethernet Connection I217-LM
>Physical Address. . . . . . . . . : 6C-C2-17-EE-EE-6D
>DHCP Enabled. . . . . . . . . . . : Yes
>Autoconfiguration Enabled . . . . : Yes
>IPv6 Address. . . . . . . . . . . : 
> 2600:4040:2b48:ce00:25e4:9527:2f2b:e571(Preferred)
>Temporary IPv6 Address. . . . . . : 
> 2600:4040:2b48:ce00:3411:b0a4:e9e7:e28f(Preferred)
>Link-local IPv6 Address . . . . . : fe80::25e4:9527:2f2b:e571%18(Preferred)
>IPv4 Address. . . . . . . . . . . : 192.168.2.146(Preferred)
>Subnet Mask . . . . . . . . . . . : 255.255.255.0
>Lease Obtained. . . . . . . . . . : Thursday, June 16, 2022 8:48:52 AM
>Lease Expires . . . . . . . . . . : Friday, June 17, 2022 8:48:51 AM
>Default Gateway . . . . . . . . . : fe80::4a5d:36ff:fecc:fe42%18
>192.168.2.254
>DHCP Server . . . . . . . . . . . : 192.168.2.254
>DHCPv6 IAID . . . . . . . . . . . : 57459223
>DHCPv6 Client DUID. . . . . . . . : 
> 00-01-00-01-23-20-9D-C9-6C-C2-17-EE-EE-6D
>DNS Servers . . . . . . . . . . . : 2600:4040:2b48:ce00::1
>192.168.2.254
>2600:4040:2b48:ce00::1
>NetBIOS over Tcpip. . . . . . . . : Enabled
>Connection-specific DNS Suffix Search List :
>fios-router.home
>  
> My Netgear router/WAP is set to autodetect IPv6 and sees it as passthrough.  
> IPv4 is double NAT, but I have the v4 interface on the Netgear set to a 
> static IP and the Verizon router is configured to treat that address as a DMZ 
> and passes all traffic directly to it (theoretically unmolested).  I used to 
> have it set to bridge mode for that port so it was only a single NAT, but 
> every time the VZ supplied router rebooted, I’d have to manually go back and 
> fix it, so I compromised and set as a DMZ instead.
>  
> In the interest of not putting my house directly on the internet without 
> protection, I do have all v6 traffic using the FiOS router’s firewall since 
> I’m not convinced that the Netgear is properly firewalling that traffic due 
> to the mode.
>  
> Thanks,
> --
> Jamie Bowden
> Senior Computer Network Technologist II
> 
> O: +1 703.842.3848
> C: +1 703.403.9745
> jamie.s.bow...@raytheon.com
> jamie.s.bow...@rtx.com
> 
> Raytheon Intelligence & Space
> Digital Technology
> 1100 Wilson Blvd.
> Suite 2000
> Arlington, VA 22209
>  
> RTX.com | LinkedIn | Twitter | Instagram
>  
> Upcoming PTO:
>  
> June 22, 2022
> July 4-8, 2022
>  
> From: NANOG  On Behalf 
> Of Christopher Morrow
> Sent: Saturday, June 11, 2022 10:05 PM
> To: nanog list 
> Subject: [External] Fwd: Congrats to AS701
>  
>  
> Looks like FIOS customers may be getting ipv6 deployed toward them, finally:
> 
> ifconfig snippet from local machine:
> inet6 2600:4040:2001:2200:73d2:6bcc:1e6b:43a1  prefixlen 64  scopeid 
> 0x0
> inet6 2600:4040:2001:2200:e87:bf36:b6cb:6ce1  prefixlen 64  scopeid 
> 0x0
>  
> ping attempt:
>   64 bytes from bh-in-f106.1e100.net (2607:f8b0:4004:c09::6a): icmp_seq=1 
> ttl=59 time=8.71 ms
>  
> 8ms from mclean, va to ashburn, va isn't wondrous, but at least it's ipv6 
> (and marginally faster than ipv4)
>  
> Congrats to the 701 folk for deploying more widely!
>   (note: I don't know exactly when this started, nor how wide it really is, 
> but progress here is welcomed by myself at least :) )
> -chris

Google Fi IPs

[John Von Essen]

Feel free to contact me off-list if your associated with Google Fi.

I’m trying to narrow down some IP abuse that I believe is coming from Google Fi 
mobile devices. The IPs are all coming up as generic Google LLC in whois, and 
they dont have any reverse DNS. I’m trying to see how I can confirm that these 
are IPs related to the Google Fi service/network, or just random Google Clould 
IPs

Here are some sample IPs:

35.187.133.202
35.187.133.204
35.187.133.200
34.116.22.76
34.116.22.74
34.116.22.72
107.178.194.231
107.178.194.233
107.178.194.235
107.178.193.10
107.178.193.12

Thanks
John

Re: OVH datacenter SBG2 in Strasbourg on fire 

[John Von Essen]

So your saying my “bot” dashboard should show a decrease in volume today? 
Interesting… I might run some stats today to see if there is a noticeable drop 
in Europe.

-John

> On Mar 10, 2021, at 10:53 AM, JORDI PALET MARTINEZ via NANOG 
>  wrote:
> 
> In addition to that, even if this is not good for many "honest" people that 
> was using the DC, we need to take it in the positive side. In my own case, 
> OVH is probably the cause of 80% of the abuse cases I report, and they never 
> react. I'm convinced I'm not the only one, as I read in other ops mailing 
> lists ...
> 
> So, the positive side is a) during some days, we can see an interesting 
> decrease in abuse cases, b) because the so many abuse cases, many OVH 
> "honest" customers are often being filtered because they share addresses with 
> the "bad guys", so it is an opportunity for them to move to alternative DCs 
> that probably are more careful about "bad guys".
> 
> A good topic for researchers :-)
> 
> Regards,
> Jordi
> @jordipalet
> 
> 
> 
> El 10/3/21 16:44, "NANOG en nombre de Andy Ringsmuth" 
>  a...@andyring.com> escribió:
> 
> 
>> On Mar 10, 2021, at 3:23 AM, Fredy Kuenzler  wrote:
>> 
>> Very sad day for our colleagues at OVH AS16276 as they lost their 
>> datacenter SBG-2 in Strasbourg/France completly („everything is destroyed“) 
>> in a fire  and the neighboring SBG1/SBG3/SBG4 at least temporary.
>> 
>> https://www.dna.fr/amp/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin
> 
>Sad to see of course, but also a little surprising that fire suppression 
> systems didn’t, well, suppress the fire.
> 
>Unless they didn’t exist?
> 
> 
>
>Andy Ringsmuth
>5609 Harding Drive
>Lincoln, NE 68521-5831
>(402) 304-0083
>a...@andyring.com
> 
>“Better even die free, than to live slaves.” - Frederick Douglas, 1863
> 
> 
> 
> 
> **
> IPv4 is over
> Are you ready for the new Internet ?
> http://www.theipv6company.com
> The IPv6 Company
> 
> This electronic message contains information which may be privileged or 
> confidential. The information is intended to be for the exclusive use of the 
> individual(s) named above and further non-explicilty authorized disclosure, 
> copying, distribution or use of the contents of this information, even if 
> partially, including attached files, is strictly prohibited and will be 
> considered a criminal offense. If you are not the intended recipient be aware 
> that any disclosure, copying, distribution or use of the contents of this 
> information, even if partially, including attached files, is strictly 
> prohibited, will be considered a criminal offense, so you must reply to the 
> original sender to inform about this communication and delete it.
> 
> 
> 

Re: Texas internet connectivity declining due to blackouts

[John Von Essen]

I just assumed most people in Texas have heat pumps- AC in the summer and 
minimal heating in the winter when needed. When the entire state gets a deep 
freeze, everybody is running those heat pumps non-stop, and the generation 
capacity simply wasn’t there. i.e. coal or natural gas plants have some 
turbines offline, etc.,. in the winter because historically power use is much 
much less. The odd thing is its been days now, those plants should be able to 
ramp back up to capacity - but clearly they haven’t. Blaming this on wind 
turbines is BS. In fact, if it weren’t for so many people in Texas with 
grid-tie solar systems, the situation would be even worse. 

And of course, the real issue is Texas’ closed grid - any other state could 
pull in more power from neighbors.

-John

> On Feb 15, 2021, at 11:34 PM, Cory Sell via NANOG  wrote:
> 
> Ercot has already released actual documentation of the outputs. Wind is NOT 
> the biggest loss here. Even if wind was operating at 100% capacity, we’d be 
> in the same boat due to gas and fossil fuel-related generation being 
> decimated. Estimated 4GW lost for wind doesn’t make up for the 30GW+ 
> estimated being lost from fossil fuels. 
> 
> I only interject because people are already pointing their fingers at 
> renewables being the cause here and trying to pawn off the blame to 
> wind/solar to further their agendas to reduce renewable energy R and 
> adoption. Sure, wind isn’t perfect, but looks like solution relied on failed 
> in a massive way.
> 
> Sent from ProtonMail Mobile
> 
> 
> On Mon, Feb 15, 2021 at 10:17 PM, Robert Jacobs  > wrote:
>> 
>> How about letting us Texans have more natural gas power plants or even let 
>> the gas be delivered to the plants we have so they can provide more power in 
>> an emergency. Did not help that 20% of our power is now wind which of course 
>> in an ice storm like we are having is shut off... Lots of issues and plenty 
>> of politics involved here.. 
>> 
>> Robert Jacobs​
>>  |   Data Center Manager
>>  
>> Direct:  832-615-7742 
>> Mobile:  281-830-2092 
>> Main:832‑615‑8000
>> Fax: 713-510-1650 <>
>> 5959 Corporate Dr. Suite 3300; Houston, TX 77036 
>>   
>>    
>>  
>>     A Certified Woman‑Owned 
>> Business 
>> 24x7x365 Customer Support: 832-615-8000 | supp...@pslightwave.com
>> 
>> ​This electronic message contains information from PS Lightwave which may be 
>> privileged and confidential. The information is intended to be for the use 
>> of individual(s) or entity named above. If you are not the intended 
>> recipient, any disclosure, copying, distribution or use of the contents of 
>> this information is prohibited. If you have received this electronic message 
>> in error, please notify me by telephone or e-mail immediately.
>> -Original Message-
>> From: NANOG  On Behalf Of 
>> Mark Tinka
>> Sent: Monday, February 15, 2021 10:06 PM
>> To: nanog@nanog.org
>> Subject: Re: Texas internet connectivity declining due to blackouts
>> 
>> 
>> 
>> On 2/16/21 04:14, Sean Donelan wrote:
>> >
>> > Poweroutage.us posted a terrific map, showing the jurisdictional 
>> > borders of the Texas power outages versus the storm related power 
>> > outages elsewhere in the country.
>> >
>> > https://twitter.com/PowerOutage_us/status/1361493394070118402
>> >
>> >
>> > Sometimes infrastructure planning failures are not due to "natural 
>> > hazards."
>> 
>> I suppose having some kind of home backup solution wouldn't be too bad right 
>> now, even though you may still not get access to services. But at least, you 
>> can brew some coffee, and charge your pulse oximetre.
>> 
>> Mark.
>> 
> 
> 

Re: Parler

[John Von Essen]

To be fair, AWS has existing contract/service clauses that are very very 
aggressive for termination. For example, if AWS contacts you regarding the 
hosting of CPEV, you have 24 hours to remove it and respond, if you dont - they 
immediately terminate the account. So the 24 hour warning for Parlor is not new 
behavior for Amazon.

Also, if you specifically read Amazon Customer Agreement 
(https://aws.amazon.com/agreement/ ), 
Section 6.1.a. and 7.2.b.ii. lay out basically what they did to Parlor.

Section 6.1.a.iii. says"

"We may suspend your or any End User’s right to access or use any portion or 
all of the Service Offerings immediately upon notice to you if we determine: 
your or an End User’s use of the Service Offerings could subject us, our 
affiliates, or any third party to liability"

Clearly if Parlor is used to covertly coordiinate a planned attack on our 
government that leads to loss of lives, AWS doesn’t want to be held liable for 
hosting that infrastructure. The above clause requires no extended notices, it 
can be immediate - so 24 hours was a favor….
-John


> On Jan 10, 2021, at 2:11 PM, Bryan Fields  wrote:
> 
> On 1/10/21 9:48 AM, Michael Thomas wrote:
>> Is it content moderation, or just giving the boot to enabling criminal 
>> activity? Would that more providers be given the boot for enabling voice 
>> spam scams, for example. Didn't one of the $n-chan's get the boot a 
>> while back? I don't seem to recall a lot of push back about that and it 
>> was pretty much the same situation, iirc.
> 
> There's legit users of parler and 8-chan.  Not every one is on the
> racist/insurrectionist/etc. sections.  And who's to say they have less of a
> right to their unpopular speech than I have to discuss retro video games?
> 
> This seems like it raises two interesting questions:
> 
> 1. When should a contracted provider be able to discontinue service with
> little to no notice to the customer if they find their content distasteful?
> 
> 2. Where do we expect legit insurrections to communicate?  Should
> AWS/Facebook/Twitter boot those calling for violent uprisings in Hong Kong
> (for example).
> 
> I suppose #2 is simply one mans freedom fighter is another criminal.
> 
> Anyone hosting with Amazon/Google/the cloud here should be really concerned
> with the timing they gave them, 24 hours notice to migrate.  Industry
> standards would seem to be at least 30 days notice.  Note this is not the
> police/courts coming to the host with notice that they are hosting illegal
> content but only the opinion of the provider that they don't want to host it.
> 
> I seem to recall a customer who was using provider IP space that sued and won
> an injunction circa 2004 against their provider allowing time to migrate. I
> remember reading the decision and was taken back by the decent grasp the judge
> had on BGP/IP space.  I can see how this might be similar.
> 
> Many years ago I was CoLo'd at a facility which shut off the racks of a
> customer at 9am on a Monday after finding said customer had poached an
> employee from the provider and was intending to compete with services the CoLo
> offered.  They physically disconnected the cross connects to these racks for
> this and banned the customer's employees from the facility.  Their counsel
> even told the customer "any contract is voidable at any time".  Basic planning
> for any company should ensure you never have all your eggs in one basket.
> Perhaps this was a bit dumb on the customers part, but they had a contract.
> 
> The cloud is just someone else's computer..
> -- 
> Bryan Fields
> 
> 727-409-1194 - Voice
> http://bryanfields.net



smime.p7s
Description: S/MIME cryptographic signature

Neustar Geo Location Data

[John Von Essen]

Anyone here have experience with Neustar’s Geo Location database feed?

And by experience, I mean, how reliable it is to reality?

I ask because I’m in the early stages of a project, and my initial take is the 
data is terrible.

I’ve stumbled across several (like a few hundred, and thats just in the US) 
/22’s and /23’s that SWIP in Arin to small regional ISPs in the US, but in the 
Neustar geo data these /22s get broken out in many continuous /28’s and /27’s 
that appear to hop across the world.

One case was a small rural WISP in St Louis, their /22 in Neustar’s data is 
spread across Brazil, Asia, Europe, I mean its all over the place. But in BGP, 
that /22 appears safe and sound coming from St Louis and also confirmed via 
traceroute.

If it were just a few ranges, I’d say no big deal, but I’m seeing massive 
issues with the data - curious as to other people’s thoughts.

Thanks
John





smime.p7s
Description: S/MIME cryptographic signature

CIDR cleanup

[John Von Essen]

Sorry if this is slightly off-topic, but I am writing some code for a custom 
GeoDNS routemap. My starting data set is a raw list of /24 subnets, no prefix 
aggregation has been done. In other words, its the entire BGP routing table in 
/24 prefixes - tagged by Geo region. Each region is its own txt file with a 
dump of /24’s. As a result, these lists are HUGE. I want to aggregate the 
prefixes as much as possible to create a smaller routemap.

So right now it looks like:

...
105.170.72.0/24 brs
105.170.73.0/24 brs
105.170.74.0/24 brs
105.170.75.0/24 brs
105.170.76.0/24 brs
105.170.77.0/24 brs
105.170.78.0/24 brs
105.170.79.0/24 brs
105.170.80.0/24 brs
105.170.81.0/24 brs
105.170.82.0/24 brs
105.170.83.0/24 brs
105.170.84.0/24 brs
…

and so on. Obviously, 105.170.72.0/24 thru 105.170.79.0/24 can be aggregated to 
105.170.72.0/21 and so on. I normally use Perl, does anyone now if there is a 
perl module that will automatically do this prefix aggregation? I tried to 
write my code to do this, and its not trivial, just lookinh for a shortcurt. I 
did a breif glance at some CIDR related Perl cpan modules, and nothing has 
jumped out.

Thanks
John





smime.p7s
Description: S/MIME cryptographic signature

Re: Orange : Propagating Bogus Saudi Telecom Announcement

[John Von Essen]

Nice find Tom…


> On Aug 24, 2020, at 3:11 PM, Tom Beecher  wrote:
> 
> Saudi Telecom ( AS 39386 ) is currently announcing Equinix NY9's IX prefix, 
> and Orange is gladly sharing that for the world to see. 
> 
> Zayo : You might want to not be using that either when you're directly 
> connected to that exchange. :)
> 
> Router: New York, NY
> Command: show route protocol bgp table inet.0 198.32.118.0/24 
>  terse exact
> 
> 
> inet.0: 833301 destinations, 5821043 routes (833250 active, 16 holddown, 88 
> hidden)
> + = Active Route, - = Last Active, * = Both
> A V DestinationP Prf   Metric 1   Metric 2  Next hopAS path
> * ? 198.32.118.0/24 B 170100 4294967294   
>5511 39386 39386 39386 39386 I
>   unverified   >64.125.29.222
> 64.125.29.220
>   ?B 170100 4294967294  5511 
> 39386 39386 39386 39386 I
>   unverified   >64.125.29.222
> 64.125.29.220
>   ?B 170100 4294967294  5511 
> 39386 39386 39386 39386 I
>   unverified   >64.125.29.220
> 64.125.29.222
>   ?B 170100 4294967294  5511 
> 39386 39386 39386 39386 I
>   unverified   >64.125.29.220
> 64.125.29.222
>   ?B 170100 4294967294  5511 
> 39386 39386 39386 39386 I
>   unverified   >64.125.29.220
> 64.125.29.222
> {master}



smime.p7s
Description: S/MIME cryptographic signature

Re: CloudFlare Issues?

[John Von Essen]

Did anyone see any collateral damage from this outside of Cloudflare? 
Specifically Azure?

I manage a very large site in Azure, and at the exact same time of the 
Cloudflare incident we saw a spike in traffic (like a DDoS or Bot), then 
followed by unusual hardware resource anomalies. We’re globally spread in 
Azure, but we only saw this in the US and Brazil.

Very coincidental, but possible.


-John

> On Jul 17, 2020, at 5:33 PM, Aaron C. de Bruyn via NANOG  
> wrote:
> 
> More digging shows high latency to CloudFlare DNS servers from Comcast in 
> Washington and Oregon as well as a few other providers (Charter, ToledoTel), 
> etc...
> 
> Sites that do resolve using other DNS servers but are hosted on CloudFlare 
> aren't loading.
> Sites that use CloudFlare for their DNS aren't resolving either.
> traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
> 
>  1  _gateway (192.168.42.254)  0.185 ms  0.109 ms  0.117 ms
>  2  pppoe-gw-208-70-52.toledotel.com 
>  (208.70.52.1)  1.896 ms  1.881 ms  
> 1.903 ms
>  3  tuk-edge-13.inet.qwest.net  
> (198.233.244.225)  4.158 ms  4.082 ms  4.071 ms
>  4  sea-brdr-03.inet.qwest.net  
> (67.14.41.154)  8.976 ms  8.949 ms  8.903 ms
>  5  * * *
>  6  ae-1-51.ear2.Seattle1.Level3.net 
>  (4.69.203.173)  4.494 ms  4.350 ms 
>  4.311 ms
>  7  4.53.154.10 (4.53.154.10)  77.622 ms  103.323 ms  103.240 ms
>  8  * * *
>  9  * * *
> 10  * * *
> 11  * * *
> 12  * * *
> 13  one.one.one.one (1.1.1.1)  87.515 ms * *
> 
> -A
> 
> On Fri, Jul 17, 2020 at 2:18 PM Aaron C. de Bruyn  > wrote:
> Anyone seeing Cloudflare DNS outages or site issues?
> 
> Affecting a bunch of sites in Washington and Oregon.
> 
> -A



smime.p7s
Description: S/MIME cryptographic signature

Anyone from Airtel or Tata on list? DNS block issue on search engine

[John Von Essen]

Starting yesterday, we’ve noticed the search engine DuckDuckGo being blocked 
via DNS in India. Specifically, users using Airtel or Tata DNS servers. Other 
search engines are fine (Bing, etc.,.). I know alot of blocking is occurring in 
India recently, but I think the net was incorrectly cast over this one. Its not 
all ISPs in India, so far just Airtel and Tata.


$ dig @122.166.234.70 duckduckgo.com

; <<>> DiG 9.10.6 <<>> @122.166.234.70 duckduckgo.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 7316
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;duckduckgo.com.IN  A

;; AUTHORITY SECTION:
rpz.airtelblack.com.262 IN  SOA rpz.airtelblack.com. 
hostmaster.rpz.airtelblack.com. 2360 3 3 2419200 604800


$ dig @59.163.126.38 duckduckgo.com

; <<>> DiG 9.10.6 <<>> @59.163.126.38 duckduckgo.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36722
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;duckduckgo.com.IN  A

;; ANSWER SECTION:
duckduckgo.com. 10800   IN  A   127.0.0.1

smime.p7s
Description: S/MIME cryptographic signature

Re: Network issues in Israel/Middle East

[John Von Essen]

Yeah, this is bad default GeoDNS logic. I'm overriding Israel now to use Europe 
now, and things are much better.

-John


> On May 26, 2020, at 11:06 AM, Dovid Bender  wrote:
> 
> John,
> 
> As others have mentioned you should be going to Europe we have a POP in Rosh 
> Hayain, IL and in Nicosia, CY. Both POP's backup to AWS Ireland. Almost all 
> of your traffic in IL is going to go through Western Europe so it makes no 
> sense to send it to India. Israel does not have any peering with its 
> neighbors.
> 
> I just did some tests from Bezeq in Petah Tiqwa
> 
> AWS Ireland
> [root@cust-219-83-123 ~]# ping 3.248.0.0
> PING 3.248.0.0 (3.248.0.0) 56(84) bytes of data.
> 64 bytes from 3.248.0.0 <http://3.248.0.0/>: icmp_seq=1 ttl=223 time=69.0 ms
> 64 bytes from 3.248.0.0 <http://3.248.0.0/>: icmp_seq=2 ttl=223 time=69.1 ms
> ^C
> 
> AWS Virginia
> [root@cust-219-83-123 ~]# ping 3.80.0.0
> PING 3.80.0.0 (3.80.0.0) 56(84) bytes of data.
> 64 bytes from 3.80.0.0 <http://3.80.0.0/>: icmp_seq=1 ttl=238 time=149 ms
> 64 bytes from 3.80.0.0 <http://3.80.0.0/>: icmp_seq=2 ttl=238 time=149 ms
> ^C
> 
> AWS Mumbai
> [root@cust-219-83-123 ~]# ping 3.6.0.0
> PING 3.6.0.0 (3.6.0.0) 56(84) bytes of data.
> 64 bytes from 3.6.0.0 <http://3.6.0.0/>: icmp_seq=1 ttl=233 time=168 ms
> 64 bytes from 3.6.0.0 <http://3.6.0.0/>: icmp_seq=2 ttl=233 time=168 ms
> ^C
> 
> AWS Milan
> [root@cust-219-83-123 ~]# ping 15.161.0.254
> PING 15.161.0.254 (15.161.0.254) 56(84) bytes of data.
> 64 bytes from 15.161.0.254 <http://15.161.0.254/>: icmp_seq=1 ttl=239 
> time=58.5 ms
> 64 bytes from 15.161.0.254 <http://15.161.0.254/>: icmp_seq=2 ttl=239 
> time=58.4 ms
> ^C
> 
> AWS London
> [root@cust-219-83-123 ~]# ping 3.8.0.0
> PING 3.8.0.0 (3.8.0.0) 56(84) bytes of data.
> 64 bytes from 3.8.0.0 <http://3.8.0.0/>: icmp_seq=1 ttl=229 time=57.2 ms
> 64 bytes from 3.8.0.0 <http://3.8.0.0/>: icmp_seq=2 ttl=229 time=57.2 ms
> ^C
> 
> AWS Frankfurt
> [root@cust-219-83-123 ~]# ping 3.120.0.0 
> PING 3.120.0.0 (3.120.0.0) 56(84) bytes of data.
> 64 bytes from 3.120.0.0 <http://3.120.0.0/>: icmp_seq=1 ttl=235 time=50.7 ms
> 64 bytes from 3.120.0.0 <http://3.120.0.0/>: icmp_seq=2 ttl=235 time=50.7 ms
> ^C
> 
> 
> It seems like you're better off going to the US over going to Mumbai!
> 
> 
> 
> On Mon, May 25, 2020 at 3:00 PM John Von Essen  <mailto:j...@essenz.com>> wrote:
> I know this is outside the scope of “North America”, but has anyone else been 
> fielding more issues related to network health/congestion in the middle east, 
> specifically Israel?
> 
> Our users in Israel are primarily served from India-based resources 
> (AWS/Azure), both of which have cloud capacity issues in India that I’m aware 
> of.
> 
> Also, the majority of our users in Israel that have been reporting slowness 
> seem to be mostly behind the ISP Bezeq. If we force them to route to Ireland 
> (which is technically farther away form a latency standpoint) things are much 
> better, so I’m wondering if just Bezeq (or everyone in Israel) is just 
> experiencing 3rd party-related network congestion to Mumbai.
> 
> Thanks
> John
> 
> 

Network issues in Israel/Middle East

[John Von Essen]

I know this is outside the scope of “North America”, but has anyone else been 
fielding more issues related to network health/congestion in the middle east, 
specifically Israel?

Our users in Israel are primarily served from India-based resources 
(AWS/Azure), both of which have cloud capacity issues in India that I’m aware 
of.

Also, the majority of our users in Israel that have been reporting slowness 
seem to be mostly behind the ISP Bezeq. If we force them to route to Ireland 
(which is technically farther away form a latency standpoint) things are much 
better, so I’m wondering if just Bezeq (or everyone in Israel) is just 
experiencing 3rd party-related network congestion to Mumbai.

Thanks
John

Akamai/CDN rate limiting

[John Von Essen]

Can someone from Akamai reach out off-list?

I work for a major search engine (not google or bing) and we’re rolling out a 
new region. One of our upstream API partners is using Akamai CDN on the front 
end. When we tried an initial rollout of the region we started to get alot of 
connection timeouts from the Akamai powered CDN, presumably due to some kind of 
rate limiting. It was also very bursty in nature. It would be great if we could 
confirm this, and figure out a way to mitigate.

Thanks
John

Any Bing engineers?

[John Von Essen]

Seeing global latency issues to Bing search API…. 

-John

Cloudflare Contacts

[John Von Essen]

Could someone from Cloudflare contact me off-list?

I work for a major search engine (not google or bing), and we just launched 
some assets in Brazil, seeing some weird behavior to Cloudflare CDN assets and 
thinking maybe we are being caught in some kind of filter/block.

Our image search traffic is proxied through a single IP, so its definitely high 
volume. We’ve never had an issue in other regions, but it could due to the 
sudden increase.

Thanks
John Von Essen

Major issues with Cloudflare DNS (specifically DNS-over-HTTPS)

[John Von Essen]

Can someone from Cloudflare contact me off-list?

I work for a major search engine (not Google) and starting yesterday, we are 
getting reports from around the world about a DNS issue. They are either not 
resolving our site, or they are getting incorrect resolution (i.e. the wrong 
IP).

The issues appears to be centered around Firefox users who have DNS-over-HTTPS 
enabled, with Cloudflare as the provider.

Thanks
John

Re: Reminiscing our first internet connections (WAS) Re: akamai yesterday - what in the world was that

[John Von Essen]

Similar….

In ’93 I had a 2400bps modem and an $40/month ISP dialup account for 10 hours a 
month - my Mac IIci was zooming!

I quickly upgraded to 9600, then 14400, then 56k. I rocked the 56k till about 
2003 - mind you all my email was over telnet/ssh/pine and websites in 2003 
still worked somewhat well on 56k.

I tried getting ISDN in the late 90s, but at the time Bell Atlantic had 
horrible pricing for ISDN.

In those early days I remember setting up a download to start before bed so it 
could run all night, then wake up the morning to see my freshly downloaded 
300KB file — assuming the phone line remained stable.

-John


> On Jan 24, 2020, at 6:26 PM, Ben Cannon  wrote:
> 
> I started what became 6x7 with a 64k ISDN line.   And 9600 baud modems…   
> 
> in ’93 or so.  (I was a child, in Jr High…)
> 
> -Ben.
> 
> 
> -Ben Cannon
> CEO 6x7 Networks & 6x7 Telecom, LLC 
> b...@6by7.net 
> 
> 
> 
> 
>> On Jan 24, 2020, at 3:21 PM, b...@theworld.com  
>> wrote:
>> 
>> 
>> On January 24, 2020 at 08:55 aar...@gvtc.com  (Aaron 
>> Gould) wrote:
>>> Thanks Jared, When I reminisce with my boss he reminds me that this 
>>> telco/ISP here initially started with a 56kbps internet uplink , lol
>> 
>> Point of History:
>> 
>> When we, The World, first began allowing the general public onto the
>> internet in October 1989 we actually had a (mildly shared*) T1
>> (1.544mbps) UUNET link. So not so bad for the time. Dial-up customers
>> shared a handful of 2400bps modems, we still have them.
>> 
>> * It was also fanned out of our office to a handful of Boston-area
>> customers who had 56kbps or 9600bps leased lines, not many.
>> 
>> -- 
>>-Barry Shein
>> 
>> Software Tool & Die| b...@theworld.com 
>>  | http://www.TheWorld.com 
>> Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
>> The World: Since 1989  | A Public Information Utility | *oo*
> 

Re: Wikipedia drops support for old Android smartphones; mandates TLSv1.2 to read

[John Von Essen]

There are really two arguments here.

1. TLSv1.0 is insecure and should never be used in an HTTPS scenario - cant 
argue with this
2. Alot of static content sites are forcing HTTPS even though “technically” 
there is nothing that needs to be secured in transit - this is where the 
argument lies.

Why does access to wikipedia need to go over https? There is no login, no 
credit card  or SSNs being transferred, etc.,. Part of the blame is google, 
they started penalize sites in their index if they didn’t do https, as a 
result, almost every website now does ssl - everything from allrecipes.com 
 to a mommy blog, literally you cant find a non-ssl 
website anymore, everybody wants the better google rank, so they all gave in 
and went 100% ssl.

There is a reason however for search engines to enforce https, its a privacy 
issue, everyone is snooping on you, so if you dont want your ISP knowing what 
your searching for (http://search.com/?q=looking+for+a+divorce+lawyer) and then 
selling that info to advertisers, you need https - and yes Wiki is sort of 
search engine.

What I foresee happening is people will come up with a 3rd party solution, 
basically, you’ll start seeing people offer http->https proxy services, it will 
be interesting to see if the content source providers try to clamp down on this 
or let it happen…

-John


> On Dec 31, 2019, at 11:11 AM, Royce Williams  wrote:
> 
> On Tue, Dec 31, 2019 at 6:12 AM Seth Mattinen  > wrote:
> On 12/31/19 12:50 AM, Ryan Hamel wrote:
> > Just let the old platforms ride off into the sunset as originally 
> > planned like the SSL implementations in older JRE installs, XP, etc. You 
> > shouldn't be holding onto the past.
> 
> 
> Because poor people anywhere on earth that might not have access to the 
> newer technology don't deserve access to Wikipedia, right? Gotta make 
> sure information is only accessible to those with means to keep "lesser" 
> people out.
> 
> This. 
> 
> I visited a rural school in South Africa around 2008. 
> 
> For many things - such as using their cellphone provider's billing 
> infrastructure to pay for third-party services via SMS - a switch to TLS 1.2 
> only would probably have no impact. 
> 
> But for educational purposes, their reliance on Wikipedia was dramatic - and 
> they could *only* get to it from outdated phones that had been donated, 
> scavenged, or cobbled together from parts.
> 
> In the intervening years, the disposable-electronics culture has probably 
> been a great boon to them, bringing better and more tech - but much of it is 
> probably still pre Android 4.4.2
> 
> But perhaps Wikipedia's decision is based on actual data. I'd love to see 
> percentages of their negotiated TLS ciphers, per country and per client type. 
> Back in 2015,  you could see them as discussed here:
> 
> https://news.ycombinator.com/item?id=10194258 
> 
> 
> ... but I'm not sure where the equivalent data would be in the new Grafana 
> data:
> 
> https://grafana.wikimedia.org/?orgId=1 
> 
> 
> Royce

Re: AWS/Route53 Issues?

[John Von Essen]

This is resolved now. It was centered around Route53 and latency based routing 
in Europe. Our AWS support rep is being tight lipped about exact details, but 
it sounds like a major event effecting all users in Europe. 

I’m just a bit hyper-sensitive now since Route53 also had a bad DDoS last month.

-John


> On Nov 22, 2019, at 8:51 AM, Jason Kuehl  wrote:
> 
> Contact your TAM with your AWS ticket number and let them know about your 
> issue. I haven't seen anything as of yet but I'm still on my way into the 
> office. (No Friday alerts yet)
> 
> On Fri, Nov 22, 2019, 8:02 AM John Von Essen  <mailto:j...@essenz.com>> wrote:
> Anyone else seeing major issues in Europe? Starting midnight, 70% of our 
> Europe traffic got redirected to the US. AWS Dashboard says “no issues” but 
> when we called im we got a vague answer that there is a none issue they are 
> working on. Not sure if its another Route53 DDoS, or something else.
> 
> John

AWS/Route53 Issues?

[John Von Essen]

Anyone else seeing major issues in Europe? Starting midnight, 70% of our Europe 
traffic got redirected to the US. AWS Dashboard says “no issues” but when we 
called im we got a vague answer that there is a none issue they are working on. 
Not sure if its another Route53 DDoS, or something else.

John

Re: Disney+ Geolocation issues

[John Von Essen]

It is amazing how much variance there is between different IP GEO data sets, we 
recently switched from MaxMind to Neustar, there was a huge difference between 
the two….

-John

AWS GeoDNS and Routing...

[John Von Essen]

If anyone from AWS Networking Engineering is here, it would be great if you 
could chime in.

I work for a very large search engine, hosted in AWS. Right around Oct 31st/Nov 
1st, we noticed a significant change/re-routing of traffic that normally goes 
to Virginia to either California or Europe. All of our stats and pixel data 
indicate that approx 6-7% of our us-east-1 inbound traffic (we use Route53, Geo 
DNS, performance based routing, so lowest latency) was migrated to us-west-1 
(Cali) and eu-west-1 (Ireland).

If we used pure Geo-based routing, I could make the argument that there was 
some major Geo IP database update, but we performance based routing based in 
lowest latency. I realize that still depends on some data sets, and maybe those 
data sets got updated, or maybe the traffic was specifically re-routed due to 
some other kind of transit issue?

This isn’t critical, but its a head scratcher…

Thanks
John Von Essen

Re: Russian government’s disconnection test

[John Von Essen]

I guess if all telecoms and carriers in Russia (or say China) are under strong 
government control/oversight, its fairly easy from a technology standpoint to 
block the outside world.

The thing that I always wonder about is the ability for citizens to bypass the 
restriction via satellite internet nowadays. I guess they need a law to make 
that illegal too, if found purchasing satellite internet gear, off to the gulag!

On the other hand, if Russia disconnected from the outside world, how would all 
their trolls and bot farms get any work done?

> On Nov 1, 2019, at 7:02 PM, Scott Weeks  wrote:
> 
> 
> 
> --- sur...@mauigateway.com wrote:
> From: "Scott Weeks" 
> 
> Anyone got any technical info on how Russia plans to execute 
> a disconnection test of the internet?  
> 
> 
> 
> Got crickets, so now I have to respond to my own post on 
> what I just found out about it.  Is that like talking to 
> yourself? :)
> 
> https://www.npr.org/2019/11/01/775366588/russian-law-takes-effect-that-gives-government-sweeping-power-over-internet
> 
> "The "sovereign Internet law," as the government calls it, 
> greatly enhances the Kremlin's control over the Web. It was 
> passed earlier this year and allows Russia's government to 
> cut off the Internet completely or from traffic outside 
> Russia "in an emergency," as the BBC reported. But some of 
> the applications could be more subtle, like the ability to 
> block a single post."
> 
> "The equipment would conduct what's known as "deep packet 
> inspection," an advanced way to filter network traffic. 
> 
> "Regardless of what the government intends, some experts 
> think it would be technically difficult for Russia to 
> actually close its network if it wanted to, because of the 
> sheer number of its international connections."
> 
> "What I found was that there were hundreds of existing 
> Internet exchange points in Russia, some of which have 
> hundreds of participants...Many of them are international 
> network providers, he says, so "basically it's challenging 
> — if not impossible, I think — to completely isolate the 
> Russian Internet."
> 
> Belson says that the requirement for Internet service 
> providers to install tracking software will very likely 
> also be challenging in practice. He adds that it will be 
> difficult to get hundreds of providers to deploy it and 
> hard to coordinate that they're all filtering the same 
> content.
> 
> scott
> 
> 
> 
> 

Re: RTG

[John Von Essen]

I too love RTG, been using it forever, appears to handle interfaces all the way 
up 10G.

Out of curiosity, are you hitting an issue that requires updating?

I get it, there are many options now, but back in the day, RTG was so simple 
and so useful, its a testament to the original product. Its a great light 
weight traffic monitor, at my old datacenter I monitored over 2000 interfaces 
(with up to 2 years of retention) from a very basic low-end single CPU box.

-John

> On Oct 30, 2019, at 8:25 AM, Drew Weaver  wrote:
> 
> Hello,
>  
> We’ve been using this product for years and years http://rtg.sourceforge.net/ 
>  to collect and store SNMP statistics.
>  
> It has been working fine for us. I haven’t really been able to find much 
> information about forks, new versions, and development happening on it.
>  
> A while back I heard that Yahoo created their own version of it but I could 
> never find it.
>  
> Does anyone know if there is a spiritual successor to RTG that pretty much 
> works the same way that is modernized?
>  
> Thanks! 
> -Drew

Contacts at Three.co.uk

[John Von Essen]

I know this is a North America list, but anyone here connected with Three or 
have a contact there?

I am investigating an issue related to the default adult filter settings that 
are becoming more common (maybe required now?) in the UK on mobile data 
networks.

I work at a large search engine, not Google or Bing, but like #4 or 5 in the 
world, and a portion of our site is being blocked by Three - and we’ve 
determined its related to their adult filter settings on mobile or mifi 
devices. We’d like to get in contact with them to understand how/why it was 
blocked and what we can do to resolve it.

Thanks
John Von Essen

Apple AS714 - peering down on the East Coast?

[John Von Essen]

Starting around July 28th, I noticed a latency spike (70ms) on some of our 
traffic to Apple (mainly api.apple-mapkit.com) coming out of Virginia. This 
traffic usually always takes some local peering, and never is higher then 
10-15ms.

I checked from AWS backbone, Cogent, Zayo, Level3, all show 70+ ms from east 
coast.

I also noticed on bgp.he.net, Apple’s IPv4 peer list dropped from 307 to 275 
also on July 28-29th.

Anyone else who peers with Apple on the east coast seeing this? Is it an outage 
or planned maintenance?

Thanks
John

AWS latency is Asia-Pacific

[John Von Essen]

Is anyone else seeing increased latency both within AWS and transit in the 
Asia-Pacific region?

We normally see 90-100ms between Aus and Sing within AWS, for the past 18 hours 
or so this has jumped up to 190ms - even for internal VPC-VPC traffic. Transit 
from Aus to Sing (3rd party endpoints) is also 190ms or so.

So far, Amazon has told me everything is fine, but that same latency test from 
a budget VPS provider in Sydney to Singapore is like 92ms, whereas on AWS its 
190ms.

I have some tickets in queue, but curious if anyone else has observed anything.

-John

Re: really amazon?

[John Von Essen]

Really??? You cant parse “User unknown”... 

Dan is simply pointed out how ridiculous it is that amazon lists a non-existent 
email address with Arin for abuse.

So yeah... really amazon?

Sent from my iPhone

> On Jul 29, 2019, at 7:07 PM, Mel Beckman  wrote:
> 
> Dan,
> 
> I don’t really have the time to parse the debug output you sent. If you want 
> me, or most others, to pay attention to your post, please provide a more 
> detailed explanation of what the deal is than “Really, amazon?”
> 
>  -mel
> 
> 
>> On Jul 29, 2019, at 4:03 PM, Dan Hollis  wrote:
>> 
>> Amazon, you really should know better.
>> 
>> Source ip: 54.240.4.4
>> 
>> https://search.arin.net/rdap/?query=54.240.4.4
>> 
>> Source Registry ARIN
>> Kind Group
>> Full Name Amazon SES Abuse
>> Handle ASA152-ARIN
>> Email email-ab...@amazon.com
>> 
> RCPT To:
>> <<< 550 #5.1.0 Address rejected.
>> 550 5.1.1 ... User unknown
> DATA
>> <<< 503 #5.5.1 RCPT first
>> 
>> Jul 29 09:47:27 yuri sendmail[14067]: x6TGlQe4014062: 
>> to=, ctladdr= (500/500), 
>> delay=00:00:01, xdelay=00:00:01, mailer=esmtp92, 
>> relay=amazon-smtp.amazon.com. [207.171.188.4], dsn=5.1.1, stat=User unknown
> 

Abuse from Vodaphone AS30722

[John Von Essen]

Anyone from Vodaphone on list?

We are experiencing a massive DDoS from three Vodafone /16’s. The DDoS is 
spread throughout the entire range.

2.38.0.0/16
2.39.0.0/16
188.216.0.0/16

We’ve had to block the entire ranges just to stay online.

Thanks
John

Comcast Outage - East Coast?

[John Von Essen]

I just saw a 40% traffic drop on my routing core (East Coast based) across all 
my BGP peers. None of my transit peers flapped or had any issues other than the 
traffic drop. Almost all the complaints of connectivity issues were people 
using Comcast, so right now thats the only common thread.

Anyone else see this? It appears to have resolved after about 5 mins.

-John

Re: Routing issues to AWS environment.

[John Von Essen]

I was just about to email the group for a related issue.

We are also seeing some funky routing/peering within the AWS network.

We primarily communicate with Verizon Media/Oath - AS10310. Verizon Media has a 
presence in Singapore, and its peered locally with AWS AS38895 - we normally 
see 8ms latency. Verizon Media also peers with AWS AS16509 in Japan, but for 
Singapore traffic, Verizon Media sends a lower MED so AWS Singapore should 
prefer that route/peer, but its not working properly on the AWS side, all of 
our traffic is going to Japan, this started early AM today.

I had Verizon Media investigate, and we gave them our AWS Singapore IP 
addresses, they confirmed that they are not receiving those 
prefixes/announcements from AWS Singapore (AS38895).

So something is broke…. hopefully if someone from AWS is reading they can 
escalate.

In my case, the AWS Singapore IP ranges in question are : 46.51.216.0/21 and 
52.74.0.0/16

-John




> On May 8, 2019, at 10:55 AM, Curt Rice  wrote:
> 
> Hi are there any AWS engineers out there? We are seeing routing problems 
> between NTT and AWS in Ashburn, Va and would like to find out which side is 
> having the problem.
>  
> Thanks,
> Curt

Re: Fibre provider in Starkville, MS

[John Von Essen]

I just took a wholesale circuit from Windstream, it was fine - the 
provisioning/delivery portion was within the Chapter 11 timeline. The Chapter 
11 thing, if you read about it, isn’t really because they are going bankrupt, 
it's more to protect them from a pending lawsuit from a hedge fund (Aurelius 
Capital) “if” the hedge fund wins in court. The hedge fun did initially win, 
but Windstream is appealing, if they lose the appeal, the Chapter 11 protection 
will prevent the hedge fund from gutting Windstream dry, which they would 
happily do! 

It all has to do with the fact that the hedge fund was unhappy that Windstream 
spin-off assets into Uniti Fiber, i.e. rich people upset about not making 
“enough” money from a deal.

-John

> On May 6, 2019, at 9:46 AM, Marshall, Quincy  
> wrote:
> 
>  Original message 
> From: Mehmet Akcin
> Date: 5/6/19 09:02 (GMT-05:00)
> To: Theo Voss
> Cc: nanog@nanog.org
> Subject: Re: Fibre provider in Starkville, MS
> 
> "hi Theo,
> 
> Looks like Earthlink (now Windstream) has fiber there. You can visit 
> www.infrapedia.com  to look at what is available.
> 
> "
> 
> 
> Hasn't windstream filed for Chapter 11/13 protection? Not certain that's the 
> best choice. 
> 
> LQ Marshall
> 
> 
> 
> 
> 
> This email has been scanned for email related threats and delivered safely by 
> Mimecast.
> For more information please visit http://www.mimecast.com 
> 

Bing news feeds stale for 5 days (api.cognitive.microsoft.com)

[John Von Essen]

Any Bing engineers on here?

I work with a major search affiliate partner, and starting this morning news 
feeds from api.cognitive.microsoft.com  
were coming in stale, nothing new in the past 5 days. However, this was only 
effecting API calls originating outside the USA.

In the US, api.cognitive.microsoft.com  
returns fresh news, but globally through GeoDNS, that URL resolves to different 
IPs, those IPs (Europe, Singapore, etc.,.) are returning 5 day old news.

Sorry if this is slightly off-topic, but we dont have a good PoC at Bing for 
this… I figure there is a very good chance someone here could escalate.

Thanks
John

Amazon AS16509 peering... how long to wait?

[John Von Essen]

I applied for peering, received an email, setup the BGP session, waited 
about a month. Then 3 weeks ago my BGP session with Amazom came up, but 
with zero routes. I assume I am in some kind of test/waiting period, but 
after three weeks, I thought I would be getting routes by now. Emails to 
the peeringdb POC have not returned anything. Anyone here from AS16509, 
can this be bumped? We are AS17185, and peering is on DE-CIX NYC.



Thanks

John

Yahoo/Oath GeoDNS Issue (AS36647)

[John Von Essen]

If anyone from Yahoo/Oath is here, please email me off-list. Have a 
GeoDNS issue with yahoo API URLs in Australia, DNS results are returning 
IPs that are not ideal for the region (like on the other side of the 
world), it so bad (excess latency), we have to override them locally 
which I really rather not have to do. This is related to operations of a 
major/global search engine platform.


In Australia, these Yahoo URLs should be resolving to Singapore, but 
instead are resolving to central USA. The issue started a few weeks 
back, prior to that DNS resolution was working to Singapore.


Thanks

John

Re: Should Netflix and Hulu give you emergency alerts?

[John Von Essen]

I don’t care if Aliens are invading or a blackhole is swallowing our sun, do 
not... I repeat, do not interrupt me watching GoT’s on HBOGo!

-John


> On Mar 8, 2019, at 6:08 PM, Aaron C. de Bruyn via NANOG  
> wrote:
> 
>> On Fri, Mar 8, 2019 at 2:36 PM Matt Hoppes 
>>  wrote:
>> No. Please no. We need less regulation. Not more. 
>> 
>> VoIP started out the same way. Very simple to start offering voip. Worked 
>> well. Then the government got involved. Now it’s a mess of requirements, 
>> warnings and reporting.
> 
> Come on now...what we really need to get everyone attention is air raid 
> sirens coupled with streaming interruptions via a simultaneous reboot of all 
> 'core routers' on the internet so people stop surfing facebook and start 
> wondering "what's up", followed by the public utilities cycling the nations 
> power grid to the morse code 'SOS'.  Oh, and this all occurs during the 
> monthly test too.
> 
> -A

Re: Cogent v6 Blackhole server issues???

[John Von Essen]

Looks like they finally fixed it.. it just sporadically came back up. My 
v4 and v6 transit sessions were never effected. Who knows...



-John

On 2/22/19 1:18 PM, Dennis Burgess via NANOG wrote:

Out of St. Louis, mine has been up since the last reboot of my router.

2001:550:0:1000::421c:802 is my peering..





Dennis Burgess, Mikrotik Certified Trainer
Author of "Learn RouterOS- Second Edition”
Link Technologies, Inc -- Mikrotik & WISP Support Services
Office: 314-735-0270  Website: http://www.linktechs.net
Create Wireless Coverage’s with www.towercoverage.com

-Original Message-
From: NANOG  On Behalf Of John Von Essen
Sent: Friday, February 22, 2019 12:15 PM
To: nanog@nanog.org
Subject: Cogent v6 Blackhole server issues???

2 days ago my IPv6 BGP session to Cogent's Blackhole server went down 
(2001:550:0:1000::421C:802), I've spent all morning emailing their NOC and I'm 
getting nowhere. Anyone else seeing this? Im in the Phila Metro area.

-John

Cogent v6 Blackhole server issues???

[John Von Essen]

2 days ago my IPv6 BGP session to Cogent's Blackhole server went down 
(2001:550:0:1000::421C:802), I've spent all morning emailing their NOC 
and I'm getting nowhere. Anyone else seeing this? Im in the Phila Metro 
area.


-John

Cisco ASR's with RSP440 engines...

[John Von Essen]

If anyone on here has experience with the ASR series running the 
RSP440-SE or -TR, please contact me off-list. I'm trying to better 
understand real world performance when it comes to handling a few full 
BGP tables on these, it would be running as very basic edge router 
primarily just doing BGP. I know the RSP440 is EOL, but the plan would 
be to upgrade to RSP880 within a year.



Thanks

John

Re: A Zero Spam Mail System [Feedback Request]

[John Von Essen]

This is great news...  

> On Feb 18, 2019, at 12:02 PM, Viruthagiri Thirumavalavan  
> wrote:
> 
> I'm leaving this mailing list too.

Can a Nanog Op please ban this guy from joining again?

No IPv6 by design to increase reliability...

[John Von Essen]

I was having a debate with someone on this. Take a critical web site, 
say one where you want 100% global uptime, no potential issues with end 
users having connectivity or routing issues getting to your IP. Would it 
be advantageous to purposely not support a  record in DNS and 
disable IPv6, only exist on IPv4?


My argument against this was "Broken IPv6 Connectivity" doesn't really 
occur anymore, also, almost all browsers and OS IP stacks implement 
Happy Eyeballs algorithm where both v4 and v6 are attempted, so if v6 
dies it will try v4. I would also argue that lack of IPv6 technically 
makes the site unreachable from native IPv6 clients, and in the event of 
an IPv4 outage, connectivity might still remain on IPv6 if the site had 
an IPv6 address (I've experienced scenarios with a bad IPv4 BGP session, 
but the IPv6 session remained up and transiting traffic...)


Thoughts?


-John

Re: ASNs decimation in ZW this morning

[John Von Essen]

Im confused as to what exactly happened and how it was implemented. I 
assume the government wanted to restrict access to sites like whatsapp, 
facebook, twitter, etc.,. So did they tell national ISPs/Mobile 
(strong-arm) to simply block access to those sites, or they did they 
tell them to completely shutdown and go dark until the protests were 
over. Im just curious as to how an ISP/Mobile would selectively block 
access under government influence, reason being... understanding how can 
help us think of ways to get around it.


For example, lets say the mobile networks null routed all traffic 
destined to twitter and facebook networks... not a complete IP shutdown. 
So a local citizen is using email from a local provider (non-gmail, 
etc.,.) and still has access to email, Twitter knows they are blocked in 
ZW, but they still try to email updates to this example citizen. If 
their networks are being null routed, they can simply deliver the email 
via an alternate network/platform.


The whole thing is very disturbing, I mean this is 2019 right? Not 1984...

-John


On 1/16/19 9:06 AM, Mark Tinka wrote:



On 16/Jan/19 15:54, Colin Johnston wrote:



I wonder how they block social media sites/whats up, is it null
routing on peering cores or filtering since did not see filtering in
place from ZIM<>UK last month...

In Africa, the majority of connectivity happens over mobile networks. So
it's easy to "fix" it, since mobile networks have some of the most
advanced DPI's in any network.

For those not aware, Emmerson Mnangagwa, the Zimbabwean president,
increased fuel prices from US$1.24/litre to US$3.11/litre for diesel,
and US$1.31/litre to US$3.31/litre for petrol. This is what led to
(violent) protests, and as such, networks being asked to shutdown services.

Mark.

Switch.com AS23005

[John Von Essen]

Can someone from Switch.com / AS23005 contact me off-list? I have an IRR route 
object conflict issue that’s attention.

John

Re: CenturyLink RCA?

[John Von Essen]

One thing that is troubling when reading that URL is that it appears 
several steps of restoration required teams to go onsite for local 
login, etc.,. Granted, to troubleshoot hardware you need to be 
physically present to pop a line card in and out, but CTL/LVL3 should 
have full out-of-band console and power control to all core devices, we 
shouldn't be waiting for someone to drive to a location to get console 
or do power cycling. And I would imagine the first step to alot of the 
troubleshooting was power cycling and local console logs.



-John



On 12/30/18 10:42 AM, Mike Hammett wrote:
It's technical enough so that laypeople immediately lose interest, yet 
completely useless to anyone that works with this stuff.




-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


*From: *"Saku Ytti" 
*To: *"nanog list" 
*Sent: *Sunday, December 30, 2018 7:42:49 AM
*Subject: *CenturyLink RCA?

Apologies for the URL, I do not know official source and I do not
share the URLs sentiment.
https://fuckingcenturylink.com/

Can someone translate this to IP engineer? What did actually happen?
From my own history, I rarely recognise the problem I fixed from
reading the public RCA. I hope CenturyLink will do better.

Best guess so far that I've heard is

a) CenturyLink runs global L2 DCN/OOB
b) there was HW fault which caused L2 loop (perhaps HW dropped BPDU,
I've had this failure mode)
c) DCN had direct access to control-plane, and L2 congested
control-plane resources causing it to deprovision waves

Now of course this is entirely speculation, but intended to show what
type of explanation is acceptable and can be used to fix things.
Hopefully CenturyLink does come out with IP-engineering readable
explanation, so that we may use it as leverage to support work in our
own domains to remove such risks.

a) do not run L2 DCN/OOB
b) do not connect MGMT ETH (it is unprotected access to control-plane,
it  cannot be protected by CoPP/lo0 filter/LPTS ec)
c) do add in your RFP scoring item for proper OOB port (Like Cisco CMP)
d) do fail optical network up

--
  ++ytti

Re: email scannering / filtering

[John Von Essen]

I've used Sendmail + MIMEDefang + SpamAssassin w/clamav for over 15 
years. And on the SA side I use all the bells and whistles available 
like DCC greylisting, all the public blacklists, there are some 3rd 
party rulesets you can subscribe to, etc.,. In the end its not as good 
as gmail, but pretty darn close.


I block at SA score 4 and above, 4-8 score I dump into a separate 
quarantine account that I check every now and again for possible errors, 
and over 8 I drop - no log or bounce.


-John

On 12/14/18 12:35 PM, Guillaume Tournat wrote:


Hello,

For MTA server, I use Postfix, with some blacklists (DNSBL).

For filtering then: SpamAssassin + Clamav works well.


Le 14/12/2018 à 12:30, David Funderburk a écrit :


What open source email filtering system is working well for you?


Regards,

David Funderburk
GlobalVision
864-569-0703

For Technical Support, please email gv-supp...@globalvision.net 
.



--
This message has been scanned for viruses and dangerous content by
*E.F.A. Project* , and is believed to be 
clean. 

Levle3's IRR db

[John Von Essen]

Whats the best way to get in contact with Level3 to make an IRR 
change... if your not a Level3 customer?


I tried emailing r...@level3.net but that bounces back as an unmonitored 
mailbox. There are dup IRR entries in Level3's db for my prefixes 
(legacy from a carrier I used over 10 years ago). My prefixes are in 
Arin's IRR and I would like that to be the only source.



-John

Re: Should ISP block child pornography?

[John Von Essen]

I block stuff all the time (like ROKSO's DROP list). The only issue with 
blocking domains of CPE is I imagine those domains change all the time 
as they get shutdown, if you block the IP (from domain lookup) its 
likely that IP maybe be legitimate in the future.


It should be stopped it at the DNS level, but even that has workarounds. 
I would think CPE is a violation of terms of "most" registrars.


-John

On 12/7/18 1:06 AM, Lotia, Pratik M wrote:


Hello all, was curious to know the community’s opinion on whether an 
ISP should block domains hosting CPE (child pornography exploitation) 
content? Interpol has a ‘worst-of’ list which contains such domains 
and it wants ISPs to block it.


On one side we want the ISP to not do any kind of censorship or 
inspection of customer traffic (customers are paying for pipes – not 
for filtered pipes), on the other side morals/ethics come into play. 
Keep in mind that if an ISP is blocking it would mean that it is also 
logging the information (source IP) and law agencies might be wanting 
access to it.


Wondering if any operator is actively doing it or has ever considered 
doing it?


Thanks.

With Gratitude,

**

*Pratik Lotia*

“Information is not knowledge.”

The contents of this e-mail message and
any attachments are intended solely for the
addressee(s) and may contain confidential
and/or legally privileged information. If you
are not the intended recipient of this message
or if this message has been addressed to you
in error, please immediately alert the sender
by reply e-mail and then delete this message
and any attachments. If you are not the
intended recipient, you are notified that
any use, dissemination, distribution, copying,
or storage of this message or any attachment
is strictly prohibited. 

Re: Monitoring service that has a human component?

[John Von Essen]

Whats your budget?

The outsourced NOC firms tend to be expensive (I've looked at them for a 
project), and they are also not that fast, so dont expect someone to 
determine if an alarm is valid within a few minutes, instead, in goes 
into their queue and waits for a tech to pick it up, so it could be 
30-60 mins.


In a perfect scenario, using freelancer/gig-economy people should be 
able to get this done quickly, but its needs to be sizeable to start and 
will involve alot of logistics, which means money.


To be honest, the best option may be to hire a developer to custom code 
really good logic that eliminates a good deal of the false positives so 
only a handful make it through.


-John

On 12/5/18 5:01 PM, David H wrote:


Hey all, was curious if anyone knows of a website monitoring service 
that has the option to incorporate a human component into the decision 
and escalation tree?  I’m trying to help a customer find a way around 
false positives bogging down their NOC staff, by having a human 
determine the difference between a real error, desired (but different) 
content, or something in between like “Hey it’s 3am and we’ve taken 
our website offline for maintenance, we’ll be back up by 6am.” 
 Automated systems tend to only know if test A, or steps A through C, 
are failing, then this is ‘down’ and do my preconfigured thing, but 
that ends up needlessly taking NOC time if the customer themselves is 
performing work on their own site, or just changed it and whatever 
content was being watched, is now gone.  So, the goal would be to have 
the end user be the first point of contact if it looks like more of a 
customer-side issue.  If they can’t be reached to confirm, THEN 
contact NOC, and unlike email alerts, keep contacting until a human 
acknowledges receipt of the alert.


Thanks

CrownCastle/Lightower/Fibertech peering...

[John Von Essen]

Anyone on the list or no someone at CrownCastle AS46887 for peering 
relationships?


They dont have anything listed on peeringdb.com.


Thanks

John

Re: Tata Scenic routing in LAX area?

[John Von Essen]

From East Coast:

root@dns1:~# traceroute 23.92.178.22

traceroute to 23.92.178.22 (23.92.178.22), 30 hops max, 60 byte packets

 1  gw-128-254.phlapalo.quonix.net (208.82.128.254)  0.657 ms  0.657 
ms  0.651 ms
 2  te0-0-2-3.nr11.b002999-2.phl01.atlas.cogentco.com (38.104.111.121)  
1.057 ms  1.118 ms  1.196 ms
 3  te0-1-0-0.rcr21.phl01.atlas.cogentco.com (154.24.50.85)  1.018 ms 
te0-1-0-0.rcr22.phl01.atlas.cogentco.com (154.24.50.89)  0.971 ms 
te0-1-0-0.rcr21.phl01.atlas.cogentco.com (154.24.50.85)  1.049 ms
 4  be2333.ccr42.jfk02.atlas.cogentco.com (154.54.5.1)  3.775 ms 
be2364.ccr41.jfk02.atlas.cogentco.com (154.54.3.141)  3.756 ms 3.756 ms
 5  be3295.ccr31.jfk05.atlas.cogentco.com (154.54.80.2)  3.767 ms 
be3294.ccr31.jfk05.atlas.cogentco.com (154.54.47.218)  3.877 ms 
be3295.ccr31.jfk05.atlas.cogentco.com (154.54.80.2)  3.772 ms

 6  38.104.74.130 (38.104.74.130)  4.807 ms  5.371 ms  5.414 ms
 7  border1-po1-bbnet1.nyj004.pnap.net (216.52.95.46)  3.360 ms 3.343 
ms  3.316 ms
 8  inapvoxcust-XX.border1.nyj004.pnap.net (74.201.136.66)  12.197 ms  
12.222 ms  12.468 ms

 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *





On 11/15/18 2:30 PM, Eric Dugas wrote:

That's quite the tour...

From Montreal, QC

traceroute to 23.92.178.22 (23.92.178.22), 30 hops max, 60 byte packets
1 67-221-x-x.ebox.net (67.221.x.x) 0.459 ms 0.431 ms 0.409 ms
2 ix-ae-10-190.tcore1.mtt-montreal.as6453.net (206.82.135.105) 5.637 
ms 5.619 ms 5.588 ms
3 if-ae-12-2.tcore1.w6c-montreal.as6453.net (64.86.31.27) 246.680 ms 
246.682 ms 246.723 ms
4 if-ae-30-2.tcore2.ct8-chicago.as6453.net (66.198.96.24) 257.231 ms 
257.275 ms 257.286 ms
5 if-ae-22-2.tcore1.ct8-chicago.as6453.net (64.86.79.2) 249.929 ms 
249.807 ms 249.989 ms
6 if-ae-29-2.tcore2.sqn-san-jose.as6453.net (64.86.21.104) 257.468 ms 
257.206 ms 257.211 ms
7 if-ae-1-2.tcore1.sqn-san-jose.as6453.net (63.243.205.1) 251.862 ms 
251.116 ms 250.928 ms
8 if-ae-18-2.tcore2.sv1-santa-clara.as6453.net (63.243.205.13) 251.988 
ms if-ae-13-2.tcore2.lvw-los-angeles.as6453.net (64.86.252.26) 254.136 
ms if-ae-38-2.tcore2.sv1-santa-clara.as6453.net (63.243.205.75) 252.838 ms
9 if-ae-7-2.tcore2.svw-singapore.as6453.net (180.87.15.25) 254.861 ms 
if-ae-0-2.tcore1.sv1-santa-clara.as6453.net (63.243.251.1) 267.270 ms 
if-ae-7-2.tcore2.svw-singapore.as6453.net (64.86.252.39) 259.345 ms
10 if-ae-20-2.tcore1.svq-singapore.as6453.net (180.87.96.21) 250.794 
ms if-et-1-2.hcore2.kv8-chiba.as6453.net (120.29.211.3) 181.610 ms 
if-ae-20-2.tcore1.svq-singapore.as6453.net (180.87.96.21) 250.683 ms
11 120.29.215.202 (120.29.215.202) 256.046 ms 
if-ae-23-2.tcore1.svw-singapore.as6453.net (180.87.67.32) 253.692 ms 
120.29.215.202 (120.29.215.202) 255.907 ms

12 * * *
13 if-ae-20-2.tcore1.svq-singapore.as6453.net (180.87.96.21) 253.551 
ms unknown.telstraglobal.net (202.127.73.101) 280.228 ms 
if-ae-20-2.tcore1.svq-singapore.as6453.net (180.87.96.21) 254.633 ms

14 120.29.215.242 (120.29.215.242) 254.595 ms 269.004 ms 265.841 ms
15 i-10850.eqnx-core02.telstraglobal.net (202.84.140.46) 248.997 ms 
249.750 ms 249.693 ms
16 i-92.eqnx03.telstraglobal.net (202.84.247.17) 247.845 ms 
unknown.telstraglobal.net (202.127.73.101) 267.627 ms 
i-92.eqnx03.telstraglobal.net (202.84.247.17) 249.147 ms

17 * i-93.sgpl-core02.telstraglobal.net (202.84.224.189) 255.787 ms *
18 bbr1.inapbb-dal-sje-1-2-4-6.dal006.pnap.net (64.95.158.182) 261.728 
ms bbr2.ae7.sje.pnap.net (64.95.158.178) 248.810 ms 
bbr1.inapbb-dal-sje-1-2-4-6.dal006.pnap.net (64.95.158.182) 261.988 ms
19 i-92.eqnx03.telstraglobal.net (202.84.247.17) 250.373 ms 245.202 ms 
bbr2.xe-1-1-1.inapbb-chg-sje-8.chg.pnap.net (64.95.159.21) 260.105 ms
20 * * bbr1.xe-0-0-1.inapbb-wdc-dal-7.wdc002.pnap.net (64.95.158.210) 
260.176 ms
21 bbr2.ae7.sje.pnap.net (64.95.158.178) 247.134 ms 251.671 ms 
bbr1.inapbb-dal-sje-1-2-4-6.dal006.pnap.net (64.95.158.182) 264.785 ms
22 bbr2.xe-1-1-1.inapbb-chg-sje-8.chg.pnap.net (64.95.159.21) 263.305 
ms * 64.95.159.45 (64.95.159.45) 287.747 ms
23 bbr1.ae7.nym007.pnap.net (64.95.158.73) 263.963 ms 
bbr1.xe-4-0-0.inapbb-chg-nym-12.nym007.pnap.net (64.95.159.18) 251.967 
ms *

24 tsr1.e6-1.nyj004.pnap.net (64.95.158.234) 267.155 ms 263.123 ms *
25 64.95.159.45 (64.95.159.45) 266.946 ms * *
26 * bbr1.ae7.nym007.pnap.net (64.95.158.73) 266.049 ms *
27 * * tsr1.e6-1.nyj004.pnap.net (64.95.158.234) 262.634 ms
28 * * *
29 * * *
30 * * *

On Nov 15 2018, at 1:43 pm, Marcus Josephson  wrote:


Anyone else seeing an odd Scenic routing in the LAX/SJE area for tata.


traceroute to 23.92.178.22 (23.92.178.22), 30 hops max, 52 byte
packets

1 if-ae-13-2.tcore2.lvw-los-angeles.as6453.net (64.86.252.34) 
180.698 ms  180.610 ms  181.712 ms

 MPLS Label=344269 CoS=0 TTL=1 S=1

2 if-ae-7-2.tcore2.svw-singapore.as6453.net (180.87.15.25) 
189.327 

Re: Zayo vs Coent

[John Von Essen]

Zayo is probably a tad better in the network quality, but… Zayo’s NCC is awful 
when it comes to fixing or resolving anything, even something as simply as add 
a default route to my BGP session. And its takes forever, like a whole day 
waiting in queue. Cogent, you can call, and 15 minutes your done.

-John

> On Nov 9, 2018, at 1:18 PM, Dovid Bender  wrote:
> 
> Hi,
> 
> We are in a facility where my only options are Cogent or Zayo. We plan on 
> getting a 10G connection for a web crawler using v4 only. Looking for 
> feedback on either or (keeping the politics out of it). 
> 
> TIA.
> 
> Dovid
> 

Re: WIndows Updates Fail Via IPv6

[John Von Essen]

I recently go a Linksys home wifi router, by default it enables ipv6 on 
the LAN. If there is no native IPv6 on the WAN side (which is my case 
since FiOS doesnt do v6 yet) the Linksys defaults to a v6 tunnel.


For the first few weeks of using the router, I had no idea alot of my 
traffic was going out via the v6 tunnel.


Then I started getting random reachability and availability issues. 
Google would not load, but Bing and Yahoo would, and so on. I thought it 
was a FiOS issue, but after digging, I discovered the v6 tunnel, 
disabled it and all my issues went away.


I dont know what Linksys uses for the v6 tunnel because its buried in 
the firmware, but any tunnel service is vulnerable to a variety of 
issues that could effect access. Its odd that it always effects Windows 
update all the time, but who knows.


-John


On 11/12/18 1:18 PM, Mark Tinka wrote:



On 11/Nov/18 18:51, Lavanauts wrote:

I’m on native IPv6 via Spectrum and have no problems with Windows 
Updates.  Could this be a tunneling issue?


I do run 6-in-4 from my backbone to my house as my FTTH provider does 
not do IPv6.


I can't imagine this to specifically be the issue, as all other IPv6 
traffic is fine, but at this point, I'm open to suggestion.


Mark.