RE: Meta outage

[Ray Van Dolson via NANOG]

On Tue, 05 Mar 2024 12:17 -0700, Michael Rathbun  wrote:

>> What I found intriguing was that I was logged out by Google Docs at 
>> the same moment FB logged me out.  Downdetector showed a number of 
>> other supposedly unrelated services with large outage report spikes 
>> at roughly the same time.

> I was logged out of Honeywell's Total Connect Comfort site (remote
> thermostat control) at the same time FB logged me out. I'm not using
> OAUTH logins anywhere.
>
> I had recently changed SSID and the thermostat wasn't accepting remote
> commands. I thought maybe the SSID change had broken it and had just
> deleted the device from the TCC site to try adding it back when I was
> logged out.
>
> It's funny timing because earlier today I was telling an employee how
> we don't like to have our maintenance overlap with vendor's
> maintenance/repair window because when something breaks while we are
> making changes we can't always tell who is at fault.

Interesting.  I have a script that makes a call to TCC every so often to read
temperature data, and it failed from 8:50AM PST to 10:30AM PST.

Ray

RE: "Permanent" DST

[Ray Van Dolson via NANOG]

I think this is essentially the bill:

https://www.congress.gov/bill/117th-congress/house-bill/69/text

Not finding anything about 15 degrees.

Ray

-Original Message-
From: NANOG  On Behalf Of Mel 
Beckman
Sent: Tuesday, March 15, 2022 12:19 PM
To: Jay R. Ashworth 
Cc: nanog@nanog.org list 
Subject: Re: "Permanent" DST

I don’t follow why cancelling DST has the effect of moving the US fifteen 
degrees to the east. Also, your subject line reads “permanent DST”, but from 
your language the bill will be permanent standard time. 

I haven’t read the bill, but I’m hoping you can explain your position more 
clearly. 

-mel via cell

> On Mar 15, 2022, at 3:13 PM, Jay R. Ashworth  wrote:
> 
> In a unanimous vote today, the US Senate approved a bill which would
> 
> 1) Cancel DST permanently, and
> 2) Move every square inch of US territory 15 degrees to the east.
> 
> My opinion of this ought to be obvious from my rhetoric.  Hopefully, 
> it will fail, because it's likely to be the end of rational time 
> worldwide, and even if you do log in UTC, it will still make your life 
> difficult.
> 
> I'm poleaxed; I can't even decide which grounds to scream about this on...
> 
> Hopefully, the House or the White House will be more coherent in their 
> decision on this engineering construct.
> 
> Cheers,
> -- jra
> 
> -- 
> Jay R. Ashworth  Baylink   
> j...@baylink.com
> Designer The Things I Think   RFC 2100
> Ashworth & Associates   
> https://urldefense.com/v3/__http://www.bcp38.info__;!!CKZwjTOV!jlq104a9OT4LH-Gk4LCElbaWSsLXzHYDHHpxEqU0OZW56655xb8Df0mA4p1wvA$
>  [bcp38[.]info]  2000 Land Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

RE: New minimum speed for US broadband connections

[Ray Van Dolson via NANOG]

Infrapedia says there is Zayo fiber across the street to the south.  Guessing a 
DIA circuit might be a budget buster though.

From: NANOG  On Behalf Of Matthew 
Petach
Sent: Wednesday, February 16, 2022 4:47 PM
To: Josh Luthman 
Cc: NANOG 
Subject: Re: New minimum speed for US broadband connections



On Wed, Feb 16, 2022 at 1:16 PM Josh Luthman 
mailto:j...@imaginenetworksllc.com>> wrote:
I'll once again please ask for specific examples as I continue to see the 
generic "it isn't in some parts of San Jose".


You want a specific example?

Friend of mine asked me to help them get better Internet connectivity a few 
weeks ago.

They live here:
https://www.google.com/maps/place/Meridian+Woods+Condos/@37.3200394,-121.9792261,17.47z/data=!4m5!3m4!1s0x808fca909a8f5605:0x399cdd468d99300c!8m2!3d37.3190694!4d-121.9818295
 
[google.com]

Just off of I-280 in the heart of San Jose.

I dug and dug, and called different companies.
The only service they can get there is the 768K DSL service they already have 
with AT

Go ahead.  Try it for yourself.

See what service you can order to those condos.

Heart of Silicon Valley.

Worse connectivity than many rural areas.   :(

Matt

Re: 95th billing and automation

[Ray Van Dolson via NANOG]

On Thu, Dec 10, 2020 at 02:28:39PM -0500, Jason Canady wrote:
> We use rtg2, which stores data in MySQL.  I use PHP to calculate
> percentiles.  It allows for most flexibility.  
> 
> > 
> > On Dec 10, 2020, at 13:29, Mehmet Akcin  wrote:
> > 
> > hi there,
> > 
> > i have asked about this in the past. What is the best tool out
> > there to do 95th percentile billing. I have decided to use
> > observium and librenms as result of responses but there seems to be
> > some kind of billing module issue with these tools (thy are
> > basically the same code). 
> > 
> > What are other systems besides observium and librenms (and old
> > fashion cacti) people are using these days with 95th billing and
> > integration with a CRM like salesforce/zoho, etc. I appreciate the
> > responses.
> > 
> > Mehmet

In a long ago past life have used nfcapd/nfsen + database and then into
Billmax for invoicing. :)

Sounds like you're after something a bit more turnkey though.

Ray

Re: modeling residential subscriber bandwidth demand

[Ray Van Dolson]

On Wed, Apr 03, 2019 at 03:45:17AM -0400, Valdis Klētnieks wrote:
> On Tue, 02 Apr 2019 23:53:06 -0700, Ben Cannon said:
> > A 100/100 enterprise connection can easily support hundreds of desktop 
> > users 
> > if not more.  It???s a lot of bandwidth even today.
> 
> And what happens when a significant fraction of those users fire up Netflix 
> with
> an HD stream?
> 
> We're discussing residential not corporate connections, I thought
> 

Yes, Enterprise requirements are certainly different, though inching
upwards with the prevalance of SaaS services like Salesforce, O365 and
file sharing services (the latter are a growing % of our traffic at
branch offices).

I feel like our rule of thumb on the Enterprise side is in the 1.5-2Mbps
per user range these days (for Internet).

Ray

RE: residential/smb internet access in 2019 - help?

[Ray Van Dolson]

Have been through something similar recently with CenturyLink extending fiber 
service to a residence where only 3Mbps DSL was available previously.

Total costs ended up being in the mid five figures range (though I don’t know 
how far they needed to extend fiber).  We amortized over a multi-year term on 
top of an already four figure MRC.  50Mbps service in the end.

Probably not worth it for most.

Ray

From: NANOG  On Behalf Of Ross Tajvar
Sent: Tuesday, March 26, 2019 8:29 PM
To: david raistrick 
Cc: North American Network Operators' Group 
Subject: Re: residential/smb internet access in 2019 - help?

This is a common problem with no good solution. Fiber buildouts are almost 
always insanely expensive. If you can get one at a more reasonable cost, or 
more likely if you can sign a contract of a sufficient length to convince the 
carrier to subsidize it, you may be able to get good service that way.

The tower thing could also work if you want to spend the time/money on building 
and maintaining it. And also provided you can get a permit, etc.

But most likely you're just out of luck.

On Tue, Mar 26, 2019, 10:44 PM david raistrick 
mailto:dr...@icantclick.org>> wrote:
folks,

I've been away from nanog for a long time - and away from the ISP world for 
longer.

Looking at a house in a new area, at copper splice box out front, bellsouth 
fiber markers as well (yes, that's usually just passing by. but it's there).  
Owners since '82 said the telephone company was AT - but the New AT 
apparently no longer offers phone or internet service there.

This is located in a semi-rural area between Ocala and Gainesville Florida 
(Micanopy, specifically).

I knew the state of residential service was in sorry shape - but from what I'm 
reading, it seems to be worse than I'd though possible.

Anyone have any suggestions for service options?  I'm cool with dark fiber, if 
it comes down to that (and can be price sanely and terminated somewhere 
useful), but it seems like there -should- still be CLEC/DLECs or just plain 
resellers in business who still have access to resources that are in the ground.

My business operates from home - so obviously quality service is a priority, 
and I'm willing to pay for it within reason.  Business plans are certainly an 
option as well.

I've confirmed with all of the known players via their front channels - att, 
windstream, centurylink, frontier, cox/comcast/spectre.

Via backchannels I've confirmed that cox has fiber in the ground 1.4 miles away 
- straight shot down a dirt road (same one with the BS fiber markers).   I have 
a lead on a couple of tower shots - but there's a big (for florida) ridge 
between us, and I might have to build 3-400ft to hit anything (speculatively).

Anyone have local area or other knowledge that might be helpful?

I'd hate to miss out on this house - it's a lot of things we love - but cell or 
sat only for internet access just isn't going to fly.


thanks guys.

...david

RE: FB?

[Ray Van Dolson]

https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_facebook_status_1106229690069442560=DwIGaQ=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho=zrKUWVShQdFllKTGbJE5kITG87q7KNJHo0bD6aETBBk=

From: NANOG  On Behalf Of Luke Guillory
Sent: Thursday, March 14, 2019 2:09 PM
To: Selphie Keller ; Mike Hammett 
Cc: NANOG list 
Subject: RE: FB?

That’s old.

By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM


Luke

Ns




From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Selphie Keller
Sent: Thursday, March 14, 2019 4:06 PM
To: Mike Hammett
Cc: NANOG list
Subject: Re: FB?

I did see this article indicating they had somehow invalidated their cache in a 
botched deployment of changes - 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_notes_facebook-2Dengineering_more-2Ddetails-2Don-2Dtodays-2Doutage_431441338919_=DwIGaQ=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho=1EmIo8GEivgILxC4jZzEdBpYWt5R9CZ5cXhtr6i55rc=

On Thu, 14 Mar 2019 at 06:18, Mike Hammett 
mailto:na...@ics-il.net>> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it 
was a route leak, but I haven't seen any other sources that aren't just quoting 
Roland. Usually there are a few independent posts out there by now.


-
Mike Hammett
Intelligent Computing Solutions
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ics-2Dil.com=DwIGaQ=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho=EAZZC6r_-2rdFCKgq9XpQy30F7OH79M6sZPNvXq0FPA=

Midwest-IX
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.midwest-2Dix.com=DwIGaQ=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho=fEoBWpTXgY7eXczzc8vo7VHbvopKqDWk6Xz2XYutL0k=

Re: Extending network over a dry pair

[Ray Van Dolson]

On Wed, Dec 12, 2018 at 01:25:32PM -0800, Nick Bogle wrote:
> A quick question for you guys; 
> 
> If you had a single dry pair (pair of copper wires originally for
> phones) to a remote site that was around 6 miles away, what would you
> use? We currently are just extending a T1 line to this site, but
> 1.5Mbps isn't cutting it anymore.  Unfortunately it's a research site
> on a federally protected wildlife preserve so we can't run any new
> infrastructure (fiber etc) and it isn't in a geographical place where
> point to point wireless is practical. We were thinking there is some
> sort of network extender that uses some form of DSL for higher
> bandwidth capacity. 
> 
> Any suggestions?

There's this[1], but only rated at one mile.

This one[2] claims it can support 15.3Mbps over a single pair.

Ray

[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amazon.com_Tupavco-2DEthernet-2DExtender-2DKit-2DRepeater-2DVDSL_dp_B01BOD8C9W_ref-3Dpd-5Fcp-5F147-5F2-3Fpd-5Frd-5Fw-3DjJF6B-26pf-5Frd-5Fp-3Def4dc990-2Da9ca-2D4945-2Dae0b-2Df8d549198ed6-26pf-5Frd-5Fr-3DYNZSNN4KVFDD0D7F28BC-26pd-5Frd-5Fr-3Dff2a9a7f-2Dfe54-2D11e8-2D9eb5-2Dcbf5e1b9be77-26pd-5Frd-5Fwg-3DYAFyN-26pd-5Frd-5Fi-3DB01BOD8C9W-26psc-3D1-26refRID-3DYNZSNN4KVFDD0D7F28BC=DwIBAg=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=TbF7NHyAPYAnOTcN0mP5L8Mx9bruJ3BQiMGiRuuEjag=1uB8i1QuuStq_4H-v8E2AvAuFwvzubQ5sfUHK81L598=
[2] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.patton.com_ethernet-2Dextender_cl1314mde_=DwIBAg=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=TbF7NHyAPYAnOTcN0mP5L8Mx9bruJ3BQiMGiRuuEjag=giCSQ1Y-mYPf-JQmTFLfqlg34eDZuCD87ScHf0sOR20=

OpenDNS Issue (US Cali or West Coast?)

[Ray Van Dolson]

Getting issues from west-coast US clients trying to access services
like O365, Yahoo, GMail, etc.  east-coast US doesn't seem to have the
same issue.

DNS servers are:

208.67.222.222
208.67.220.220

Switching to Google (8.8.8.8) fixes the issue.

Anyone else seeing this?

Ray

PS: Also posted to dns-operations

Re: Oct. 3, 2018 EAS Presidential Alert test

[Ray Van Dolson]

Anecdotally, we had staff feeding off of both AT and VZW IP-based
metrocells get the alert message.

Ray

On Wed, Oct 03, 2018 at 12:53:57PM -0700, mike.l...@gmail.com wrote:
> Iphone, vzw, silicon valley, rcvd.
> 
> Interesting question though... I wonder if people on micro-cells
> and/or wifi calling don’t get the alerts. That would be extremely
> dumb and irresponsible of the cell phone carriers, so its likely the
> case :)
> 
> In rural America where cell coverage may not exist but the customer
> may have PTMP wireless internet and is using a microcell and/or wifi
> calling over the internet, if they dont get the alert, that could be
> catastrophic. Something along the lines of the Santa Rosa, CA fires
> catastrophic.
> 
> I wonder if that is the case.
> 
> -Mike

Re: GTV-ETH-2-COAX - Is this HomePNA?

[Ray Van Dolson]

On Tue, Jan 09, 2018 at 10:57:49AM -0800, Ray Van Dolson wrote:
> Looking at doing a one-off extension over RG6 and have these devices in
> hand.  Anyone know if they're HPNA?  Manual I have found doesn't
> specify, but frequency ranges don't appear to be MoCA (but also don't
> appear to be HPNA 1.3).
> 
> Comparing to these:
> 
> Ray

Apologies for the corporatized email links.  Stumbled across the manual
for this device and appears to be UPA DHS based (Powerline -- extended
to work over coax).

Ray

GTV-ETH-2-COAX - Is this HomePNA?

[Ray Van Dolson]

https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amazon.com_GefenTV-2DEthernet-2DExtender-2DDiscontinued-2DManufacturer_dp_B0013LYMQ8=DwIBAg=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=MLZzcgCKfcPGBwKCi3lSUygoJ78g6KFaevQZoryCq9s=HwKmGRftJEcyn2of9m9-zXwj2WV33LsB0QM-dB4cgWU=

Looking at doing a one-off extension over RG6 and have these devices in
hand.  Anyone know if they're HPNA?  Manual I have found doesn't
specify, but frequency ranges don't appear to be MoCA (but also don't
appear to be HPNA 1.3).

Comparing to these:

https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amazon.com_TRENDnet-2DMid-2DBand-2DTransmission-2DDistances-2DTPA-2D311_dp_B00684E0UI=DwIBAg=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=MLZzcgCKfcPGBwKCi3lSUygoJ78g6KFaevQZoryCq9s=yQVnjc4gedaqBM_23_96Q5brEIPekkOjPUE6GRQDT8s=

Ray

Re: Alternatives to ISE?

[Ray Van Dolson]

On Sun, Dec 03, 2017 at 02:39:27PM +, Christopher J. Wolff wrote:
> I've about reached my limit with the dumpster fire that is Cisco's
> Identity Service Engine.  Are there any reliable alternatives that do
> endpoint classification, central web auth, and .1x auth?

What version of ISE are you running?  What are your main frustrations
with it?

Ray

Re: Nationwide AT BVOIP/SIP Outage

[Ray Van Dolson]

On Mon, Apr 03, 2017 at 12:06:26PM -0400, Cary Wiedemann wrote:
> All,
> 
> Our AT BVOIP service is down nationwide.  Our account managers are
> frantically looking into it but we don't have an official statement yet.
> 
> Symptoms vary from no ringing (sourced from MegaPath), ring then drop
> (sourced from Verizon/T-Mobile), or "The call you are attempting to place
> is not allowed from this line" (sourced from AT Wireless).
> 
> Anyone else experiencing this or have any explanations?  It's supposedly
> affecting the entire platform.  Anyone on-list from AT who can comment?
> 
> Our PNT private line circuits are all up, it seems the BVOIP phone switch
> is just hard down.
> 
> Sorry, I know NANOG isn't the best place for outage discussions; but the
> puck.nether.net Outages list seems to be broken today and AT is a
> gigantic North American phone provider.
> 
> Thank you.
> 
> - Cary

We're an ATT BVoIP/SIP user but not experiencing any issues.

Southern California.

Ray

.gov / census.gov contact

[Ray Van Dolson]

One of our external hide NATs has been blocked by the census.gov WAF.
Would someone contact me off-list about this or help point me to the
proper POC?  Am thinking responsibility may lie at the higher .gov
level...

TIA,
Ray

Re: SoCal FIOS outage(?) / static IP readdressing

[Ray Van Dolson]

On Wed, Jan 04, 2017 at 01:52:15PM -0800, Paul B. Henson wrote:
> > From: valdis.kletni...@vt.edu
> > Sent: Wednesday, January 04, 2017 6:49 AM
> > 
> > Even if nothing else happens, calling in and reporting the problem *does*
> > (or at least it *should*) set the clock running for any SLA-related
> > compensation.
> 
> I'm pretty sure FIOS doesn't have any contractual SLA's. I suppose if you
> call and whine enough you might get a billing credit, but as another poster
> pointed out, it's generally not worth it.
> 

Have been evaluating going to more consumerish-grade circuits like this
at remote locations, but this scenario is one that has kept me sticking
with the more traditional (and more expensive) SLA-bound circuits.

Ray

Re: Death of the Internet, Film at 11

[Ray Van Dolson]

https://urldefense.proofpoint.com/v2/url?u=http-3A__hub.dyn.com_dyn-2Dblog_dyn-2Dstatement-2Don-2D10-2D21-2D2016-2Dddos-2Dattack=DQIBAg=n6-cguzQvX_tUIrZOS_4Og=r4NBNYp4yEcJxC11Po5I-w=iGvkbfzRJPqKO1A6YGa-c1m0RBLNkRk03hCjvVGTH3k=bScBNFncB3kt_cG0L3iys0mfXBmwwUR7A8rIDmi94D4=
 

On Sat, Oct 22, 2016 at 04:48:01PM -0500, Mike Hammett wrote:
> Until Dyn says or someone says Dyn said, everything is assumed. 
> 
> - Original Message -
> 
> From: "Peter Baldridge"  
> To: "Jean-Francois Mezei"  
> Cc: nanog@nanog.org 
> Sent: Saturday, October 22, 2016 4:45:13 PM 
> Subject: Re: Death of the Internet, Film at 11 
> 
> On Sat, Oct 22, 2016 at 1:47 PM, Jean-Francois Mezei 
>  wrote: 
> > Generic question: 
> > 
> > The media seems to have concluded it was an "internet of things" that 
> > caused this DDoS. 
> > 
> > I have not seen any evidence of this. Has this been published by an 
> > authoritative source or is it just assumed? 
> 
> Flashpoint[0], krebs[1], arstechnica[2]. I'm not sure what credible 
> looks like unless they release a packet but this is probably 
> consensus. 
> 
> > Has the type of device involved been identified? 
> 
> routers and cameras with shitty firmware [3] 
> 
> > Is it more plausible that those devices were "hacked" in the OEM 
> > firmware and sold with the "virus" built-in ? That would explain the 
> > widespread attack. 
> 
> The source code has been released. krebs [4], code [5] 
> 
> > Also, in cases such as this one, while the target has managed to 
> > mitigate the attack, how long would such an attack typically continue 
> > and require blocking ? 
> This is an actual question that hasn't been answered. 
> 
> > Since the attack seemed focused on eastern USA DNS servers, would it be 
> > fair to assume that the attacks came mostly from the same region (aka: 
> > devices installed in eastern USA) ? (since anycast would point them to 
> > that). 
> 
> Aren't heat maps just population graphs? 
> 
> > BTW, normally, if you change the "web" password on a "device", it would 
> > also change telnet/SSH/ftp passwords. 
> 
> Seems like no one is doing either. 

Re: google and amazon wierdness via HE right now

[Ray Van Dolson]

On Fri, Apr 22, 2016 at 01:22:27PM -0400, Ken Chase wrote:
> and of course the second I post it all fixes itself. NANOG works! Thanks!
> 
> (was going on for about 10-15 min)
> 
> On Fri, Apr 22, 2016 at 01:21:47PM -0400, Ken Chase said:
>   >
>   >From toronto - something odd - mtr to google.com (google.com 
> (172.217.3.142))
>   >
>   > 5. v638.core1.tor1.he.net  
>   > 6. 100ge7-2.core1.nyc4.he.net  
>   > 7. 100ge11-1.core1.par2.he.net 
>   > 8. 10ge3-2.core1.zrh1.he.net   
>   > 9. ???
>   >
>   >par is paris, zrh is zurich?
>   >
>   >same base path for hitting my EC2 nodes... Cant imagine this is just 
> affecting Toronto
>   >HE customers.
>   >
>   >EC2 node is in 107.20/14 
>   >
>   >/kc
> 
> /kc

There's some chatter about Amazon issues on the Outages mailing list as
well.  Definitely seems like something blipped.

Ray

Re: GeoIP information

[Ray Van Dolson]

I don't believe anyone is either.  We looked at it as well and after
reviewing logs from our authoritative DNS server responsible for our
in-addr.arpa zones, we saw zero queries for LOC records.

Ray

On Fri, Sep 25, 2015 at 10:43:13AM -0400, Clay Curtis wrote:
> I don't believe anyone is actually using the LOC RR, but maybe I'm wrong.
> This seems like the best way to store this type of data.  I could see CDNs
> being able to leverage this along with edns-client-subnet to decrease page
> load times significantly.  How is this still an issue?  I mean, we have the
> means to fix this.  Whoever a reverse zone is delegated to could easily
> update the LOC record to provide this info.  They can make the LOC record
> as "fuzzy" as they feel comfortable by zeroing out the minutes and seconds,
> as the LOC record is just a set of GPS coordinates.  Who better to report
> the physical location of a network than that network's operators.  I think
> country code would be a nice addition to the LOC record though.
> 
> Sincerely,
> 
> Clay Curtis
> 
> On Fri, Sep 25, 2015 at 6:36 AM, Stephen Satchell  wrote:
> 
> > https://tools.ietf.org/html/rfc1876 (EXPERIMENTAL)
> >
> > There appears to be a way of associating a subnet in the IN-ADDR.ARPA
> > domain to a FQDN, which could then be queries for LOC data.  For single
> > addresses, the domain owner could opt to include location data for their
> > domain.  For subnets, the operator can include location data at their
> > option.
> >
> > Also, I would add one more field to the LOC RR:  country code.  This would
> > be a two-byte value that is the standard two character ASCII country code.
> > When missing, a value of binary zero would be returned on query.
> >
> >
> 

Re: GeoIP information

[Ray Van Dolson]

On Thu, Sep 24, 2015 at 09:41:42PM -0400, William Herrin wrote:
> On Thu, Sep 24, 2015 at 9:33 PM, Ian Clark  wrote:
> > Where do GeoIP companies get their data, if not whois records?
> 
> I would assume that they query whois for one of their sources. They
> don't have to enter any contract with ARIN to do so but they also
> can't promptly collect any sizable portion database that way. That
> isn't the same as signing up for bulk whois access
> (https://www.arin.net/resources/request/bulkwhois.html).
> 
> I imagine they also do traceroutes to identify the last known location
> in the route.
> 
> Regards,
> Bill Herrin

I assumed it must be based off of WHOIS.  The IP space I'm working with
is in the midwest (US).  The address associated with it is from our
primary IP block out here in California, which it would have only been
able to gather from WHOIS.  If it had gone off the last hop, presumably
it would have seen that as something a little closer to the real
location rather than *exactly* where our primary environment is. :)

Ray

Re: GeoIP information

[Ray Van Dolson]

On Thu, Sep 24, 2015 at 08:47:56PM -0400, William Herrin wrote:
> On Thu, Sep 24, 2015 at 7:29 PM, Roland Dobbins  wrote:
> > On 25 Sep 2015, at 5:58, Ian Clark wrote:
> >> Any advice would be awesome!
> > There is no inherent correlation between IP addressing and geopolitical
> > boundaries.
> 
> Maxmind does not concur.
> 
> Regards,
> Bill Herrin

I've recently SWIP'd some IP space to see if Maxmind would pick up the
new location.  48 hours later it hasn't (just via their free, web-based
query tool).  Perhaps I need to be more patient.

Ray

Akamai Geolocation Secret Sauce?

[Ray Van Dolson]

Anyone familiar with how Akamai does its geolocation?  Presumably they
do more than Maxmind/WHOIS, but I suppose one or both of those could
factor in?

For those of you with ARIN IP space, do you typically SWIP things to
yourself to help clarify the locations where the IP space physically
resides to feed into Geolocation databases?

Thanks,
Ray

Re: West Coast FIOS disconnect

[Ray Van Dolson]

Had a BGP blip with our Verizon circuit around 1620 PDT.

On Thu, May 28, 2015 at 08:40:16PM -0400, Bill Patterson wrote:
 Seems to be a pretty widespread Verizon issue along the west coast and
 majority of the eastern US, at least according to down detector.
 On May 28, 2015 8:12 PM, James Laszko jam...@mythostech.com wrote:
 
  Is anyone else seeing wide spread Verizon FIOS disconnections from the
  world?  Started about an hour ago and extremely spotty. Seeing hundreds of
  customers with impacted connections that die at the LAX Verizon-GNI hub.
 
 
  James Laszko
  Mythos Technology Inc
 
  Sent from my iPhone

Re: SAS Drive Enclosure

[Ray Van Dolson]

MD1200 is a great bet then.

Other options -- SuperMicro has lots:

http://www.supermicro.com/products/chassis/2U/?chs=216

Quanta:

http://www.quantaqct.com/Product/Rack-Systems/Rackgo-X/JBODs/JBR-p247c77c86c88c92

On Wed, May 27, 2015 at 01:06:09PM +, Graham Johnston wrote:
 I am primarily wanting something that will act like a DELL MD1200,
 SAS connected to a server, then run a clustered filesystem on the
 server(s) which will serve up NFS or iSCSI to client devices.
 
 Graham Johnston
 Network Planner
 Westman Communications Group
 204.717.2829
 johnst...@westmancom.com
 think green; don't print this email.
 
 -Original Message-
 From: Jameson, Daniel [mailto:daniel.jame...@tdstelecom.com] 
 Sent: Tuesday, May 26, 2015 3:11 PM
 To: Ray Van Dolson; Graham Johnston
 Cc: 'nanog@nanog.org'
 Subject: RE: SAS Drive Enclosure
 
 What are you thinking for connectivity,  Ethernet,  FiberChannel,
 Infiniband ...  Building *Storage Nodes* or in need of just drive
 connectivity?
 
 
 -Original Message-
 From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ray Van Dolson
 Sent: Tuesday, May 26, 2015 2:53 PM
 To: Graham Johnston
 Cc: 'nanog@nanog.org'
 Subject: Re: SAS Drive Enclosure
 
 On Tue, May 26, 2015 at 07:19:59PM +, Graham Johnston wrote:
  I am looking for information about SAS drive enclosures, is there a 
  list like NANOG that covers that area of IT?
  
  I am specifically looking for an enclosure that can handle 12 or more 
  drives, I am looking to create a clustered file system between 
  multiple servers and would like to avoid a drive enclosure that only 
  works with a very small number of approved drives.  I am looking to 
  support traditional HDDs as well as SSDs.
 
 There were discussions at some point about setting up a
 storage-centric list via SNIA or something else fairly 'neutral'.
 Never really materialized, however.
 
 Lists like lopsa-tech and the LISA/USENIX SAGE list are general
 enough you might get some good responses.
 
 WRT your question, we've had good luck with the Dell MD1200 line of
 JBODs.
 
 Ray

Re: SAS Drive Enclosure

[Ray Van Dolson]

On Tue, May 26, 2015 at 07:19:59PM +, Graham Johnston wrote:
 I am looking for information about SAS drive enclosures, is there a
 list like NANOG that covers that area of IT?
 
 I am specifically looking for an enclosure that can handle 12 or more
 drives, I am looking to create a clustered file system between
 multiple servers and would like to avoid a drive enclosure that only
 works with a very small number of approved drives.  I am looking to
 support traditional HDDs as well as SSDs.

There were discussions at some point about setting up a storage-centric
list via SNIA or something else fairly 'neutral'.  Never really
materialized, however.

Lists like lopsa-tech and the LISA/USENIX SAGE list are general enough
you might get some good responses.

WRT your question, we've had good luck with the Dell MD1200 line of
JBODs.

Ray

Re: Voip encryption

[Ray Van Dolson]

On Thu, Apr 09, 2015 at 11:04:06AM -0400, Christopher Morrow wrote:
 On Thu, Apr 9, 2015 at 6:21 AM, Simon Brilus sbri...@blueyonder.co.uk wrote:
  Hi - I have a PCIDSs requirement to encrypt VoIP over a 3rd party VPLS
  network. Has anyone dealt with this. I'd really not use VPN's over the VPLS
  so am looking at hardware WAN encrypters.
 
 wait, you don't want to do some VPN thing over the VPLS network links,
 but you think that hardware wan encrypters are going to work on the
 VPLS links? Did you plan on installing one of these devices at the
 carrier facility? and at all the other possible hops along the way?
 
 or were you hoping that the encrypter would not muddle with the L2
 payload, but leave the L2 headers intact?
 
  Any guidance appreciated.
 

Lost the original post, but why not SIP+TLS  SRTP?

Ray

OT - Small DNS appliances for remote offices.

[Ray Van Dolson]

Hopefully not too far off topic for this list.

Am looking for options to deploy DNS caching resolvers at remote
locations where there may only be minimal infrastructure (FW and Cisco
equipment) and limited options for installing a noisier, more power
hugnry  servers or appliances from a vendor.  Stuff like Infoblox is
too expensive.

We're BIND-based and leaning to stick that way, but open to other
options if they present themselves.

Am considering the Soekris net6501-50.  I can dump a Linux image on
there with our DNS config, indudstrial grade design, and OK
performance.  If the thing fails, clients will hopefully not notice due
to anycast which will just hit another DNS server somewhere else on the
network albeit with additional latency.  We ship out a replacement
device rather than mucking with trying to repair.

There's also stuff like this[1] which probably gives me more horsepower
on my CPU, but maybe not as reliable.

Maybe I'm overengineering this.  What do others do at smaller remote
sites?  Also considering putting resolvers only at hub locations in
our MPLS network based on some latency-based radius.

Ray

[1] http://www.newegg.com/Mini-Booksize-Barebone-PCs/SubCategory/ID-309

OT - Verizon/ATT Cell/4G Signal Booster/Repeater

[Ray Van Dolson]

Hi all;

Looking to improve cell reception for mixed ATT/Verizon users on the
first floor of one of our buildings.

Starting to dig into this and coming across items like this one at
Amazon[1], but thought some of you out there might have recommendations
for something that has worked well for you and has been reliable.

Am in a position to run cable from the roof to the floor in question.

Thanks,
Ray

[1] 
http://www.amazon.com/Wilson-Electronics-Indoor-Cellular-Booster/dp/B00IWW9AB8/ref=lp_2407782011_1_1?s=wirelessie=UTF8qid=1418671553sr=1-1

Re: .mil postmaster Contacts?

[Ray Van Dolson]

On Wed, Oct 29, 2014 at 10:43:34AM -0400, Chuck Church wrote:
 
 -Original Message-
 From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Alain Hebert
 Sent: Wednesday, October 29, 2014 9:14 AM
 To: nanog@nanog.org
 Subject: Re: .mil postmaster Contacts?
 
  Might be related to the news (CNN this morning) about the WH network being
 exploited for a few days now.
  They might be going after some .mil to and the tightening up of those
 networks may cause disruption.
 
 
 I think it has to do with DNSSEC.  The google DNS FAQ mentions (along with
 someone else who emailed me off-list) checking DNSVIZ for issues.  So
 looking at:
 http://dnsviz.net/d/disa.mil/dnssec/
 
 seems to indicate some issues.   RRSET TTL MISMATCH I think they all are.
 Any DISA people on here?  Using a non-Google DNS (which I guess isn't doing
 DNSSEC validation) does resolve the names fine.
 
 Chuck

I saw the same errors in dnsviz, but was unsure if they were sufficient
to cause lookup failures (they were warnings only).

# dig @8.8.8.8 disa.mil MX +dnssec

;  DiG 9.8.2rc1-RedHat-9.8.2-0.23.rc1.el6_5.1  @8.8.8.8 disa.mil MX 
+dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; -HEADER- opcode: QUERY, status: NOERROR, id: 9111
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;disa.mil.  IN  MX

;; ANSWER SECTION:
disa.mil.   20039   IN  MX  5 indal.disa.mil.
disa.mil.   20039   IN  MX  0 pico.disa.mil.
disa.mil.   20039   IN  MX  10 dnipro.disa.mil.
disa.mil.   20039   IN  RRSIG   MX 8 2 86400 2014112128 
2014102228 40608 disa.mil. 
lC2W9knYgviYJUKMYw9FJueUk4cR19spu7QsX3novmYrlOI70F0Rrzxm 
adU17tvfq1vbtzgYH0FriGIMdywPu/ssO7mK4KGhDj7pkQCcJZzlbrMe 
OlJOcC9mQcjgb6nt5KREBaIGzTGY0gA7AM6X2Ft/t9ZdsE/K+jNejgEc 4+M=

I see the ad flag in the query response flags, so am thinking this
lookup succeeded and was validated?

I do note that once we disabled DNSSEC on our resolvers we were able to
push mail out to these domains.  May have been coincidental -- needs
further testing.

Ray

Re: .mil postmaster Contacts?

[Ray Van Dolson]

It *might* have been.  Things cleared up yesterday.  I initially
thought it was the result of disabling DNSSEC on our primary resolvers,
but am less certain that was the fix now as I don't see any issues
with their config (per dnsviz).

Ray

On Mon, Oct 27, 2014 at 09:03:15PM -0400, Chuck Church wrote:
 You sure it's not a DNS issue?  I've had problems resolving various
 *.disa.mil sites today.  Google DNS claims they don't exist.
 
 Chuck
 
 -Original Message-
 From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ray Van Dolson
 Sent: Monday, October 27, 2014 1:52 PM
 To: nanog@nanog.org
 Subject: .mil postmaster Contacts?
 
 We're seeing issues deliving email to certain .mil domains.  MX hosts for
 these domains are not responding on port 25 and have verified from
 off-network as well.
 
 Anyone else seeing the same or can point me to a technical POC to start
 with?
 
 navy.mil, usmc.mil, uscg.mil are just a few that seem to be having issues.
 
 Ray

.mil postmaster Contacts?

[Ray Van Dolson]

We're seeing issues deliving email to certain .mil domains.  MX hosts
for these domains are not responding on port 25 and have verified from
off-network as well.

Anyone else seeing the same or can point me to a technical POC to start
with?

navy.mil, usmc.mil, uscg.mil are just a few that seem to be having
issues.

Ray

Re: Connectivity issue between Verizon and Amazon EC2 (NTT issue?)

[Ray Van Dolson]

On Tue, Jul 22, 2014 at 07:40:26PM -0700, Ray Van Dolson wrote:
 On Tue, Jul 22, 2014 at 05:29:55PM -0700, Ray Van Dolson wrote:
  On Mon, Jul 21, 2014 at 10:54:59PM -0700, Ray Van Dolson wrote:
   Others appear to be having similar issues.  Seems like Verizon is
   pointing at AWS:
   
   https://forums.aws.amazon.com/thread.jspa?messageID=558094
   
   Ray
   
   On Mon, Jul 21, 2014 at 08:56:27PM -0700, Tim Heckman wrote:
Realized I sent the reply to Roland. Apologies.

Here it is in full:



I am seeing the same issue between AWS US-WEST 2 and Hurricane
Electric's Fremont 2 location (Linode). Looks to be deep within
Amanzon's network based on changes in latency in a simple trace
route.

I would provide an mtr, however my network configuration is
something mtr doesn't support.

Cheers!  -Tim
  
  Update on this:
  
  - We have a ticket open with both AWS and Verizon.
  - AWS has responded and felt the issue was with Verizon, but notified
their network team and asked them to investigate further.
  - Nothing back from Verizon yet (anyone here have a Verizon NOC
contact?)
  
  In the interim, the issue persists.
  
 
 Further update -- Verizon indicates that the issue is related to
 saturation on a peering link between themselves and NTT.  Verizon is
 pointing to the NTT side as the source of the saturation / congestion.
 
 We don't have a direct customer relationship with NTT so am hoping
 someone on this list may be able to pass this information along or
 investigate on our behalf.
 
 Ray

To close the loop on this one, Amazon made a change for us and shifted
their peering point from NTT Ashburn to NTT Dallas.

This helped out tremendously, and although we still see times where
things are slower, it's at least not 100KB/sec slow. :)

Appreciate all the responses received.

Ray

Re: Connectivity issue between Verizon and Amazon EC2

[Ray Van Dolson]

On Mon, Jul 21, 2014 at 08:47:36PM -0700, Ray Van Dolson wrote:
 On Tue, Jul 22, 2014 at 10:41:25AM +0700, Roland Dobbins wrote:
  
  On Jul 22, 2014, at 10:31 AM, Ray Van Dolson rvandol...@esri.com wrote:
  
   We're seeing poor performance (very slow download speeds -- 
   100KB/sec) to certain EC2 instances via our Verizon hosted
   circuits.
  
  Have you tried dorking around with your MTU to see if that makes a
  difference?
 
 Not in a position to easily test that tonight (PDT) but will do so
 tomorrow.
 

Monkeyed with the MTU -- 500, 100, 1200 -- no real observed difference
and still seeing lots of retransmits.

Ray

Re: Connectivity issue between Verizon and Amazon EC2

[Ray Van Dolson]

On Mon, Jul 21, 2014 at 10:54:59PM -0700, Ray Van Dolson wrote:
 Others appear to be having similar issues.  Seems like Verizon is
 pointing at AWS:
 
 https://forums.aws.amazon.com/thread.jspa?messageID=558094
 
 Ray
 
 On Mon, Jul 21, 2014 at 08:56:27PM -0700, Tim Heckman wrote:
  Realized I sent the reply to Roland. Apologies.
  
  Here it is in full:
  
  
  
  I am seeing the same issue between AWS US-WEST 2 and Hurricane
  Electric's Fremont 2 location (Linode). Looks to be deep within
  Amanzon's network based on changes in latency in a simple trace
  route.
  
  I would provide an mtr, however my network configuration is
  something mtr doesn't support.
  
  Cheers!  -Tim

Update on this:

- We have a ticket open with both AWS and Verizon.
- AWS has responded and felt the issue was with Verizon, but notified
  their network team and asked them to investigate further.
- Nothing back from Verizon yet (anyone here have a Verizon NOC
  contact?)

In the interim, the issuer persists.

Thanks,
Ray

Re: Connectivity issue between Verizon and Amazon EC2 (NTT issue?)

[Ray Van Dolson]

On Tue, Jul 22, 2014 at 05:29:55PM -0700, Ray Van Dolson wrote:
 On Mon, Jul 21, 2014 at 10:54:59PM -0700, Ray Van Dolson wrote:
  Others appear to be having similar issues.  Seems like Verizon is
  pointing at AWS:
  
  https://forums.aws.amazon.com/thread.jspa?messageID=558094
  
  Ray
  
  On Mon, Jul 21, 2014 at 08:56:27PM -0700, Tim Heckman wrote:
   Realized I sent the reply to Roland. Apologies.
   
   Here it is in full:
   
   
   
   I am seeing the same issue between AWS US-WEST 2 and Hurricane
   Electric's Fremont 2 location (Linode). Looks to be deep within
   Amanzon's network based on changes in latency in a simple trace
   route.
   
   I would provide an mtr, however my network configuration is
   something mtr doesn't support.
   
   Cheers!  -Tim
 
 Update on this:
 
 - We have a ticket open with both AWS and Verizon.
 - AWS has responded and felt the issue was with Verizon, but notified
   their network team and asked them to investigate further.
 - Nothing back from Verizon yet (anyone here have a Verizon NOC
   contact?)
 
 In the interim, the issue persists.
 

Further update -- Verizon indicates that the issue is related to
saturation on a peering link between themselves and NTT.  Verizon is
pointing to the NTT side as the source of the saturation / congestion.

We don't have a direct customer relationship with NTT so am hoping
someone on this list may be able to pass this information along or
investigate on our behalf.

Ray

Re: Muni Fiber and Politics

[Ray Van Dolson]

My municipality (Loma Linda, CA) doesn't offer anything free, but does
provide fiber connectivity (Layer 3) to residents in some portions of
the city.  There were plans at one point to make it available more
broadly, but nearly eight years later I still am not in an area which
has access nor do I think there has been great progress in the
build-out efforts for whatever reasons (costs, lack of demand, etc.).

Ray

On Mon, Jul 21, 2014 at 03:26:54PM -0500, Aaron wrote:
 Do you have an example of a municipality that gives free internet
 access to it's residents?
 
 
 On 7/21/2014 2:26 PM, Matthew Kaufman wrote:
 I think the difference is when the municipality starts throwing in
 free or highly subsidized layer 3 connectivity free with every
 layer 1 connection
 
 Matthew Kaufman
 
 (Sent from my iPhone)
 
 On Jul 21, 2014, at 12:08 PM, Blake Dunlap iki...@gmail.com wrote:
 
 My power is pretty much always on, my water is pretty much always on
 and safe, my sewer system works, etc etc...
 
 Why is layer 1 internet magically different from every other utility?
 
 -Blake
 
 On Mon, Jul 21, 2014 at 1:38 PM, William Herrin b...@herrin.us wrote:
 On Mon, Jul 21, 2014 at 10:20 AM, Jay Ashworth j...@baylink.com wrote:
 Over the last decade, 19 states have made it illegal for municipalities
 to own fiber networks
 Hi Jay,
 
 Everything government does, it does badly. Without exception. There
 are many things government does better than any private organization
 is likely to sustain, but even those things it does slowly and at an
 exorbitant price.
 
 Muni fiber is a competition killer. You can't beat city hall; once
 built it's not practical to compete, even with better service, so
 residents are stuck with only the overpriced (either directly or via
 taxes), usually underpowered and always one-size-fits-all network
 access which results. As an ISP I watched something similar happen in
 Altoona PA a decade and a half ago. It was a travesty.
 
 The only exception I see to this would be if localities were
 constrained to providing point to point and point to multipoint
 communications infrastructure within the locality on a reasonable and
 non-discriminatory basis. The competition that would foster on the
 services side might outweigh the damage on the infrastructure side.
 Like public roads facilitate efficient transportation and freight
 despite the cost and potholes, though that's an imperfect simile.
 
 Regards,
 Bill Herrin

Connectivity issue between Verizon and Amazon EC2

[Ray Van Dolson]

I'm short some important details on this one, but hopefully can fill in
more shortly.

We're seeing poor performance (very slow download speeds -- 
100KB/sec) to certain EC2 instances via our Verizon hosted circuits.
The issue is reproducible on both our production Gigabit circuit as
well as a consumer grade Verizion FIOS line.

Speeds are normal (10MB/sec plus) via non-Verizon circuits we've
tested.

Source IP's are in the 198.102.62.0/24 range and destination on the EC2
side is 54.197.239.228.  I'm not sure in which availability zone the
latter IP sits, but hope to find out shortly.

MTR traceroute details are as follows:

Host   Loss%   Snt Drop   Avg  Best 
 Wrst StDev
1. 198.102.62.253   0.0%   5260   0.2   0.2 
  0.5   0.0
2. 152.179.250.141  0.0%   5260  14.1   7.0 
 19.4   3.6
3. 140.222.225.135 37.5%   526  197   7.7   6.8 
 35.8   1.9
4. 129.250.8.85 0.0%   5260   8.1   7.4 
 11.7   0.3
5. 129.250.2.229   10.3%   525   54  11.4   7.1 
 85.7   9.6
6. 129.250.2.169   41.5%   525  218  63.0  45.5 
130.7  10.3
7. 129.250.2.1540.2%   5251  59.9  44.5 
 69.0   4.0
8. ???
9. 54.240.229.967.8%   525   41  76.6  71.3 
119.9   8.6
54.240.229.104
54.240.229.106
10. 54.240.229.2 6.9%   525   36  74.7  
71.6 109.1   4.9
54.240.229.4
54.240.229.20
54.240.229.8
54.240.229.14
54.240.228.254
54.240.229.16
54.240.229.10
11. 54.240.229.174   5.5%   525   29  76.0  
71.7 109.0   7.3
54.240.229.162
54.240.229.160
54.240.229.170
54.240.229.172
54.240.229.168
54.240.229.164
12. 54.240.228.167  94.5%   525  495  76.4  
71.7 126.0  11.6
54.240.228.169
54.240.228.165
54.240.228.163
13. 72.21.220.1085.1%   525   27  75.2  
71.3 112.6   6.8
205.251.244.12
72.21.220.8
205.251.244.64
72.21.220.96
205.251.244.8
72.21.220.6
205.251.244.4
14. 72.21.220.45 9.0%   525   47  74.0  
71.6 199.5   8.5
72.21.220.149
72.21.220.29
72.21.220.125
72.21.220.37
72.21.220.61
72.21.220.2
72.21.220.69
15. 72.21.222.3310.5%   525   55  73.4  
71.5  87.1   1.5
205.251.245.65
72.21.222.149
72.21.222.35
72.21.220.29
72.21.222.131
72.21.222.147
72.21.220.37
16. 205.251.245.65  93.9%   525  492  73.1  
72.2  76.2   1.2
72.21.222.35
72.21.222.131
17. ???
18. ???
19. 216.182.224.79  13.5%   524   71  77.9  
72.4 101.2   5.4
216.182.224.81
216.182.224.95
216.182.224.77
20. 216.182.224.81  94.1%   524  492  77.9  
72.8  93.0   6.3
216.182.224.95
216.182.224.77
21. ???

The 140.222.225.135 shows up in the traceroutes via our Verizon
Business FIOS line as well.

Will be opening a ticket with both Verizon and AWS to assist, but
hoping someone out there can take a look or chime in.  Feel free to
reply off list.

Thanks,
Ray

Re: Connectivity issue between Verizon and Amazon EC2

[Ray Van Dolson]

On Tue, Jul 22, 2014 at 10:41:25AM +0700, Roland Dobbins wrote:
 
 On Jul 22, 2014, at 10:31 AM, Ray Van Dolson rvandol...@esri.com wrote:
 
  We're seeing poor performance (very slow download speeds -- 
  100KB/sec) to certain EC2 instances via our Verizon hosted
  circuits.
 
 Have you tried dorking around with your MTU to see if that makes a
 difference?

Not in a position to easily test that tonight (PDT) but will do so
tomorrow.

Ray

Re: Connectivity issue between Verizon and Amazon EC2

[Ray Van Dolson]

Others appear to have repoted this.  Seems like Verizon is pointing at
AWS:

https://forums.aws.amazon.com/thread.jspa?messageID=558094

Ray

On Mon, Jul 21, 2014 at 08:56:27PM -0700, Tim Heckman wrote:
 Realized I sent the reply to Roland. Apologies.
 
 Here it is in full:
 
 
 
 I am seeing the same issue between AWS US-WEST 2 and Hurricane Electric's
 Fremont 2 location (Linode). Looks to be deep within Amanzon's network based 
 on
 changes in latency in a simple trace route.
 
 I would provide an mtr, however my network configuration is something mtr
 doesn't support.
 
 Cheers! 
 -Tim
 
 On Jul 21, 2014 8:34 PM, Ray Van Dolson rvandol...@esri.com wrote:
 
 I'm short some important details on this one, but hopefully can fill in
 more shortly.
 
 We're seeing poor performance (very slow download speeds -- 
 100KB/sec) to certain EC2 instances via our Verizon hosted circuits.
 The issue is reproducible on both our production Gigabit circuit as
 well as a consumer grade Verizion FIOS line.
 
 Speeds are normal (10MB/sec plus) via non-Verizon circuits we've
 tested.
 
 Source IP's are in the 198.102.62.0/24 range and destination on the EC2
 side is 54.197.239.228.  I'm not sure in which availability zone the
 latter IP sits, but hope to find out shortly.
 
 MTR traceroute details are as follows:
 
 Host   Loss%   Snt Drop   Avg
  Best  Wrst StDev
 1. 198.102.62.253   0.0%   5260   0.2 
  
 0.2   0.5   0.0
 2. 152.179.250.141  0.0%   5260  14.1 
  
 7.0  19.4   3.6
 3. 140.222.225.135 37.5%   526  197   7.7 
  
 6.8  35.8   1.9
 4. 129.250.8.85 0.0%   5260   8.1 
  
 7.4  11.7   0.3
 5. 129.250.2.229   10.3%   525   54  11.4 
  
 7.1  85.7   9.6
 6. 129.250.2.169   41.5%   525  218  63.0
  45.5 130.7  10.3
 7. 129.250.2.1540.2%   5251  59.9
  44.5  69.0   4.0
 8. ???
 9. 54.240.229.967.8%   525   41  76.6
  71.3 119.9   8.6
 54.240.229.104
 54.240.229.106
 10. 54.240.229.2 6.9%   525   36  74.7
  71.6 109.1   4.9
 54.240.229.4
 54.240.229.20
 54.240.229.8
 54.240.229.14
 54.240.228.254
 54.240.229.16
 54.240.229.10
 11. 54.240.229.174   5.5%   525   29  76.0
  71.7 109.0   7.3
 54.240.229.162
 54.240.229.160
 54.240.229.170
 54.240.229.172
 54.240.229.168
 54.240.229.164
 12. 54.240.228.167  94.5%   525  495  76.4
  71.7 126.0  11.6
 54.240.228.169
 54.240.228.165
 54.240.228.163
 13. 72.21.220.1085.1%   525   27  75.2
  71.3 112.6   6.8
 205.251.244.12
 72.21.220.8
 205.251.244.64
 72.21.220.96
 205.251.244.8
 72.21.220.6
 205.251.244.4
 14. 72.21.220.45 9.0%   525   47  74.0
  71.6 199.5   8.5
 72.21.220.149
 72.21.220.29
 72.21.220.125
 72.21.220.37
 72.21.220.61
 72.21.220.2
 72.21.220.69
 15. 72.21.222.3310.5%   525   55  73.4
  71.5  87.1   1.5
 205.251.245.65
 72.21.222.149
 72.21.222.35
 72.21.220.29
 72.21.222.131
 72.21.222.147
 72.21.220.37
 16. 205.251.245.65  93.9%   525  492  73.1
  72.2  76.2   1.2
 72.21.222.35
 72.21.222.131
 17. ???
 18. ???
 19. 216.182.224.79  13.5%   524   71  77.9
  72.4 101.2   5.4
 216.182.224.81
 216.182.224.95
 216.182.224.77
 20. 216.182.224.81  94.1%   524  492  77.9
  72.8  93.0   6.3
 216.182.224.95
 216.182.224.77
 21. ???
 
 The 140.222.225.135 shows up in the traceroutes via our Verizon
 Business FIOS line as well.
 
 Will be opening a ticket with both Verizon and AWS to assist, but
 hoping someone out there can take a look or chime in.  Feel free to
 reply off list.
 
 Thanks,
 Ray

Looking for an Amazon EC2 East Contact

[Ray Van Dolson]

Seeing pretty consistent packet loss to/from instances in EC2 East
(54.80 IPs) from various vantage points.

Working through normal support channels, but looking for a contact to
help expedite.

Thanks,
Ray

Re: Is the FBI's DNSSEC broken?

[Ray Van Dolson]

On Fri, Aug 30, 2013 at 10:27:36PM +, John Levine wrote:
 I don't claim to be a big DNSSEC expert, but this looks just plain
 wrong to me, and unbound agrees, turning it into a SERVFAIL.
 
 Here's a lookup that succeeds, an A record for mail.ic.fbi.gov:
 
 $ dig @ns1.fbi.gov mail.ic.fbi.gov a +dnssec
 
 ;; -HEADER- opcode: QUERY, status: NOERROR, id: 7222
 ;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 7, ADDITIONAL: 1
 ;; WARNING: recursion requested but not available
 
 ;; OPT PSEUDOSECTION:
 ; EDNS: version: 0, flags: do; udp: 65235
 ;; QUESTION SECTION:
 ;mail.ic.fbi.gov. IN  A
 
 ;; ANSWER SECTION:
 mail.ic.fbi.gov.  600 IN  A   153.31.119.142
 mail.ic.fbi.gov.  600 IN  RRSIG   A 7 4 600 20131124123847 
 20130826123847 32497 fbi.gov. 
 dYs+1bPdO+8y3T5ij8qSn0BvTDv7X51wi++HV681rKzlK5SLKrZiGryV 
 ow67iO30CWwztI3d5oCF7/6bEn3NetWq9IajeM19aorIdJMA6tAp1BQI 
 EZMTcCsnInSIn2IRb3V2MXXOBx6r6wMt7ptNfp/Tro89h2K7q+Pgp0O2 WdU=
 
 ;; AUTHORITY SECTION:
 fbi.gov.  600 IN  NS  ns3.fbi.gov.
 fbi.gov.  600 IN  NS  ns5.fbi.gov.
 fbi.gov.  600 IN  NS  ns4.fbi.gov.
 fbi.gov.  600 IN  NS  ns2.fbi.gov.
 fbi.gov.  600 IN  NS  ns1.fbi.gov.
 fbi.gov.  600 IN  NS  ns6.fbi.gov.
 fbi.gov.  600 IN  RRSIG   NS 7 2 600 20131124123847 
 20130826123847 32497 fbi.gov. 
 l/AcT+Pmr/5yosWyvP3zbFIJE7f07F+AA8eh1X3qv8ulw9FbC0DhZfSo 
 1f5ctD6DIb613ButzKG01PdMzIknMroraOyGyRcAq27qYXzKRE0cTqhv 
 UWz15jLa7N7YKYccR8Hmt6GY1DJitY41EwQP7Z2Fpac9yPTRnybc4mTS 4eY=
 
 Here's a query for the same name, but for  which it doesn't have:
 
 $ dig @ns1.fbi.gov mail.ic.fbi.gov  +dnssec
 
 ;  DiG 9.8.3-P4  @ns1.fbi.gov mail.ic.fbi.gov  +dnssec
 ; (2 servers found)
 ;; global options: +cmd
 ;; Got answer:
 ;; -HEADER- opcode: QUERY, status: NOERROR, id: 41056
 ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 1
 ;; WARNING: recursion requested but not available
 
 ;; OPT PSEUDOSECTION:
 ; EDNS: version: 0, flags: do; udp: 65235
 ;; QUESTION SECTION:
 ;mail.ic.fbi.gov. IN  
 
 ;; AUTHORITY SECTION:
 fbi.gov.  600 IN  SOA ns1.fbi.gov. dns-admin.fbi.gov. 
 2013082601 7200 3600 2592000 43200
 95RIPFTKTJC9I7J8HDAIA7CM6L279FSR.fbi.gov. 43200   IN NSEC3 1 0 10 BBAB 
 97S2G907NEFOJ79P721E4FEQ9LR3IT1S A RRSIG
 fbi.gov.  600 IN  RRSIG   SOA 7 2 600 20131124123847 
 20130826123847 32497 fbi.gov. 
 QgsdhUT7AHic8tJv39br+994eoyJ4c8/SuQr35dRudceE/bYyZV26IPI 
 4qnR8Cy35WoepW12bhhhY0Ug26Qy81KWcWHYPw0Wa7g5Ig8Pw27l8gCV 
 J7NDY6O5jTb4MMc9THTPKEvXjeX/YE4060HrbJXo1U93qhdILkGTvno7 3hA=
 
 Shouldn't there be some more stuff there in the authority section,
 like an NSEC3 and RRSIG for mail.ic.fbi.gov?
 
 Am I missing something, or is it broken?  The server says it's from
 Ultradns.
 
 R's,
 John

Hi John;

I don't think you're alone on this!  Ref this thread (an issue we ran
into with accepting mail from ic.fbi.gov due to DNSSEC validation
failure) from July[1].

Have done my best to get someone's attention to fix the issue, but so
far no joy.

Ray

[1] https://lists.isc.org/pipermail/bind-users/2013-July/091140.html

fbi.gov hostmaster contact?

[Ray Van Dolson]

Looking for a DNS/hostmaster contact for fbi.gov for troubleshooting a
DNSSEC issue[1].

Have tried the usual hostmaster alias with no luck.

Thanks,
Ray

[1] https://lists.isc.org/pipermail/bind-users/2013-July/091142.html 

Re: Wired access to SMS?

[Ray Van Dolson]

On Tue, Oct 09, 2012 at 03:35:37PM -0400, William Herrin wrote:
 Hi Folks,
 
 I'm looking for a way to do wireline access to send and receive
 cellular phone short message service (SMS) messages. Despite all my
 google-fu, I have had limited luck finding anyone that meets my needs,
 so I'm hoping someone here has found the path through. My main
 criteria are:
 
 
 1. Low quantity, high reliability. I'll want a few dozen phone numbers
 and effectively I'll be sending to and receiving from phones I own.
 2. Wireline delivery to Honolulu and Northern Virginia. Dynamically
 move numbers between the two locations for failover purposes.
 3. U.S. based carrier. Tying in to the SMS system via Europe isn't
 acceptable to my customer.
 4. Solution must reach phones on all U.S. cellular carriers.
 5. Price is a very distant fifth criteria to the preceding four.
 
 I can consider Internet based systems where the provider uses U.S.
 based facilities and ties in to a U.S. phone network, provided that my
 standards of reliability and redundancy are met by their
 infrastructure.
 
 Alternately, I can also consider a wireless carrier that can provide
 two SIM-based phones with the same phone number for sending and
 receiving SMS messages. I'd put the sims in a pair of modems and
 manage deduplication of the received messages in software.
 
 
 Has anybody had any luck with this kind of requirement? Which vendors
 should I talk to and who at the vendor?
 
 Thanks,
 Bill Herrin

We use the MultiTech MultiModem iSMS SF-100G linked up to an ATT
Wireless account.

It has a RESTful API and can handle both transmission and reception of
text messages.

There are probably SaaS options out there, but have never explored.

Thanks,
Ray

Re: Wired access to SMS?

[Ray Van Dolson]

On Tue, Oct 09, 2012 at 06:17:26PM -0400, William Herrin wrote:
 On Tue, Oct 9, 2012 at 6:13 PM, Ray Van Dolson rvandol...@esri.com wrote:
  On Tue, Oct 09, 2012 at 03:35:37PM -0400, William Herrin wrote:
  Alternately, I can also consider a wireless carrier that can provide
  two SIM-based phones with the same phone number for sending and
  receiving SMS messages. I'd put the sims in a pair of modems and
  manage deduplication of the received messages in software.
 
  We use the MultiTech MultiModem iSMS SF-100G linked up to an ATT
  Wireless account.
 
  It has a RESTful API and can handle both transmission and reception of
  text messages.
 
 Hi Ray,
 
 Have you figured out how to get ATT to give you two SIMs with the
 same phone number? I'm using a different set of multitech modems now
 but I need the same, I guess the terminology is SMS long code, at
 both sites.
 
 Regards,
 Bill Herrin

Sorry, Bill -- not something we've had a need for so have never tried.
:)

Ray

Re: guys != gender neutral

[Ray Van Dolson]

On Thu, Sep 27, 2012 at 02:57:36PM -0400, Andrew D Kirch wrote:
 I really wish people would get over themselves and get to work.
 Work is a place where things get done, not where people piss and
 moan about every single perceived slight they can come up with.
 
 Andrew

I only wish you had used 'guys' instead of 'people' :)

Ray

Re: MXLogic outage

[Ray Van Dolson]

On Wed, Aug 08, 2012 at 04:39:04PM +, Blake Pfankuch wrote:
 We are the same way.  Phones going nuts ringing as we are an MXLogic
 partner.  I am slowly getting email with about a 2-3 hour delay right
 now.  Anyone know any more?
 
 -Original Message-
 From: Duane Toler [mailto:deto...@gmail.com] 
 Sent: Wednesday, August 08, 2012 10:34 AM
 To: nanog@nanog.org
 Subject: MXLogic outage
 
 Probably old news by now, but MXLogic folks are having some major
 issues today and not reliably receiving inbound mail.  Several of our
 customers are talking with MXLogic about it.
 
 FYI.

What MX servers are your affected domains using?

Ours are:

208.65.145.3
208.65.145.2
208.65.144.2
208.65.144.3

And no obvious delays currently.

Ray

Re: The day SORBS goes away ...

[Ray Van Dolson]

On Mon, Apr 09, 2012 at 09:50:00AM -0700, Brian Keefer wrote:
 
 
 On Apr 7, 2012, at 4:41 PM, TR Shaw wrote:
  
  As for SORBS, most competent mail admins dropped its use a long
  time ago. I thought when Proofpoint took it over things would
  change (I actually thought they would dump the SORBS name because
  of bad karma) but it hasn't happened.
 
 Out of curiosity, has anyone other than the OP and one other
 gentlemen on the 4th had a serious issue? Do we know whether the
 issues from the 4th have been resolved? I'm wondering whether this is
 a chronic issue, or if folks are just extrapolating from one
 complaint.
 
 I looked back through the archives for the last year and the only
 other SORBS mentions were in July and August of last year.

I last worked at an ISP back in 2006, so this may not be relevant
today.  I do remember however relying pretty much exclusively on
Spamhaus.  Originally used SORBS too but found they were overly
aggressive on what they'd add to their RBL.  Maybe great if you're an
individual user, but not so great to be blocking all of yahoo/gmail,
etc as an ISP.

Don't think they were as frustrating to deal with as Spamcop though :)

Ray

Re: Time Warner Telecom problems

[Ray Van Dolson]

On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote:
 Gizmodo is reporting problems at Time Warner Telecom  we're suffering
 from it too and calls to the NOC have not been answered so far...  does
 anyone have any further information?
 
 http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us

FWIW, my home TWC connection dropped this morning for about 15 minutes
(Southern California around 6:30AM'ish).  Still could ping the default
gateway, but packets weren't traversing much beyond that.

Didn't investigate further, just headed into work.

Ray

Re: Apple updates - Affect on network

[Ray Van Dolson]

On Wed, Oct 12, 2011 at 01:10:08PM -0700, Zachary McGibbon wrote:
 With all of Apple's updates today (MacOS, iOS, Apps, etc) we saw a big
 increase on one of our links to our ISP at 1pm Eastern.
 
 Did anyone else notice significant traffic jumps on their networks?

That's an impressive jump.  Do you have some netflow data showing the
target subnets that were being hit?

Ray