Re: Famous operational issues

[Adam Kennedy via NANOG]

While we're talking about raid types...

A few acquisitions ago, between 2006-2010, I worked at a Wireless ISP in
Northern Indiana. Our CEO decided to sell Internet service to school
systems because the e-rate funding was too much to resist. He had the idea
to install towers on the schools and sell service off that while paying the
school for roof rights. About two years into the endeavor, I wake up one
morning and walk to my car. Two FBI agents get out of an unmarked towncar.
About an hour later, they let me go to the office where I found an entire
barrage of FBI agents. It was a full raid and not the kind you want to see.
Hard drives were involved and being made redundant, but the redundant
copies were labeled and placed into boxes that were carried out to SUVs
that were as dark as the morning coffee these guys drank. There were a lot
of drives, all of our servers were in our server room at the office. There
were roughly five or six racks of varying amounts of equipment in each.

After some questioning and assisting them in their cataloging adventure,
the agents left us with a ton of questions and just enough equipment to
keep the customers connected. CEO became extremely paranoid at this point.
He told us to prepare to move servers to a different building. He went into
a tailspin trying to figure out where he could hide the servers to keep
things going without the bank or FBI seizing the assets. He was extremely
worried the bank would close the office down. We started moving all network
routing around to avoid using the office as our primary DIA.

One morning I get into the office and we hear the words we've been
dreading: "We're moving the servers". The plan was to move them to a tower
site that had a decent-sized shack on site. Connectivity was decent, we had
a licensed 11GHz microwave backhaul capable of about 155mbps. The site was
part of the old MCI microwave long-distance network in the 80s and 90s. It
had redundant air conditioners, a large propane tank, and a generator
capable of keeping the site alive for about three days. We were told not to
notify any customers, which became problematic because two customers had
servers colocated in our building. We consolidated the servers into three
racks and managed to get things prepared with a decent UPS in each rack.
CEO decided to move the servers at nightfall to "avoid suspicion". Our
office was in an unsavory part of town, moving anything at night was
suspicious. So, under the cover of half-ass darkness, we loaded the racks
onto a flatbed truck and drove them 20 minutes to the tower. While we
unloaded the racks, an electrician we knew was wiring up the L5-20 outlets
for the UPS in each rack. We got the racks plugged in, servers powered up,
and then the two customers came that had colocated equipment. They got
their equipment powered up and all seemed ok.

Back at the office the next day we were told to gather our workstations and
start working from home. I've been working from home ever since and quite
enjoy it, but that's beside the point.

Summer starts and I tell the CEO we need to repair the AC units because
they are failing. He ignores it, claiming he doesn't want to lose money the
bank could take at any minute. About a month later, a nice hot summer day
rolls in and the AC units both die. I stumble upon an old portable AC unit
and put that at the site. Temperatures rise to 140F ambient. Server
overheat alarms start going off, things start failing. Our colocation
customers are extremely upset. They pull their servers and drop service.
The heat subsides, CEO finally pays to repair one of the AC units.

Eventually, the company declares bankruptcy and goes into liquidation.
Luckily another WISP catches wind of it, buys the customers and assets, and
hires me. My happiest day that year was moving all the servers into a
better-suited home, a real data center. I don't know what happened to the
CEO, but I know that I'll never trust anything he has his hands in ever
again.

Adam Kennedy
Systems Engineer
adamkenn...@watchcomm.net | 800-589-3837 x120 <800-589-3837;120>
Watch Communications | www.watchcomm.net

3225 W Elm St, Suite A
Lima, OH 45805





On Tue, Feb 23, 2021 at 8:55 PM brutal8z via NANOG  wrote:

> My war story.
>
> At one of our major POPs in DC we had a row of 7513's, and one of them had
> intermittent problems. I had replaced every piece of removable card/part in
> it over time, and it kept failing. Even the vendor flew in a team to the
> site to try to figure out what was wrong. It was finally decided to replace
> the whole router (about 200lbs?). Being the local field tech, that was my
> Job. On the night of the maintenance at 3am, the work started. I switched
> off the rack power, which included a 2511 terminal 

Re: DNS cache Validation

[Adam Kennedy via NANOG]

I wrote a script to expose stats from unbound to SNMP and built a Cacti
template for that. Recently started moving the DNS stats to feed into
Telegraf that pushes to an InfluxDB server, then built a dashboard in
Grafana. We track DNS RTT for a few queries, number of drops, number of
rejects, various record type requests per second, etc. We also have a
Nagios plugin that checks each of our DNS cache resolving servers scattered
across the network to ensure they can resolve a handful of popular domains.
Adam Kennedy
Systems Engineer
adamkenn...@watchcomm.net | 800-589-3837 x120 <800-589-3837;120>
Watch Communications | www.watchcomm.net

3225 W Elm St, Suite A
Lima, OH 45805





On Mon, May 18, 2020 at 11:47 PM Justin Wilson (Lists) 
wrote:

> What are you folk doing to validate your DNS cache server configs
> and operation? In other words, what are you doing to make sure they are
> performing well, not just alive.
>
> Justin
> —
> https://blog.j2sw.com

Re: WIKI documentation Software?

[Adam Kennedy via NANOG]

We've been using BookStack. It's easy for staff to use and understand. We
gave each department their own "shelf" in there and can assign rights to
shelves so managers of the departments can add their own
books/chapters/pages. Once you dive in you'll see how it's organized but
it's a really solid platform. Supports LDAP auth as well. Great platform,
we've loved it.

https://www.bookstackapp.com/
Adam Kennedy
Systems Engineer
adamkenn...@watchcomm.net | 800-589-3837 x120 <800-589-3837;120>
Watch Communications | www.watchcomm.net

3225 W Elm St, Suite A
Lima, OH 45805





On Sat, Mar 14, 2020 at 7:09 PM  wrote:

> I've been using MoinMoin wiki for years.  It hasn't been updated for
> quite a while, but it has worked really well for me, is trivial to
> install, uses text file backend so no need for a database, allows for
> hierarchical structure, is pretty fast, is very very light weight and
> extensible, built on python and free.
>
> I don't know if there is a docker container, but I'm thinking of
> building one.
>
> If you want something simple, stable, older, small and usable you might
> take a look at MoinMoin.
>
> If you want a docker container, ask and I'll probably build one.
>
> Geoff
>
>
> On 3/14/20 2:35 PM, Gavin Henry wrote:
> > I think DokuWiki does this and as an added bonus saves all as text files.
>
>

Re: Are network operators morons? [was: CloudFlare issues?]

[Adam Kennedy via NANOG]

Now with that out of the way...  The mentality of everyone working together
for a Better Internet (tm) is sort of a mantra of WISPA and WISPs in
general. It is a mantra that has puzzled me and perplexed my own feelings
as a network engineer. Do I want a better overall experience for my users
and customers? Absolutely. Do I strive to make our network the best...
pause... in the world? Definitely. Should I do the same to help a
neighboring ISP, a competitor? This is where I scratch my head. You would
absolutely think that we would all want a better overall Internet. One that
we can depend on in times of need. One that we can be proud of. But we are
driven, unfortunately, by our C-level execs to shun the competition and do
whatever we can to get a leg up on everyone else. While this is good for
the bottom line it is not exactly a healthy mentality to pit everyone
against each other. It causes animosity between providers and we end up
blaming each other for something simple and then claim they are stupid. A
mistake that may be easy to make, a mistake that we have probably made
ourselves a few times, perhaps a mistake we can learn to shrug off.

I believe there probably is a happy medium we can all meet, sort of our own
ISP DMZ, where we can help one another in the simple mistakes or cut each
other some slack in those difficult times. I like to think NANOG is that
place.

--

Adam Kennedy, Network & Systems Engineer

adamkenn...@watchcomm.net

*Watch Communications*

(866) 586-1518






On Tue, Jun 25, 2019 at 8:50 AM Matthew Walster  wrote:

>
>
> On Tue, 25 Jun 2019, 14:31 Patrick W. Gilmore,  wrote:
>
>> I must be old. All I can think is Kids These Days, and maybe Get Off My
>> BGP, er Lawn.
>>
>
> Maybe they ought to [puts on shades] mind their MANRS.
>
> M (scuttling away)
>
>>

Re: NG Firewalls & IPv6

[Adam Kennedy via NANOG]

We've been using DHCP-PD with Sophos SG/XG on a couple Comcast connections
and it works fine. It will even go through all your firewall objects and
automatically change the IPv6 prefix from the old to new if the prefix from
PD changes.

--

Adam Kennedy, Network & Systems Engineer

adamkenn...@watchcomm.net

*Watch Communications*

(866) 586-1518





On Wed, Apr 4, 2018 at 2:41 PM, Chuck Anderson  wrote:

> Also, IPv6 BGP support was only introduced in PanOS 8.  But everything
> works fine here too.
>
> On Wed, Apr 04, 2018 at 10:47:45AM +, Dan Kitchen wrote:
> > We run PaloAlto dual stack with no problems at all, that’s full dynamic
> routing with OSPF and BGP, web filtering, IPS, VPN access using
> GlobalProtect, etc.
> >
> > I must admit GlobalProtect IPv6 support was only introduced in PanOS 8
> which was a little late in my opinion – but it was delivered and works.
> >
> >
> >
> >
> > Dan Kitchen
> > Managing Director
> > razorblue | IT Solutions for Business
> >
> > ddi:0330 122 7143 |  t: 0333 344 6 344 | e: dkitc...@razorblue.com
>  | w: razorblue.com
> >
> > Legal and address information for all Razorblue Group companies can be
> found
> > at www.razorblue.com/contact.
> >
> > From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joe Klein
> > Sent: 02 April 2018 23:58
> > To: NANOG list 
> > Subject: NG Firewalls & IPv6
> >
> > WARNING: This e-mail originated from outside the Razorblue Group
> corporate network
> >
> > All,
> >
> > At security and network tradeshows over the last 15 years, I have asked
> > companies if their products supported "IPv6". They all claimed they did,
> > but were unable to verify any successful installations. Later they told
> me
> > it was on their "Roadmap" but were unable to provide an estimated year,
> > because it was a trade secret.
> >
> > Starting this last year at BlackHat US, I again visited every product
> > booth, asking if their products supported dual-stack or IPv6 only
> > operations. Receiving only the same unsupported answers, I decided to
> focus
> > on one product category.
> >
> > To the gurus of the NANOG community, What are your experiences with
> > installing and managing Next Generations firewalls? Do they support IPv6
> > only environments? Details? Stories?
> >
> > If you prefer not to disparage those poor product companies, please
> contact
> > me off the list.
> >
> > Thanks,
> >
> > Joe Klein
>

Re: NG Firewalls & IPv6

[Adam Kennedy via NANOG]

We've deployed about a dozen Sophos SG and XG firewalls with IPv6 on WAN,
LAN and VPN with great success. The XG is the firmware with the more modern
appearance and a couple latest-gen features. But the SG is just as "next
gen" and still has good IPv6 capability.

--

Adam Kennedy, Network & Systems Engineer

adamkenn...@watchcomm.net

*Watch Communications*

(866) 586-1518





On Wed, Apr 4, 2018 at 1:44 AM, Jima  wrote:

> Hey Joe,
>
> I don't know how next-gen they'd be considered, but I've had reasonably
> good luck with Cisco ASA (v9+), and to a lesser degree Juniper ScreenOS
> (v6.3+). Modern-ish ASA does v6-only pretty well; ScreenOS has more
> v4-dependent nuances, that I've found.
>
> I do like the NAT64 support in ASA (although it sadly doesn't support the
> Well-Known Prefix) -- no love in ScreenOS, as far as I've ever found.
>
> - Jima
>
> > On Apr 2, 2018, at 16:58, Joe Klein  wrote:
> >
> > All,
> >
> > At security and network tradeshows over the last 15 years, I have asked
> > companies if their products supported "IPv6". They all claimed they did,
> > but were unable to verify any successful installations. Later they told
> me
> > it was on their "Roadmap" but were unable to provide an estimated year,
> > because it was a trade secret.
> >
> > Starting this last year at BlackHat US, I again visited every product
> > booth, asking if their products supported dual-stack or IPv6 only
> > operations. Receiving only the same unsupported answers, I decided to
> focus
> > on one product category.
> >
> > To the gurus of the NANOG community, What are your experiences with
> > installing and managing Next Generations firewalls? Do they support IPv6
> > only environments? Details? Stories?
> >
> > If you prefer not to disparage those poor product companies, please
> contact
> > me off the list.
> >
> > Thanks,
> >
> > Joe Klein
> >
> > "inveniet viam, aut faciet" --- Seneca's Hercules Furens (Act II, Scene
> 1)
> > PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8
>