Re: Perhaps it's time to think about enhancements to the NANOG list...?

2021-03-20 Thread Rich Kulawiec
On Sat, Mar 20, 2021 at 12:46:57PM -0600, David Siegel wrote:
> The board has been thinking about enhancements to the NANOG list for a
> couple of years now, with the goal of creating a modern interface that the
> younger generation of engineers will be more comfortable using.

This isn't a valid goal.  It's fine that some people can't handle mailing
lists -- but then they shouldn't be network engineers or system admins,
because (a) using mailing lists is a fundamental skill required in those
fields and (b) anyone can't master such a rudimentary task in relatively
short order really isn't equipped to be an engineer/admin.  (Just like
someone who can't do binary arithmetic or grasp multi-step processes
shouldn't be an engineer/admin.  This doesn't make them bad people,
it just makes them people who are unlikely to be successful in the field.)

> Those of you that have attended recent NANOG members meetings may recall
> that we are currently beta testing a new community interface called
> discourse as part of our NANOG modernization strategic initiative.

Discourse is a MAJOR downgrade from the functionality of mailing lists.

Oh, it's shiny and pretty and all that, but it's not a good tool for
serious professional or even amateur communication.  (And, of particular
interest to *this* list, it performs extremely poorly -- if at all --
when confronted with (a) network outages and congestion and (b) attacks
and abuse.  Two of the *many* significant advantages of properly-run
mailing lists are that they continue to function plausibly well under
highly adverse conditions and that there are numerous, well-understood
tactical and strategic mechanisms for defending them.)


Re: Perhaps it's time to think about enhancements to the NANOG list...?

2021-03-20 Thread Rich Kulawiec
On Thu, Mar 18, 2021 at 03:28:31PM -0700, Matthew Petach wrote:
> If only we had some way to segregate out different topics
> of interest or disinterest, so that people who weren't interested
> in questions about bandwidth availability could unsubscribe
> from those topics, and only subscribe to the topics that *did*
> interest them...

1. Anyone who's using a professional-quality mail client like mutt already
has this capability.  There's no need to modify the mailing list to
accomodate people who choose to use any of the lesser mail clients that
don't facilitate basic threading, filtering, and sorting.

2. This is a low-traffic list, so even without appropriate mail client
support it's really not a big deal.


Re: OVH datacenter SBG2 in Strasbourg on fire ????

2021-03-12 Thread Rich Kulawiec
On Fri, Mar 12, 2021 at 02:46:51PM +, David Hubbard wrote:
> After sending them abuse reports for years with only an increase in
> malicious traffic, I have no expectation of anything they do getting
> better or being for the benefit of the internet as a whole.

This is a shared experience. It's abundantly clear that OVH is either
acting in concert with the abusers, or they *are* the abusers.  It doesn't
really matter which, the operational outcome is the same in either case.

The best course of action is to remove them from your (generic you)
view of the Internet via whatever means are most expedient.


Re: OVH datacenter SBG2 in Strasbourg on fire ????

2021-03-10 Thread Rich Kulawiec
If you give people the means to hurt you, and they do it, and
you take no action except to continue giving them the means to
hurt you, and they take no action except to keep hurting you,
then one of the ways you can describe the situation is "it isn't
scaling well".
--- Paul Vixie (on NANOG)

1. I have every OVH network block that I'm aware of permanently firewalled
out.  I recommend the same to everyone else, unless you're doing research.

2. When something burns that's not supposed to burn, that was designed
not to burn, that was built not to burn, that was operated not to burn,
then one of the distinct possibilities is that it wasn't an accident.


Re: Texas internet connectivity declining due to blackouts

2021-02-23 Thread Rich Kulawiec
On Mon, Feb 22, 2021 at 08:44:32PM +0200, Saku Ytti wrote:
> On Mon, 22 Feb 2021 at 20:28, Rich Kulawiec  wrote:
> > right: artificial sweeteners are safe, WMDs were in Iraq, and Anna Nicole
> Hope you meant to write 'unsafe', as the conspiracy theory is that
> aspartame is unsafe, the science says it is safe.

Those last three points are a quote from a movie -- which is why I included
the shout-out to Levon Helm (warning, spoilers, the quote's at about 2:00):

Shooter: Levon Helm as Mr Rate - YouTube

I included them as a joke because anyone who disputes AGW at this
point *is* a joke and should be laughed out of the room.

Less snarktastically, a very good starting point for people who want to
understand the science of global warming is this document:

Climate Change 2013: The Physical Science Basis

It's exhaustive in its coverage (which is why it's 1500+ pages) and reading
it will require basic literacy in math/stat and physical science.  But it's
part of the required homework for anyone trying to understand this topic.
The 2021 version is now in preparation and if things go well, it should be
out mid-to-late summer.

Another highly useful document is:

Fourth National Climate Assessment

which also clocks in at 1500+ pages.  This document has both a broader focus
(for example, it discusses impacts and mitigation) and a narrower focus
(it's US-centric).  It's also written for a more general audience and
requires less math/science background for comprehension, so I recommend
that anybody who struggles to get through the one above try this instead.

Also: this one is arguably more useful for NANOG/operational/planning
purposes.  I think it'd be a good read for anyone who's trying to figure
out what's going to happen to their physical assets/locations, or for
anyone who's trying to plan where to put things and how to build them.

Additional resources:

Climate Change and Infrastructure, Urban Systems, and Vulnerability
Technical Report for the US DoE
Thomas Wilbanks and Steven J. Fernandez

(This one is also useful for NANOG denizens.  Chapter 5 on risk mitigation
strategies is particularly interesting.)

The Encyclopedia of Global Warming and Climate Change, 2ed 
S. George Philander, editor

(A general reference.  Having this handy while reading the IPCC report
I mentioned above can be helpful.)

Atmospheric Thermodynamics - Elementary Physics and Chemistry
Gerald R. North and Tatiana L. Erukhimova

(You'll need integral and differential calculus for this one, and a previous
course in introductory thermodynamics will help.  This is not about climate
-- or weather -- per se, but it provides some of the fundamental science
necessary to understand both.)

Attribution of Extreme Weather Events in the Context of Climate Change
Committee of Extreme Weather Events and Climate Change Attribution

(Attribution science is relatively new but is making rapid progress.  The
ability to look back and demonstrate causal relationships is going to be
invaluable as we look forward.)


Re: Texas internet connectivity declining due to blackouts

2021-02-22 Thread Rich Kulawiec
On Mon, Feb 22, 2021 at 05:48:06PM +, Mel Beckman wrote:
> Sorry Global Warmists,
Right.  Sure.  Also, the earth is 6,000 years old (and flat), the moon
landings were faked, creationism is real, dinosaurs and humans co-existed,
vaccines cause autism, Elvis is alive, does that line go?  Oh,
right: artificial sweeteners are safe, WMDs were in Iraq, and Anna Nicole
married for love. [shout-out to Levon Helm]

This trash doesn't deserve rebuttal: it deserves ridicule.


Re: Texas internet connectivity declining due to blackouts

2021-02-22 Thread Rich Kulawiec
On Wed, Feb 17, 2021 at 10:34:35AM -0800, Sabri Berisha wrote:
> With apologies to those on the list who still use mutt/pine etc. 

1. "still"?  Competent professionals with security awareness use text-only
email clients as a matter of basic self-defense.  I trust it's obvious
why those of us who are responsible for systems/networks/data need to
protect ourselves in order to protect the resources we run.

2. It's pretty easy to handle attachments gracefully while using mutt.
By default it checks ~/.mailcap, and in that file one can specify external
programs to handle various attachment types, e.g.:

text/html; w3m -dump -T text/html %s | less
application/pdf; /usr/bin/evince %s

Of course this needs to be done carefully, since it subjects the user
to attacks against those applications carried via attachments, but
that's where judicious choices on the part of the user come in --
choices as in "which attachment types, which applications, and
whether or not to exercise this capability on a per-message basis".


Re: Texas internet connectivity declining due to blackouts

2021-02-22 Thread Rich Kulawiec
On Tue, Feb 16, 2021 at 12:23:22PM +, Bret Clark wrote:
> Texas doesn't generally experience this type of extreme cold.

That was then; this is now.

As scientist Jeff Masters put it most of a decade ago:

The atmosphere I grew up with no longer exists.  My new motto
with regards to the weather is, "expect the unprecedented."

In the years since he's said that we've seen a number of unprecedented
events: Sandy, Harvey, California wildfires, last year's midwest derecho,
and so on.  This event in Texas is just another one; there will be more;
they'll get worse.

We should probably get ready for that.


Re: Famous operational issues

2021-02-16 Thread Rich Kulawiec
On Tue, Feb 16, 2021 at 01:37:35PM -0600, John Kristoff wrote:
> Which examples would make up your top three?

Morris worm, November 1988.  Much confusion and eventually the realization
the John Brunner had called it from 13 years out ("The Shockwave Rider", 1975).
But sloppy coding meant it could be defeated with one line of /bin/sh.


Re: Texas internet connectivity declining due to blackouts

2021-02-16 Thread Rich Kulawiec
On Tue, Feb 16, 2021 at 04:17:15AM +, Robert Jacobs wrote:
> How about letting us Texans have more natural gas power plants or even
> let the gas be delivered to the plants we have so they can provide more
> power in an emergency.  Did not help that 20% of our power is now wind
> which of course in an ice storm like we are having is shut off... Lots
> of issues and plenty of politics involved here..

First, wind-generation-is-responsible-for-this is a canard that's
already been debunked elsewhere in this thread.

Second, wind generation works just fine all winter in places like
Quebec and the Alps because they design, build, and operate it to.
Various de-icing solutions have been available for years, and market
competition is continuously making them better and cheaper.

Third, trying to slap a fossil fuel band-aid on a problem whose root
cause is...wait for it...fossil fuels...while certainly a tempting option,
is not a viable long-term solution.


Re: Past policies versus present and future uses

2021-01-26 Thread Rich Kulawiec
On Mon, Jan 25, 2021 at 11:26:51AM -0500, Rob McEwen wrote:
> Is DDoS-Guard without blame? Probably not, but them hosting some occasional

You might wish to scroll back up to the message I sent here on January 21
with the Subject "DDOS-Guard"  and note the list of domains that I provided.

That's not a network with "occasional" issues, that's a network with
pervasive issues.

> By these SAME standards, many other large and famous
> networks should lose most or much of their IPs too!

Yes, that's exactly what should happen.  "Large and famous" operations,
by their very nature, have plenty of money to spend on large, trained,
competent, empowered, 24x7 abuse staff as well as on customer screening
-- and should do that.  Those that don't should not have their problematic
allocations confiscated: they should have *all* their allocations confiscated.

Why?  Well, first because there are no acceptable excuses for running
an operation like that.  NONE.  And second, because when those operations
refuse to pay the costs of keeping abusers out, you know who *does* pay
for that?

We do.


Re: opportunistic email encryption by the MTA (not MUA)

2021-01-16 Thread Rich Kulawiec

While I agree pretty much entirely with everything you've expressed,
there is another force in the world quietly chugging away to make
sure that email privacy remains largely hypothetical...and that is:
cloud computing.

A lot of people have outsourced their mail service to cloud operations,
so even if the mail servers on both ends do everything "right", which
(to your point) might include a refusal to transmit messages unless an
over-the-wire encryption method is agreed on, messages thus sent are
available in plaintext on both sides of the transmission and thus can
be inspected/collected/etc. at will by the operators of the cloud [1].
Or by anyone who's penetrated the cloud.

Note that while use of PGP/similar to encrypt the message itself helps
with this, that doesn't stop traffic analysis on either side of the


[1] Which includes the people working there and working for them,
as well as the people working there and not working for them.

Re: Looking for hosted SMTP service for small ISP

2021-01-14 Thread Rich Kulawiec
On Thu, Jan 14, 2021 at 03:45:06PM -0700, Grant Taylor via NANOG wrote:
> Be mindful that there is and has been a concerted effort to block parts of
> SendGrid.  There's even a relatively new RBL -- which started because of
> SendGrid -- specifically for blocking some of their worse customers. Many
> people in the effort have been advocating for simply blocking SendGrid
> completely.

I recommend the latter course.  I have spam samples from SendGrid
going back to 2010 and continuing to the present, including spam to
never-existed addresses.  They've had 10 years to figure out how not
to spam -- something that should take less than 10 minutes -- and they
still haven't managed it.


Re: Re Parler

2021-01-14 Thread Rich Kulawiec
On Thu, Jan 14, 2021 at 11:01:19AM -0700, Keith Medcalf wrote:
> This result will only come to pass if Parler wins their lawsuit (which is 
> likely)

The first hearing in this case was held today.

Per reporting by Katherine Long of the Seattle Times, during
that hearing Parler's attorney:

- forgot the name of Parler's CEO

- stated that he's unfamiliar with some of the terminology
because he's not on social media

- admitted that he filed a day late because he needed to
update his PACER account

This is the same attorney who filed Parler's complaint -- the one that
names Twitter as a defendant in section 5 while omitting them from the
title page of the complaint.  Here, look for yourself:


Read the first sentence of section 5.  It's on page 3.  Oh heck,
let me save you the trouble:

"Thus, AWS is in violation of Section 1 of the Sherman
Antitrust Act in combination with Defendant Twitter."

I am not an attorney but my general understanding is that if you wish
to file a civil complaint against multiple defendants that you should
actually go through the trouble of naming them all as defendants on the
complaint (and serving them).


Re: DoNotPay Spam?

2021-01-13 Thread Rich Kulawiec
On Wed, Jan 13, 2021 at 05:06:15PM -0500, Robert Webb wrote:
> Anyone else getting spam from DoNotPay everytime they send an email to the
> list?

This is solvable by permanently blocking all traffic from Mailgun in
your MTA.  This should be a good start and may suffice:


Re: Parler

2021-01-10 Thread Rich Kulawiec

Given that people on Parler are currently discussing/planning attacks
against Amazon/Google/Apple/etc.'s facilities and personnel, this seems wise.


Re: WhatsApp's New Policy Has...

2021-01-09 Thread Rich Kulawiec
On Fri, Jan 08, 2021 at 01:31:56PM -0600, Dave Phelps wrote:
> Keybase was purchased by Zoom (
> >From what I've gathered, Zoom is too tight with, owned by, or run by China,
> so I believe there was a similar mass exodus from Keybase for lack of trust.

I've been maintaining a page of relevant links concerning Zoom since
late winter 2020.  It's here:


I need to add a link there concerning the complaint filed in the EDNY,
USA v. Xinjiang Jin (JIN).  As pointed out by File411, there are repeated
references in that complaint to "under 1 minute", as in:

Employee-1 explained that "The current requirement" -- apparently
referring to Company-1's internal restrictions -- "is that domestic
engineers cannot access the data of us clusters" -- indicating
that PRC-based software engineers were not permitted to access user
data stored on U.S.-based servers.  JIN responded "Net Security's
requirement is that [the employer] must have the authority to
directly handle it, and it must be handled within one minute.
For example, including U.S. users, if the issue of June 4th is
being discussed in a meeting, it must be handled within one minute
of [the meeting being reported], otherwise will be [rate] as
security non-compliant."

("June 4th" refers to Tiananmen Square - June 4, 1989.)

It's unclear yet exactly what this means/implies, but my working assumption
for the moment is that everything passing through Zoom is being made
available in real or close-to-real time to the PRC.

Also in the complaint:

JIN wrote in an electronic messages to other individuals who are
Company-1 employees stating that, even if other U.S. social media
and search companies had no business in the PRC, they still terminated
accounts and posted at the request of the "CN zf".  Based on open
source information and my training and experience, the "CN" in "CN zf"
refers to "China" (the PRC) and "zf" is shorthand for zhengfu,
a Chinese word for government.


Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study

2021-01-07 Thread Rich Kulawiec
On Thu, Jan 07, 2021 at 01:27:07AM +1300, Mark Foster wrote:
> I respect this in principle, but hyperbole serves no-one - a smartphone
> only creates a "morass of privacy/security issues" if you let it.

You can't be serious.  Have you paid *any* attention to what's been
going on in this ecosystem for the past N years?  It's not as bad as the
raging dumpsterfire in the IoT, but it's still bad.

Why would I want to give myself security/privacy issues (that I currently
don't have and thus don't need to solve/manage on an ongoing basis) in
exchange for functionality I don't need or want?

> A basic smartphone can be had for less than $100 USD, which would give
> you calling, text messaging and emergency alerts.

It would also give me a much less sturdy device and one that chews up its
battery doing things that I have no use for.  I [sometimes] use my phone
for critical communications in hostile environments, so anything that
doesn't increase the probability that it will work is just baggage.
And as a bonus it would cost me more every time I lose or destroy one.


Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study

2021-01-06 Thread Rich Kulawiec
On Mon, Jan 04, 2021 at 09:08:06PM -0600, Billy Crook wrote:
> Then again how many people would benefit from adding this to online
> streaming, but don't already have cellphones that have emergency alert
> popups that get their attention.  The kind of people who don't have
> smartphones are going to be the ones still watching bunny ears television
> anyway.

I've worked in science and technology for a long time, and I've chosen not
to have a smartphone.  Not that I have to justify this decision to you,
but: (1) I spend a fair amount of my time in environments with poor/no
connectivity (2) I participate in activities that are likely to result
in the destruction or loss of the phone and (3) I use my phone...for
phone calls and the occasional text.  So spending $40 rather than $800,
avoiding the morass of privacy/security issues involved in a smartphone,
and maximizing available battery life seems like the best move.

I know others who've made the same decision for similar reasons.


Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study

2021-01-03 Thread Rich Kulawiec
On Sun, Jan 03, 2021 at 03:26:07AM -0500, Valdis Kl??tnieks wrote:
> Meanwhile, this causes yet another problem - if Hulu has to be able to
> know what alerts should be piped down to my device, this now means that
> every single police and public safety agency has to be able to send the
> alerts to Hulu (and every other streaming company) - and do this securely.

And then there's another problem (that I'm going to bet you've already
thought of, given what you've written here): Hulu and every other
streaming company need to be able to authenticate the alerts from
all those different agencies.  Those agencies also need to secure
their sending infrastructure...and good luck with that.

And then there's another problem, which is that once all those different
agencies have this facility, they're going to (ab)use it as they see fit.
I've noticed that over the last decade or so that weather alerts I've
received are covering increasingly-less-severe events, e.g., we've
slowly gone from "there's a tornado on the ground" to "there's going
to be a thunderstorm".  And at this particular point in history, I can
think of one person who would be using this every five minutes simply
because it's there.


Re: NDAA passed: Internet and Online Streaming Services Emergency Alert Study

2021-01-02 Thread Rich Kulawiec
On Fri, Jan 01, 2021 at 05:07:22PM -0500, Sean Donelan wrote:
> Not later than 180 days after the date of
> enactment of this Act, and after providing public notice and
> opportunity for comment, the Commission shall complete an
> inquiry to examine the feasibility of updating the Emergency
> Alert System to enable or improve alerts to consumers provided
> through the internet, including through streaming services.

I do hope that they consider the certainty that this will be subjected
to abuse the moment it goes live.


Re: Are the days of the showpiece NOC office display gone forever?

2020-12-30 Thread Rich Kulawiec
On Tue, Dec 22, 2020 at 10:41:43PM -0700, Wayne Bouchard wrote:
> And if the last 15 years has shown us anything, it is that when you
> can't get past the auto-attendant and talk to a real human, and if
> that person can't talk to you like a person instead of reading scripts
> at you, your stress levels go way up as does your desire to break
> things. Automation in customer service (or excessive emphasis on
> procedures) is a really nice way of taking a five minute problem and
> turning it into an hour long ordeal.

There are some easy methods for service/support organizations to decrease
the pain that this inflicts on people reporting problems.

For example, one thing that I've taught people to do is to make liberal
use of procmail in order to sort incoming traffic to role accounts.
It requires diligence, but that diligence is repaid many times over by
how it expedites dealing with problems.  A simple example of this is
that when a problem report is received at the RFC 2142 security@ role
address, and it's clueful, well-written, and important, a procmail rule
gets created for the sending address so that all future messages from
that address are prioritized...because it obviously came from someone who
knows what the heck they're doing and did us a favor by telling us that
we have a problem.  Chances are that any future messages from them will
be similarly helpful and that if we respond to those quickly we may be
able to forestall a lot more messages that aren't going to be as clueful.

The opposite thing is done with clueless/misdirected/etc. reports:
they're not discarded, but they go into the low-priority queue.

Everything else goes somewhere in the middle.

Repeated hundreds or thousands of times over many years, this builds a
ruleset that pre-sorts messages rather well.  It's not perfect, it's not
foolproof, but it helps us *and* it helps lower the frustration level of
people sending clueful messages, because it better positions us to read,
act on, and respond to those.  Those people are catching our mistakes,
the least we can do is try to pay attention.

(Hint: a useful way to begin building such a ruleset is to grab all the
addresses from NANOG, dnsops, outages, etc. and pre-load the ruleset
with them...because traffic received at role accounts from participants
in these mailing lists is probably useful.)


Re: "Hacking" these days - purpose?

2020-12-14 Thread Rich Kulawiec
On Mon, Dec 14, 2020 at 09:58:01AM -0500, Tom Beecher wrote:
> Questionable cloud / VPS / hosting companies are great for spammers and
> botnet C, but not so great for DDoS "ion cannons". You still need a large
> volume of geographically diverse endpoints for those to be effective.

To piggyback on this: when launching a DDoS, diversity along multiple
axes is helpful: geography, topology, connectivity, operating system, etc.
Each additional form of diversity slightly raises the bar for defenders.

Also, every compromised device may be a source of useful/saleable data,
or the gateway to more of the same or to more valuable targets or to the
compromise of people.  The IoT is particularly fertile ground for this
because to a very good first approximation, "IoT security" is an oxymoron.


Re: Weather Service faces Internet bandwidth shortage, proposes limiting key data

2020-12-10 Thread Rich Kulawiec
On Thu, Dec 10, 2020 at 09:48:25AM -0500, Jared Mauch wrote:
> I miss weather underground before it became slow as molasses with
> openstreetmap and other things.

As do I, and the demise of took away one of the alternatives.
I spent some time earlier this year unsuccessfully trying to contact
the people behind in the hopes of restarting it as a very
lightweight site that could be read in a text-only web browser *and*
scripted -- since I think it would be handy to be have a weather
site that could easily be scraped for custom uses such as "cron job that
sends me a brief 3-day forecast for locations X Y Z every day at 6 AM".


Re: The Real AI Threat?

2020-12-10 Thread Rich Kulawiec
On Thu, Dec 10, 2020 at 12:34:33AM +, Mel Beckman wrote:
> So don???t be fooled by Siri and Google voice response. There is no
> intellect there, only pattern matching. Which we???ve been doing with
> machines since the Jacquard Loom.

On this particular point: many years ago, some of us at Purdue discussed
this at great length and eventually coined the term "ad-hockery" --
which found its way into the New Hacker's Dictionary.  The gist of
the idea is that it's possible to craft a sufficient number of ad hoc
rules, plug them into a pattern matcher, and present a modestly plausible
appearance of "intelligence"...  when in fact nothing resembling actual
intelligence is involved.  Such systems are brittle and demonstrate
it when presented with input not covered by those ad hoc rules, which
is why they're often made progressively less so by repeated tweaking.
(Also known as "release 3.0" and accompanied by prose touting it as
an innovative upgrade.)

But to borrow Mel's phrasing, even a very large collection of ad hoc
rules that performs its task tolerably well is no more intelligent than
the loom.


Fwd: Weather Service faces Internet bandwidth shortage, proposes limiting key data

2020-12-10 Thread Rich Kulawiec
- Forwarded message from Dave Farber  -

> From: Dave Farber 
> Date: Thu, 10 Dec 2020 15:47:44 +0900
> Subject: [IP] Weather Service faces Internet bandwidth shortage, proposes 
> limiting key data
> Weather Service faces Internet bandwidth shortage, proposes limiting key data
> The National Weather Service is proposing to place limits on accessing its
> life-saving weather data in a bid to fix Internet outages.
> By Jason Samenow and Andrew Freedman


This seems like a problem that this group could solve rather rapidly with 
incremental expense.  It also seems like one that's very much worth solving.


Re: Technology risk without safeguards

2020-11-06 Thread Rich Kulawiec

/Friday afternoon

On Thu, Nov 05, 2020 at 09:05:34AM -0800, William Herrin wrote:
> Following staff home and picking them off with a rifle is so much
> cheaper and carries a better probability of success.

So does following them home and leaving them brand new unopened large
bottles of Woodford Reserve.  I highly recommend this approach for anyone
who has selected me as a target and promise that I will duly report on
its progressive deleterious effects.  For accuracy, repeated trials
over an extended period of time may be necessary but this is an ordeal
I'm selflessly prepared to undertake for the sake of science.


p.s.1: I've worked in high-energy EM environments twice, in two different
contexts.  The safety measures were thorough and rigorous: it would have
been very hard to screw up and even if any of us had, the inverse-square
law would probably have saved us from serious harm.

p.s.2: The large quantities of power conduits, cables, shelving, racks,
HVAC ductwork, etc. that are typical of datacenters constitute a haphazard
but modestly effective EM shield, as measured on an ad hoc basis by anyone
who tries to receive external signals inside them (even when everything
is powered down) will quickly discover. Thus an attempt to pull off a
movie villain-grade underground attack designed to fry a staff member
would likely require that the victim stand still on a selected spot
(on the lowest-level floor) with a minimal amount of metal under it.
I recommend that prospective attackers use the Wile E. Coyote (Sooper
Genius) methodology, draw a large X on that spot, and install a sign
that says "Free Birdseed".  I'm certain this will work.

Re: Virginia voter registration down due to cable cut

2020-10-19 Thread Rich Kulawiec
On Sat, Oct 17, 2020 at 07:44:01PM -0400, Sean Donelan wrote:
> In the USA, absent clear and convincing evidence otherwise, I expect any
> outages will be due to the normal things that cause outages on election day.

One of those things is the chronic underfunding of the systems/personnel
involved.  (This isn't the case everywhere of course but it's the case in
a lot of places.)  This leads to operations that are working -- barely --
and thus not resistant to stress, outages, and mistakes.

We could argue that given the criticality of this particular function
that resources should be more generously allocated, but unfortunately
that's often an unsuccessful argument.


Re: Cogent emails

2020-09-22 Thread Rich Kulawiec
On Tue, Sep 22, 2020 at 07:58:42AM -0500, J. Hellenthal via NANOG wrote:
> geeks@nanog works just fine 

Yes, it works just fine for *that* purpose.   However, *this* has a
different purpose:

Shining a light on ambulance chasers - Noction


Re: Cogent emails

2020-09-22 Thread Rich Kulawiec
On Mon, Sep 21, 2020 at 06:30:24PM -0600, Grant Taylor via NANOG wrote:
> Is this simply being aggregated by a NANOG member / subscriber and thus
> something unofficial?

That's exactly right.  Whether NANOG itself ever wants to do anything
with the results is entirely up to them.


Re: Cogent emails

2020-09-21 Thread Rich Kulawiec
On Mon, Sep 14, 2020 at 12:45:32PM -0400, Dovid Bender wrote:
> Is anyone starting to get the "cogent emails" again?

Reminder: forwarding these sorts of things (with full headers please) to:

will cause them to be compiled into a list.


Re: BGP route hijack by AS10990

2020-08-27 Thread Rich Kulawiec
On Mon, Aug 03, 2020 at 08:57:53AM -0400, Tom Beecher wrote:
> Telia made a mistake. They owned it and will endeavor to do better. What
> more can be asked?

Figure out how that mistake happened -- what factors led to it?  Then make
changes so that it can't happen again, at least not in that particular
way.  (And if those changes are applicable to more than this isolated
case: excellent.  In that case, share them with all of us so that maybe
they'll keep us from repeating the error.)  "Stopping myself from making
the same mistake twice" has probably been the most effective thing
I've ever done.


Re: (updated) COVID-19 fast/small resources page

2020-05-14 Thread Rich Kulawiec
Update on:

On Mon, Mar 23, 2020 at 10:42:32PM -0400, Rich Kulawiec wrote:
> It's here:

I've been updating this every 24-48 hours.  It now includes every
applicable case/test tracker I'm aware of and links to quite a few
articles and papers.  (I may break the latter out into a second page;
there's obviously a lot of activity and it's starting to become difficult
to figure out what's most germane.  I've long since pushed journalism
links to a second page, because there are too many.)  I've gotten helpful
feedback from the NANOG and REN-ISAC folks (thanks!) and have added
some things in response to that.

If there are other relevant and reliable resources that would assist
the community, I'd be happy to add them.  Please ping me off-list.


Re: Abuse Desks

2020-04-29 Thread Rich Kulawiec
On Tue, Apr 28, 2020 at 12:40:12PM -0400, Matt Corallo via NANOG wrote:
> Please don't use this kind of crap to send automated "we received 3 login 
> attempts on our SSH box..wa" emails.
> This is why folks don't have abuse contacts that are responsive to real 
> issues anymore.

[ "you" = rhetorical "you", throughout ]

No, the reason that folks don't have responsive abuse contacts is that
they're some combination of:

- lazy
- cheap [1]
- incompetent
- unprofessional
- actively supporting the abusers

A "we received 3 login attempts on our SSH box" complaint should be read,
investigated, and acted on.  It means that something is going on that
shouldn't, and so for your own sake, as well as for the well-being of
your Internet neighbors, you should find out what that is.

That "for your own sake" clause is often overlooked.  An incoming abuse
complaint is sometimes the canary in the coal mine.  Ignoring it because
it appears to be trivial at first glance is extremely foolish.

The lesson of the 75-cent accounting error is now 34 years old.  This would
be a really good time to learn from it.

You should also consider that -- thanks to the negligence and incompetence of
many abuse desks -- a lot of people simply don't bother reporting incidents
any more.  Thus what presents to you, on the surface, as "we received 3
login attempts on our SSH box" may in fact be one isolated report of
a much larger incident.  It thus requires your immediate attention, if you
want to even pretend to be a responsible, competent professional.

Incidentally, an excellent way to reduce the number of "we received 3
login attempts on our SSH box" complaints is to deal with all of them,
thus decreasing incident occurence, which will of course result in a
corresponding decrease in complaints.  An even better way is to run
your operation in such a way that you detect and deal with as many
such things as possible before anybody needs to file a complaint.
After all, if they can see the traffic arriving on their side, you can
see it leaving on yours.


[1] I note, for example, that Charter -- which is mentioned in the
original message in this thread -- currently has a market capitalization
of 116.63 billion dollars.  Surely they could spare a tiny fraction of
that to appropriately staff a 24x7 multi-lingual abuse desk with senior
engineers and empower/equip them to do what needs to done.  That's
what a professional operation would do.

Re: mail admins?

2020-04-26 Thread Rich Kulawiec
On Thu, Apr 23, 2020 at 10:47:18AM -0700, William Herrin wrote:
> One of the annoyances with both those guys and the swedish folks is
> that they're not sending messages to the return path, they're
> responding to the header from address. Mailman at NANOG never sees it.
> It doesn't pass through their servers.

Yeah, I know: I've seen similar things on the Mailman-powered mailing
lists that I've run.

> Even if mailman saw it, mailman doesn't use VERP. It has to scan the
> message to match the subscriber and that doesn't consistently work.
> The subscriber address forwards to another address, the second address
> bounces and the bounce message doesn't necessarily contain the
> original subscriber address.
> To identify these jokers the ops will probably have to send a unique
> message to each subscriber with crafted headers. That can be folded in
> to a message that would go to the list anyway but the capability isn't
> baked in to mailman.

I get all this, but there are other methods that should help narrow it down.
For example (and I really do mean "example", this might or might not
be useful in this case): one of the things I've done is to (1) grab the
subscriber list (2) reduce it to domains (3) look up the MXs for every
domain -- via a script (4) sort/uniq (5) see if any match the domain
that appears to be the culprit.  (Sometimes works.)  Another approach is
to look up the MX's for the culprit and see if those match any on the
just-generated list.  (Usually doesn't work.)  Still another is to map
the MX list to IP addresses, sort by address, then grab the IP addresses
relevant to the culprit (from A, NS, MX) and see if those are local to
any of the ones on the list.  (Sometimes works.)

All of these are hit-or-miss but I've found that pursuing them usually
results in some insight, if not an answer.  If nothing else it often
allows the "exoneration" of some number of subscriber addresses, which
reduces the scope of the problem and may render it tractable via other
methods.  But to your point, VERP or an equivalent tactic may be the
only way to really diagnose/repair some of these.


Re: mail admins?

2020-04-26 Thread Rich Kulawiec
On Thu, Apr 23, 2020 at 07:56:30PM -0700, Michael Thomas wrote:
> $SHINYNEWSITE has only to entice you to enter your reused password which
> comes out in the clear on the other side of that TLS connection.?? basically
> password phishing. you can whine all you like about how stupid they are, but
> you know what... that is what they average person does. that is reality. js
> exploits do not hold a candle to that problem.

Two equally large problems -- neither of which have anything to do with
encryption in transport -- are backend security and password strength.
In the former case, we've seen an ongoing parade of security breaches
and subsequent dataloss incidents.  That parade is still going on.
In the latter case, despite years of screaming from the rooftops, despite
myriad enforcement attempts in code, despite another parade of incidents,
despite everything, we still have people using "password" as a password.

As a side note, I've found it nearly impossible to get users to
understand that there is a qualitative and quantitative difference
between "password used for brokerage account" and "password used for
little league baseball mailing list".

The minor problem of passwords-over-the-wire pales into insignificance
compared to these (and others -- but that's a long list).


Re: mail admins?

2020-04-23 Thread Rich Kulawiec
[ Bunch of replies to messages in thread bundled here. ]

On Tue, Apr 21, 2020 at 06:28:48PM -0400, Bryan Fields wrote:
> It's a mailman list, so should work.  If not reach out
> to the communications committee.

All mailing lists should support that, regardless of what's running them.
Mailman, thankfully, makes it easy for configuring it by default.

Other topics:

- I've received erroneous bounces from as well.
It should be possible to track down the culprit via Mailman's logs
and the MTA's logs.

- There is zero point in obfuscating email addresses in archives or
anywhere else on the 'net.  None.  There hasn't been any point for
most of twenty years. It's cargo cult "privacy" and that capability
should be excised from Mailman's source code, because its presence
unfortunately encourages people to indulge in a worst practice.

- I also volunteered (in 2018? not sure, need coffee) to help out
with -owner technical issues, but never heard anything back.

- One of the queries that I've sent but not had an answer to is
whether the entire archives are available in "mbox" format.  (Including
the older ones.)  If they are, then it should be pretty easy to fold
the old pre-Mailman archives into the current with-Mailman archives.


Re: 24x7 vs 24x7x365 Re: Constant Abuse Reports / Borderline Spamming from RiskIQ

2020-04-17 Thread Rich Kulawiec
(since it's Friday and we're all stressed)

I can't believe that out of everything I wrote that we're going to discuss
the semantics of this, but then again: yes I can.  I should have known.
I should have known.  I. Should. Have. Known.  *bangs head on desk*
*reaches for scotch*  Alrighty then:

24x7 means every hour of the week, as in "24 by 7".

24x365 means every hour of the year. (modulo those with 366 days
but please let's not go there because this is bad enough)
(oh wait, too late, someone upthread already went there)
(and then leap seconds reared their ugly head, oh good grief)

24x7x365 thus means every hour of 7 years.  YES, I know, I know.  NO.  I will not go there.  Nor will you.  Just stop.
I swear I will turn this car around *right now*.

Yeah, I know it's in common use.  Like any number of other things in
common use (e.g., "going forward" -- really?  like there's another
direction to go?) it's...annoying.

I suspect that someone who just wasn't thinking started this in an
attempt to out-promote people who merely said 24x7 or 24x365, and it
propagated outwards.  If that hypothesis is correct and there is thus
a patient 0 for this epidemic, I very much want to find them and pummel
them with a bag of Oxford commas.


Re: Constant Abuse Reports / Borderline Spamming from RiskIQ

2020-04-17 Thread Rich Kulawiec
On Wed, Apr 15, 2020 at 11:33:58PM -0400, Ross Tajvar wrote:
> Can you give some examples of the things you mention above? I'm not doing
> much in terms of customer filtering and would be interested to hear what
> others consider best practice.

Sure.  These are just examples and are by no means exhaustive.  Also,
some will work better than others depending on who you are, what services
you offer, where you are, etc.  There's no substitute for human judgment
seasoned with experience.

1. Let's start with a timely one.  Whenever there's a national or global
crisis, scammers begin registering domains to exploit it.  For instance:

Thousands of COVID-19 scam and malware sites are being created on a 
daily basis

[I'll omit the long rant about why ICANN is responsible for this and
should be ashamed of what they've not only allowed, but encouraged.]

That story contains a link to a repository where somebody is tracking
these.  I pulled that list a month ago and there were 7500 entries.
Now there are over 25,000.   (Caveat for anybody doing the same: note
carefully the methodology.  There are legitimate domains/subdomains/hosts
in there, although they're rapidly being swamped by the bogus ones.  So don't
just blindly use the data: filter out the 1-2% of legitimate entries.)

So, if it's April 2020, and a customer comes to you and wants to set up
web service for a domain or fifty that have "covid", "corona", "virus",
etc., in their names: they're probably up to something.

2. There are longstanding versions of (1) as well.  Domains with strings
in them like "bulk", "seo", "credit", etc., or domains with variations
on the names of financial institutions, or domains which are typos of
well-known domains, etc., are all suspect.  *That doesn't mean they're
all bogus.*  It just means that a human being should give them closer
scrutiny before the process goes forward.

3. Look at the diversity of their domains.  This sort of is a rehash
of what I said in (2), but: if all their domains are about one or
two topics, yeah, it's probably someone with a business and a hobby
or something like that.  But if they have domains that suggest they're
running 17 different businesses, then look closer.

4. Look at whether they've been, that is, where they were hosted
previously, by checking their DNS history.  If they've hopped through
four different hosts in the last seven months, something is going on.
(Note: a few months ago a bunch of cheap VPS services all simultaneously
ceased operations.  If they were on one of those, then they may have just
been caught up in the mess, so don't count that against them.)

5. Check Spamhaus.

6. Find out how many domains they have.  People doing legitimate things
may have 5 or 17 or something like that.  People who have 5,000 are up
to something.  (Note: I've been doing research in this area for many
years.  I know of zero instances where registrants with thousands of
domains were doing something legitimate.  There may still be a
counterexample out there, but I haven't seen it yet.)

7. MLM (multi-level marketing) is a red flag.  So is Bitcoin

8. A business putatively located in Iowa but with contact email
addresses or is dubious.  Same for other
incongruous information: it might really be okay, or it might
be a hint that they're up to something.

Most of these are just indicators: they're not definitive.  And there
are counterexamples all over the place.  Plus, this list isn't exhaustive:
like I said they're just examples.  That's why I said at the beginning
that there's no substitute for human judgment seasoned with experience.
That takes time and probably more than a few bad experiences.  But it's
worth it, because it's easier to solve problems before you have them.


Re: Constant Abuse Reports / Borderline Spamming from RiskIQ

2020-04-15 Thread Rich Kulawiec
On Mon, Apr 13, 2020 at 12:11:44PM -0700, Matt Corallo via NANOG wrote:
> I don???t really get the point of bothering, then. AWS takes about
> ~forever to respond to SES phishing reports, let alone hosting abuse,
> and other, cheaper, hosts/mailers (OVH etc come up all the time) don???t
> bother at all. Unless you want to automate ???1 report = drop customer???,
> you???re saying that we should all stop hosting anything?

No, you don't have to stop hosting anything/everything.  But there are
all kinds of things that can be done to detect problematic customers
before you sign them up and once they're in place.  None of those
things are panaceas but all of them done in combination (a) reduce
the chances that you'll have a mess to clean up later and (b) enhance
one's reputation as a place NOT to go for dubious activities, which
in turn discourages future miscreants from trying to get in the door.


Re: Constant Abuse Reports / Borderline Spamming from RiskIQ

2020-04-15 Thread Rich Kulawiec
[ Copied to Jonathan @ RiskIQ because I don't believed he's subscribed. ]

On Mon, Apr 13, 2020 at 11:14:11PM +0530, Kushal R. wrote:
>  All abuse reports that we receive are dealt within 48 business
> hours. As far as that tweet is concerned, it???s pending for 16 days
> because they have been blocked from sending us any emails due to the sheer
> amount of emails they started sending and then our live support chats.  >

There's a lot to unpack here, both for you and for RiskIQ.

Let's start with you.

Your home page says that you host over 100,000 web sites.
Your home page says that you have over 10,000 customers.
Your home page says that you have 24x7x365 support.

(Which is wrong, by the way.  It's either 24x7 or 24x365
or maybe 24x7x52 depending on what you're trying to express.
There is no such thing as 24x7x365.  But let's press on:)

Given all that, why don't you have have a 24-hour abuse desk that is
empowered to act immediately on reports?  Do you not understand that --
as Suresh has pointed out -- the lifetime of many abusive activities
is measured in hours and that a 48-hour turnaround is far too slow
to be effective?

Your "about" page says that you're a leading web hosting company.

Alright then: *lead*.  Show us that you're one of the best at this.
Be one of the operations that we can point to and say "this is
how it's supposed to be done".

Because right now you're the opposite of that.

Also: don't use  Use abuse@, per RFC 2142.  There is
zero reason not to go along with the standard.  If you want to alias
it internally fine, but at least get this rudimentary thing right.
*This is why we have standards*.

By the way: did you know that there are multiple COVID-19 scammers
who have set up shop on your service in past few weeks?  I'm very
curious as to why a "leading web hosting company" would allow such
a thing to happen, given that much of it's trivial to prevent.

And now: RiskIQ, it's your turn.

If an operation has exhibited the competence to read and implement
RFC 2142, and thus has a working abuse@ address that goes to some
combination of people and automation that deals with abuse reports,
then that's the one you should be using.  If it has a security@ address
then that's appropriate for those kinds of events.  And while there
are obviously cases where it's appropriate to send to both, it's never
appropriate to send this stuff to role accounts like sales@ or info@
or anything like that.  So: knock it off.

What about operations that haven't done that?  Okay, that's where you
look up their registered contacts.   There is of course no reason for
addresses like when abuse@ will do perfectly fine for
everyone on this planet but if that's what has to be done, then (a) use it
and (b) try to convince them to use abuse@ like competent people who
have read RFC 2142 do.  We'll all be happy if you succeed.

Sending reports repeatedly may make you feel better by venting your
frustration, but it won't solve the problem.  (Now, if new information
arrives about a report you've already filed, then a supplemental message
is appropriate.)  Bombarding people either means you're (a) annoying
people who were already doing something or (b) annoying people who were
never going to do anything anyway.  So knock that off too.

Bugging people in live support chats is probably equally pointless.
So if you're doing that: stop.

(Actually: given my experience over the past few decades "live
support chats" are pretty much pointless, but that's a whole
'nother problem and if I have to deliver *that* rant, I'll
need scotch before noon.  So again, pressing on:)

As to naming-and-shaming on the web or Twitter or wherever: sure, if
you want.  But if you're going to do that then it's probably worth doing
a bit more formally, a la Spamhaus, with a web page that has a
unique URL and supporting evidence and an explanation and so on.
(Do keep in mind that operations like Twitter are transient and thus
not a good choice if you're trying to create a permanent record.)


Re: Constant Abuse Reports / Borderline Spamming from RiskIQ

2020-04-13 Thread Rich Kulawiec
On Mon, Apr 13, 2020 at 07:55:37PM +0530, Kushal R. wrote:
> We understand these reports and deal with them as per our policies and 
> timelines but this constant spamming by them from various channels is not 
> appreciated.

Quoting from:
which is dated 9:15 AM 4/13/2020:

5 #phishing URLs on admin12.find-textbook[.]com were reported
to @Host4Geeks (Walnut, CA) from as far back as 16 days ago,
and they are all STILL active

16 days is unacceptable.  If you can't do better than that -- MUCH
better -- then shut down your entire operation today as it's unworthy of
being any part of the Internet community.


Re: [EXT] Shining a light on ambulance chasers - Noction

2020-04-02 Thread Rich Kulawiec
On Wed, Mar 25, 2020 at 08:13:58PM -0400, Chuck Anderson wrote:
> Let's start a public blacklist, sort of like a RBL reputation block
> list or, but for companies to "never to do business with"
> for spamming.

So it shall be done.  Nominations accepted at:

Of course, someone/something is going to harvest that address
and self-nominate, but that's a feature, not a bug.

Send either references that I can check by going through my archives
of this list, i.e. grep'able strings or send messages themselves,
preferably quoted with full headers.  I'll figure out how to publish it.

Note that this isn't a high priority at the moment, so it may be
a while before I plow through what's received.


Re: free collaborative tools for low BW and losy connections

2020-03-30 Thread Rich Kulawiec
On Mon, Mar 30, 2020 at 06:30:16AM -0500, Joe Greco wrote:
> Actual text traffic has been slowly dying off for years as webforums
> have matured and become a better choice of technology for nontechnical
> end users on high speed Internet connections.

My view is that the move to web forums is a huge downgrade.  Mailing lists
are vastly superior.

Here's a (large) snip from
(which I wrote) with a comment from me shoved in the middle.


1. Mailing lists require no special software: anyone with a sensible mail
client can participate. Thus they allow you to use *your* software with the
user interface of *your* choosing rather than being compelled to learn 687
different web forums with 687 different user interfaces, all of which range
from "merely bad" to "hideously bad".

2. Mailing lists are simple: learn a few basic rules of netiquette and a couple
of Internet-wide conventions, and one's good to go. Web forums are complicated
because all of them are different. In other words, participating in 20
different mailing lists is just about as easy as participating in one; but
participating in 20 different web forums is onerous.

3. They impose minimal security risk.

4. They impose minimal privacy risk.

Points 3 and 4 stand in stark contrast to the security and privacy risks
imposed on users of web forums and "social" media, especially the latter.

5. Mailing lists are bandwidth-friendly -- an increasing concern for people on
mobile devices and thus on expensive data plans. Web forums are

6. Mailing lists interoperate. I can easily forward a message from one list to
another one. Or to a person. I can send a message to multiple lists. I can
forward a message from a person to this list. And so on. Try doing this with
web forum software A on host B with destinations web forum software X and Y on
hosts X1 and Y1. Good luck with that.

7. They're asynchronous: you don't have to interact in real time. You can
download messages when connected to the Internet, then read them and compose
responses when offline.

8. As a result of 7, They work reasonably well even in the presence of multiple
outages and severe congestion. Messages may be delayed, but once everything's
up again, they'll go through. Web-based forums simply don't work.

9. They're push, not pull, so new content just shows up. Web forums require
that you go fishing for it.

10. They scale beautifully.

11. Mailing lists -- when properly run -- are highly resistant to abuse.
Web forums, because of their complexity, are highly vulnerable to software
security issues as well as spam/phishing and other attacks.

[ I'm going to interject a comment here that's not on the web
page I'm quoting myself from.  There are, of course, counter examples
to this.  There is a very busy very well-known mailing list that
is an absolute cesspool of trivially-blockable spam.  Hence
the phrase "when properly run", becase when that's done spam incidents
should be at the 1 per year level or less.  It's not that hard.]

12. They handle threading well. And provided users take a few seconds to edit
properly, they handle quoting well.

13. They're portable: lists can be rehosted (different domain, different host)
rather easily.

14. They can be freely interconverted -- that is, you can move a list hosted by
A using software B on operating system C to host X using software Y on
operating system Z.

15. They can be written to media and read from it. This is a very non-trivial
task with web forums.

16. The computing resources require to support them are minimal -- CPU, memory,
disk, bandwidth, etc.

17. Mailing lists can be uni- or bidirectionally gatewayed to Usenet. (The main
Python language mailing list is an example of this.) This can be highly useful.

18. They're easily archivable in a format that is simple and likely to be
readable long into the future. Mail archives from 10, 20, even 30 or more years
ago are still completely usable. And they take up very little space. (Numerous
tools exist for handling Unix "mbox" format: for example, "grepmail" is a
highly useful basic search tool. Most search engines include parsers for email,
and the task of ingesting mail archives into search engines is very well

19. You can archive them locally...

20. ...which means you can search them locally with the software of *your*
choice. Including when you're offline. And provided you make backups, you'll
always have an archive -- even if the original goes away. Web forums don't
facilitate this. (Those of who've been around for a while have seen a lot of
web-based discussions vanish forever because a host crashed or a domain expired
or a company went under or a company was acquired or someone made a mistake or
there was a security breach or a government confiscated it.)



Re: free collaborative tools for low BW and losy connections

2020-03-29 Thread Rich Kulawiec
On Wed, Mar 25, 2020 at 05:27:41PM +, Nick Hilliard wrote:
> nntp is a non-scalable protocol which broke under its own weight. Threaded
> news-readers are a great way of catching up with large mailing lists if
> you're prepared to put in the effort to create a bidirectional gateway.  But
> that's really a statement that mail readers are usually terrible at handling
> large threads rather than a statement about nntp as a useful media delivery
> protocol.

Some mail readers are terrible at that: mutt isn't.

And one of the nice things about trn (and I believe slrn, although
that's an educated guess, I haven't checked) is that it can save
Usenet news articles in Unix mbox format, which means that you can
read them with mutt as well.  I have trn set up to run via a cron job
that executes a script that grabs the appropriate set of newsgroups,
spam-filters them, saves what's left to a per-newsgroup mbox file that
I can read just like I read this list.

Similarly, rss2email saves RSS feeds in Unix mbox format.  And one of
the *very* nice things about coercing everything into mbox format is
that myriad tools existing for sorting, searching, indexing, etc.


Re: free collaborative tools for low BW and losy connections

2020-03-25 Thread Rich Kulawiec
On Wed, Mar 25, 2020 at 09:59:53AM -0600, Grant Taylor via NANOG wrote:
> Something that might make you groan even more than NNTP is UUCP.  UUCP
> doesn't even have the system-to-system (real time) requirement that NNTP
> has.  It's quite possible to copy UUCP "Bag" files to removable media and
> use sneaker net t transfer things. 

I was remiss not to mention this as well.  *Absolutely* UUCP still has
its use cases, sneakernetting data among them.  It's been a long time
since "Never underestimate the bandwidth of a station wagon full of tapes"
(Dr. Warren Jackson, Director, UTCS) but it still holds true for certain
values of (transport container, storage medium).


Re: free collaborative tools for low BW and losy connections

2020-03-25 Thread Rich Kulawiec

One of the tools that we've had for a very long time but which is
often overlooked is NNTP. It's an excellent way to move information
around under exactly these circumstances: low bandwidth, lossy
connections -- and intermittent connectivity, limited resources, etc.

Nearly any laptop/desktop has enough computing capacity to run an
NNTP server and depending on the quantity of information being moved
around, it's not at all out of the question to do exactly that, so that
every laptop/desktop (and thus every person) has their own copy right
there, thus enabling them to continue using it in the absence of any

Also note that bi- or unidirectional NNTP/SMTP gateways are useful.

It's not fancy, but anybody who demands fancy at a time like this
is an idiot.  It *works*, it gets the basics done, and thanks to
decades of development/experience, it holds up  well under duress.


(updated) COVID-19 fast/small resources page

2020-03-23 Thread Rich Kulawiec
It's here:

There's now a link to Job Snijders' "Internet Operations During
Pandemics" PDF, better coverage of mapping/tracking, links to
every US state's public health agency, links to Canada and Mexico's
CDC-equivalents, etc.  I also fixed the character encoding.
I may move the commentary at the bottom elsewhere: I'm trying to
keep this page very lightweight, which is why there are no graphics,
scripting, or anything else.

Comments/fixes/additions to me, off-list please.


Re: COVID-19 vs. our Networks

2020-03-21 Thread Rich Kulawiec
On Sat, Mar 21, 2020 at 04:42:51AM +0200, Mark Tinka wrote:
> All I'm saying is at the moment, there is no empirical information to
> suggest that Netflix will break what's left of the Internet. Nor is
> there any empirical information suggesting that singling them out will
> help keep it going.

My remarks weren't about Netflix or any other particular service.
(FWIW, I agree with you on both quoted points about the lack of
evidence.  Maybe it'll arrive.  Maybe it won't.)

I was trying to speak, perhaps unsuccessfully, in broader terms about
trying to best position ourselves for challenges that we may not
see coming despite the aggregated millenia of expertise here.  I think
at this particular point in time we need to be ready for anything,
for a very nebulous and possibly quite surprising value of "anything".


Re: COVID-19 vs. our Networks

2020-03-20 Thread Rich Kulawiec
On Fri, Mar 20, 2020 at 10:00:15AM -0500, Mike Hammett wrote:
> Because they're trying to be a responsible Internet citizen instead of just 
> telling everyone else to bugger off. 
> Perhaps if more entities tried to be responsible instead of entitled, the 
> Internet wouldn't be as bad as it is? 


In all the decades that I've been here (on the 'nets), the saddest change
I've seen is the lack of responsibility on the part of people who have,
by virtue of their positions, been given incredible power.  This is the
time for those people to step up and (try to) do the right thing.

None of us know what's going to be needed.  How could we?  We could guess,
and we *are* guessing, but we don't really know because we're sailing
off the edge of the map now.

In those circumstances, the virtue of frugality -- a sensible thing
at any time -- now becomes a necessity.  Every single one of us should
be doing whatever we can to prepare for the unknown, and conserving
resources is one part of that.

"Everything we do before a pandemic will seem alarmist.
Everything we do after will seem inadequate."
--- Michael Leavitt, former HHS Secretary

As I write this, doctors and nurses are working without PPE, risking
their own wellbeing to try to save patients.  We're not being asked
to do anything like that.  Hopefully we still have enough left to rise
to the comparatively minor challenge in front of us.


Re: COVID-19 vs. our Networks

2020-03-18 Thread Rich Kulawiec
On Wed, Mar 18, 2020 at 03:43:37AM -0600, Keith Medcalf wrote:
> So you failed because you did not require the person making the decision
> to take responsibility for their decision.  That is, your organization
> has a severely flawed process wherein the "R" for making the decision is
> not the same person as has the "R" for the repercussions.

The use of "you/your" here and throughout is misplaced and inappropriate.

Also: this not an isolated or unique experience.  It's this way pretty
much everywhere in the US now.  And I can disapprove of it, you can
disapprove of it, we can all disapprove of it, but like I said, until
money is completely removed from the calculation, this is how it will be.
Critiques of process and role and organization and everything else are
interesting, maybe even correct --  but will change nothing.


Re: COVID-19 vs. our Networks

2020-03-17 Thread Rich Kulawiec
On Tue, Mar 17, 2020 at 11:35:59AM -0700, Owen DeLong wrote:
> Anything in the healthcare vertical that is outside of the medical
> providers control/ownership is a result of the medical provider
> buying into that model on some level. STOP DOING THAT.  (How am I
> suddenly reminded of the old adage ???Doctor, doctor, it hurts when I
> do this!??)
> I understand how the allure of lower costs and the frustration of ???every
> vendor does this, we can???t find one who doesn???t??? plays out. However,
> the only way ???every vendor does it??? will continue is if every vendor
> continues to be able to make sales without changing.

Fought this battle, lost this battle.


Because the people with the authority to make purchasing decisions are
not the people who will be on the phone to some vendor's tech support at
3 AM on a Sunday morning, frantically pleading with them to fix a problem
because they really need that piece of equipment to work right now.

Decisions are no longer based on the greater good or on anticipating worst
case scenarios or on maximizing preparedness or anything that we might
hope they're based on.  They're based, coldly and calculatingly, on money.

If you want this to change -- and I sure would like it to change --
then money needs to be entirely removed from that calculation.  That is
a problem whose solution lies outside the scope of NANOG.

Meanwhile, I've updated this:


to include some more resources, including CORD-19, which compiles tens
of thousands of papers on the virus in one place.  I've also included
a link to the relevant Folding@Home project -- which could probably
use as much CPU as you can throw at it.


Re: COVID-19 vs. our Networks

2020-03-17 Thread Rich Kulawiec
On Tue, Mar 17, 2020 at 08:38:28AM -0700, Mike Bolitho wrote:
> Anybody who works in the healthcare vertical will tell you just how
> bad medical devices are to work with from an IT perspective.

Medical devices are appallingly bad to work with from an IT perspective.

They're designed and built to work in idealized environments that don't
exist, they make unduly optimistic assumptions, they completely fail to
account for hostile actors, and whenever possible they are gratuitously
incompatible to ensure vendor lock-in.

That's the good news.   Here's the bad news: in about 2-3 weeks, when
our health care systems are stretched to the breaking point, there will
be a window of opportunity for adversaries to maximize the damage.


Re: COVID-19 vs. our Networks

2020-03-14 Thread Rich Kulawiec
On Sat, Mar 14, 2020 at 05:17:01PM -0400, wrote:
> On March 14, 2020 at 14:49 (Rich Kulawiec) wrote:
>  > 
>  > 2. Find all the phone chargers, laptop chargers, USB sticks, cables,
>  > everything.  If you're not already obsessive about keeping things
>  > charged, get that way.
> You're really expecting power interruptions due to the virus (in the
> US)?

No.  I'm guessing that people may be called upon to pack up their gear
and work in a place they didn't expect to be working.  Or that they may
need to improvise solutions in a hurry with whatever they've got handy.
So having a bag full of chargers and cables and adapters and everything
else just seems like good/easy/cheap preparation.  (I think a lot of us
already have such a bag, it's just that entropy sets in and the way it
was once packed it probably not the way it's packed today.)

I'm not really "expecting" anything other than suboptimal conditions.


Re: COVID-19 vs. our Networks

2020-03-14 Thread Rich Kulawiec
On Sat, Mar 14, 2020 at 11:01:48AM -0700, Mike Bolitho wrote:
> Third, the trouble we had was a third party service having congestion
> issues. 

This is a tiny sample of what's coming.  We're all about to be tested
in a major way, and lots of latent problems are about to become real,
pressing problems.  So:

1. Get some rest.  Stock up (judiciously, don't hoard) on supplies
including medications, fluids, food, etc.

2. Find all the phone chargers, laptop chargers, USB sticks, cables,
everything.  If you're not already obsessive about keeping things
charged, get that way.

3. Make sure your role addresses are up-to-date and working:


and whatever else is appropriate.  Make sure that eyeballs are watching
everything that comes in there and anticipate that some people -- under
stress and anxious -- will send things to the wrong place.

Same for your phone contacts.  And make sure frontline support personnel
have the ability and judgment to rapidly escalate, do not allow urgent
needs to get lost in some ticketing system.

4. Make sure your WHOIS contacts on networks and domains are up-to-date
and working.  Same for your phone contacts.

5. Identify any spare resources that you can lend out.  Identify any
resources that you can guess will be needed.

6. Everyone who can telecommute should be telecommuting right now.
If you need hands on-site, and of course lots of people will, keep
those people separated from others.  Make sure hands-on people know
how to sanitize equipment, tools, etc.

7. Find time in the midst of this for self-care.  You can't help
anybody if you're exhausted.  Take a shower, watch dog videos, do
whatever you need to in order to stay functional.

Here's a resource page that I threw together with a little help
from some epidemiologists.  It's short, plain HTML so it should
load very fast, and of course because it's short it's probably missing
things.  Send suggestions to me off-list.


Re: idiot reponse

2020-02-27 Thread Rich Kulawiec
On Thu, Feb 27, 2020 at 12:25:27AM +, Mark Rousell wrote:
> This (or what it appears to be) is happening on an increasing number of
> mail lists. It's not many but it's there I don't know who is behind it
> or why, but it's an increasing annoyance.

There is a partial fix for this, at least for anyone using Mailman to run
their lists (e.g., nanog):

Set Mailman so that all new subscribers are moderated by default.

Either new subscriber X will one day send real content to the list
or they won't.   If it's the latter, then it is very simple to use
Mailman's interface to simultaneously (a) approve the message for
distribution and (b) clear their moderation flag.  If it's the
former, then the message will only be seen by the list-owners and
won't bother everyone on the list. [1]

This doesn't help with copies that are sent directly to list-members,
however.  The fix for that is for responsible list owners (a) to
be available at the -owner address (per RFC 2142 and decades of best
practices) so that they can field problem reports and (b) to use Mailman
to (a) unsubscribe the errant address and (b) ban it.  I'd also recommend
that they (c) publicly announce such actions with an "administrivia" Subject
line on-list so that list members can take corresponding actions in their
own mail systems.

If nanog-owner isn't responding then that's a serious lapse and
needs to be corrected immediately.  Doing so is a fundamental part
of basic mailing list administration.

I'd also strongly recommend that list-owners have Mailman configured
to notify them of all subscribe/unsubscribe events and/or to require
manual list-owner approval for subscriptions.  Interposing human
beings in the process doesn't solve this problem but it provides
the opportunity to detect and quash it early on.


[1] Note that this is also a partial defense against accounts which
are hijacked and turned into bots.  Given that -- on most mailing lists
and especially on large ones -- the overwhelming majority of subscribers
will *never* send any traffic, nothing is lost by doing this.  But on
the day when an account is hijacked and suddenly starts sending large
amounts of traffic, none of of it will get through to the mailing list.

Re: Tell me about AS19111

2020-02-06 Thread Rich Kulawiec
On Thu, Feb 06, 2020 at 09:08:35AM +0100, Pierfrancesco Caci wrote:
> You would sound much more credible if you'd step down the high horse and
> stop insulting the very same people you're supposed to work with.

You're concerned with policing his tone instead of dealing with the
massive security failure -- on the part of *many* of us -- that this

If I have something horrible going on with a service/server/network/etc.
that I'm responsible for and I don't catch it, then I'm grateful to
anyone who reports it -- because they've caught my mistake, which is
helpful to me and to everyone impacted by it.  I'll worry about my
bruised ego later, it won't be the first time.  Or the last.


Re: FYI - Suspension of Cogent access to ARIN Whois

2020-01-07 Thread Rich Kulawiec
On Tue, Jan 07, 2020 at 04:54:22PM -0600, Mike Hammett wrote:
> That said, if there's a stern warning about Cogent abusing the system,
> maybe their customers finding out is a good thing for the overall
> community. ;-)

And that is what I would suggest: reply to all queries with a notice
that explains what is happening, why it's happening, and provides
contact information for Cogent executives: preferably their *personal*
email addresses and phone numbers.


Re: Iran cuts 95% of Internet traffic

2019-12-29 Thread Rich Kulawiec

And this is why, despite all the disdainful remarks labeling such
things as "antiquated", mailing lists and Usenet newsgroups are vastly
superior to web sites/message boards/ when it comes to facilitating
many-to-many communications between people.  Why?  Well, there are many
reasons, but one of the applicable ones in this use case is that their
queues can be written to media, physically transported in/out, and then
injected either into an internal or external network seamlessly modulo the
time delay.  And because the computing resources required to handle this
are in any laptop or desktop made in the last decade, probably earlier.

If you're trying to get information in/out of a society that is raising
network barriers to realtime communication, then you need methods that
don't rely on a network and aren't realtime.


Re: FCC proposes $10 Million fine for spoofed robocalls

2019-12-19 Thread Rich Kulawiec
[ Re-sent with proper headers.  My apologies for the typo'd previous version. ]

On Thu, Dec 19, 2019 at 11:34:48AM -0800, William Herrin wrote:
> I don't want to start an arms race with the spam callers, I want to
> end it. That means: jump directly to something they can't easily
> defeat.

It is at this point that I am reminded of the wisdom of former FTC
Commissioner Orson Swindle, who was testifying before Congress on
the subject of spam when he said "We need a couple of good hangings here."
It was true in 2003 (which is I believe when he said it) and it's still
true now.  Fines, whatever they are, will be evaded and bargained down,
companies will be dissolved and reconstituted, money will be laundered,
and the problem will persist.


[no subject]

2019-12-19 Thread Rich Kulawiec
Subject: Re: FCC proposes $10 Million fine for spoofed robocalls

On Thu, Dec 19, 2019 at 11:34:48AM -0800, William Herrin wrote:
> I don't want to start an arms race with the spam callers, I want to
> end it. That means: jump directly to something they can't easily
> defeat.

It is at this point that I am reminded of the wisdom of former FTC
Commissioner Orson Swindle, who was testifying before Congress on
the subject of spam when he said "We need a couple of good hangings here."
It was true in 2003 (which is I believe when he said it) and it's still
true now.  Fines, whatever they are, will be evaded and bargained down,
companies will be dissolved and reconstituted, money will be laundered,
and the problem will persist.


We've lost another innovator

2019-11-27 Thread Rich Kulawiec
- Forwarded message from Russ Allbery  -

> From: Russ Allbery 
> Date: Tue, 26 Nov 2019 20:56:23 -0800
> Subject: Brian Kantor has died
> Slashdot reported, via a contributor from the 44Net amateur radio mailing
> list, that Brian Kantor died suddenly in his home last week.
> Brian was the co-inventor of NNTP and the co-author of RFC 977, with Phil
> Lapsley.  I never met him in person, but have had several opportunities to
> chat with him electronically, including as recently as last month (via
> NNTP and netnews newsgroups, of course).  He will be missed.

I never met Brian either, but have "known" him electronically for decades.
His was a voice I always paid attention to, even when I disagreed with
what he had to say.  I'm sorry he's gone, and I'll miss him.


Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019

2019-11-11 Thread Rich Kulawiec
On Fri, Nov 08, 2019 at 01:43:41PM -0500, Mark Stevens wrote:
> Reading Syniverse's cause of trouble (lame excuse) tells me their data
> handling processes are poor and seemingly shady since I do not buy reason
> for the trouble.

Agreed.  So how many other messages have been delayed, lost, forwarded
incorrectly, or...sold to third parties?

(Note that I'm not saying Syniverse did that last one.  What I'm saying
is that an operation like this inevitably affords plenty of opportunities
for employees to engage in a little freelance capitalism of their own
or for third parties to simply help themselves.)


Re: Russian government???s disconnection test

2019-11-05 Thread Rich Kulawiec
On Sat, Nov 02, 2019 at 09:18:36AM -0700, Mike Bolitho wrote:
> The very fact that there are
> AWS/Azure/Google Cloud data centers located around the globe makes anything
> hosted there even more resilient, not less (and for the most part, I still
> prefer on prem DC so I'm not even pushing "To the cloud!").

No, this fact makes everything far less resilient, because it means
"one stop shopping" for attackers.  It also makes the available attacker
budget much greater, since the ROI increases every time more resources
are concentrated in fewer places.


Re: Unable to email anyone from my primary domain name; thanks Google Mail and G Suite.

2019-10-25 Thread Rich Kulawiec
[ Again, just commenting on one point. ]

On Thu, Oct 24, 2019 at 01:21:12PM -0700, Mark Milhollan wrote:
> My experience says that: their system has learned that your system(s)
> continued to send messages that their user (yes you, but they don't know
> that) did not want, i.e., you left it marked as SPAM or deleted it without
> reading the message, or at least not enough was noted as not SPAM *and* read
> (aka displayed, and not for half a second either) so as to influence their
> AI, which makes mistakes and will never correct them if not fed correcting
> info.

[ Note that the proper term is "spam": it's not an acronym and should
never be fully capitalized. ]

It is a worst practice in mail systems engineering to allow input from
putative users into any decision-making process *absent* manual review
of each piece of data by very clueful humans, e.g., curated abuse reports.

Consider: either that input is coming from the person who owns the
account or it's not.

If it's coming from a person, then there is an extremely high probability that
it's coming from someone who doesn't have the slightest idea what they're
doing.  (If users, en masse, knew what they were doing and were even
modestly diligent, then spam and phishing would not be serious problems.
But they are; they flourish because users are careless, lazy, stupid, etc.
They've been proving this daily by the hundreds of millions for decades.)
Systems which do this are built on the laugable assumption that the
aggregate opinions of a hundred million idiots are somehow magically
more valid than the opinions of one.

If it's not coming from a person, then it's coming from the new owner
of the account.  That new owner is hostile by definition, so allowing
input from them is an even worse idea than allowing input from users,
who may only be hostile most of the time.  (Keep in mind that email
accounts are compromised by the billions -- e.g., Yahoo --  and that
systems are likewise so.  And of course anyone who has compromised a
system can avail themselves of any/all email credentials stored on or
traversing that system.  Those who doubt the scale on which both are
happening are invited to peruse the darknet marketplace of their choice
and observe that accounts/systems are for sale in bulk quantities from
a wide variety of sellers.)


Re: Unable to email anyone from my primary domain name; thanks Google Mail and G Suite.

2019-10-24 Thread Rich Kulawiec
[ I'm just going to focus on one point. ]

On Wed, Oct 23, 2019 at 06:18:46PM -0600, Constantine A. Murenin wrote:
> it is revealed that Postmaster Tools cannot tell me anything at all, with
> all tabs and screens being 100% blank, allegedly because I'm not actually a
> mass email sender (I don't send hundreds of emails a day or whatnot), and
> they're too afraid that I'll figure out why my mail doesn't actually go
> through, instead of signing up for G Suite.

There is a persistent mythos -- a worst practice, actually -- among many
operations that obfuscating the reasons why messages are rejected is useful.
This is wrong.

Consider: either the sender is benign (as in this case) or they are not.

If they are benign, then denying the information necessary to understand
and solve the problem helps no one.  It's also counter to the decades of
cooperation and mutual assistance that have built the Internet.

If they're not benign, then either they don't care enough to acquire
this information or they do.  If they don't care, then providing the
information doesn't hurt, because it'll be ignored anyway.  If they do
care, then they WILL get it, whether by conducting research or by
breaching security or by the simpler/cheaper path of paying someone
on the inside off.  (If you're going to tell me that everyone who
works AT Google is working FOR Google, then I'm going to tell you that
you're naive and clueless.  If I were in the large-scale spam-for-hire
business, I'd have already planted my own people there a long time ago.)

Best practice when rejecting mail traffic is to (a) provide at least
a semblance of a reason why and (b) a remediation path that includes
escalation to real live human beings.  All mail systems (except for
the edge case of those which accept everything) make rejection errors
and that must be accounted for in design and operation.


Is anybody else getting spam from

2019-10-22 Thread Rich Kulawiec
I'm guessing -- because spammer Ben Reynolds (
wrote to me about voice/data services -- that it's possible they've
been scraping addresses from here.


Re: Update to BCP-38?

2019-10-09 Thread Rich Kulawiec
On Tue, Oct 08, 2019 at 10:03:16AM -0700, William Herrin wrote:
> Limiting the server banner so it doesn't tell an adversary the exact
> OS-specific binary you're using has a near-zero cost and forces an
> adversary to expend more effort searching for a vulnerability.

Why would they bother performing that search?  Why not use their botnets
to throw every exploit they have at a service and see if anything works?
That's easier and cheaper and faster than being selective.  It also --
if they have happen to have a working exploit -- blows right past
(announced) versions, whether real, fake, or elided.

Brute force is cheap, analysis is expensive.

Case in point: every mail server I have eyeballs on was probed by
attackers trying to exploit the recent exim vulnerability -- no matter
what MTA they're running, no matter that they all announce the MTA and
version, no matter anything.  I doubt I'm alone in observing this.

Even a diligent, capable attacker -- someone who is willing to invest
the time and effort to ascertain what's running which service, down
to the version -- could save themselves some homework by launching an
attack like the one in the first paragraph above, examining the results,
and using those to greatly reduce their search space.  It's easy, it's
cheap, it's fast, it's automated, and it yields no clues as to where
the followup (version-specific) attack is going to come from.


Re: Update to BCP-38?

2019-10-08 Thread Rich Kulawiec
On Tue, Oct 08, 2019 at 01:35:16PM +0100, Mike Meredith via NANOG wrote:
> You've ignored step 1 - identifying critical information that needs
> protecting. It makes sense to protect information that needs protecting and
> don't lose sleep over information that doesn't need protecting. Not many of
> us are planning an invasion of a Nazi-infected Europe any time soon.

We are heading toward a restatement of Kerckhoff's principle/Shannon's maxim,
the latter of which can be paraphrased as "design systems assuming that
your adversary will know as much about them as you do".

Not that I'm advocating publishing all internal design documents, but systems
whose security is predicated on the secrecy of those are brittle and likely
to be badly compromised.  Better to assume that enemies know or can find out
everything and design/build accordingly.


Re: Automated Abuse Reports

2019-10-08 Thread Rich Kulawiec
On Mon, Oct 07, 2019 at 05:28:08PM -0700, Matt Corallo wrote:
> Is it time to have ARIN add a ???abuse contact available only after 
> captcha??? option?

No.  Captchas are a worst practice and should never be used.


Re: "Using Cloud Resources to Dramatically Improve Internet Routing"

2019-10-07 Thread Rich Kulawiec
On Mon, Oct 07, 2019 at 04:42:11PM +0200, Stephane Bortzmeyer wrote:
> Otherwise, an impressive amount of WTF. My favorite: "while
> communication by servers ___on the ground___ might take hundreds of
> milliseconds, in the cloud the same operation may take only one
> millisecond from one machine to another" 

My favorite: "The researchers expect their cloud-based system will be more
secure than the Internet is today [...]"  Apparently they're blissfully
unaware that there is no such thing as "cloud security".


Re: Corporate Identity Theft: Azuki, LLC -- AS13389,

2019-08-13 Thread Rich Kulawiec
On Mon, Aug 12, 2019 at 04:11:00PM -0400, Ross Tajvar wrote:
> Seems like submitting a fraud request to ARIN is more effective than
> writing a novel and sending it to NANOG, and doesn't require the latter...

But if he didn't fully document his assertion(s), then he would be faced
with a plethora of replies decrying the lack of substantiating evidence.
Better to lay the case out in detail so that everyone can see the work
and so that anyone who cares to can check it for themselves.

And -- given Ron's long history of thorough documentation -- there are
some of us who are willing to take his word for it and make operational
decisions based on what he reports, independent of what ARIN decides to
do or not do, or when it decides to do it.


Re: User Unknown (WAS: really amazon?)

2019-08-12 Thread Rich Kulawiec
On Sun, Aug 04, 2019 at 12:12:48AM -0700, Stephen Satchell wrote:
> "The rules" have been around for years, and are codified in the RFCs
> that are widely published and available to all at zero cost.  (That
> wasn't always true, as it wasn't until the DDN Protocol Handbook volumes
> were published in 1985 that the RFCs were available to everyone.  I seem
> to recall there was an FTP site that provided the RFC documents before
> that, but my memory is hazy on that.)

IIRC, the CSnet CIC provided an RFC-by-mail service in the mid to late 1980's.
It allowed anyone to request any RFC by number, e.g., sending it "rfc123" would
result in a response containing that RFC.  I also share your recollection of
an earlier FTP site but a few minutes of checking old documents hasn't turned
up its name and it's fallen out of long-term memory, at least for the moment.

> During my career as a web server admin, mail admin, and network admin, I
> followed "the rules" strictly.   As the main abuse contact during my
> time at a web hosting company, my postmaster@ and abuse@ contact
> addresses were according to Hoyle, and published with the company ASN,
> netblock, and domain registration records.

I've done the same -- imperfectly, to be sure, I've certainly tried.
Half my grump with Amazon here is that they have, for all practical
purposes, unlimited money and unlimited personnel.  They should be the
go-to example for How To Do It Right.  They should be the model (or one
of the models) that we're all trying to emulate, the gold standard that
we can all point to.

But they're not.

The other half of my grump is that they're enormous, and therefore capable
of inflicting enormous damage.  The larger an operation, the more critical
it is that abuse/security/ be fully supported, highly responsive,
empowered to act decisively, etc.

But they're not.

And I have yet to see anyone from Amazon (a) admit this and (b) ask for help
fixing it.


Re: really amazon?

2019-07-31 Thread Rich Kulawiec
On Thu, Aug 01, 2019 at 12:54:07AM +0300, Scott Christopher wrote:
> Rich Kulawiec wrote: 
> > On Wed, Jul 31, 2019 at 11:13:48PM +0300, Scott Christopher wrote:
> > > Because it will get spammed if publicly listed in WHOIS.
> > 
> > Yes.  It will.  Are you telling us that Amazon, with its enormous financial
> > and personnel resources, doesn't have ANYBODY on staff who knows how to
> > properly manage an abuse@ address -- part of which includes dealing
> > with that exact problem?
> They do, but it's just time-consuming and inefficient. You can't spam-filter
> the content of abuse@ obviously.

Actually, yes, you can -- but probably not in the way you're thinking, because
if you do it *that* way you will break [some of the] required functionality.

> But in addition to spam, random (read: non-technical) people will send
> complaints outside of the usual purview of spam, network abuse, DMCA,
> etc. They find some FAQ on the web telling them to determine the PoC on
> and then they start firing crap.

This is not my first day on the job.  I'm aware of what shows up at
role addresses.  However, handling the problems you enumerate here is a
straightforward (albeit occasionally tedious) matter that any operations
engineer above entry-level should be able to handle.  Doubly so because
people like me have done them the favor of writing about it (here and
elsewhere), so they can use our experience without needing to repeat
our numerous mistakes.

> I prefer openness and transparency and the general spirit of WHOIS but, in 
> practice,
> you really do need the limit the PoC information to a trusted group of 
> insiders.

First, there's no such thing as a trusted group of insiders.

Second, even if such a group existed, limiting PoC information to
them is impossible.  Think about it.

Third, besides WHOIS PoC, RFC 2142 (and decades of best practices)
specify abuse@, postmaster@, etc.  My expectation is that anyone
equipped with baseline competence will be fully prepared to handle
traffic to those addresses (as applicable) effectively at whatever
scale their operation requires.


Re: really amazon?

2019-07-31 Thread Rich Kulawiec
On Wed, Jul 31, 2019 at 11:13:48PM +0300, Scott Christopher wrote:
> Because it will get spammed if publicly listed in WHOIS.

Yes.  It will.  Are you telling us that Amazon, with its enormous financial
and personnel resources, doesn't have ANYBODY on staff who knows how to
properly manage an abuse@ address -- part of which includes dealing
with that exact problem?


Re: really amazon?

2019-07-31 Thread Rich Kulawiec

Yes, this is egregious, but on the other hand even when the abuse
reporting mechanisms are working my experience has been that they
emit no response (other than -- maybe -- boilerplate) and take no
action, so it's not terribly surprising.


Re: netstat -s

2019-07-19 Thread Rich Kulawiec
On Wed, Jul 17, 2019 at 05:54:49PM -0700, Randy Bush wrote:
> do folk use `netstat -s` to help diagnose on routers/switches?

I (mostly) use it on firewalls, but yes, it's something I turn
to fairly often (along with other incantations of netstat, plus
lsof and other tools).


Re: Twitter security team?

2019-07-18 Thread Rich Kulawiec
On Thu, Jul 18, 2019 at 12:45:25PM -0600, Ken Gilmour wrote:
> I have evidence and can't contact anyone due to
> the lack of an appropriate form and the fact that the security@ email
> address doesn't work.

Of course I'm not surprised that the ignorant newbies running Twitter
can't manage this: who wouldn't be, given their atrocious track record?
But for everyone else:

[ engage soapbox ]

RFC 2142 was published in 1997, and most of the role addresses it
specifies were in relatively common use prior to that.

Yet -- nearly every day -- this list carries traffic from someone
attempting to help/warn/etc. some allegedly professional operation
that has its fingers firmly lodged in its ears in a desperate attempt
to prevent basic communication and expects people who are already
trying to provide them with free consulting services to jump through
various annoying  hoops in order to do so.

RTFRFC, folks, and implement it.  It's operations 101.  It's something you
should have done in the first hour of the first day, before you turned on
the rest of your stuff.  It's not hard.  And when a day like this comes
for your operation, which it will, it may save you considerable pain,
time, and/or money.

[ soapbox off - for now ;) ]


Re: SHAKEN/STIR Robocall Summit - July 11 2019 at FCC

2019-07-10 Thread Rich Kulawiec
On Mon, Jul 08, 2019 at 06:54:51PM -0600, Keith Medcalf wrote:
> This is because DKIM was a solution to a problem that did not exist.

This is correct.  We have always known the IP address of the connecting
MTA, therefore we have always known the network it resides in, therefore
we have always known who is responsible for what transits that connection.

Worse, this (poorly) attempts to wallpaper over the problems of
compromised systems/accounts.  Do recall that not long ago we learned that
EVERY Yahoo account was compromised.  Anyone who thinks that Microsoft
or Google or Comcast or anyone else are doing any better is naive:
it's not a question of whether they've also suffered mass compromises,
only a question of how many and when they'll publicly admit it.

This isn't surprising.  The real underlying problems here are tough and
expensive, thus it's far easire to do (nearly) meaningless feel-good work,
declare the problems solved, and engage in a round of self-congratulation.
It *appears*, and that's a preliminary assessment on my part, that
SHAKEN/STIR is following this same track.


Re: CloudFlare issues?

2019-06-25 Thread Rich Kulawiec
On Mon, Jun 24, 2019 at 09:39:13PM -0400, Ross Tajvar wrote:
> A technical one - see below from CF's blog post:
> "It is unfortunate that while we tried both e-mail and phone calls to reach
> out to Verizon, at the time of writing this article (over 8 hours after the
> incident), we have not heard back from them, nor are we aware of them
> taking action to resolve the issue."

Which is why an operation the size of Verizon should be able to manage
the trivial task of monitoring its RFC 2142 role addresses 24x7 with a
response time measured in minutes.   And not just Verizon: every large
operation should be doing the same.  There is no excuse for failure
to implement this rudimentary operational practice.

[ And let me add that a very good way to deal with mail sent to those
addresses is to use procmail to pre-sort based on who it's from.  Every
time a message is received from a new source, a new procmail rule should
be added to classify it appropriately.  Over time, this makes it very
easy to identify traffic from clueful people vs. traffic from idiots,
and thus to readily discern what needs to be triaged first. ]


Re: Russian Anal Probing + Malware

2019-06-23 Thread Rich Kulawiec
On Fri, Jun 21, 2019 at 05:13:35PM -0700, Ronald F. Guilmette wrote:
> Is there anybody on this list who keeps firewall logs and who
> DOESN'T have numerous hits recorded therein from one or more
> of the following IP addresses?

Well, I *did*, but having noticed their activities and grown tired of
them, I now just drop their traffic on the floor (and log it).

They are one of several operations that I've noticed who have taken it
upon themselves to poke at open (and closed) ports without bothering
to ask.  Assuming for a moment the most charitable interpretation of
their collective actions -- that they are earnest researching problems
with the intention of helping to solve them -- this is still highly
problematic for two reasons:

1. They didn't ask permission.

2. Whether they realize it or not, they're building a target.  When,
not if, their results database(s) are compromised, they will have
furnished the attackers with a comprehensive target list, painstakingly
gathered at no cost to them and thoughtfully annotated with whatever
metadata has been collected.


Re: Google Post Master experiences

2019-06-03 Thread Rich Kulawiec

This is probably best asked on the mailop list (where some Google
personnel hang out): subscribe via


Re: PSA: change your account logins

2019-05-31 Thread Rich Kulawiec
On Fri, May 31, 2019 at 01:17:19PM +, Richard wrote:
> When I have looked into this type of issue for my unique addressing
> some did trace back to back-end db hacks (e.g., adobe), but I found
> that the most likely culprit was the 3rd-party bulk mailer that
> handled the organization's marketing mail. It could be a non-zeroed
> disk thrown into the trash or an inside job, but it almost always
> traced back to one or two bulk mailing companies. 

FYI, I've been running numerous experiments in this area for many years
using unique non-guessable non-typo'able addresses.  Explaining the
results in full would take many pages, so let me summarize: 3rd party
bulk mailers leak like sieves.  "How?" remains an open question: could be
that they're selling, could be that they have security issues, could be
that insiders are selling on their own, could be any number of things:
it's really not possible to say.  But they are unquestionably leaking.
This is hardly surprising: many of them are spammers-for-hire, many of
them use invasive tracking/spyware, and none of them actually care in
the slightest about privacy or security -- after all, it's not *their*
data, why should they?

Which are some of the many reasons that outsourcing your mailing lists
is a terrible idea, doubly so when it's quite easy to run your own with
Mailman (or equivalent).


Re: Spamming of NANOG list members

2019-05-24 Thread Rich Kulawiec
On Fri, May 24, 2019 at 06:34:25PM +0300, Scott Christopher wrote:
> and
> mangle email addresses in the headers but do nothing about email addresses
> that are quoted / attributed in the body.

There is zero, as in 0.0, point in mangling/obfuscating/etc. email
addresses in forlon and misguided and ultimately futile attempts to keep
spammers from getting their hands on them.  I wrote about this extensively
a few years ago so please let me cite myself in these two messages [1]:

On the other hand, there are a lot of reasons NOT to mangle/obfuscate/etc.
email addresses, including the use of archives by people who come along
later and are trying to track down authors of messages of interest.


[1] As long as those are, there's still more: as one thought experiment,
consider how many of the addresses on this very list can be correctly
deduced by using simple constructions based on real names.  By example,
let's suppose John Smith at is on this list.  We could
readily guess:

and similar variations, and if you compare that to the results of

egrep "^From: " nanog | sort -u

you'll quickly see that a very simple script could come up with roughly
half the addresses on this list immediately.

One of the implications of this, given the widespread adoption of
uniform algorithmic generation of email addresses by much of the
corporate and government and nonprofit  worlds, is that an
attacker who has very little knowledge of the corpus of valid email
addresses at any such entity can make a first-order pass at enumerating
them by combining a script such as the one I posited above with lists
of the 1000 most common first and last names in the appropriate locale.
Of course if the attacker has even a small sample of known-valid
addresses, then it's not necessary to use the myriad variations that
such a script would generate, only the one that appears to be in use
at the target.

Re: Spamming of NANOG list members

2019-05-24 Thread Rich Kulawiec
On Fri, May 24, 2019 at 08:17:31AM -0700, Brian Kantor wrote:
> Anne, the way that such addresses are often harvested is that one of
> the spammers (or his agent) becomes a member of the list and simply
> records the addresses of persons posting to the list.  They then
> get spammed.

I rather suspect that's exactly what's happening here.  I've gotten three,
but a colleague who is subscribed but has never posted has gotten zero,
despite sharing the same email infrastructure and thus precisely the
same configuration.


Re: FCC Hurricane Michael after-action report

2019-05-14 Thread Rich Kulawiec
On Mon, May 13, 2019 at 11:48:02PM -0500, wrote:
> One of my takeaways from that article was that burying fiber underground
> could likely have avoided many/most of these fiber cuts, though I???m
> not familiar enough with the terrain to know how feasible that is.

I suspect that may not be possible in (parts of) Florida.

However, even in places where it's possible, fiber installation is
sometimes miserably executed.  Like my neighborhood.  A couple of
years ago, Verizon decided to finally bring FIOS in.  They put in the
appropriate calls to utility services, who dutifully marked all the
existing power/cable/gas/etc. lines and then their contractors (or
sub-sub-contractors) showed up.

The principle outcome of their efforts quickly became clear, as one
Comcast cable line after another was severed.   Not a handful, not even
dozens: well over a hundred.  They managed to cut mine in three places,
which was truly impressive.  (Thanks for the extended outage, Verizon.)
After this had gone on for a month, Comcast caught on and took the
expedient route of just rolling a truck every morning.  They'd park at
the end of the road and just wait for the service calls that they knew
were coming.  Of course Comcast's lines were not the only victims of
this incompetence and negligence.  Amusingly, sometimes Verizon had to
send its own repair crews for their copper lines.

There's a lot more but let me skip to the end result.  After inflicting
months of outages on everyone, after tearing up lots of lawns, after all
of this, many of the fiber conduits that are allegedly underground: aren't.


Re: EXERCISE: 2019 IAA Planetary Defence Conference - Day 5 Scenario

2019-05-08 Thread Rich Kulawiec
On Wed, May 08, 2019 at 10:11:10AM -0400, Sean Donelan wrote:
> Many exercise designers could use help coming up with useful Internet
> disaster sub-plots.  Bad enough to inject stress into the exercise, but not
> extinction.
> All ISP tech support agents are infected, and become brain eating zombies.

We call that "Tuesday".


p.s. On a more serious note, disaster exercises that include partial
failures of emergency response infrastructure are often quite challenging.
As I write this, the IT infrastructure of Baltimore is down due to
a ransomware attack.  As a consequence, while 911 is functional,
fire department computers are down.  If a significant event requiring
BCFD happened tonight, it would be challenging for them to coordinate
a large-scale response.

Re: Packetstream - how does this not violate just about every provider's ToS?

2019-04-27 Thread Rich Kulawiec
On Fri, Apr 26, 2019 at 06:31:08PM -0700, William Herrin wrote:
> On Fri, Apr 26, 2019 at 6:06 PM John Levine  wrote:
> > I assumed that something this sleazy would be offshore, but their
> > terms of service say they're in Los Angeles.
> >
> They tricked you. [snip]

Also, unless I'm misreading their site, they expect users to download/run
an application program of unknown provenance and function, from an operation
that has gone to great lengths to conceal its location and principals.
What could possibly go wrong?


Re: Comcast storing WiFi passwords in cleartext?

2019-04-26 Thread Rich Kulawiec
On Fri, Apr 26, 2019 at 07:06:40PM +0300, T??ma Gavrichenkov wrote:
> Also, I've seen people who use the same password (sometimes with few easily
> reversible modifications) for virtually all the purposes, from the WiFi
> router up to their e-mail and banking accounts.

This is one of the many risks incurred here: password re-use is
amazingly common (sometimes, as you note, with a few easily reversible
modifications).  Accruing a database full of these means building a
target, and the bigger it is, the more valuable target it will become.
Also, given that this is a public mailing list, lots of people who didn't
know the target existed last week could certainly know it now.

I hear all the arguments in favor of convenience but it's worth noting
that making things convenient for support ops in this fashion has the
unpleasant side effect of making them convenient for attackers.


Re: Comcast storing WiFi passwords in cleartext?

2019-04-24 Thread Rich Kulawiec
On Wed, Apr 24, 2019 at 03:13:33PM +, Benjamin Sisco wrote:
> The bigger concern should be the cleartext portion of the subject.

Yes, and the availability of all this to anyone who hacks Comcast
customer support.


P2P [was: Special Counsel Office report web site]

2019-04-18 Thread Rich Kulawiec
On Thu, Apr 18, 2019 at 12:56:03PM +, Kain, Rebecca (.) wrote:
> I can???t believe p2p isn???t used more, even inside companies.  It does have 
> legit uses

It does, and some of the use cases for it are quite compelling.  However,
there is often deep mistrust associated with it: years of propaganda from
the copyright lobby have fostered the impression that it is inherently
malicious.  That can be very difficult to overcome: it's in the
same class of mythos as "all ICMP traffic is bad", and well, lots of
us have spent lots of time over lots of years trying to get past that one.
Getting P2P accepted looks like a much bigger hill to climb.


Re: Special Counsel Office report web site

2019-04-18 Thread Rich Kulawiec
On Wed, Apr 17, 2019 at 09:02:52PM -0400, Sean Donelan wrote:
> The Special Counsel's report is expected to be posted [...]

Not quite.  A *version* of the report that has been redacted by
the President's hand-picked obedient lackey will be posted.

I suspect that the full report will find its way to us via other means.


Re: Contacts wanted: OVH, DigitalOcean, and Microsoft (Deutschland)

2019-03-27 Thread Rich Kulawiec
On Mon, Mar 18, 2019 at 05:02:38PM -0700, Ronald F. Guilmette wrote:
> I generated the following survey, on the fly, last night,
> based on a simple reverse DNS scan of the evidently relevant addrdess
> ranges:
> As anyone who isn't as blind as a bat can easily see, there's a bit of a
> pattern here.  

I finally found time to check this out.  And I have to ask: how in the
heck did anybody accept this operation as a customer?  Because it's
obvious on inspection -- of the information in that paste -- that they're
abusers.  Let me 'splain.

First, domains in certain TLDs should be considered as -- at best --
dubious until proven otherwise, because those TLDs are well-known as
abuse magnets.  Every domain in this sample falls in that category.
Anyone making mass use of domains in those TLDs is up to something

Second, anyone making mass requests for PTR records for random subdomains 
is up to something abusive.

Third, anyone mass-registering domains whose names are permutations of
each other is up to something abusive.  (I'm not talking about someone
registering a couple of domains that are plausible typos of a primary one
or engaging in defensive registrations across a few TLDs.  Look at the
list, this is obviously quite different from those cases.)

Fourth, anyone mass-registering domains whose names are intended
to be typo'd and/or misread is up to something abusive.

Anybody doing all of the above is not only up to something abusive,
but they're standing on a rooftop screaming it through a bullhorn.

The word "mass" is key throughout not only because it is a highly reliable
indicator of ensuing abuse but because its nature makes detecting this
up front quite easy.  Once I got to it, it took me less than a minute
of scanning that list to determine that there is absolutely no way I
would accept this operation as a customer.  I recognize that not everyone
everyone has my experience in this area, but surely every operation should
have someone equipped with modest experience and and a skeptical eye who
screens new customers, and, at *minimum*, puts them on hold while some
due diligence takes place.  It's much easier (and cheaper) to refuse
service to operations like this than to deal with the fallout that
will inevitably ensue.  It's also much better for the rest of us.

So: how did these people ever get in the door?


Re: Contacts wanted: OVH, DigitalOcean, and Microsoft (Deutschland)

2019-03-20 Thread Rich Kulawiec
On Tue, Mar 19, 2019 at 09:17:23AM -0700, Eric Kuhnke wrote:
> Absolutely unrelated to Ronald's original post, but it's ironic that the
> abuse@ address is itself heavily "abused", by commercial copyright
> enforcement companies which think it's a catch-all address for things which
> are not operationally related to the health of a network [snip]

I've seen this movie and have implemented various mitigation approaches
to it -- none of which constitute a "solution" but all of which help.

1. Block the addresses originating this traffic.  There's no need for
staff/processes on the receiving end to put up with spam.  (If it's UBE,
then it's spam -- by definition.  The content and intention are irrelevant.)

2. Use procmail to redirect it where it needs to go.

3. Set up (non-public) Mailman-operated mailing lists for each role
account and use the moderation queue on those as a throttling tool.
(This works best in conjunction with (2).  Let procmail do some of
the heavy/straightforward lifting and sort the rest out later.)
This also makes it easy to archive everything by subscribing an
address that's an append-only mailbox.

4. Funnel the output of (2) and/or (3) into one of the many ticketing
systems with priority assigned based on the characteristics of the
senders as observed over time.  


Re: Contacts wanted: OVH, DigitalOcean, and Microsoft (Deutschland)

2019-03-19 Thread Rich Kulawiec
On Tue, Mar 19, 2019 at 09:23:34AM -0400, Jeff McAdams wrote:
> We would prefer, but don't require, that you use the web form because that
> is integrated into the workflow of the groups that respond to those
> reports.  

Why isn't abuse@ integrated into the workflow?  It darn well should be,
(a) given that RFC 2142 has been "on the books" for 22 years and
(b) given that methods for handling incoming abuse (or bug, or outage,
or other) reports via email to role accounts are numerous and reliable.

To be clear: if you want to offer a web form in addition to an abuse@
address (or a security@ address, or a postmaster@ address) that's fine.
But web forms are a markedly inferior means of communication and are
clearly not a substitute for well-known/standardized role addresses that
route to the appropriate people/processes.


Re: Should Netflix and Hulu give you emergency alerts?

2019-03-11 Thread Rich Kulawiec
> Just wait until your connected home speakers, smart smoke detector, smart
> refrigerator, smart tv, cell phone, IP streaming box, satellite receiver,
> cable box, home security panel and your Fitbit all go off warning you
> of the cancellation of an Amber alert at 1:30am, because the good folks
> at AlertReady.Ca and Pelmorex think that everything needs to go out at
> highest precedence, because, well,  think of the children!


> And thus, in the first week the system was alive, alarm fatigue set in, the
> government confirmed that it cannot be trusted, and I revoked their
> privilege to use my personal devices for stuff I don't want.

This is why the service(s) should use confirmed opt-in on a per-device basis
and offer sufficient granularity that alerts are only sent to the people who
need/want them on the devices they need/want them on.  To fabricate some

Tornado alert: why, yes, I live in southwestern Kansas so definitely
send those to my home device.

Silver alert: nope.  I live in Queens and don't go out much and don't
drive, so I won't be on the road to see the license plate you're trying
to tell me about.  Never send me these.

Coastal flooding alert: maybe.  I live 130 miles from the coast and
at 550 feet, so any coastal flooding even that would affect me will be
beyond catastrophic.  So don't send that to my home device.  However,
I'm vacationing at the beach right now so send it to my mobile.

This will eliminate some of the alarm fatigue as well as reducing the
transmission requirements.  It's just a rather straightforward exercise
in database management.


Re: Should Netflix and Hulu give you emergency alerts?

2019-03-10 Thread Rich Kulawiec
A side point:

On Sat, Mar 09, 2019 at 02:04:33PM -0500, Sean Donelan wrote:
> Wireless Emergency Alerts (WEA), i.e., mobile phone alerts, are less than 10
> years old. And mostly on the high-end expensive cell phones and the most
> expensive carriers. People on NANOG may use mostly expensive smartphones,
> but not everyone can afford smartphones.

That's an excellent point that's often lost among people who work in
our industry.  Not everyone is so wealthy as to afford the luxury of
a smartphone.  And not everyone can use one.  And not everyone wants one.

The first two items also happen to describe the people who are most
vulnerable to disasters and have the most difficulty getting assistance
recovering from them: the poor and the elderly.


Re: WIndows Updates Fail Via IPv6 - Update!

2019-03-05 Thread Rich Kulawiec
On Mon, Mar 04, 2019 at 08:04:12AM +1100, Mark Andrews wrote:
> ICMP is NOT optional.

I've pointed folks at this for years:

ICMP Packet Filtering v1.2


Re: Announcing: "dumpsterfire", the mailing list for IoT security/privacy issues

2019-01-11 Thread Rich Kulawiec
On Thu, Jan 10, 2019 at 10:57:02AM -0600, J. Hellenthal via NANOG wrote:
> Unfortunately I don???t see this as having very much connectivity where I am 
> at.

It's not the best-connected or most powerful server, however it's been
running a bunch of public/private mailing lists for many years and
for that purpose, it's sufficed nicely.   (That's one of the many major
advantages of mailing lists over web forums: they don't need much in
the way of connectivity, bandwidth, or horsepower.)  Sure, I'd like
to have bigger/better/faster/more, but since I'm paying for this
out of my own pocket...


Re: Announcing: "dumpsterfire", the mailing list for IoT security/privacy issues

2019-01-11 Thread Rich Kulawiec
On Fri, Jan 11, 2019 at 08:23:31AM -0800, Yang Yu wrote:
>   * no HTTPS

HTTPS isn't needed for this application.  I'll probably add it anyway
when I have a chance, but there are other things ahead of it.

>   * archive is returning HTTP 403

That is exactly what you should expect to see when a Mailman archive is empty,
as it is for any new mailing list.  Now it's not.  So now you won't.


  1   2   3   4   5   >