Re: Scanning the Internet for Vulnerabilities

[Matt Palmer]

On Mon, Jun 20, 2022 at 02:18:30AM +, Mel Beckman wrote:
> When researchers, or whoever, claim their scanning an altruistic service,
> I ask them if they would mind someone coming to their home and trying to
> open all the doors and windows every night.

If there were a few hundred people with nefarious intent trying to open your
doors and windows every night, someone doing the same thing with altruistic
intent might not be such a bad thing.

- Matt

Re: AWS and IPv6

[Matt Palmer]

On Sun, Nov 28, 2021 at 02:10:40PM -0800, William Herrin wrote:
> On Sun, Nov 28, 2021 at 1:18 PM Karl Auer  wrote:
> > On Sun, 2021-11-28 at 12:53 -0800, Michael Thomas wrote:
> > > I was reading their howto yesterday and it seems they are only
> > > allocating a /64? Why?
> >
> > That's a /64 *per subnet*...
> >
> > But the size of a VPC's IPv6 CIDR block does seem to be fixed at /56.
> > Would have been nice to see /48 instead.
> 
> To what purpose? You can't alter the VPC routing of any of the IP
> addresses (v4 or v6) assigned to an AWS VPC.

Which is, fundamentally, half the problem with IPv6 in AWS.  I'd have much
preferred that they'd added the ability to do actually-useful IPv6 routing
rather than IPv6-only subnets, which strikes me as more of a toy than
something *actually* useful.

- Matt

Re: Redploying most of 127/8 as unicast public

[Matt Palmer]

On Wed, Nov 17, 2021 at 11:29:49PM +, Jay R. Ashworth wrote:
> This seems like a really bad idea to me; am I really the only one who noticed?
> 
> https://www.ietf.org/id/draft-schoen-intarea-unicast-127-00.html
> 
> That's over a week old and I don't see 3000 comments on it, so maybe it's just
> me.  So many things are just me.

I didn't comment because I was laughing too hard to be able to reliably use
the keyboard.

- Matt

Re: Redploying most of 127/8 as unicast public

[Matt Palmer]

On Wed, Nov 17, 2021 at 01:45:04PM -1000, scott wrote:
> On 11/17/2021 1:29 PM, Jay R. Ashworth wrote:
> > This seems like a really bad idea to me; am I really the only one who 
> > noticed?
> > 
> > https://www.ietf.org/id/draft-schoen-intarea-unicast-127-00.html
>
> https://github.com/schoen/unicast-extensions
> 
> --
> 
> Fixing the odd nooks and crannies still mildly broken in IPv4, by:
> 
>  * Making class-e (240/4), 0/8, 127/8, 224/4 more usable
>  * Adding 419 million new IPs to the world
>  * Fixing zeroth networking
>
>  * Improving interoperability with multiple protocols and tunnelling
>technologies
>  * Supplying tested patches and tools that address these problems
> 
> --
> 
> Some of these are hardcoded in ASICs, I believe.  Change that! ;)

Probably easier to change the ASICs than it'll be to get those "tested
patches" they've apparently written deployed to the millions of Windows XP
boxes still out there.

- Matt

Re: IPv6 woes - RFC

[Matt Palmer]

On Mon, Sep 06, 2021 at 08:38:44AM +0200, Xavier Beaudouin via NANOG wrote:
> Hello,
> 
> 
> > I absolutely HATE testing, developing and supporting IPv4+IPv6, more
> > than doubling my time, adding 3rd stack would actually not increase
> > cost that much, it's the 1=>2 which is fantastically expensive. And
> > costs are transferred to customers.
> 
> Dual stack is doubling dev time ? Ok... this is quite strange...

Nope, entirely normal and expected.  Coding for one case is very
straightforward:

Do thing
Do other thing
Do this as well

Coding for two cases involves identifying each step which needs to be done
differently for the two cases and coding up the things to be done
differently:

if case1:
  Do thing this way
elif case2:
  Do same thing this other way
Do other thing the same way regardless
if case1:
  Do this as well
elif case2:
  Don't do anything

Adding a third case is *usually* not as hard, as you've identified at least
most of the places that need to be done differently for different cases, and
if you're lucky then some of the doings can be the same both ways:

if case1:
  Do thing this way
elif case2:
  Do same thing this other way
elif case3:
  Do same thing in *yet another* way
Do other thing the same way for everyone
if case1 or case3:
  Do this as well
elif case2:
  Don't do anything

It's not just program control flow either; there's also data structures and
data storage to think of.  That can end up being even *more* complicated
than the control flow, especially if you're interoperating with other
systems that need to consume the same data.

For example, in the IPAM software world, imagine you named a field
"ip_address", that contains an IPv4 address, and now you want to add IPv6. 
But that database is also used by some other software (that you might not
even control), so you can't just rename it to "ipv4_address" and change all
the references in your code.  Do you add "ipv6_address" and hope that
everyone figures out that "ip_address" *means* "ipv4_address"?  What about
in the future, when you have the first device that is IPv6-only...  what
goes into that "ip_address" field?

- Matt

Re: CNAME records in place of A records

[Matt Palmer]

On Sun, Nov 08, 2020 at 08:01:12PM -0500, Rob McEwen wrote:
> except - don't forget that the root of a domain (that domain without "www."
> or any other label) - cannot have a CNAME as the "A" record - fwiw...

Yes.  I didn't think that was something that needed to be explained on NANOG,
though.

- Matt

Re: CNAME records in place of A records

[Matt Palmer]

On Fri, Nov 06, 2020 at 05:07:26AM -0500, Dovid Bender wrote:
> Sorry if this is a bit OT. Recently several different vendors (in
> completely different fields) where they white label for us asked us to
> remove A records that we have going to them and replace them with CNAME
> records. Is there anything *going around* in the security aranea  that has
> caused this?

The closest thing to a *security* issue I can think of is IP agility in the
face of DDoS attacks -- most booter-style attacks are dumb as rocks, and
null-routing the target IP and moving all the customers on that IP to
another one is the easiest solution.

However, there are many *other* great reasons to get customers to CNAME onto
their SaaS vendors, including:

* No need to coordinate routine renumbering events;
* IPv6 support;
* CAA record (SSL cert issuance) support; and
* no doubt a bunch of other reasons I've forgotten for the moment.

Basically, if you sign up for a SaaS that uses your own domain and they
*don't* give you a CNAME target to point at, I'd be very cautious, because
they're either *very* new to the game, or they're probably also
operationally deficient in a lot of other areas, too.

- Matt

Re: Clever use of a decommissioned Datacenter

[Matt Palmer]

On Thu, Jul 23, 2020 at 12:20:10PM -0500, Aaron Wendel wrote:
> We decommissioned 200,000 sq ft of DC space this year to turn into a
> marijuana grow.  Very similar power and cooling requirements.

Phat pipes of a whole different kind.

- Matt

Re: Friday Reminder: Web Site Security

[Matt Palmer]

On Fri, May 15, 2020 at 07:24:51PM -0400, Valdis Klētnieks wrote:
> And yes, I know that automated systems can't use passphrases.. so remember to
> check to see if you can use 'force-command=' in the known hosts file so that 
> the
> key can only issue one command.  (yes, this means that if the automation host 
> has
> to do a dozen different things, it needs a dozen keypairs.  Security is 
> always tradeoffs.)

No need for trade-offs here; you can have a `command=` (it's not
`force-command=`) wrapper script that validates the command that was sent
(via `$SSH_ORIGINAL_COMMAND`) and does an `exec` if it's on the "approved"
list.  One key, many commands, any command you don't allow gets blocked.

- Matt

Re: Abuse Desks

[Matt Palmer]

On Wed, Apr 29, 2020 at 12:24:01PM +0530, Mukund Sivaraman wrote:
> On Tue, Apr 28, 2020 at 11:40:16PM -0700, Matt Corallo wrote:
> > Sadly dumb kids are plentiful. If you have to nag an abuse desk every
> > time they sell a server to a kid who’s experimenting with nmap for the
> > first time then we’ll end up exactly where we are - abuse contacts
> > are not a reliable way to get in touch with anyone, and definitely not
> > a reliable way to do so fast or with any reasonably large
> > network. Please don’t clog the otherwise-useful system.
> > 
> > If you have trouble sleeping at night, I’d recommend the
> > “PasswordAuthentication no” option in sshd_config.
> 
> Yes we use that, and PermitRootLogin no and an AllowUsers list.
> 
> I asked in my first email, if with security practices as above and use
> of fail2ban to filter attempts, should we just ignore this problem and
> think that nobody is ultimately reponsible to get rid of this activity?

In theory, no.  In practice, unfortunately, yes.

The typical service provider has so much low-level "noise" going on that if
everyone reported everything to them, they'd semi-literally drown. 
Certainly, there's no possible way they could economically handle all that
abuse reporting -- hiring all the people to examine, determine the veracity
of, and act upon the reports would cost a fortune, because you better
believe there's no One True Format for automated abuse notifications, nor
will there ever likely be one, so it's all humans, all the time.

Now, you could argue that they should clean up their network so they don't
have that volume of abuse reports coming in -- and you'd be right, in
theory.  But there's a *lot* of low-level stuff that it isn't practical to
stop, in and of itself.

Consider your own reports.  What is it, exactly, that you expect a provider
to do with your report of a few failed SSH login attempts to stop the
activity?  Say it's a residential ISP, or an IaaS provider.  They have only
a few very large hammers at their disposal -- they can (maybe) filter the
destination port, filter your destination IP, or disconnect the customer. 
Any of those will very possibly result in a support call, or lost customer. 
That's a very large cost you're expecting them to pay for something which
has, let's face it, cost you practically nothing.

Forcing disconnection for a port scan is also, by the way, a *great* way to
create an absolute gold-plated A+ denial-of-service: send in a
plausible-looking report of shenanigans to the ISP of someone you don't
like, and *boom* their Internet connection's cut off.  WINNAH!

So what are you left with, action-wise?  An ISP could keep a tally of abuse
reports by customer, and take action on whoever's at the top of the pile,
but that would then require a large and expensive army of humans to receive,
check, classify, and record all incoming abuse reports.  Do *you* want to
pay $1,000/month for your home Internet connection to cover the cost of all
those extra ISP staff?  Because, as I said before, there's no One True
Format for reporting abuse, and there never will be.

Not that it would work, anyway -- any sort of "threshold" system for abuse
ends up being gamed, anyway.  You only need to look at how Twitter users
with an axe to grind gang up to send in malicious reports about some other
Twitter they don't like, which trips Twitter's "lots of reports => autoban"
logic, to see how that would end if you started applying it to Internet
abuse reporting.

Finally, because nobody is ever convinced by rhetoric, here's an appeal to
your self-interest: "crying wolf" is never a good idea.  In the event that
you *do* have a real problem that needs to be dealt with some time in the
future, do you want to have your e-mail address, IP address, and whatever
else associated with a thousand previous GWF ("goober with firewall")
reports?  Any abuse desk who has seen your hundreds of previous unactionable
reports will almost certainly round-file that important one, and then you're
*really* up the creek sans paddle.  Far better to keep your powder dry and
be ready for when you actually need assistance from whoever you're
contacting.

- Matt

Re: mail admins?

[Matt Palmer]

On Sun, Apr 26, 2020 at 05:10:56PM -0700, Michael Thomas wrote:
> 
> On 4/26/20 5:07 PM, Matt Palmer wrote:
> > On Sun, Apr 26, 2020 at 07:59:24AM -0700, Michael Thomas wrote:
> > > On 4/26/20 7:32 AM, Rich Kulawiec wrote:
> > > > On Thu, Apr 23, 2020 at 07:56:30PM -0700, Michael Thomas wrote:
> > > > > $SHINYNEWSITE has only to entice you to enter your reused password 
> > > > > which
> > > > > comes out in the clear on the other side of that TLS connection.?? 
> > > > > basically
> > > > > password phishing. you can whine all you like about how stupid they 
> > > > > are, but
> > > > > you know what... that is what they average person does. that is 
> > > > > reality. js
> > > > > exploits do not hold a candle to that problem.
> > > > Two equally large problems -- neither of which have anything to do with
> > > > encryption in transport -- are backend security and password strength.
> > > > In the former case, we've seen an ongoing parade of security breaches
> > > > and subsequent dataloss incidents.  That parade is still going on.
> > > > In the latter case, despite years of screaming from the rooftops, 
> > > > despite
> > > > myriad enforcement attempts in code, despite another parade of 
> > > > incidents,
> > > > despite everything, we still have people using "password" as a password.
> > > > 
> > > > As a side note, I've found it nearly impossible to get users to
> > > > understand that there is a qualitative and quantitative difference
> > > > between "password used for brokerage account" and "password used for
> > > > little league baseball mailing list".
> > > > 
> > > > The minor problem of passwords-over-the-wire pales into insignificance
> > > > compared to these (and others -- but that's a long list).
> > > Um, those are exactly the consequences of passwords over the wire. If you
> > > didn't send "password" over the wire, nobody could guess that's your
> > > password on your banking site.
> > I guess that's why best practices for authentication encourage the adoption
> > of HTTP Digest authentication.  No password over the wire == no problems!
> 
> Which exactly zero deployment. And you need to store the plain-text password
> on the server side. What could possibly go wrong?

But you said that *passwords on the wire* were the biggest problem.  Digest
auth solves that.  Also, you don't have to store the plain-text password.

- Matt

Re: mail admins?

[Matt Palmer]

On Sun, Apr 26, 2020 at 07:59:24AM -0700, Michael Thomas wrote:
> 
> On 4/26/20 7:32 AM, Rich Kulawiec wrote:
> > On Thu, Apr 23, 2020 at 07:56:30PM -0700, Michael Thomas wrote:
> > > $SHINYNEWSITE has only to entice you to enter your reused password which
> > > comes out in the clear on the other side of that TLS connection.?? 
> > > basically
> > > password phishing. you can whine all you like about how stupid they are, 
> > > but
> > > you know what... that is what they average person does. that is reality. 
> > > js
> > > exploits do not hold a candle to that problem.
> > Two equally large problems -- neither of which have anything to do with
> > encryption in transport -- are backend security and password strength.
> > In the former case, we've seen an ongoing parade of security breaches
> > and subsequent dataloss incidents.  That parade is still going on.
> > In the latter case, despite years of screaming from the rooftops, despite
> > myriad enforcement attempts in code, despite another parade of incidents,
> > despite everything, we still have people using "password" as a password.
> > 
> > As a side note, I've found it nearly impossible to get users to
> > understand that there is a qualitative and quantitative difference
> > between "password used for brokerage account" and "password used for
> > little league baseball mailing list".
> > 
> > The minor problem of passwords-over-the-wire pales into insignificance
> > compared to these (and others -- but that's a long list).
> 
> Um, those are exactly the consequences of passwords over the wire. If you
> didn't send "password" over the wire, nobody could guess that's your
> password on your banking site.

I guess that's why best practices for authentication encourage the adoption
of HTTP Digest authentication.  No password over the wire == no problems!

- Matt

Re: mail admins?

[Matt Palmer]

On Thu, Apr 23, 2020 at 07:47:58PM -0700, Michael Thomas wrote:
> On 4/23/20 7:35 PM, Matt Palmer wrote:
> > While I do think webauthn is a neat idea, and solves at least one very real
> > problem (credential theft via phishing), you do an absolutely terrible job
> > of making that case.
> 
> see RFC 4876, it is not about phishing. not even a little bit. Never has
> been.

Whilst I do *absolutely* agree with you that "A Configuration Profile Schema
for Lightweight Directory Access Protocol (LDAP)-Based Agents" is "not about
phishing, not even a little bit", I'm not entirely sure how it advances your
argument.

- Matt

Re: mail admins?

[Matt Palmer]

On Thu, Apr 23, 2020 at 06:31:04PM -0700, Michael Thomas wrote:
> Passwords over the wire are the *key* problem of computer security. Nothing
> else even comes close.

Hmm, a bold claim, but I'm confident the author will have strong support for
their position.

> One only needs to look at the LinkedIn salting problem

That was a stored password problem, not a passwords-over-the-wire problem,
but OK.  I'm sure we'll be back on track shortly.

> to know how trivial it is to exploit password reuse.

Not sure how exploiting password reuse causes problems with passwords over
the wire.  Halfway through the paragraph now, still haven't seen anything
talking about passwords over the wire.  No doubt the next sentence will
address the claim in detail, though.

> They are a big company and they still absolutely failed.

Starting to think that maybe there isn't going to be the solid justification
for the topic sentence that I'd originally assumed.

> There are a trillion smaller sites who are just as vulnerable, and all it
> takes is one.

A trillion smaller sites that are just as vulnerable... to passwords over the
wire?

Wait, this is the end of the paragraph.  How odd, not a single statement in
support of the assertion.  Perhaps it's not, in fact, true, then, that
passwords over the wire are the *key* problem of computer security.

While I do think webauthn is a neat idea, and solves at least one very real
problem (credential theft via phishing), you do an absolutely terrible job
of making that case.

- Matt

Re: mail admins?

[Matt Palmer]

On Thu, Apr 23, 2020 at 09:10:37AM -0700, Michael Thomas wrote:
> javascript is a hell of a lot safer than downloading native apps on your
> phone, for example.

Because those are, of course, the *only* two possible options for accessing
information.

- Matt

Re: mail admins?

[Matt Palmer]

On Thu, Apr 23, 2020 at 04:30:28PM -0700, Michael Thomas wrote:
> Ironically it seems that the way to disable javascript is to install a
> browser extension.

Nope.  chrome://settings/content/javascript for Chromium, about:config ->
javascript.enabled in Firefox.

- Matt

Re: Constant Abuse Reports / Borderline Spamming from RiskIQ

[Matt Palmer]

[Hideously mangled quoting fixed]

On Tue, Apr 14, 2020 at 02:51:55PM +0530, Kushal R. wrote:
> Matt Palmer wrote:
> > On Mon, Apr 13, 2020 at 11:14:11PM +0530, Kushal R. wrote:
> > > All abuse reports that we receive are dealt within 48 business hours.
> >
> > At eight business hours per calendar day, and five business days per
> > (typical) calendar week, 48 business hours is...  a week and a bit,
> > calendar wise.
>
> We are a 24x7 operation.  

Then why not just say "withing 48 hours", rather than the weaselish "48
business hours"?  Makes it seem like you're trying to clever-word yourself
an alibi.

- Matt

Re: Constant Abuse Reports / Borderline Spamming from RiskIQ

[Matt Palmer]

On Mon, Apr 13, 2020 at 11:14:11PM +0530, Kushal R. wrote:
>  All abuse reports that we receive are dealt within 48 business hours.

At eight business hours per calendar day, and five business days per
(typical) calendar week, 48 business hours is... a week and a bit, calendar
wise.

- Matt

Re: RIPE our of IPv4

[Matt Palmer]

On Tue, Nov 26, 2019 at 05:26:44PM -0500, b...@theworld.com wrote:
> If the commitment really was to spread IPv6 far and wide IPv6 blocks
> would be handed out for free, one per qualified customer (e.g., if you
> have an IPv4 allocation you get one IPv6 block free), or perhaps some
> trivial administrative fee like $10 per year.

It has been some time since I had to deal with RIRs directly, but my
understanding was that if you had an IPv4 allocation, you got a reasonably
sized chunk of IPv6 alongside for free.  Not even an extra $10/year.  FREE!

Looking at ARIN's fee schedule
(https://www.arin.net/resources/fees/fee_schedule/), it does seem like that
is still the case:

> For organizations holding both ARIN-issued IPv4 and IPv6 allocations, the
> fee is based on the larger of the two service categories.

So you only need to pay extra for your IPv6 numbers if you've got a lot more
of them than you've got IPv4.  The only situation in which I could imagine
that happening is if you were a (very) late-start eyeball network that had a
tiny IPv4 allocation (and a *lot* of CGNAT), but were planning on handing
out IPv6 /48s to every customer.

- Matt

Re: IPv4 and Auctions

[Matt Palmer]

On Thu, Oct 24, 2019 at 09:30:20AM -0400, Matt Hoppes wrote:
> On 10/24/19 9:25 AM, Chris Adams wrote:
> > Once upon a time, Matt Hoppes  said:
> > > You don’t suddenly just not need a/8 or suddenly not need a/21.
> > 
> > You don't "suddenly just" do lots of things, because things change over
> > time; no need to attribute malicious intent.  For example, a former
> > employer had a /17, a /18, and a /19 at one point (and even that was
> > kind of tight).  The business has changed though, and now they have a
> > /23 and two /24s, and sold the rest.
> 
> I get that -- but if you don't need a /18, don't the ARIN rules require they
> be returned?

Well, there's need, and then there's "need".  If you've sprayed your
organisation's IP addresses all across your /18, then you "need" that whole
/18, and you're going to have to undertake a whole renumbering fiesta to
change that, which is going to be costly both in direct labour costs, as
well as disruption to services.

Since there's no requirement for all the addresses to be publicly reachable,
there's no way for anyone outside of your organisation to tell that only one
/24 in each /21 is actually being used, and even then each /24 only has 4-5
hosts in it.  Thus, nobody at ARIN (or anywhere else) is going to come
beating down your door to demand you give back that big block you're only
sparsely using.

Now, imagine that suddenly you can get Real Money for the unused addresses. 
It might, at some point, turn out to make financial sense to spend the money
to renumber into a smaller block and get phat sacks of mad cheddar for the
addresses you can free up.  If you want to take a charitable look at it, the
money that organisations get for "selling" their unused space can be seen
rather as compensation for undertaking the work necessary to free up that
space.  The invisible hand of the market at work!

You can also, of course, see it as pointless busy-work that would have been
better spent just rolling out IPv6 and being done with it.  Your mileage, as
the sticker says, may vary.

- Matt

Re: Automated Abuse Reports

[Matt Palmer]

On Mon, Oct 07, 2019 at 05:28:08PM -0700, Matt Corallo wrote:
> Because people seem to include “you tried three to log
> in three times and got the password wrong” in their definition of abuse,
> I’ve had to provide bogus abuse contacts (and include actual abuse
> comments in the comments section).  I’ve tried reaching out to other
> operators suggesting that they cut it out, as they are the reason many
> operators do not provide functional abuse contacts, usually with a
> response of “so stop attacking my serverz!!!1one”.

Fight fire with fire, and report their abuse of your abuse contact as abuse
to their (or their upstream's) abuse contact?  More seriously, though, if
someone insists on reporting (automatically or otherwise) seriously bogus
"abuse" a couple of times, I don't think it's unreasonable to block / filter
that specific reporter from further calls on your attention.

What would also be reasonable, I think, is to require automatically
generated abuse reports to themselves be automatically processable.  That
requires some way of *detecting* automatic generation, of course, but it
might encourage a few people to adopt machine-readable abuse reports.

> Is it time to have ARIN add a “abuse contact available only after captcha”
> option?

I'd prefer if it wasn't, because it seems somewhat of a "baby/bathwater"
solution, just to deal with a few GWFs (GsWF?).

> On the flip side, I run a Tor exit node (as well as bridge nodes, which
> appear to be used exclusively by Chinese, Russian, and Iranian IPs,
> indicating Tor is, contrary to popular belief, used in large part for its
> intended purpose).

[Rearranging this paragraph because it is a side-bar from the main issue]

I suspect that if you checked the stats on your guard nodes (if you have
any) the stats would be somewhat different.  Bridges, by their nature, are
less likely to be used by miscreants because they're harder to get (have to
ask BridgeDB) and nobody gets all of them.  If you're just out to anaonymise
your source IP and you're in a location that doesn't imprison and torture
you for reading the news, it's a lot easier to just use a regular
guard/middle/exit circuit.

I'm not calling this out because I'm a Tor-hater; on the contrary, I too run
Tor nodes (bridges and relays; no exit nodes).  It's just that it's somewhat
misleading to suggest that Tor isn't the tool of choice for people who'd
prefer to cause mischief without anyone knowing what they're up to.  I
support Tor because the benefits outweigh the unpleasantness it
(necessarily) permits.

- (Another) Matt

Re: IPv6 Pain Experiment

[Matt Palmer]

On Sat, Oct 05, 2019 at 04:36:50PM -0400, b...@theworld.com wrote:
> 
> On October 4, 2019 at 15:26 o...@delong.com (Owen DeLong) wrote:
>  > 
>  > OK… Let’s talk about how?
>  > 
>  > How would you have made it possible for a host that only understands 
> 32-bit addresses to exchange traffic with a host that only has a 128-bit 
> address?
> 
> A bit in the header or similar (version field) indicating extending
> addressing (what we call IPv6, or similar) is in use for this packet.

How does that allow the host that only understands 32-bit addresses to
exchange traffic with a host which sets this header bit?

- Matt

Re: IPv6 Pain Experiment

[Matt Palmer]

On Fri, Oct 04, 2019 at 11:48:33PM +, Michel Py wrote:
> > Owen DeLong wrote :
> > How would you have made it possible for a host that only understands 32-bit 
> > addresses to exchange traffic with a host that only has a 128-bit address?
> 
> With some kind of NAT mechanism, naturally.

That word "naturally" is doing an *awful* lot of work there.

> > How would you have made a 128-bit address more human-readable? Does it 
> > really matter?
> 
> I have found it difficult to talk hex with people from other countries.
> 
> Try to say FACEB00C to someone who does not speak your langage.
> Foxtrox Alpha Charlie Echo Bravo Zero Zero Charlie does not go through either.
> 250.206.176.192 works all the time. Everyone gets the numbers.

That word "Everyone" is doing an *awful* lot of work there.

All this hand-waving is creating quite a breeze.  Think I'll put on a
sweater.

- Matt

Re: IPv6 Pain Experiment

[Matt Palmer]

On Thu, Oct 03, 2019 at 03:20:50PM +, Naslund, Steve wrote:
> Can you imagine keeping those v6 addresses in your head the same way?

I don't have to imagine, I do it on a daily basis.  Doesn't seem to cause me
any grief.

In my experience, IPv4 addresses which need to be used directly on a regular
basis tend to follow certain patterns or rules of thumb ("default gateway is
on the (first|last) address in the range", for instance), and the same thing
tends to happen with similarly situated IPv6 addresses.

- Matt

Re: This DNS over HTTP thing

[Matt Palmer]

On Wed, Oct 02, 2019 at 05:45:57AM -0400, Valdis Klētnieks wrote:
> On Wed, 02 Oct 2019 01:55:13 -0600, "Keith Medcalf" said:
> > It is a common fallacy that TLS connections are authenticated.  The vast
> > majority of them are not authenticated in any meaningful fashion and all 
> > that
> > can be said about TLS is that it provides an encrypted connection between 
> > the
> > two communicating applications.  This is perhaps why it is call *transport*
> > layer security ...
> 
> Another major disconnect is that TLS validates the hostname that the browser
> decided to connect to, not the host you thought you were connecting to..

Sadly, the W3C is stonewalling on the WebMindReading API.

- Matt

Re: AWS issues with 172.0.0.0/12

[Matt Palmer]

On Tue, Oct 01, 2019 at 04:50:33AM -0400, Jim Popovitch via NANOG wrote:
> On 10/1/2019 4:09 AM, Christopher Morrow wrote:
> > possible that this is various AWS customers making iptables/firewall 
> > mistakes?
> >"block that pesky rfc1918 172/12 space!!"
> 
> AWS also uses some 172/12 space on their internal network (e.g. the network
> that sits between EC2 instances and the AWS external firewalls)

Does AWS use 172.0.0.0/12 internally, or 172.16.0.0/12?  They're different
things, after all.

- Matt

Re: Should ISP block child pornography?

[Matt Palmer]

On Sat, Dec 08, 2018 at 06:26:21PM -0800, Owen DeLong wrote:
> Which is it…
> 
> It’s being reported on NPR as “Australia required Apple and others to
> remove encryption protections from their devices.”
> 
> That’s a massively different (and arguably even worse) outcome than
> “Australia is requiring Apple and others to provide decryption technology
> to law enforcement.”

Part of the problem is... nobody really knows.  There's very little
meaningful oversight or judicial review of the whole system, and the law is
*very* badly written, vague and without any clear guidance as to what is
*actually* required.  It doesn't even define things like "systemic
weakness", which is the standard by which a required change is judged when
determining whether it is within the scope of the law: anything which
doesn't introduce a "systemic weakness" is a-OK.

I'd say lawyers are going to make a fortune out of arguing this, except
as I said, there's very little judicial oversight.  Someone who is asked to
do something which they think introduces a systemic weakness is basically
SOL if the Attorney General and Communications Minister disagree.

- Matt

Re: Ticketmaster?

[Matt Palmer]

On Sun, Dec 03, 2017 at 07:34:29PM -0800, Doug Barton wrote:
> On 12/02/2017 02:39 PM, Ryan Gard wrote:
> > *Oh, you must be sharing your IP with everyone else in your area*
> 
> CGNAT by any chance?

... and yet:

$ dig www.ticketmaster.com 

; <<>> DiG 9.10.3-P4-Debian <<>> www.ticketmaster.com 
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28358
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.ticketmaster.com.  IN  

;; AUTHORITY SECTION:
ticketmaster.com.   2560IN  SOA a1-157.akam.net. 
tmhostmaster.ticketmaster.com. 2991260818 600 600 1048576 2560

Re: Russian diplomats lingering near fiber optic cables

[Matt Palmer]

On Thu, Jun 01, 2017 at 02:02:46PM -0400, Sean Donelan wrote:
> There must be a perfectly logical explanation  Yes, people in the
> industry know where the choke points are. But the choke points aren't always
> the most obvious places. Its kinda a weird for diplomats to show up there.

Maybe they're not *actually* Russian diplomats, but instead undercover
backhoes using Russian diplomatic credentials.

- Matt

Re: Russian diplomats lingering near fiber optic cables

[Matt Palmer]

On Thu, Jun 01, 2017 at 12:20:54PM -0700, Eric Kuhnke wrote:
> That said, a pretty quick way to get on some homeland security watch lists
> would be to hang around a cable landing station beach location with a big
> DSLR camera, and appear uninterested in the beach...

I think regardless of what you appear to be interested in, hanging around a
beach with a big DSLR is likely to get you on one list or another.

- Matt

Re: Please run windows update now

[Matt Palmer]

On Tue, May 16, 2017 at 08:12:41PM -0400, valdis.kletni...@vt.edu wrote:
> On Tue, 16 May 2017 16:41:36 -0600, "Keith Medcalf" said:
> > Of course Microsoft knew, since they wrote in the backdoor in the first
> > place.  That is why when informed by their employers that the backdoor was
> > going to be made public, they could undo the changes they had introduced so
> > rapidly.
> 
> Do you have any actual evidence or citations that in fact, this was an
> intentionally inserted backdoor?

You'll have to speak up, he can't hear you over the rustling of the tin
foil.

- Matt

Re: did facebook just DoS me?

[Matt Palmer]

On Tue, Apr 04, 2017 at 09:47:23PM -0300, Kurt Kraut wrote:
> But one of my collegues quickly realized the incoming MAC ADDRESS was the
> actual Facebook router we have a peering at a internet exchange. So indeed
> the traffic came from their network.

If you've got a bilateral peering session with Facebook, presumably you have
some sort of technical contact there that you can reach out to and ask
"WTF?".  That would seem to be a good first step.

- Matt

Re: Serious Cloudflare bug exposed a potpourri of secret customer data

[Matt Palmer]

On Sat, Feb 25, 2017 at 07:21:48AM +, Mike Goodwin wrote:
> Useful information on potentially compromised sites due to this:
> 
> https://github.com/pirate/sites-using-cloudflare

"This list contains all domains that use Cloudflare DNS"

That's only marginally more useful than saying "any domain matching /^.*$/";
plenty of domains use Cloudflare's DNS without using the proxy service (and
it is, barely, possible to use the proxy service which had the bug without
using the DNS service).

- Matt

-- 
A byte walks into a bar and orders a pint. Bartender asks him "What's
wrong?" The byte says "Parity error." Bartender nods and says "Yeah, I
thought you looked a bit off."

Re: SHA1 collisions proven possisble

[Matt Palmer]

On Thu, Mar 02, 2017 at 03:42:12AM +, Nick Hilliard wrote:
> James DeVincentis via NANOG wrote:
> > On top of that, the calculations they did were for a stupidly simple
> > document modification in a type of document where hiding extraneous
> > data is easy. This will get exponentially computationally more
> > expensive the more data you want to mask. It took nine quintillion
> > computations in order to mask a background color change in a PDF.
> > 
> > And again, the main counter-point is being missed. Both the good and
> > bad documents have to be brute forced which largely defeats the
> > purpose. Tthose numbers of computing hours are a brute force. It may
> > be a simplified brute force, but still a brute force.
> > 
> > The hype being generated is causing management at many places to cry
> > exactly what Google wanted, “Wolf! Wolf!”.
> 
> The Reaction state table described in
> https://valerieaurora.org/hash.html appears to be entertainingly accurate.

With particular reference to the "slashdotter" column.

- Matt

Re: SHA1 collisions proven possisble

[Matt Palmer]

On Tue, Feb 28, 2017 at 01:16:23PM -0600, James DeVincentis via NANOG wrote:
> The CA signing the cert actually changes the fingerprint

The what?  RFC5280 does not contain the string "finger".

> (and serial number, which is what is checked on revocation lists)

The CA doesn't "change" the serial number (a CSR doesn't have a place to
even ask for a serial), they pick one, and while it's *supposed* to be at
least partially random, given the largely appalling state of CA operations
(and, even worse, the competence of the auditors who are supposed to be
making sure they're doing the right thing), I'd be awfully surprised if
there wasn't at least one CA in a commonly-used trust store which was
issuing certificates with predictable serial numbers.

> Beyond that, SHA1 signing of certificates has long been deprecated and no
> new public CAs will sign a CSR and cert with SHA1.

Except all the ones that the payment industry (there's a group with no stake
in good security, huh?) have managed to convince browsers to allow
(thankfully, they get a good counter-cryptanalysis over them first), and all
the ones that have been issued "by mistake" to inconsequential organisations
like, say, HMRC (which just appear in CT logs, and the vigilance of the
community finds and brings to the attention of trust stores).

- Matt

-- 
 I remember going to my first tutorial in room 404. I was most upset
when I found it.

Re: SHA1 collisions proven possisble

[Matt Palmer]

On Mon, Feb 27, 2017 at 01:15:28AM -0500, Patrick W. Gilmore wrote:
> On Feb 26, 2017, at 21:16, Matt Palmer <mpal...@hezmatt.org> wrote:
> > Even better: I want a CA cert.  I convince a CA to issue me a regular,
> > end-entity cert for `example.com` (which I control) in such a way that I can
> > generate another cert with the same SHA1 hash, but which has `CA:TRUE` for
> > the Basic Constraints extension.
> > 
> > Wham!  I can now generate certs for *EVERYONE*.  At least until someone
> > notices and takes away my shiny new toy...
> 
> In the example above, the CA knows the SHA-1 hash of the cert it issued.
> (We are assuming there is a CA which still does SHA-1.) How do you get
> that CA to believe the two OTHER certs with DIFFERENT hashes you have to
> create so you can have two docs with the same hash?

This is doable because the data that goes into the cert is mostly
predictable (or attacker controlled).  The public key is provided by the
attacker; the subject and sAN extension is attacker-provided, and a lot of
the rest is predictable (issuer, policy OIDs, etc etc).  Even the notBefore
and notAfter fields are, to some degree, predictable; you can measure the
average issuance delay to figure out when you need to submit your CSR(s) to
get favourable timestamps, and many CAs "round" values, to make that easier. 
Thus, constructing a "dummy" TBSCertificate ("to-be-signed certificate") to
play with isn't difficult.

The mitigation to a chosen prefix attack (which is what this is) is to make
the serial number a random number.  The CA/Browser Forum Baseline
Requirements (the agreed-upon rules between CAs and browsers) say that you
need to put "at least 64 bits of randomness" into the serial number field,
which supposedly mitigates this.  Unfortunately, that isn't a property you
can easily determine by observation, and I'll bet dollars to donuts it isn't
audited, so whether that would stand up to a determined attacker is a
Rumsfeldian question..

- Matt

Re: SHA1 collisions proven possisble

[Matt Palmer]

On Sun, Feb 26, 2017 at 05:41:47PM -0600, Brett Frankenberger wrote:
> On Sun, Feb 26, 2017 at 12:18:48PM -0500, Patrick W. Gilmore wrote:
> > I repeat something I've said a couple times in this thread: If I can
> > somehow create two docs with the same hash, and somehow con someone
> > into using one of them, chances are there are bigger problems than a
> > SHA1 hash collision.
> > 
> > If you assume I could somehow get Verisign to use a cert I created to
> > match another cert with the same hash, why in the hell would that
> > matter?  I HAVE THE ONE VERISIGN IS USING.  Game over.
> > 
> > Valdis came up with a possible use of such documents. While I do not
> > think there is zero utility in those instances, they are pretty small
> > vectors compared to, say, having a root cert at a major CA.
> 
> I want a google.com cert.  I ask a CA to sign my fake google.com
> certificate.  They decline, because I can't prove I control google.com.

Even better: I want a CA cert.  I convince a CA to issue me a regular,
end-entity cert for `example.com` (which I control) in such a way that I can
generate another cert with the same SHA1 hash, but which has `CA:TRUE` for
the Basic Constraints extension.

Wham!  I can now generate certs for *EVERYONE*.  At least until someone
notices and takes away my shiny new toy...

- Matt

-- 
[M]ost of the other people here [...] drive cars that they have personally
built (starting with iron ore, charcoal, and a Malaysian turn-signal tree)
[...] but I wimp out on all of those points.  Sometimes there are advantages
to paying somebody else to do it for you.  -- Matt Roberds, in the Monastery

Re: Chinese root CA issues rogue/fake certificates

[Matt Palmer]

On Wed, Sep 07, 2016 at 04:15:47PM -0700, Eric Kuhnke wrote:
> Further update on all known suspicious activity from Wosign:
> 
> https://wiki.mozilla.org/CA:WoSign_Issues
> 
> Seriously, what level of malice and/or incompetence does one have to rise
> to in order to be removed from the Mozilla (and hopefully Microsoft and
> Chrome) trusted root CA store?  Is this not sufficient?

At this point, it's pretty clear that WoSign as an operational CA is going
to be no more, at least as far as Mozilla is concerned.  The number of
issues is immense, and nobody on m.d.s.p is arguing in favour of keeping the
root (except WoSign).  The other major trust stores are completely opaque as
to their process, but a root pulled from Mozilla is practically dead in the
water.

The problem is that just pulling the root is extremely damaging -- to
Mozilla, and to the ecosystem.  If a root gets pulled, all the sites that
are currently using a WoSign-issued cert "stop working".  Since plenty of
people use WoSign certs (in China, as well as their "free" issuance
offering), a lot of sites go dead all at once.  Since users cannot stand to
not have their dancing kitten gifs, they'll barge through any barrier you
put in place, whether that be clicking past warnings or switching to another
browser.  Mozilla doesn't want to lose (more) market share, and training
people to click past security warnings is a really, really dumb move.

There are a number of things that could be done to reduce the mess of a
pulled root, but many of them involve the cooperation of the CA being
pulled, and it's highly unlikely that they'd be in a cooperative mood.

The relevant discussion at the moment is around how best to cause
WoSign to no longer be trusted, *without* causing collateral damage (or at
least minimising it).  Certificate Transparency can help, maybe, but
CT isn't a live query mechanism, and shipping a giant whitelist of all valid
WoSign certs is... large.

Honest Achmed had the right idea.

- Matt

Nit-pickers' corner: Chrome uses the OS trust store; Google doesn't run its
own trust store for Chrome, although it does maintain *something* for
Android.  Chrome has a cert blacklist, and its own EV treatment criteria,
but no trust store as such.

Re: Software for tracking network related projects and activities

[Matt Palmer]

On Thu, Sep 01, 2016 at 11:07:33PM -0600, Manuel Marín wrote:
> We are currently using RT for tracking tasks related to network operations
> like BGP configuration change requests, circuit/ports activation, support
> tickets, etc, but when trying to track multiple activities that involve
> multiple departments, the RT (Request Tracker) system does not provide all
> the tools as a project/tasks management system. I was wondering if you can
> share what you use for tracking network related projects and activities.

RT and a couple of custom fields always worked fine for me and my org.

- Matt

Re: Chinese root CA issues rogue/fake certificates

[Matt Palmer]

On Wed, Aug 31, 2016 at 06:49:17PM -0700, Lyndon Nerenberg wrote:
> > On Aug 31, 2016, at 6:36 PM, Matt Palmer <mpal...@hezmatt.org> wrote:
> > 
> > Thanks, Netscape.  Great ecosystem you built.
> 
> Nobody at that time had a clue how this environment was going to scale,
> let alone what the wide-ranging security issues would be.

Nor did they particularly trouble themselves to find out.

> And where were you back then, not saving us from our erroneous path ...

You're going to go with that one, are you?  Good for you.

- Matt


signature.asc
Description: Digital signature

Re: Chinese root CA issues rogue/fake certificates

[Matt Palmer]

On Wed, Aug 31, 2016 at 09:33:18PM -0700, George William Herbert wrote:
> > On Aug 31, 2016, at 6:36 PM, Matt Palmer <mpal...@hezmatt.org> wrote:
> > there's just wy too many sites using WoSign (and StartCom) for the
> > CAs' roots to just be pulled.  Sad, but true.
> 
> Not even.  Pull away.

Not going to happen.  Feel free to argue otherwise in the appropriate
venues, but you're tilting at windmills, IMO.

> > I'd be surprised if most business continuity people could even name their
> > cert provider, and most probably don't even know how certs come to exist or
> > that they *can* be made useless on a wide scale by the actions of,
> > seemingly, an unrelated third party.
> 
> Not in my neck of the woods.  If you have a drought of good ones in your
> area my consulting company calls that an opportunity...

How the hell do you get from "the world does not work that way" to "please
pitch me your consulting services"?

- Matt

Re: Chinese root CA issues rogue/fake certificates

[Matt Palmer]

On Wed, Aug 31, 2016 at 10:45:48AM -0800, Royce Williams wrote:
> Hypothetically, it would be an interesting strategy for a CA to
> publicly demonstrate this level of competence:
> 
> https://www.schrauger.com/the-story-of-how-wosign-gave-me-an-ssl-certificate-for-github-com
> 
> ... while at the same time taking over another large install base like
> StartSSL's (an install base fueled by offering free certs).
> 
> If one got caught doing something naughty, one could buy time by A)
> playing the incompetence card a few times, and B) having a large
> enough deployment that it becomes non-trivial for the browsers/OSes to
> revoke you outright.

Honest Achmed's business model wins again!

I'm pretty sure that's how this is going to go down here, too, incidentally
-- there's just wy too many sites using WoSign (and StartCom) for the
CAs' roots to just be pulled.  Sad, but true.

> Also, this is a cautionary tale about certificate diversity.
> 
> Because of relative issuer stability, orgs have had the luxury of
> depending wholly on a single cert supplier. The risk/continuity folks
> might want to model some "one of our major certificate issuers just
> got globally revoked" scenarios - if they haven't already.

I'd be surprised if most business continuity people could even name their
cert provider, and most probably don't even know how certs come to exist or
that they *can* be made useless on a wide scale by the actions of,
seemingly, an unrelated third party.  It's a system nearly without
precedent, when you think about it.  In fact, my gut feel is that, if they
really understood the system, most risk/continuity folks would scream "are
you f**king kidding me?  That's ridiculous!".

Thanks, Netscape.  Great ecosystem you built.

- Matt


-- 
Talk about unlucky. D'you know, if I fell in a barrel of tits I'd come out
sucking me thumb.
-- Seen on the 'net:
 http://thelawwestofealingbroadway.blogspot.com/2006/01/bang-to-rights.html

Re: Why the internal network delays, Gmail?

[Matt Palmer]

On Sat, Aug 27, 2016 at 01:25:42AM -, John Levine wrote:
> In article 
>  you 
> write:
> >I was working within the limits of what I had available.
> 
> Here's the subscription page for mailop.  It's got about as odd
> a mix of people as nanog, ranging from people with single user linux
> machines to people who run some of the largest mail systems in
> the world, including Gmail:
> 
> https://chilli.nosignal.org/cgi-bin/mailman/listinfo/mailop

I know they're mailops, and not tlsops, but surely presenting a cert that
didn't expire six months ago isn't beyond the site admin's capabilities?

- Matt

Re: www.RT.com bad dns record

[Matt Palmer]

On Thu, Jul 07, 2016 at 06:36:23PM -0700, Ca By wrote:
> On Thursday, July 7, 2016, Spencer Ryan  wrote:
> 
> > Dotted-quad notation is completely valid, and works fine.
> >
> > https://en.wikipedia.org/wiki/IPv6_address#Presentation
> >
> > http://[:::37.48.108.112] loads fine in my browsers.
> 
> It may be legit on your network, but people generally don't do that If
> they publish a  record, it usually has a legit v6 address in it.

That is a legit IPv6 address.  That it won't work on a host that is IPv6-only
is a different issue, and one I agree is probably an unexpected and unwanted
side effect.

- Matt

Re: NAT firewall for IPv6?

[Matt Palmer]

On Fri, Jul 01, 2016 at 09:28:54PM -0500, Edgar Carver wrote:
> Hello NANOG community. I was directed here by our network administrator
> since she is on vacation. Luckily, I minored in Computer Science so I have
> some familiarity.

Well played, Tay.  Well played.

For everyone else:

https://twitter.com/SwiftOnSecurity/status/749062835687174144

- Matt

Re: SNMP "bridging"/proxy?

[Matt Palmer]

On Fri, May 20, 2016 at 04:43:07PM -0700, Nathan Anderson wrote:
> Is anybody out there aware of a piece of software that can take data from
> an arbitrary source and then present it, using a MIB or set of OIDs of
> your choosing, as an SNMP-interrogatable device?

Many, many years ago, I wrote a fairly minimal SNMP agent in Ruby
(http://theshed.hezmatt.org/rubysnmpd/index.html), which was
explicitly designed to be easily extensible (it didn't do anything useful by
itself).  If your team's into Ruby, that might be a possibility.

Otherwise, net-snmp does come with some fairly decent extensibility
capabilities these days.  It has a variety of mechanisms by which you can
feed extra data into it, and in general I'd recommend going down that route.

- Matt

Re: IPv6 traffic percentages?

[Matt Palmer]

On Wed, Jan 20, 2016 at 01:14:42PM +0100, nanog-...@mail.com wrote:
> Would those with IPv6 deployments kindly share some statistics on their 
> percentage of IPv6 traffic?

https://twitter.com/discourse/status/679808652128030720

We're a smallish content source.

- Matt

Re: Nat

[Matt Palmer]

On Sun, Dec 20, 2015 at 08:11:53PM -0700, Keith Medcalf wrote:
> > I agree that a /48 or /56 being reserved for business
> > customers/sites is reasonable.  But for residential use, I'm having a hard
> > time believing multi-subnet home networks are even remotely common outside
> > of networking folk such as the NANOG members.  A lot of recent IPv4
> > devices
> > such as smart TVs have the ability to auto-discover things they can talk
> > to
> > on the network.  If we start segmenting our home networks to keep toasters
> > from talking to thermostats, doesn't this end up meaning your average home
> > user will need to be proficient in writing FW rules?  Bridging an entire
> > house network isn't that bad.
> 
> I have (currently) 6 network segments.  One for my "trusted" clients, one
> for my "trusted" servers, one for the "entertainment" systems, one for
> "dirty & untrustworthy" computers (such as those from $dayjob), one for
> "clean" WiFi, and one for dirty WiFi.  If there were any additional
> classes of devices, they would live in their own subnets as well.

If suspect you probably come under the "networking folk such as NANOG
members" exception to the general assertion.

- Matt

Re: Nat

['Matt Palmer']

On Sun, Dec 20, 2015 at 10:54:49PM -0500, Chuck Church wrote:
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Matt Palmer
> >Depends on how many devices you have on it.  Once you start filling your
> >home with Internet of Unpatchable Security Holes devices, having everything
> >on a single ethernet >segment might start to get a little...  noisy.
> 
> >Thankfully, IPv6 has well-defined multicast scopes, which makes it
> >trivially easy to do cross-L2-segment service discovery without needing to
> >resort to manually berking around >with firewall rules.
> 
> If your home is full of unpatched or compromised hosts, and they're using
> these well-defined multicast scopes, doesn't that mean they can now
> communicate and infect one another?

No, multicast for discovery doesn't necessarily mean that the application
traffic can also pass.  The discovery multicast packets could be filtered at
any point within the network, also.

However, access control isn't what you asked about.  You claimed that
multiple L2 segments broke service discovery, and I refuted that point.

> For years I've seen people on this list
> insist on "NAT/PAT != firewall".   Well, a router routing everything it sees
> is even less of a firewall.

Correct.  However, nowhere did I suggest that a router should be routing
absolutely everything it sees.

> I'm really not trying to be argumentative here,

And yet, you're doing an awfully good job of being argumentative, about a
subject you really don't seem to know a whole lot about.

> but I'm just having a hard time believing Joe Sixpack will be applying
> business networking principals such as micro-segmenting to a home network
> with 3 to 7 devices on it.  If anything, these complexities we keep
> adding/debating such as DHCP vs RA, prefix delegation, etc are only slowing
> down the general deployment of IPv6.

Yes, it's a pity that people who refuse to learn about the new features that
IPv6 provides keep trying to shoehorn IPv6 into their legacy mindset, but
there's not a whole lot the rest of us can do about that.

- Matt

Re: Nat

[Matt Palmer]

On Sun, Dec 20, 2015 at 09:23:04PM -0500, Chuck Church wrote:
>   I agree that a /48 or /56 being reserved for business
> customers/sites is reasonable.  But for residential use, I'm having a hard
> time believing multi-subnet home networks are even remotely common outside
> of networking folk such as the NANOG members.  A lot of recent IPv4 devices
> such as smart TVs have the ability to auto-discover things they can talk to
> on the network.  If we start segmenting our home networks to keep toasters
> from talking to thermostats, doesn't this end up meaning your average home
> user will need to be proficient in writing FW rules?  Bridging an entire
> house network isn't that bad.

Depends on how many devices you have on it.  Once you start filling your
home with Internet of Unpatchable Security Holes devices, having everything
on a single ethernet segment might start to get a little...  noisy.

Thankfully, IPv6 has well-defined multicast scopes, which makes it trivially
easy to do cross-L2-segment service discovery without needing to resort to
manually berking around with firewall rules.

- Matt

Re: Nat

[Matt Palmer]

On Fri, Dec 18, 2015 at 07:30:35PM +0300, Ahmed Munaf wrote:
> > On Dec 17, 2015, at 8:47 PM, Nick Hilliard  wrote:
> > 
> > On 17/12/2015 17:36, Ahmed Munaf wrote:
> >> we are using ESP 20 
> > 
> > You haven't said what you mean by "better".  This could mean "faster" or
> > "copes with more sessions" or "cheaper".  If your ISP is large, then it
> > might be "cost per user is lower" or "able to cope with the number of 
> > users".
> 
> I mean by better, it handle more sessions and cheaper 

At what scale?  You probably won't get much cheaper than an rPi and
netfilter at home scale, but I wouldn't want to run a national ISP on one. 
Also, by "cheaper" do you just mean capex, or is opex a concern?

- Matt

Re: IPv6 Cogent vs Hurricane Electric

[Matt Palmer]

On Thu, Dec 03, 2015 at 04:58:08PM -0800, Matthew Petach wrote:
> Or, if you feel that Cogent's stubborn insistence on partitioning the
> global v6 internet shouldn't be rewarded with money, pay someone *other*
> than cogent for IPv6 transit and also connect to HE.net; that way you
> still have access to cogent routes, but you also send a subtle economic
> nudge that says "hey cogent-- trying to get into the tier 1 club by
> partitioning the internet isn't a good path for long-term sucess".

Sadly, anyone you pay for transit to Cogent routes is going to be giving
Cogent their cut, so it's not a perfect signal to Cogent that we'd prefer to
have one IPv6ternet rather than two.  At the very least, configure the
routers to that any routes you learn via HE are preferenced, and announce
your routes as preferring HE, so that Cogent gets as little of the traffic
as possible.

- Matt

Re: Bluehost.com

[Matt Palmer]

On Wed, Nov 25, 2015 at 02:24:05PM -0500, Andrew Kirch wrote:
> remember folks, redundancy is the savior of all f***ups.

Except for the fuckups that the redundancy *caused*...

- Matt

Re: DHCPv6 PD & Routing Questions

[Matt Palmer]

On Fri, Nov 20, 2015 at 01:35:55PM -0800, Jim Burwell wrote:
> My questions are:
> 
> 1) Does the DHCPv6 protocol include any standards/mechanisms/methods for
> managing routes to prefixes it delegates, or does it consider this
> outside of its function?  (I suspect the latter)

It's considered outside of function.  It makes a lot of sense, from the
*protocol's* viewpoint, not to go constraining itself in any way.

*Implementations*, on the other hand, appear to have kinda dropped the ball,
insofar as none of the OSS DHCPv6 servers that can do PD appear to have put
any thought into what to do with the prefixes delegated.

> 2) What are the most common ways of managing the routing of delegated
> prefixes in the ISPs routing domain?  Has a standard method/best
> practice emerged yet?  Routing protocols?  IPv6 RAs?

I hacked some code into ISCP DHCPD to give called scripts sufficient knowledge
to add routes to the local machine's routing table:


http://www.hezmatt.org/~mpalmer/blog/2014/11/20/multi-level-prefix-delegation-is-not-a-myth-ive-seen-it.html

(Holy crap, I published that post almost exactly a year ago today...)

More recently, I'm doing some work with a production containerised
environment, and I decided to use RAs to propagate /64 routes amongst the
container hosts and immediate upstream router (the upstream router has the
whole /48 routed to it, and the router then gets the RAs to know which
machine to send the /64 to).  It seems to work rather well.  If I had any
more complicated a setup, I'd definitely have broken out the OSPF or
something.

> One obvious answer would be routing protocols.  In my brief googling,
> I've seen a forum post that seems to indicate that Comcast makes use of
> RIPng on their CPE to propagate routing information for prefixes
> delegated to it.  Can someone confirm this?  This would seem as good a
> method as any to do this, albeit with obvious security concerns.

I can't confirm Comcast's use of anything in particular, but I'd certainly
consider it a possibility.  In an ISP environment, I think I'd prefer for my
routing to *not* be under the control of anything that the customer can get
their fingers into, but I'm sure there's suitable filters in place to stop a
customer trying to announce all of 2000::/3...

> What's the best way to implement a DHCPV6 PD client on a Linux router? 
> Dibbler seems to do everything except route propagation (asks for PD,
> puts PD address on local NIC if asked).  Anything better out there?

Well, I'm quite partial to the solution I hacked up for ISC DHCPD, but it's
hard to argue that I'm an unbiased observer.  

- Matt

Re: DNSSEC and ISPs faking DNS responses

[Matt Palmer]

On Sat, Nov 14, 2015 at 05:32:41PM +1100, Mark Andrews wrote:
> In message <20151114044614.ga4...@hezmatt.org>, Matt Palmer writes:
> > On Fri, Nov 13, 2015 at 10:51:52AM +0100, Bj�rn Mork wrote:
> > > So what do we do? We currently point the blocked domains to addresses of
> > > a web server with a short explanation.  But what if the domains were
> > > signed?  We could let validating servers return SERVFAIL.  But I'd
> > > really prefer avoiding that for the simple reason that there is no way
> > > to distinguish that SERVFAIL from one caused by e.g. a domain owner
> > > configuration error.
> > 
> > Perhaps we need to expand RCODE to be the full octet, and indicate "blocked
> > for legal reasons" with RCODE value 25.
> 
> Rcode's were expanded to 12 bits back in 1999.  See RFC 2671.

I didn't feel it was worth looking beyond RFC1035 for an off-the-cuff joke.

- Matt

Re: DNSSEC and ISPs faking DNS responses

[Matt Palmer]

On Fri, Nov 13, 2015 at 09:54:28AM +, a.l.m.bu...@lboro.ac.uk wrote:
> > BTW, the proposed law, being done by lawyers, will have the list of
> 
> you say law but this idea of blocking all competitors to the states
> lotto sounds very unlawful and anti-competitive  - yes, I can
> understand states or countries blocking ALL gambling , thats a simple
> 'we dont allow it here' , but to say 'yes, you can access just ours'
> well, in EU I dont think that would ever fly.

Sweden's still part of the EU, isn't it?  ("Systembolaget", if you need a
search term).

- Matt

Re: DNSSEC and ISPs faking DNS responses

[Matt Palmer]

On Fri, Nov 13, 2015 at 10:51:52AM +0100, Bjørn Mork wrote:
> So what do we do? We currently point the blocked domains to addresses of
> a web server with a short explanation.  But what if the domains were
> signed?  We could let validating servers return SERVFAIL.  But I'd
> really prefer avoiding that for the simple reason that there is no way
> to distinguish that SERVFAIL from one caused by e.g. a domain owner
> configuration error.

Perhaps we need to expand RCODE to be the full octet, and indicate "blocked
for legal reasons" with RCODE value 25.

- Matt

Re: Cloud backups versus lightning strikes

[Matt Palmer]

On Wed, Aug 19, 2015 at 08:44:03PM -0400, Sean Donelan wrote:
 As the saying goes, cloud computing is just someone else's computer. Always
 backup your cloud backups... in your backup.

This was data loss on GCE persistent disks (equivalent to AWS EBS), not
archival storage.  Hopefully very few people are using persistent disks as
*backup* storage (if nothing else, it's more expensive).

The general point is still valid, though -- *any* single instance of data is
vulnerable to loss, whether it's on your computer, or someone else's. 
Backups are never optional.

- Matt

Re: Working with Spamhaus

[Matt Palmer]

On Tue, Jul 28, 2015 at 11:41:08PM -0600, Bryan Tong wrote:
 Yes that is part of it.
 
 There are other blocks they listed as well.

Well, http://www.spamhaus.org/sbl/query/SBL263089 has a fair amount of shady
stuff going on, and http://www.spamhaus.org/sbl/listings/esited.com gives a
pretty decent history of what Spamhaus has been doing.  Note the
(escalation) entries in there, which indicates a lack of interest on
esited.com's part in fixing any of the problems.

- Matt

Re: Working with Spamhaus

[Matt Palmer]

On Tue, Jul 28, 2015 at 11:13:02PM -0500, Larry Sheldon wrote:
 On 7/28/2015 22:57, Bryan Tong wrote:
 
 Yes I have followed all of the procedures. I will continue to wait to see
 if there is any change.
 
 Would you please send me the address range in question--I would like to see
 what they told you to do.

I suspect that http://www.spamhaus.org/query/ip/199.87.233.245 may be part
of it (although it indicates a /21 blocked, not a /17).

- Matt

-- 
One of the Rules Of Flight is, or should be: Pullout Altitude Is Not A
Signed Quantity.
-- Anthony de Boer, in the monastery

Re: SEC webpages inaccessible due to Firefox blocking servers with weak DH ciphers

[Matt Palmer]

On Fri, Jul 17, 2015 at 07:14:17PM +, Michael O Holstein wrote:
 making 99% of the web secure is better than keeping an old 1% working
 
 A fine idea, unless for $reason your application is among the 1% ..
 nevermind the arrogance of the I'm sorry Dave sort of attitude.

First they came for SSLv2, and I said nothing because...

 As an example .. we have a vendor who, in the current release (last 3
 months) still requires weak ciphers in authentication responses.  That
 was mostly okay until another vendor (with more sense) wanted to auth the
 same way but only permitted strong ciphers.

So get up your vendors to update their stuff, and *preferably* before a
super-critical hole is found in protocols that should have ideally died a
natural death years ago.  TLS 1.2, AES, and SHA-256 aren't exactly OMFG
new! at this stage of the game.

Also, take this as a learning experience: next time, make sure RFPs and
contracts include an undertaking to maintain compatibility with reasonably
recent standards, and financial penalties for the vendor if their failure to
do so results in operational problems for you.

- Matt

-- 
aren't they getting rarer than amigas now?  just without all that fuzzy
good times nostalgia?
-- Ron Lee, in #debian-devel, on Itanic

Re: SEC webpages inaccessible due to Firefox blocking servers with weak DH ciphers

[Matt Palmer]

On Fri, Jul 17, 2015 at 10:26:22AM +0200, Alexander Maassen wrote:
 Ok, it is good to think about security, but not giving you any chance to
 make exceptions is simply forcing users to use another browser in order to
 manage those devices, or to keep an old machine around that not gets
 updated.

Hey, if those DRACs can't get updated to not use piss-weak ciphers, what's
the problem with having one more machine laying around unpatched to manage
them?

- Matt

Re: Anycast provider for SMTP?

[Matt Palmer]

On Mon, Jun 15, 2015 at 05:07:22PM -0700, Dave Taht wrote:
 On Mon, Jun 15, 2015 at 5:00 PM, Randy Bush ra...@psg.com wrote:
  What about IPv6? We have a plan! We plan to be dead before customers
  demand IPv6.
  I am pretty sure the authors are still alive(?).
 
  and customer demand for ipv6 still holds strong, right?
 
 Does seem to be on the uptick!

It's certainly stronger than it has *ever* been before.

- Matt

-- 
I am cow, hear me moo, I weigh twice as much as you. I'm a cow, eating
grass, methane gas comes out my ass. I'm a cow, you are too; join us all!
Type apt-get moo.

Re: Android (lack of) support for DHCPv6

[Matt Palmer]

On Wed, Jun 10, 2015 at 10:31:25AM +0200, Sander Steffann wrote:
 I don't think it is unreasonable. If the network doesn't support the
 features you need then let the user know (grey out the feature and add a
 note that says broken network).  It will put pressure on the network
 department to fix their DHCPv6 implementation.

Because that has worked so well in the past, as opposed to the helpdesk
person who receives the complaint shrugging their shoulders and saying, I
dunno, that's just the way it is.

- Matt

Re: Android (lack of) support for DHCPv6

[Matt Palmer]

On Tue, Jun 09, 2015 at 02:56:26PM -0700, Owen DeLong wrote:
 Further, the cellular companies would do well to be more adaptive to the
 capabilities that exist in the hardware rather than insisting that they
 choose the solution and the hardware makers must adapt.

Hahahahahaha!

Fun fill in the blanks game: We're the phone company, __ ___'_  __ .

- Matt

-- 
FreeFrag The most secure computer in the world is one not connected to the
internet.  Thats why I recommend Telstra ADSL.
-- bash.org/?168859

Re: AWS Elastic IP architecture

[Matt Palmer]

The question that Matthew Kaufman proposed was specifically asking about app
architecture deployments, so what Facebook is choosing to do is entirely
germane.

- Matt

On Mon, Jun 01, 2015 at 02:43:27PM -0400, Todd Underwood wrote:
 fb is not a 'cloud provider'.
 
 it's orthogonal to the question.
 
 t
 
 On Mon, Jun 1, 2015 at 2:36 PM, Ca By cb.li...@gmail.com wrote:
 
  On Mon, Jun 1, 2015 at 10:49 AM, Matthew Kaufman matt...@matthew.at
  wrote:
 
   On 6/1/2015 12:06 AM, Owen DeLong wrote:
  
   ... Here’s the thing… In order to land IPv6 services without IPv6
  support
   on the VM, you’re creating an environment where...
  
  
   Let's hypothetically say that it is much easier for the cloud provider if
   they provide just a single choice within their network, but allow both v4
   and v6 access from the outside via a translator (to whichever one isn't
   native internally).
  
   Would you rather have:
   1) An all-IPv6 network inside, so the hosts can all talk to each other
   over IPv6 without using (potentially overlapping copies of) RFC1918
   space... but where very little of the open-source software you build your
   services on works at all, because it either doesn't support IPv6 or they
   put some IPv6 support in but it is always lagging behind and the bugs
  don't
   get fixed in a timely manner. Or,
  
 
 
  Facebook selected IPv6-only as outlined above
 
  http://blog.ipspace.net/2014/03/facebook-is-close-to-having-ipv6-only.html
 
 
  
   2) An all-IPv4 network inside, with the annoying (but well-known) use of
   RFC1918 IPv4 space and all your software stacks just work as they always
   have, only now the fraction of users who have IPv6 can reach them over
  IPv6
   if they so choose (despite the connectivity often being worse than the
  IPv4
   path) and the 2 people who are on IPv6-only networks can reach your
   services too.
  
   Until all of the common stacks that people build upon, including
   distributed databases, cache layers, web accelerators, etc. all work
   *better* when the native environment is IPv6, everyone will be choosing
  #2.
  
   And both #1 and #2 are cheaper and easier to manage that full dual-stack
   to every single host (because you pay all the cost of supporting v6
   everywhere with none of the savings of not having to deal with the
   ever-increasing complexity of continuing to use v4)
  
   Matthew Kaufman
  
  
 
 

-- 
Designing an effective undergrad CS degree is hard. It's no wonder so many
ivy-league schools have more or less given up and turned into Java
Certification shops.-- Steve Yegge,
  http://steve-yegge.blogspot.com/2007/06/rich-programmer-food.html

Re: AWS Elastic IP architecture

[Matt Palmer]

On Mon, Jun 01, 2015 at 10:49:09AM -0700, Matthew Kaufman wrote:
 On 6/1/2015 12:06 AM, Owen DeLong wrote:
 ... Here’s the thing… In order to land IPv6 services without IPv6 support
 on the VM, you’re creating an environment where...
 
 Let's hypothetically say that it is much easier for the cloud provider if
 they provide just a single choice within their network, but allow both v4
 and v6 access from the outside via a translator (to whichever one isn't
 native internally).
 
 Would you rather have:
 1) An all-IPv6 network inside, so the hosts can all talk to each other over
 IPv6 without using (potentially overlapping copies of) RFC1918 space... but
 where very little of the open-source software you build your services on
 works at all, because it either doesn't support IPv6 or they put some IPv6
 support in but it is always lagging behind and the bugs don't get fixed in a
 timely manner. Or,

I'd much rather have this.  In fact, I'm building this at the moment.  It
simplifies so many things, like not having to worry about address
assignment, choosing appropriately-sized subnets, address re-use, etc. 
Having direct access to everything from the outside world without having to
deal with NAT/VPN/a jump box makes so many things smoother, too.  Everything
I've deployed (and yes, all the components are OSS) has dealt with IPv6 just
fine, and everything I've considered deploying claims IPv6 support.  I've
had to submit one patch for fixing an IPv6 bug, and because it's OSS, I can
do that!

- Matt

Re: AWS Elastic IP architecture

[Matt Palmer]

On Mon, Jun 01, 2015 at 11:30:00AM -0400, Christopher Morrow wrote:
 I don't get why
 'ipv6 address on my vm' matters a whole bunch (*in a world where v4 is
 still available to you I mean),

It simplifies infrastructure management considerably.  Having to balance
between how many subnets will I ever need? vs how many machines could I
end up with in a subnet? is something I never thought would become
annoying, until I had the opportunity to not worry about it...  then it was
frustrating to have to go back to it.  Not having to use a VPN/NAT/jump box
to hit all my infrastructure seems like a small benefit, but it saves having
to maintain a VPN/jump box (and all its attendant annoyances).  Oh, yeah,
never having to faff around with split-horizon DNS management... Family Guy
Tooth Fairy on YouTube.  grin

In short, there's a whole pile of dodgy hacks we deploy almost without
thinking about it, because that's just how things are done, to work around
limitations in IPv4 deployments.  Having IPv6 everywhere *within* the
infrastructure makes all of those hacks disappear, and like most things we
just do because we have to, you don't realise how much of a PITA they were
until they're gone.

- Matt


-- 
And Jesus said unto them, And whom do you say that I am?  They replied,
You are the eschatological manifestation of the ground of our being, the
ontological foundation of the context of our very selfhood revealed. And
Jesus replied, What?  -- Seen on the 'net

Re: AWS Elastic IP architecture

[Matt Palmer]

On Sun, May 31, 2015 at 10:46:02PM -0400, Christopher Morrow wrote:
 So... ok. What does it mean, for a customer of a cloud service, to be
 ipv6 enabled?

IPv6 feature-parity with IPv4.

My must-haves, sorted in order of importance (most to least):

 o Is it most important to be able to terminate ipv6 connections (or
 datagrams) on a VM service for the public to use?
 
 o Is it most important to be able to address ever VM you create with
 an ipv6 address?
 
 o Is it most important to be able to talk to backend services (perhaps
 at your prem) over ipv6?

If, by backend services, you mean things like RDS, S3, etc, this is in the
right place.

 o Is it most important that administrative interfaces to the VM
 systems (either REST/etc interfaces for managing vms or 'ssh'/etc) be
 ipv6 reachable?
 
 I don't see, especially if the vm networking is unique to each
 customer, that 'ipv6 address on vm' is hugely important as a
 first/important goal. I DO see that landing publicly available
 services on an ipv6 endpoint is super helpful.

Being able to address VMs over IPv6 (and have VMs talk to the outside world
over IPv6) is *really* useful.  Takes away the need to NAT anything.

 Would AWS (or any other cloud provider that's not currently up on the
 v6 bandwagon) enabling a loadbalanced ipv6 vip for your public service
 (perhaps not just http/s services even?) be enough to relieve some of
 the pressure on other parties and move the ball forward meaningfully
 enough for the cloud providers and their customers?

No.  I'm currently building an infrastructure which is entirely v6-native
internally; the only parts which are IPv4 are public-facing incoming service
endpoints, and outgoing connections to other parts of the Internet, which
are proxied.  Everything else is talking amongst themselves entirely over
IPv6.

- Matt

-- 
After years of studying math and encountering surprising and
counterintuitive results, I came to accept that math is always reasonable,
by my intuition of what is reasonably is not always reasonable.
-- Steve VanDevender, ASR

Re: AWS Elastic IP architecture

[Matt Palmer]

On Sun, May 31, 2015 at 01:38:05AM +1000, Andras Toth wrote:
 Perhaps if that energy which was spent on raging, instead was spent on
 a Google search, then all those words would've been unnecessary.
 
 Official documentation:
 http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-internet-facing-load-balancers.html#internet-facing-ip-addresses

Congratulations, you've managed to find exactly the same info as Owen
already covered:

Load balancers in a VPC support IPv4 addresses only.

and

Load balancers in EC2-Classic support both IPv4 and IPv6 addresses.

- Matt

Re: Fixing Google geolocation screwups

[Matt Palmer]

On Wed, May 06, 2015 at 10:56:22AM +1000, Mark Andrews wrote:
 In message 20150505210746.gh22...@hezmatt.org, Matt Palmer writes:
  On Tue, May 05, 2015 at 12:03:23PM -0400, Luan Nguyen wrote:
   There's a form here - https://support.google.com/websearch/contact/ip
   But google is pretty smart, its systems will learn the correct geolocation
   over time...
  
  That'd be quite a trick, given that the netblock practically can't be used
  at all with Google services.
 
 One would expect support.google.com to not be geo blocked just like
 postmaster@ should not be filtered.  That said they can always
 disable IPv6 temporarially (or just firewall off the IPv6 instance
 of support.google.com and have the browser fallback to IPv4) and
 reach support.google.com over IPv4 to lodge the complaint.

I was specifically responding to the suggestion that Google would
automagically learn the correct location of the netblock, presumably based
on the characteristics of requests coming from the range.  Being explicitly
told that a given netblock is in a given location (as effective, or
otherwise, as that may be) doesn't really fit the description of systems
[learning] the correct geolocation over time.

- Matt

-- 
Skippy was a wallaby. ... Wallabies are dumb and not very trainable...  The
*good* thing...is that one Skippy looks very much like all the rest,
hence...one-shot Skippy and plug-compatible Skippy.  I don't think they
ever had to go as far as belt-fed Skippy  -- Robert Sneddon, ASR

Re: Fixing Google geolocation screwups

[Matt Palmer]

On Tue, May 05, 2015 at 12:03:23PM -0400, Luan Nguyen wrote:
 There's a form here - https://support.google.com/websearch/contact/ip
 But google is pretty smart, its systems will learn the correct geolocation
 over time...

That'd be quite a trick, given that the netblock practically can't be used
at all with Google services.

- Matt

Re: ADSL Line Extenders

[Matt Palmer]

On Tue, Apr 28, 2015 at 05:24:39PM -0400, Jean-Francois Mezei wrote:
 A search on Google yields many products dating back to the days of
 ADSL-1 advertising 1mbps profiles, but a few seem more recent and
 support ADSL2+ (not sure if any support VDSL2).
 
 Are these thing out of date and no longer deployed ? Were they ever
 effective, or just vapourware that didn't really improve things ?

Wikipedia[1] suggests they're a real thing, and have real-world uses.  I can
imagine that a big disincentive to installing them would be powering them,
although line power would probably be enough for one or two hops.  Getting
access to the line in the right places to install them might be a challenge,
too (or at least be somewhat expensive).  No doubt telcos also don't want to
install them because they can make a whole lot more money for less effort
by forcing customers onto wireless.

- Matt

[1] https://en.wikipedia.org/wiki/ADSL_loop_extender

Re: Galaxy S6 is IPv6 on all US National Mobile carriers

[Matt Palmer]

On Mon, Apr 13, 2015 at 09:42:07PM -0400, Jared Mauch wrote:
  On Apr 13, 2015, at 9:02 PM, Christopher Morrow morrowc.li...@gmail.com 
  wrote:
  On Mon, Apr 13, 2015 at 7:30 PM, Will Dean w...@willscorner.net wrote:
  Reddit started using CloudFlare late last year, so they should able to
  serve content up over v6.
  
  nice!
 
 Sorry to rain on your parade:
 
 dhcp-7f01:~ jared% host -t  www.reddit.com.
 www.reddit.com has no  record

should be able to serve != are serving.

- Matt

-- 
If you are a trauma surgeon and someone dies on your table, [...] everyone
would know you did your best.  When someone does something truly stupid
with their system and it dies and you can't resuscitate it, you must be
incompetent or an idiot.  -- Julian Macassey, in the Monastery

Re: Any google network admins out there?

[Matt Palmer]

Or, to answer your question more simply: No.

- Matt

On Fri, Apr 03, 2015 at 11:39:36AM +0100, Pedro Cavaca wrote:
 https://support.google.com/websearch/answer/86640?hl=en
 
 On 3 April 2015 at 04:53, Randy na...@afxr.net wrote:
 
  I've started to get some message today from google claiming that my
  computer or network was sending automated queries, and they are blocking me.
  I'm not sending automated queries, Ive logged all of my outbound traffic
  and there is only my browser traffic going to google.
 
  I'm not responsible for any one else on my network since it is owned by
  my ISP, and solely blocking me based on what some one else with an ip
  address close to mine is not an acceptable practice to have for an address
  used for personal web browsing.
  I would like to know if there is any way to get into contact with google
  about this other then by legal means?
 
 

-- 
How many Apple Newton users does it take to change a lightbulb?
Foux. There to eat lemons, axe gravy soup.
-- Seen on the 'net

Re: FIXED - Re: Broken SSL cert caused by router?

[Matt Palmer]

On Sat, Mar 28, 2015 at 09:05:38AM -0700, Mike wrote:
 On 03/27/2015 10:34 AM, Frank Bulk wrote:
 Glad you figured that out.
 
 I've used three SSL evaluation websites to help me with intermediate 
 certificate issues:
 https://www.ssllabs.com/ssltest/analyze.html (will show the names and 
 details of the certs, missing or not
 https://www.wormly.com/test_ssl (quick SSL tester, will point out if 
 intermediate certificate is missing)
 https://www.digicert.com/help/ (will show a green chain link between certs 
 when they're all there *and* in order)
 
 I went back to Frank's list and did some additional testing. I have a
 different server which was set up the same way as the previous one
 discussed, and I thought I would use the above tools and see if my problem
 would have been identified by any of them. I am sorry to report, no, none of
 these either caught the problem either.

Are you able to share the URL of the misconfigured site?  It would be
interesting to examine exactly what's going on.

- Matt

-- 
The main advantages of Haynes and Chilton manuals are that they cost $15,
where the factory manuals cost $100 and up, and that they will tell you how
to use two hammers, a block of wood, and a meerkat to replace special tool
no. 2-112-A-- Matt Roberds in asr.

Re: HTTPS redirects to HTTP for monitoring

[Matt Palmer]

On Sun, Jan 18, 2015 at 08:05:18PM +, Kelly Setzer wrote:
 I don't know if you're referring to HSTS.

No, HSTS is separate to certificate pinning.  Certificate pinning would, in
fact, cause Chrome to freak out in the presence of an HTTPS-intercepting
proxy, but that's what it's supposed to do.  I doubt that organisations
regressive enough to do HTTPS-MitM would be enlightened enough to allow
Chrome to be installed, though.

 If not, it's worth noting in
 this thread.  As I understand HSTS, session decryption is still possible
 on sites that send the 'Strict-Transport-Security' header.  See:
 https://tools.ietf.org/html/rfc6797

Yes, HSTS allows interception; it would, on the other hand, prevent the
downgrade attack which the OP was suggesting as one option to allow
organisational monitoring of web requests and responses.

 I suspect it's only a matter of time before browsers become suspicious by
 default, requiring that HTTPS responses be signed and requiring that SSL
 certificates come from trusted sources.

That sounds like what has been the case since... forever.

 All of this points to the deficiency of the existing commercial
 certificate authority system.  The fact that organizations can easily
 purchase software specifically designed to subvert encrypted communication
 channels is proof that HTTPS security is an illusion.

What does the existence of a HTTPS proxy have to do with the deficiency of
existing CAs?  Yes, CAs have issued intermediate CA certificates to MitM
boxes (Trustwave has been caught doing it; I'm sure others have done it,
too).  However, the standard mechanism for doing this sort of thing is a
locally-issued root CA certificate, which is installed in the corporate SOE
as a trusted root.  That is, actually, *exactly* how the TLS certificate
system is supposed to work -- root CA certificate is marked as trusted, thus
everything issued therefrom is considered OK.

That this is possible is not proof that HTTPS security is an illusion;
it's simply another demonstration that if the bad guy has control over your
machine, it isn't your machine any more.  If TLS wasn't vulnerable to this
particular mode of subversion, I'm sure there'd be products out there that
would hook into the core of the browser and grab the requests before they
got into the encrypted channel and re-route them to the proxy, and it would
be that software, rather than the local root CA certificate, which would be
installed in the corporate SOE.

- Matt

Re: Tech Laptop with DB9

[Matt Palmer]

On Tue, Nov 11, 2014 at 12:34:23PM +0300, Stepan Kucherenko wrote:
 I want to reiterate on AirConsole because it IS amazing. I don't even
 grab a laptop when I go onsite anymore, just an AirConsole, its
 usb-serial cable and a tablet.

My, that *is* a rather snazzy piece of kit.  I'm almost sad that I don't
really deal with serial console kit any more.  *Almost*.

- Matt

-- 
Debian is geared towards building long-term stable systems; this really only
comes at the expense of newbie user-friendliness.  It's the same reason that
building a treehouse is easy, but building a steel-reinforced-concrete
bunker is hard. -- Don Werve

Re: Tech Laptop with DB9

[Matt Palmer]

On Mon, Nov 10, 2014 at 11:05:39AM -1000, joel jaeggli wrote:
 ftdi chipsets work on both mac and windows devices.

As long as it's FTDI and not FTDI...

- Matt

-- 
Once one has achieved full endarkenment, one is happy to have an entirely
nonfunctional computer
-- Steve VanDevender, ASR

Re: Tech Laptop with DB9

[Matt Palmer]

On Mon, Nov 10, 2014 at 12:57:49PM -0800, Max Clark wrote:
 On Mon, Nov 10, 2014 at 12:55 PM, Job Snijders j...@instituut.net wrote:
  Do you have a specific application that would prohibit the use of USB?
 
 It's purely for convenience and forgetfulness.

Cable ties.  They're my forget-me-not.

- Matt

-- 
Alas, slideware often reduces the analytical quality of presentations. In
particular, the popular PowerPoint templates (ready-made designs) usually
weaken verbal and spatial reasoning, and almost always corrupt statistical
analysis. -- http://www.edwardtufte.com/tufte/books_pp

Re: Tech Laptop with DB9

[Matt Palmer]

On Mon, Nov 10, 2014 at 03:15:38PM -0800, Kate Gerry wrote:
 The bonus about the adapter that I linked is that they use legit chips.

If only supply chain security were that easy.

- Matt

Re: DDOS, IDS, RTBH, and Rate limiting

[Matt Palmer]

On Sat, Nov 08, 2014 at 10:37:45PM -0500, Jon Lewis wrote:
 On Sun, 9 Nov 2014, Roland Dobbins wrote:
 But this kind of thing punishes the victim.  It's far better to do
 everything possible to *protect* the target(s) of an attack, and
 only use D/RTBH as a last resort.
 
 I'm sure it's not always the case, but in my experience as a SP, the
 victim virtually always did something to instigate the attack

Like have the temerity to have a successful online store.  Or be featured in
the mainstream media for providing information during a natural disaster. 
The bastards.  I've dealt with far more DDoS attacks that were for the
purposes of extortion or lulz than were due to the recipient instigating
the attack.  Perhaps that's a function of not attempting to cater to the
lowest common denominator.

- Matt

Re: Linux: concerns over systemd adoption and Debian's decision to switch [OT]

[Matt Palmer]

On Sat, Oct 25, 2014 at 01:55:43PM -0700, Matthew Petach wrote:
 On Sat, Oct 25, 2014 at 10:22 AM, Stephen Satchell l...@satchell.net
 wrote:
  Oh, and I hate binary logs.  Period.  If you can't stand plain text,
  then try XML.  At least humans have a *chance* to read it without having
  to make fancy reader tools.

 Completely agree on this point--but I fail
 to see why it has to be one or the other?
 Why can't systemd have a --text flag to
 tell it to output in ascii text mode for those
 of us who prefer it that way?

Because Lennart knows better than you.

- Matt

-- 
Igloo I remember going to my first tutorial in room 404. I was most upset
when I found it.

Re: Linux: concerns over systemd adoption and Debian's decision to switch [OT]

[Matt Palmer]

On Sat, Oct 25, 2014 at 02:41:55PM -0700, Peter Baldridge wrote:
 On Sat, Oct 25, 2014 at 1:55 PM, Matthew Petach mpet...@netflight.com wrote:
  Why can't systemd have a --text flag to
  tell it to output in ascii text mode for those
  of us who prefer it that way?

 ^
This | is not what that | does
v

 It does.
 
 --no-pager

Unless I'm grossly misreading the manpages that Google things are relevant.

- Matt

-- 
Just because we work at a University doesn't mean we're surrounded by smart
people.
-- Brian Kantor, in the monastery

Re: Linux: concerns over systemd adoption and Debian's decision to switch

[Matt Palmer]

On Wed, Oct 22, 2014 at 10:05:30PM -0500, Jeffrey Ollie wrote:
 To achieve the level of integration that timedated has with the rest
 of systemd would require more than just putting code into timedatectl
 to write out /etc/ntpd.conf and starting a service.  timedated talks
 to networkd (that
 DHCP server that everyone is hating on as well) in real-time to
 determine the state of the network and to get any NTP servers that
 were sent in DHCP packets.  To do that for chronyd or ntpd in the same
 way would require code changes and the systemd developers didn't want
 to do the work,

This is the core problem with systemd, in my mind -- and what has gotten
Linus, amongst other people, so thoroughly cheesed off with the systemd
devs.  They don't play well with other children.  They don't appear
particularly interested in reusing any existing code, because it's a lot
more fun to write new code.  I'm a strong proponent of Joel Spolsky's views
on rewrites (sorry, no URL, I'm on the train) and I don't doubt that all the
same problems will come to haunt systemd on its way from being the new kid
on the block to being legacy code[1].

- Matt

[1] A computer industry term which means, it works.

Re: Linux: concerns over systemd adoption and Debian's decision to switch

[Matt Palmer]

On Thu, Oct 23, 2014 at 12:12:13PM -0400, Jim Popovitch wrote:
 On Thu, Oct 23, 2014 at 12:04 PM, Mike. the.li...@mgm51.com wrote:
  GNU/Linux is morphing into GNU/systemd 
 
 Let's start calling it Systemd/Linux... that will get RMS on their
 case real fast. :-)

I don't think they've done anything to deserve *that*...

- Matt

-- 
I don't do veggies if I can help it.-- stevo
If you could see your colon, you'd be horrified.-- Iain Broadfoot
If he could see his colon, he'd be management.  -- David Scheidt

Re: Linux: concerns over systemd adoption and Debian's decision to switch

[Matt Palmer]

On Thu, Oct 23, 2014 at 10:56:40AM -0400, Randy wrote:
 I've enjoyed kernel hot patches (ksplice) until now.
 
 So my primary concern is that updates to systemd appears to require
 a full reboot:
 
 http://forums.fedoraforum.org/showthread.php?t=300166
 
 Is systemd really like a 2nd 'kernel' -- demanding mass reboots
 every time a security issue is discovered?

But it's OK, because it reboots so *fast*!

- Matt

-- 
[O]ne of the requirements for writing code in Python seems to be that one
spends as least as much time promoting the language as coding in it.
-- Matt Roberds, in the Monastery

Re: Linux: concerns over systemd adoption and Debian's decision to switch

[Matt Palmer]

On Thu, Oct 23, 2014 at 04:17:14PM -0400, Miles Fidelman wrote:
 Matt Palmer wrote:
 On Thu, Oct 23, 2014 at 12:12:13PM -0400, Jim Popovitch wrote:
 On Thu, Oct 23, 2014 at 12:04 PM, Mike. the.li...@mgm51.com wrote:
 GNU/Linux is morphing into GNU/systemd 
 Let's start calling it Systemd/Linux... that will get RMS on their
 case real fast. :-)
 I don't think they've done anything to deserve *that*...
 
 PulseAudio, systemd - yeah... maybe they have :-)

Of all the PoetteringStuff out there, PA's the one that's caused me the
least problems.  I know that I'm in some vanishingly small minority there,
because it seems to cause problems for everyone around me.  One of my
colleagues needs to jiggle the wires every time he wants to use gHangouts
because PA gets itself in a tizzy.

Now *Avahi*, on the other hand... FML.

- Matt

Re: Linux: concerns over systemd [OT]

[Matt Palmer]

On Wed, Oct 22, 2014 at 12:00:52PM +0100, Daniel Ankers wrote:
 On 22 October 2014 11:34, na...@jack.fr.eu.org wrote:
  Before leaving Debian, things to think:
  - will systemd be officialy the only system available ?
  - if so, won't we get a way to bypass that ?
 
 And one other thought... is it really that bad?
 
 Personally I like it a lot better than sysV plus inittab plus daemontools.

When it was init+daemontools, I could hold my nose over the binary logging
and consider using it.  Now it's taking over cron and all manner of other
things, there's no way in hell I'm letting it onto my systems.

As to the issue of will it only be systemd, the problem is that as the
officially-blessed option, that's the one that'll get the universal support,
so if you want to run something else, some things will mysteriously not
work, and package maintainers won't care nearly as much.

Bypassing systemd is a whole hell of a lot harder than switching out
sysvinit for something else, because systemd does so many things, and many
other things are being modified to absolutely depend on things that only
systemd provides -- GNOME's the big one, but docker is closely tied to
systemd too, I believe, I think udev needs systemd now (or has it been
incorporated into systemd?  I can't keep all this straight) and I'm pretty
sure I've heard of other things deprecating non-systemd ways of doing
things.

The *really* damaging part of it, though, is that as systemd grows
to overshadow the things it re-implements (udev, dbus, etc) it starves the
alternatives of light and they quickly fall behind and are no longer viable
as ways to avoid systemd.  That isn't systemd's fault, per se, but it does
make it much harder over time to avoid getting caught in the gaping maw.

- Matt

Re: Linux: concerns over systemd adoption and Debian's decision to switch

[Matt Palmer]

On Tue, Oct 21, 2014 at 09:40:30AM -0400, valdis.kletni...@vt.edu wrote:
 On Tue, 21 Oct 2014 14:44:57 +0900, Randy Bush said:
  systemd is insanity.  one would have hoped that deb and others would
  know better.  sigh.
 
 It started as a replacement init system.  I suspected it had jumped
 the shark when it sprouted an entirely new DHCP and NTP service.  And this
 was confirmed when I saw this:
 
 Leading up to this has been cursor rendering support, keyboard mapping
 support, screen renderer, DRM back-end, input interface, and dozens of other
 commits.
 
 http://www.phoronix.com/scan.php?page=news_itempx=MTgwNzQ
 
 When your init system is worrying about cursor rendering, you have truly
 fallen victim to severe feature bloat.  I guess Jamie Zawinski was right:
 Every program attempts to expand until it can read mail.

Ooooh...

/me submits a patch to systemd to provide localhost:25 and
//usr/sbin/sendmail emulation...

- Matt

-- 
The real art of conversation is not only to say the right thing at the
right place but to leave unsaid the wrong thing at the tempting
moment. -- Dorothy Nevill

Re: Linux: concerns over systemd adoption and Debian's decision to switch

[Matt Palmer]

On Tue, Oct 21, 2014 at 07:20:12PM -0500, Jimmy Hess wrote:
 On Tue, Oct 21, 2014 at 8:40 AM,  valdis.kletni...@vt.edu wrote:
 [snip]
  It started as a replacement init system.  I suspected it had jumped
  the shark when it sprouted an entirely new DHCP and NTP service.  And this
 
 Yikes.   What's next?   Built-in DNS server + LDAP/Hesiod + Kerberos +
 SMB/Active Directory  client and server + Solitaire + Network
 Neighborhood functionality built into the program ?

You missed font renderer. 
https://technet.microsoft.com/library/security/ms14-058

- Matt

-- 
A friend is someone you can call to help you move. A best friend is someone
you can call to help you move a body.

Re: wifi blocking [was Re: Marriott wifi blocking]

[Matt Palmer]

On Tue, Oct 07, 2014 at 09:36:26PM -0400, valdis.kletni...@vt.edu wrote:
 On Tue, 07 Oct 2014 20:10:44 -0500, Jimmy Hess said:
 
  The only way to legally block cell phone RF would likely be on behalf
  of the licensee   In other words, possibly, persuade the cell
  phone companies to allow this,   then  create an approved special
  local cell tower  all their phones in the same building will by
  default connect to  in preference to any other,  which will also  not
  receive any calls or messages   or allow any to be sent.
 
 I wonder how many customers the cell phone company will attract by doing that.

Getting paid by third parties to abuse your customers seems to be working
well for certain other industries.

- Matt

-- 
You keep using that word.  I do not think it means what you think it means.
-- Inigo, The Princess Bride

Re: large BCP38 compliance testing

[Matt Palmer]

On Fri, Oct 03, 2014 at 03:20:58PM -0400, Alain Hebert wrote:
 On the 1st of January 2015:

That's quite short notice.  Perhaps we could delay it by exactly three
months?

- Matt

Re: AWS EC2 us-west-2 reboot

[Matt Palmer]

On Wed, Oct 01, 2014 at 11:01:37AM -0700, Grant Ridder wrote:
 For those interested, this is the Xen bug they were fixing with the reboots
 http://xenbits.xen.org/xsa/advisory-108.html

Ouch.  Good thing Bashpocalypse is still capturing everyone's attention...

Interestingly, Amazon *didn't* discover this bug, which makes one wonder why
they, out of all the big Xen-based providers out there, got a heads-up in
advance of the embargo end.  If I was a big provider who didn't get advance
notice, I'd be somewhat miffed.

- Matt

-- 
If you are a trauma surgeon and someone dies on your table, [...] everyone
would know you did your best.  When someone does something truly stupid
with their system and it dies and you can't resuscitate it, you must be
incompetent or an idiot.  -- Julian Macassey, in the Monastery

Re: Scotland ccTLD?

[Matt Palmer]

On Wed, Sep 17, 2014 at 10:02:45AM +0200, Tei wrote:
 http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2#Decoding_table
 
 GO [...] seems to be free :D

1600 Amphitheatre Parkway... the newest independent state.

- Matt

Re: Scotland ccTLD?

[Matt Palmer]

On Tue, Sep 16, 2014 at 01:01:24PM -0400, Barry Shein wrote:
 .PC, for Picts (I believe it's available.) But I doubt that would fly.

They could abolish all taxes and fund the entire country just on domain name
sales.

 I don't know anything about Scotland's attitude toward being
 identified with the Picts, however. Perhaps that's a nonsensical idea.

They've always been a bit picty about that sort of thing.

- Matt

Re: Carrier Grade NAT

[Matt Palmer]

On Wed, Jul 30, 2014 at 08:05:28PM -0400, valdis.kletni...@vt.edu wrote:
 On Wed, 30 Jul 2014 16:39:14 -0700, Owen DeLong said:
 
  I was talking about Amazon, not AWS. Yes, AWS would help too, but in terms 
  of
  the Alexa list, Amazon would swing the percentage meaningfully. I don’t 
  know to
  what extent AWS would swing the percentage.
 
 There's probably not much stuff that individually is in the Alexa top 100, but
 collectively AWS probably has a half million or so hosted entities that
 together would end up at the bottom end of the Top 50 if not better.
 
 Of course, then the question becomes what percentage of those half million
 entities are ready to go once AWS flips the switch

Given that almost all of them will be using ELB, which is just a reverse
proxy, where AWS controls the A records that get returned, I'd say that most
of them would Just Work.  The ones that don't will fail only because they're
assuming that the IP address they get sent via HTTP header is IPv4, but
plenty of sites don't even look, and most of the rest wouldn't need much
more than a regex update and/or DB column size change.

- Matt

-- 
The real art of conversation is not only to say the right thing at the
right place but to leave unsaid the wrong thing at the tempting
moment. -- Dorothy Nevill

Re: Carrier Grade NAT

[Matt Palmer]

On Tue, Jul 29, 2014 at 11:42:31AM -0500, Chris Boyd wrote:
 On Jul 29, 2014, at 10:23 AM, Mikael Abrahamsson wrote:
  If law enforcement comes along without port numbers then you give them a
  list of subscribers behind that IP at the time.  Use port block
  allocation and keep track of the blocks to reduce logging load.
 
 There's probably going to be some interesting legal fallout from that
 practice.  As an ISP customer, I'd be furious to find out that my
 communications had been intercepted due to the bad behavior of another
 user.

Then you'll no doubt be happy to know that you're very, very unlikely to
ever find out.

- Matt

Re: Many players make up application performance (was Re: Richard Bennett, NANOG posting, and Integrity)

[Matt Palmer]

On Tue, Jul 29, 2014 at 05:25:47PM +, McElearney, Kevin wrote:
 Performance is a two way street (as are shakedowns)

It takes two to lie, Marge: one to lie, and one to listen.

- Matt

Re: Carrier Grade NAT

[Matt Palmer]

On Tue, Jul 29, 2014 at 06:19:31PM -0400, Lee Howard wrote:
 Thanks for sharing your experience; it's very unusual to get the
 perspective of an operator running CGN (on a broadband ISP; wireless has
 always had it).
 
 On 7/29/14 5:28 PM, Tony Wicks t...@wicks.co.nz wrote:
 
 OK, as someone with experience running CGNAT to fixed broadband customers
 in
 general, here are a few answers to common questions. This is based on the
 setup I use which is CGNAT is done on the BNG (Cisco ASR1K6).
 
 1. APNIC ran out of IPv4 a couple of years ago, so unless you want to pay
 USD $10+ per IP then CGNAT is the only option.
 
 Eh, a bit over US$7 now, but whatever. Higher in APNIC.
 
 2. IPv6 is nice (dual stack) but the internet without IPv4 is not a viable
 thing, perhaps one day, but certainly not today (I really hate clueless
 people who shout to the hills that IPv6 is the solution for today's
 internet access)
 
 It's viable, it's just not a substitute for IPv4 yet.
 Except for specific scenarios.  For instance, you mention gaming below; if
 two users are playing on Xbox ONE, they can use IPv6 and they're off the
 CGN.  Or if a bank has blacklisted an IPv4 address on the CGN, but the
 bank is dual-stack, some users can still get there.
 Of course, that snowballs.
 
 3. 99.99% of customers don't notice they are transiting CGNAT, it just
 works.
 
 Surprised it's that high.
 
 4. You need to log NAT translations for LI purposes. (IP
 source/destination,
 Port source/destination, time) Surprisingly this does not produce that
 big a
 database burden. However as Cisco's Netflow NAT logging is utterly useless
 you need to use syslog and this ramps up the ASR CPU a bit.
 
 Can you quantify?
 The log entry has to be at least:
 32 bits   source address
 16 bits source port
 32 bits destination address
 16 bits destination port
 64 bits? timestamp
 ---
 160 bits = 20 bytes per flow
 You have to log the end of the flow, too, right?  Another 20 bytes?
 40 bytes per flow.  Not including syslog severity and message text.

You can get it down a bit smaller, if you're OK with having to find the
records again to update them at the end of the connection (either TCP FIN,
or UDP mapping timeout):

32 bits NAT endpoint ip
16 bits NAT endpoint port
32 bits dest ip
16 bits dest port
32 bits start timestamp
32 bits end timestamp
16 bits customer ID (you could store the customer's internal IP, but that's
bigger)

That's 22 bytes per flow (maybe 24 if you're planning on having more than
64ki customers in your CGNAT's lifetime).

You could drop the timestamps by another 16 bits each if you don't mind
reducing granularity (if you guarantee you won't reuse a given IP/port pair
for, say, 30 seconds, you can define the timestamp to be, say, 15 second
increments) and/or changing the epoch -- 15 second granularity + rolling
epoch every week = 16 bit timestamps do just fine.

 As I recall, a site like cnn.com opens 80 flows, so 3200 bytes of log data.
 If, as you say in #6, 10,000 customers = 200,000 active translations,
 that's 8,000,000 bytes of syslog. . . per second?  Not sure if active
 indicates how fast those sessions churn.
 180 days of log retention would be. . . 124TB of data.  Per 10,000 users.

Of course, getting anything back *out* of that again in any sort of
reasonable timeframe would be... optimistic.  I suppose if you're storing it
all in hadoop you can map/reduce your way out of trouble, but that's going
to mean a lot of equipment sitting around doing nothing for 99.99% of the
time.  Perhaps mine litecoin between searches?

 7. CGNAT protects your customers from all sorts of nasty's like small DDOS
 attacks and attacks on their crappy CPE
 8. DDOS on CGNAT pool IP's are a pain in the rear and happen often.
 
 Between #7 and #8, do they balance out?

I'd doubt it.  A customer getting DDoS'd counts against their usage limit;
you can't bill traffic pointed at a CGNAT address against any particular
customer.  grin

- Matt

-- 
If only more employers realized that people join companies, but leave
bosses. A boss should be an insulator, not a conductor or an amplifier.
-- Geoff Kinnel, in the Monastery