Re: Apple devices spoofing default gateway?

[Curtis, Bruce]

We are running 8.5 and 1815s and I don’t think we are seeing this problem.

We do have a very small number of 1810s and did see some strange behavior but 
it doesn’t seem to match this problem description.

Is proxy arp disabled on the default gateway device?  That could potentially 
interact strangely with the features mentioned in earlier posts and mentioned 
below.

> On Mar 14, 2019, at 4:40 PM, Simon Lockhart  wrote:
> 
> On Thu Mar 14, 2019 at 04:19:04PM -0500, Jimmy Hess wrote:
>> Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy
>> for Wake on Demand ---  When a device goes to sleep,  the Proxy that runs on
>> various Apple devices is supposed to seize all the IP and MAC addresses that
>> device had registered, so it can wait for an incoming TCP SYN, (and if one's
>> received,  then signal the sleeping device to wake up and process the
>> connection.)
> 
> That's a very interesting observation - when we talk to the users of the
> Apple devices, they quite often say that the device was 'asleep' when it
> was sending these 'spoofed' ARP responses.

The "Information About Passive Clients” section of this document

https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-5/config-guide/b_cg85/wlan_interfaces.html

says:

"Wireless LAN controllers currently act as a proxy for ARP requests. Upon 
receiving an ARP request, the controller responds with an ARP response instead 
of passing the request directly to the client. This scenario has two advantages:

• The upstream device that sends out the ARP request to the client will 
not know where the client is located.

• Power for battery-operated devices such as mobile phones and printers 
is preserved because they do not have to respond to every ARP requests."


  Perhaps that function on version 8.5 is interacting incorrectly with the 
Apple Sleep Proxy feature on the Apple devices.

"When a sleep proxy sees an IPv4 ARP or IPv6 ND Request for one of the sleeping 
device's addresses, it answers on behalf of the sleeping device, without waking 
it up, giving its own MAC address as the current (temporary) owner of that 
address.”

https://en.wikipedia.org/wiki/Bonjour_Sleep_Proxy

https://discussions.apple.com/thread/2160614

> 
>> (Or perhaps they wanted to have a feature to let someone  AirPlay from a
>> different VLAN than another device?)
> 
> Cisco Wireless does claim to have some features to 'help' Bonjour / mDNS
> to work better. I wonder if one of those features is misbehaving.
> 
> Simon


---
Bruce Curtis bruce.cur...@ndsu.edu
Certified NetAnalyst II701-231-8527
North Dakota State University

Re: Analysing traffic in context of rejecting RPKI invalids

[Jakob Heitz (jheitz) via NANOG]

If at least one ROA matches a route, then the route is valid.
This is to cover the case when more than one AS is authorized to
originate a particular prefix.

https://tools.ietf.org/html/rfc6811
Page 5:
   o  NotFound: No VRP Covers the Route Prefix.

   o  Valid: At least one VRP Matches the Route Prefix.

   o  Invalid: At least one VRP Covers the Route Prefix, but no VRP
  Matches it.

BTW, this rule allows you to issue a ROA authorizing AS0 to originate
your complete address space.

Why would you do that? Suppose you own an address space, but you only
want to announce a portion of it to the internet. If you issue a ROA
for the unannounced portion authorizing your own ASN, then an attacker
can announce that portion and prepend your ASN. The attacker can thus
hijack your unannounced space and appear valid by RPKI!

To prevent that, you issue a ROA for your complete address space
authorizing AS0 and your BGP announced space authorizing your own ASN.

An AS path containing AS0 is malformed, being treated as withdraw.
https://tools.ietf.org/html/rfc7607

Regards,
Jakob.

-Original Message-
Date: Wed, 13 Mar 2019 11:17:22 -0400
From: Steve Meuse 
>
>
> Thanks for the update, but based on that description I'm not certain
> that you implemented the same thing that pmacct built, which IMO is
> what is needed by those considering deploying a drop-invalids policy.
> (Perhaps you omitted mentioning that ability in your description but
> included it in your implementation.)
>
>
Thanks Jay, you are correct. As we were talking through the logic we
realized we missed that bit. Internally, we're working though the logic to
understand if there is a covering route, is that route valid, and if not,
will we recurse and look for another covering route that is valid?

Either way, we'll be updating our software with that functionality shortly.

-Steve

Re: FB?

[Robert Webb]

No one looks at dates on Facebook posts.

On Thu, Mar 14, 2019, 17:10 Luke Guillory  wrote:

> That’s old.
>
>
>
> By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM
>
>
>
>
>
> Luke
>
>
>
> Ns
>
>
>
>
>
>
>
> *From:* NANOG [mailto:nanog-boun...@nanog.org] *On Behalf Of *Selphie
> Keller
> *Sent:* Thursday, March 14, 2019 4:06 PM
> *To:* Mike Hammett
> *Cc:* NANOG list
> *Subject:* Re: FB?
>
>
>
> I did see this article indicating they had somehow invalidated their cache
> in a botched deployment of changes -
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>
>
>
> On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>
> So what happened at Facebook today? I saw one article quoting Roland
> saying it was a route leak, but I haven't seen any other sources that
> aren't just quoting Roland. Usually there are a few independent posts out
> there by now.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>
>
>

Re: Apple devices spoofing default gateway?

[Simon Lockhart]

On Thu Mar 14, 2019 at 04:19:04PM -0500, Jimmy Hess wrote:
> Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy
> for Wake on Demand ---  When a device goes to sleep,  the Proxy that runs on
> various Apple devices is supposed to seize all the IP and MAC addresses that
> device had registered, so it can wait for an incoming TCP SYN, (and if one's
> received,  then signal the sleeping device to wake up and process the
> connection.)

That's a very interesting observation - when we talk to the users of the
Apple devices, they quite often say that the device was 'asleep' when it
was sending these 'spoofed' ARP responses.

> (Or perhaps they wanted to have a feature to let someone  AirPlay from a
> different VLAN than another device?)

Cisco Wireless does claim to have some features to 'help' Bonjour / mDNS
to work better. I wonder if one of those features is misbehaving.

Simon

Re: FB?

[Ross Tajvar]

The cache invalidation thing is incorrect according to an Facebook SWE I
talked to. He wouldn't tell me what it actually was though, basically
saying "you have to know our infrastructure to understand and I can't tell
you that."

On Thu, Mar 14, 2019, 5:28 PM cosmo  wrote:

> Yes, evidently someone screenshotted it and it was making the rounds on
> social media this morning, sans the date.
>
> So now back to other theories.
>
> On Thu, Mar 14, 2019 at 2:16 PM Jeff Shultz 
> wrote:
>
>> The date on that is 2010.
>>
>> On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller 
>> wrote:
>> >
>> > I did see this article indicating they had somehow invalidated their
>> cache in a botched deployment of changes -
>> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>> >
>> > On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>> >>
>> >> So what happened at Facebook today? I saw one article quoting Roland
>> saying it was a route leak, but I haven't seen any other sources that
>> aren't just quoting Roland. Usually there are a few independent posts out
>> there by now.
>> >>
>> >>
>> >>
>> >> -
>> >> Mike Hammett
>> >> Intelligent Computing Solutions
>> >> http://www.ics-il.com
>> >>
>> >> Midwest-IX
>> >> http://www.midwest-ix.com
>> >>
>>
>>
>> --
>> Jeff Shultz
>> Central Office Technician
>> SCTC
>> (503) 769-2125
>> Go Big  Ask for Gig
>>
>> --
>> Like us on Social Media for News, Promotions, and other information!!
>>
>>
>> 
>> 
>> 
>> 
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _ This message
>> contains confidential information and is intended only for the individual
>> named. If you are not the named addressee you should not disseminate,
>> distribute or copy this e-mail. Please notify the sender immediately by
>> e-mail if you have received this e-mail by mistake and delete this e-mail
>> from your system. E-mail transmission cannot be guaranteed to be secure
>> or
>> error-free as information could be intercepted, corrupted, lost,
>> destroyed,
>> arrive late or incomplete, or contain viruses. The sender therefore does
>> not accept liability for any errors or omissions in the contents of this
>> message, which arise as a result of e-mail transmission. _
>>
>>

Re: Apple devices spoofing default gateway?

[Jimmy Hess]

On Thu, Mar 14, 2019 at 7:29 AM Simon Lockhart  wrote:

> Apple devices, but what's more strange is that we're only seeing it where
> those Apple devices are connected to Cisco 1810 and 1815 APs, and where those
> APs are connected to a Cisco WLC running v8.5 software. If we downgrade the
> WLC to v8.2 the problem goes away (but v8.2 doesn't support 1815 APs, so we

Apple's Bonjour protocols include something called Apple Bonjour Sleep Proxy
for Wake on Demand ---  When a device goes to sleep,  the Proxy that
runs on various
Apple devices is supposed to seize all the IP and MAC addresses that
device had registered,
so it can wait for an incoming TCP SYN, (and if one's received,  then
signal the
sleeping device to wake up and process the connection.)

Bonjour and the related mDNS protocols used for AirPlay/AirPrint/etc
are built on Link-Local
multicast.   I wonder what would happen if  some random Wireless LAN
controllers malfunctioned,
and decided that it would like to ignore that Link-Local restriction
and proxy those packets b/w
subnets anyways, as if they had been unrestricted multicast or
Unicast,   Possibly with the
source IP address on registration Mangled to or  "gateway'd"  from the
 router's  IP address.

(Or perhaps they wanted to have a feature to let someone  AirPlay from
a different VLAN than
another device?)

Either way  playing around with the source IP address on the
Link-local m/c packets
might accidentally  get them a  Default Gateway IP address  registered
with other workstations
as a mobile device that's gone to sleep and needs a proxy.

-- 
-JH

Re: FB?

[cosmo]

Yes, evidently someone screenshotted it and it was making the rounds on
social media this morning, sans the date.

So now back to other theories.

On Thu, Mar 14, 2019 at 2:16 PM Jeff Shultz  wrote:

> The date on that is 2010.
>
> On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller 
> wrote:
> >
> > I did see this article indicating they had somehow invalidated their
> cache in a botched deployment of changes -
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
> >
> > On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
> >>
> >> So what happened at Facebook today? I saw one article quoting Roland
> saying it was a route leak, but I haven't seen any other sources that
> aren't just quoting Roland. Usually there are a few independent posts out
> there by now.
> >>
> >>
> >>
> >> -
> >> Mike Hammett
> >> Intelligent Computing Solutions
> >> http://www.ics-il.com
> >>
> >> Midwest-IX
> >> http://www.midwest-ix.com
> >>
>
>
> --
> Jeff Shultz
> Central Office Technician
> SCTC
> (503) 769-2125
> Go Big  Ask for Gig
>
> --
> Like us on Social Media for News, Promotions, and other information!!
>
>
> 
> 
> 
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
> _ This message
> contains confidential information and is intended only for the individual
> named. If you are not the named addressee you should not disseminate,
> distribute or copy this e-mail. Please notify the sender immediately by
> e-mail if you have received this e-mail by mistake and delete this e-mail
> from your system. E-mail transmission cannot be guaranteed to be secure or
> error-free as information could be intercepted, corrupted, lost,
> destroyed,
> arrive late or incomplete, or contain viruses. The sender therefore does
> not accept liability for any errors or omissions in the contents of this
> message, which arise as a result of e-mail transmission. _
>
>

Re: FB?

[cosmo]

Looks like Google recently posted their post-mortem of their outage on the
12th
https://status.cloud.google.com/incident/storage/19002

On Thu, Mar 14, 2019 at 1:21 PM Suresh Ramasubramanian 
wrote:

> That's a 2010 outage that someone dug out and was doing the rounds as a
> new one
>
> --srs
>
> --
> *From:* NANOG  on behalf of cosmo <
> clinton.mie...@gmail.com>
> *Sent:* Thursday, March 14, 2019 9:50 PM
> *To:* Bryan Holloway
> *Cc:* nanog@nanog.org
> *Subject:* Re: FB?
>
> Facebook pushed an update to their code that manages cookies, that had a
> rather severe bug in it that resulted in a large flood of requests to their
> database servers. To deal with this load, they had to prevent all writes
> and then slowly allow people back on.
>
> I saw the writeup for it last night but cannot seem to find it now! Grrr.
> Did I dream it?
>
> On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway  wrote:
>
>>
>> On 3/14/19 9:06 AM, Tom Beecher wrote:
>> > As much as I wanted to crack jokes because I cannot stand Facebook (the
>> > product), much love to all you FB engineers that went through (and are
>> > probably still going through) much hell.
>> >
>>
>> +1 on both counts.
>>
>> We've all been there; no bueno.
>>
>

RE: FB?

[Ray Van Dolson]

https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_facebook_status_1106229690069442560&d=DwIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho&s=zrKUWVShQdFllKTGbJE5kITG87q7KNJHo0bD6aETBBk&e=

From: NANOG  On Behalf Of Luke Guillory
Sent: Thursday, March 14, 2019 2:09 PM
To: Selphie Keller ; Mike Hammett 
Cc: NANOG list 
Subject: RE: FB?

That’s old.

By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM


Luke

Ns




From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Selphie Keller
Sent: Thursday, March 14, 2019 4:06 PM
To: Mike Hammett
Cc: NANOG list
Subject: Re: FB?

I did see this article indicating they had somehow invalidated their cache in a 
botched deployment of changes - 
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.facebook.com_notes_facebook-2Dengineering_more-2Ddetails-2Don-2Dtodays-2Doutage_431441338919_&d=DwIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho&s=1EmIo8GEivgILxC4jZzEdBpYWt5R9CZ5cXhtr6i55rc&e=

On Thu, 14 Mar 2019 at 06:18, Mike Hammett 
mailto:na...@ics-il.net>> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it 
was a route leak, but I haven't seen any other sources that aren't just quoting 
Roland. Usually there are a few independent posts out there by now.


-
Mike Hammett
Intelligent Computing Solutions
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.ics-2Dil.com&d=DwIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho&s=EAZZC6r_-2rdFCKgq9XpQy30F7OH79M6sZPNvXq0FPA&e=

Midwest-IX
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.midwest-2Dix.com&d=DwIGaQ&c=n6-cguzQvX_tUIrZOS_4Og&r=r4NBNYp4yEcJxC11Po5I-w&m=IHR1veHNjVYVktL31OQ_tgBUNHO5Uf3ACrvIVAW5cho&s=fEoBWpTXgY7eXczzc8vo7VHbvopKqDWk6Xz2XYutL0k&e=

Re: FB?

[Selphie Keller]

Yeah I just saw that date and that is odd, I got the link yesterday from
somewhere and didn't notice the date was old.

They do mention the configuration change issue in this one though that is
dated today 14th -
https://www.cbsnews.com/news/facebook-instagram-down-wednesday-facebook-blames-server-configuration-for-longest-ever-outage/

>From my understanding yesterday is they invalidated their world cache and
all the traffic hit their backend quickly overwhelming their servers. It
was strange that they didn't really talk about the issue just some brief
messages on their twitter saying it wasn't ddos.




On Thu, 14 Mar 2019 at 15:08, Luke Guillory 
wrote:

> That’s old.
>
>
>
> By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM
>
>
>
>
>
> Luke
>
>
>
> Ns
>
>
>
>
>
>
>
> *From:* NANOG [mailto:nanog-boun...@nanog.org] *On Behalf Of *Selphie
> Keller
> *Sent:* Thursday, March 14, 2019 4:06 PM
> *To:* Mike Hammett
> *Cc:* NANOG list
> *Subject:* Re: FB?
>
>
>
> I did see this article indicating they had somehow invalidated their cache
> in a botched deployment of changes -
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>
>
>
> On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>
> So what happened at Facebook today? I saw one article quoting Roland
> saying it was a route leak, but I haven't seen any other sources that
> aren't just quoting Roland. Usually there are a few independent posts out
> there by now.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>
>
>

Re: FB?

[Jeff Shultz]

The date on that is 2010.

On Thu, Mar 14, 2019 at 2:07 PM Selphie Keller  wrote:
>
> I did see this article indicating they had somehow invalidated their cache in 
> a botched deployment of changes - 
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>
> On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>>
>> So what happened at Facebook today? I saw one article quoting Roland saying 
>> it was a route leak, but I haven't seen any other sources that aren't just 
>> quoting Roland. Usually there are a few independent posts out there by now.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>


-- 
Jeff Shultz
Central Office Technician
SCTC
(503) 769-2125
Go Big  Ask for Gig

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: FB?

[cosmo]

Ah-ha, that is indeed the write-up I saw. 8 years old!


On Thu, Mar 14, 2019, 2:07 PM Selphie Keller 
wrote:

> I did see this article indicating they had somehow invalidated their cache
> in a botched deployment of changes -
> https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/
>
> On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:
>
>> So what happened at Facebook today? I saw one article quoting Roland
>> saying it was a route leak, but I haven't seen any other sources that
>> aren't just quoting Roland. Usually there are a few independent posts out
>> there by now.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>>

RE: FB?

[Luke Guillory]

That’s old.

By Robert Johnson on Thursday, September 23, 2010 at 7:29 PM


Luke

Ns




From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Selphie Keller
Sent: Thursday, March 14, 2019 4:06 PM
To: Mike Hammett
Cc: NANOG list
Subject: Re: FB?

I did see this article indicating they had somehow invalidated their cache in a 
botched deployment of changes - 
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/

On Thu, 14 Mar 2019 at 06:18, Mike Hammett 
mailto:na...@ics-il.net>> wrote:
So what happened at Facebook today? I saw one article quoting Roland saying it 
was a route leak, but I haven't seen any other sources that aren't just quoting 
Roland. Usually there are a few independent posts out there by now.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Re: FB?

[Selphie Keller]

I did see this article indicating they had somehow invalidated their cache
in a botched deployment of changes -
https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919/

On Thu, 14 Mar 2019 at 06:18, Mike Hammett  wrote:

> So what happened at Facebook today? I saw one article quoting Roland
> saying it was a route leak, but I haven't seen any other sources that
> aren't just quoting Roland. Usually there are a few independent posts out
> there by now.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>

Re: FB?

[Suresh Ramasubramanian]

That's a 2010 outage that someone dug out and was doing the rounds as a new one

--srs


From: NANOG  on behalf of cosmo 

Sent: Thursday, March 14, 2019 9:50 PM
To: Bryan Holloway
Cc: nanog@nanog.org
Subject: Re: FB?

Facebook pushed an update to their code that manages cookies, that had a rather 
severe bug in it that resulted in a large flood of requests to their database 
servers. To deal with this load, they had to prevent all writes and then slowly 
allow people back on.

I saw the writeup for it last night but cannot seem to find it now! Grrr. Did I 
dream it?

On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway 
mailto:br...@shout.net>> wrote:

On 3/14/19 9:06 AM, Tom Beecher wrote:
> As much as I wanted to crack jokes because I cannot stand Facebook (the
> product), much love to all you FB engineers that went through (and are
> probably still going through) much hell.
>

+1 on both counts.

We've all been there; no bueno.

Re: Apple devices spoofing default gateway?

[J. Hellenthal via NANOG]

Right on!

https://www.tracewrangler.com/

> On Mar 14, 2019, at 13:13, Mel Beckman  wrote:
> 
> You asked if anyone else has seen this. It’s possibly going on in other 
> networks but nobody is noticing. What symptoms brought the problem to your 
> attention?
> 
> You can sanitize the packet captures by limiting them to just the headers. 
> The payloads are likely not useful for troubleshooting anyway, since this 
> seems to be a Layer 2 problem. You asked for help, and sanitized packets 
> would help people help you :)
> 
> -mel
> 
>> On Mar 14, 2019, at 10:02 AM, Simon Lockhart  wrote:
>> 
>> On Thu Mar 14, 2019 at 12:53:01PM +, Mel Beckman wrote:
>>> Can you post some packet captures? 
>> 
>> I have some packet captures, but as they're from a live network, I'd rather
>> not post them publicly.
>> 
>>> I was a network engineer on the WiFi network at SFO, for both passengers and
>>> baggage scanners, with several hundred APs. Several times we were misled by
>>> packet captures that seemed to show client traffic causing network problems,
>>> such as packet storms, but which ultimately always had some more mundane
>>> cause, like a failed DHCP server or flapping switch interface. 
>> 
>> Sure - we're rattling every possible other cause we can think of, including
>> using alternative DHCP server software vendor, etc. The only thing that's
>> reliably making the problem go away is running the APs against WLC version 
>> 8.2.
>> 
>>> The particular SFO network I worked on has Juniper switching and Aruba APs,
>>> so it???s not directly applicable to your ecosystem. But the complexities of
>>> interpreting packet captures may apply.
>> 
>> I'm the sort of person who has copies of RFCs printed out on his desk. I'm 
>> fairly experienced at interpreting packet captures :)
>> 
>> Simon
> 


— 

J. Hellenthal

The fact that there's a highway to Hell but only a stairway to Heaven says a 
lot about anticipated traffic volume.

Re: Apple devices spoofing default gateway?

[Mel Beckman]

You asked if anyone else has seen this. It’s possibly going on in other 
networks but nobody is noticing. What symptoms brought the problem to your 
attention?

You can sanitize the packet captures by limiting them to just the headers. The 
payloads are likely not useful for troubleshooting anyway, since this seems to 
be a Layer 2 problem. You asked for help, and sanitized packets would help 
people help you :)

 -mel

> On Mar 14, 2019, at 10:02 AM, Simon Lockhart  wrote:
> 
> On Thu Mar 14, 2019 at 12:53:01PM +, Mel Beckman wrote:
>> Can you post some packet captures? 
> 
> I have some packet captures, but as they're from a live network, I'd rather
> not post them publicly.
> 
>> I was a network engineer on the WiFi network at SFO, for both passengers and
>> baggage scanners, with several hundred APs. Several times we were misled by
>> packet captures that seemed to show client traffic causing network problems,
>> such as packet storms, but which ultimately always had some more mundane
>> cause, like a failed DHCP server or flapping switch interface. 
> 
> Sure - we're rattling every possible other cause we can think of, including
> using alternative DHCP server software vendor, etc. The only thing that's
> reliably making the problem go away is running the APs against WLC version 
> 8.2.
> 
>> The particular SFO network I worked on has Juniper switching and Aruba APs,
>> so it???s not directly applicable to your ecosystem. But the complexities of
>> interpreting packet captures may apply.
> 
> I'm the sort of person who has copies of RFCs printed out on his desk. I'm 
> fairly experienced at interpreting packet captures :)
> 
> Simon

Re: Apple devices spoofing default gateway?

[Simon Lockhart]

On Thu Mar 14, 2019 at 12:53:01PM +, Mel Beckman wrote:
> Can you post some packet captures? 

I have some packet captures, but as they're from a live network, I'd rather
not post them publicly.
 
> I was a network engineer on the WiFi network at SFO, for both passengers and
> baggage scanners, with several hundred APs. Several times we were misled by
> packet captures that seemed to show client traffic causing network problems,
> such as packet storms, but which ultimately always had some more mundane
> cause, like a failed DHCP server or flapping switch interface. 

Sure - we're rattling every possible other cause we can think of, including
using alternative DHCP server software vendor, etc. The only thing that's
reliably making the problem go away is running the APs against WLC version 8.2.
 
> The particular SFO network I worked on has Juniper switching and Aruba APs,
> so it???s not directly applicable to your ecosystem. But the complexities of
> interpreting packet captures may apply.

I'm the sort of person who has copies of RFCs printed out on his desk. I'm 
fairly experienced at interpreting packet captures :)

Simon

Re: FB?

[cosmo]

Facebook pushed an update to their code that manages cookies, that had a
rather severe bug in it that resulted in a large flood of requests to their
database servers. To deal with this load, they had to prevent all writes
and then slowly allow people back on.

I saw the writeup for it last night but cannot seem to find it now! Grrr.
Did I dream it?

On Thu, Mar 14, 2019 at 8:42 AM Bryan Holloway  wrote:

>
> On 3/14/19 9:06 AM, Tom Beecher wrote:
> > As much as I wanted to crack jokes because I cannot stand Facebook (the
> > product), much love to all you FB engineers that went through (and are
> > probably still going through) much hell.
> >
>
> +1 on both counts.
>
> We've all been there; no bueno.
>

Re: FB?

[Bryan Holloway]

On 3/14/19 9:06 AM, Tom Beecher wrote:
As much as I wanted to crack jokes because I cannot stand Facebook (the 
product), much love to all you FB engineers that went through (and are 
probably still going through) much hell.




+1 on both counts.

We've all been there; no bueno.

Re: FB? / AS 200020 leak

[Job Snijders]

Hi,

On Thu, Mar 14, 2019 at 02:04:39PM +, Jeroen Wunnink wrote:
> The route-leak was something different that seems to have mainly hit
> west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in
> the *NOG communities still digging at the complete details of that
> right now, but it currently points to have originated from AS200020,
> impacting a few large upstreams for a short period of time.

Here are some details of prefixes affected (courtesy of Doug Madory).
The percent at the beginning is the percentage of the peering sources
that saw each prefix leaked. Last column is the AS_PATH

March 14th, 2019 - 16:43 UTC was the start of the BGP leak incident.

The leak was very serious in terms of negative impact, it affected many
West European access providers (for instance AS 1136 has over 50% of
Dutch access market).

Kind regards,

Job

70.6% 92.68.0.0/14 KPN B.V. NL ... 200020 1136
70.6% 92.64.0.0/14 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 1136
70.4% 93.154.64.0/18 KPN B.V. NL ... 200020 1136
70.4% 93.154.0.0/18 KPN B.V. NL ... 200020 1136
70.4% 86.88.0.0/13 KPN B.V. NL ... 200020 1136
70.4% 86.80.0.0/13 KPN B.V. NL ... 200020 1136
70.4% 84.84.0.0/14 KPN B.V. NL ... 200020 1136
70.4% 84.80.0.0/14 KPN B.V. NL ... 200020 1136
70.4% 81.206.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 81.204.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 80.61.0.0/16 Customers NL ... 200020 1136
70.4% 80.60.0.0/16 Customers NL ... 200020 1136
70.4% 77.62.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 77.60.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 77.170.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 77.168.0.0/15 KPN B.V. NL ... 200020 1136
70.4% 77.164.0.0/14 KPN B.V. NL ... 200020 1136
70.4% 77.160.0.0/14 KPN B.V. NL ... 200020 1136
70.4% 62.12.0.0/20 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 1136
70.4% 46.145.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 46.144.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 31.161.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 31.160.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 145.7.128.0/17 KPN B.V. NL ... 200020 1136
70.4% 145.133.0.0/16 KPN B.V. NL ... 200020 1136
70.4% 145.132.0.0/16 KPN B.V. NL ... 200020 1136
70.1% 89.200.64.0/18 KPN Mobile The Netherlands B.V. NL ... 200020 1136
70.1% 89.200.0.0/18 KPN Mobile The Netherlands B.V. NL ... 200020 1136
70.1% 83.232.32.0/19 KPN Mobile The Netherlands B.V. NL ... 200020 1136
70.1% 83.232.128.0/17 KPN B.V. NL ... 200020 1136
70.1% 83.232.0.0/19 KPN Mobile The Netherlands B.V. NL ... 200020 1136
70.1% 83.232.0.0/17 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 82.171.96.0/19 Customers NL ... 200020 1136
70.1% 82.171.64.0/19 Customers NL ... 200020 1136
70.1% 82.171.32.0/19 Customers NL ... 200020 1136
70.1% 82.171.192.0/18 Customers NL ... 200020 1136
70.1% 82.171.128.0/18 Customers NL ... 200020 1136
70.1% 82.171.0.0/19 Customers NL ... 200020 1136
70.1% 82.170.128.0/17 Customers NL ... 200020 1136
70.1% 82.170.0.0/17 Customers NL ... 200020 1136
70.1% 82.169.96.0/20 Customers NL ... 200020 1136
70.1% 82.169.80.0/20 Customers NL ... 200020 1136
70.1% 82.169.64.0/20 Customers NL ... 200020 1136
70.1% 82.169.32.0/19 Customers NL ... 200020 1136
70.1% 82.169.224.0/19 Customers NL ... 200020 1136
70.1% 82.169.192.0/19 Customers NL ... 200020 1136
70.1% 82.169.176.0/20 Customers NL ... 200020 1136
70.1% 82.169.160.0/20 Customers NL ... 200020 1136
70.1% 82.169.144.0/20 Customers NL ... 200020 1136
70.1% 82.169.128.0/20 Customers NL ... 200020 1136
70.1% 82.169.112.0/20 Customers NL ... 200020 1136
70.1% 82.169.0.0/19 Customers NL ... 200020 1136
70.1% 82.168.64.0/18 Customers NL ... 200020 1136
70.1% 82.168.240.0/20 Customers NL ... 200020 1136
70.1% 82.168.224.0/20 Customers NL ... 200020 1136
70.1% 82.168.208.0/20 Customers NL ... 200020 1136
70.1% 82.168.192.0/20 Customers NL ... 200020 1136
70.1% 82.168.160.0/19 Customers NL ... 200020 1136
70.1% 82.168.128.0/19 Customers NL ... 200020 1136
70.1% 82.168.0.0/18 Customers NL ... 200020 1136
70.1% 82.136.224.0/19 KPN B.V. NL ... 200020 1136
70.1% 82.136.192.0/19 KPN B.V. NL ... 200020 1136
70.1% 80.60.224.0/20 Customers Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 80.60.224.0/19 Customers Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 77.173.128.0/17 Customers NL ... 200020 1136
70.1% 77.173.0.0/17 Customers NL ... 200020 1136
70.1% 77.172.128.0/17 Customers NL ... 200020 1136
70.1% 77.172.0.0/17 Customers NL ... 200020 1136
70.1% 62.41.128.0/17 KPN B.V. NL ... 200020 1136
70.1% 62.41.0.0/17 KPN B.V. NL ... 200020 1136
70.1% 62.25.32.0/19 KPN B.V. NL ... 200020 1136
70.1% 62.25.0.0/19 KPN B.V. NL ... 200020 1136
70.1% 62.21.192.0/18 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 62.21.128.0/18 KPN B.V. NL ... 200020 1136
70.1% 62.207.128.0/17 KPN B.V. NL ... 200020 1136
70.1% 62.207.0.0/17 KPN B.V. Amsterdam Provincie Noord-Holland NL ... 200020 
1136
70.1% 62.133.96.0/19 KPN Mobile The Netherlands B.V. NL ... 200020

Re: FB? - route leak AS200020

[Pim van Stam]

> On 14 Mar 2019, at 15:04, Jeroen Wunnink  wrote:
> 
> The route-leak was something different that seems to have mainly hit 
> west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in the *NOG 
> communities still digging at the complete details of that right now, but it 
> currently points to have originated from AS200020, impacting a few large 
> upstreams for a short period of time.
>  
> So unless this leak caused a catastrophic cascade in FB’s network somehow, it 
> seems to be unrelated.
> It looked like a valid suspect because timing was very similar between the 
> start of the FB outage and the leak.

Hello,

I’m an engineer at AS200020. We indeed had a route leak yesterday between 16:45 
and 17:07 UTC.
This was caused by a mistake in the configuration of one of ou core routers by 
one of our engineers.
We leaked full table to one of our transits, which is apparently accepted. 
We’ve seen some prefixes from a couple of AS numbers coming in. Facebook was 
not one of these AS’es. As far as I can tell it’s not related at all to the 
major incidents at Facebook.

We are very sorry for any inconvenience people could have experienced.
If you received complaints in the mentioned time period and want to know if it 
could be related, please let me know.

Best regards,

Pim van Stam
NBIP-naWas / AS200020

Re: FB?

[Tom Beecher]

As much as I wanted to crack jokes because I cannot stand Facebook (the
product), much love to all you FB engineers that went through (and are
probably still going through) much hell.

On Thu, Mar 14, 2019 at 9:58 AM Jason Suter  wrote:

>
> I found this article
> 
>  but
> no real answers.
>
> On Thu, Mar 14, 2019 at 9:36 AM Kain, Rebecca (.)  wrote:
>
>> So what happened yesterday?
>>
>>
>>
>> *From:* NANOG  *On Behalf Of *Mike Hammett
>> *Sent:* Thursday, March 14, 2019 8:29 AM
>> *To:* Roland Dobbins 
>> *Cc:* nanog@nanog.org
>> *Subject:* Re: FB?
>>
>>
>>
>> Do you have a link to the clarification? With the high jitter of news,
>> all I'm finding is people parroting the original statement.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>>
>> Midwest-IX
>> http://www.midwest-ix.com
>>
>>
>> --
>>
>> *From: *"Roland Dobbins" 
>> *To: *nanog@nanog.org
>> *Sent: *Thursday, March 14, 2019 7:23:00 AM
>> *Subject: *Re: FB?
>>
>> On 14 Mar 2019, at 19:17, Mike Hammett wrote:
>>
>> > I saw one article quoting Roland saying it was a route leak, but I
>> > haven't seen any other sources that aren't just quoting Roland.
>>
>> That was the result of a miscommunication; a clarification has been
>> issued, FYI.
>>
>> 
>> Roland Dobbins 
>>
>>
>>
>

Re: FB?

[Jeroen Wunnink]

The route-leak was something different that seems to have mainly hit 
west-Europe between 16:52 UTC to 17:08 UTC. There’s a few people in the *NOG 
communities still digging at the complete details of that right now, but it 
currently points to have originated from AS200020, impacting a few large 
upstreams for a short period of time.

So unless this leak caused a catastrophic cascade in FB’s network somehow, it 
seems to be unrelated.
It looked like a valid suspect because timing was very similar between the 
start of the FB outage and the leak.




Jeroen Wunnink
Sr. Manager - Integration Engineering

www.gtt.net

[id:image001.png@01D37331.D1301F60]


From: NANOG  on behalf of "Kain, Rebecca (.)" 

Date: Thursday, 14 March 2019 at 14:36
To: Mike Hammett , Roland Dobbins 

Cc: "nanog@nanog.org" 
Subject: RE: FB?

So what happened yesterday?

From: NANOG  On Behalf Of Mike Hammett
Sent: Thursday, March 14, 2019 8:29 AM
To: Roland Dobbins 
Cc: nanog@nanog.org
Subject: Re: FB?

Do you have a link to the clarification? With the high jitter of news, all I'm 
finding is people parroting the original statement.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Roland Dobbins" 
mailto:roland.dobb...@netscout.com>>
To: nanog@nanog.org
Sent: Thursday, March 14, 2019 7:23:00 AM
Subject: Re: FB?

On 14 Mar 2019, at 19:17, Mike Hammett wrote:

> I saw one article quoting Roland saying it was a route leak, but I
> haven't seen any other sources that aren't just quoting Roland.

That was the result of a miscommunication; a clarification has been
issued, FYI.


Roland Dobbins mailto:roland.dobb...@netscout.com>>

Re: FB?

[Jason Suter]

I found this article

but
no real answers.

On Thu, Mar 14, 2019 at 9:36 AM Kain, Rebecca (.)  wrote:

> So what happened yesterday?
>
>
>
> *From:* NANOG  *On Behalf Of *Mike Hammett
> *Sent:* Thursday, March 14, 2019 8:29 AM
> *To:* Roland Dobbins 
> *Cc:* nanog@nanog.org
> *Subject:* Re: FB?
>
>
>
> Do you have a link to the clarification? With the high jitter of news, all
> I'm finding is people parroting the original statement.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>
> --
>
> *From: *"Roland Dobbins" 
> *To: *nanog@nanog.org
> *Sent: *Thursday, March 14, 2019 7:23:00 AM
> *Subject: *Re: FB?
>
> On 14 Mar 2019, at 19:17, Mike Hammett wrote:
>
> > I saw one article quoting Roland saying it was a route leak, but I
> > haven't seen any other sources that aren't just quoting Roland.
>
> That was the result of a miscommunication; a clarification has been
> issued, FYI.
>
> 
> Roland Dobbins 
>
>
>

RE: FB?

[Kain, Rebecca (.)]

So what happened yesterday?

From: NANOG  On Behalf Of Mike Hammett
Sent: Thursday, March 14, 2019 8:29 AM
To: Roland Dobbins 
Cc: nanog@nanog.org
Subject: Re: FB?

Do you have a link to the clarification? With the high jitter of news, all I'm 
finding is people parroting the original statement.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Roland Dobbins" 
mailto:roland.dobb...@netscout.com>>
To: nanog@nanog.org
Sent: Thursday, March 14, 2019 7:23:00 AM
Subject: Re: FB?

On 14 Mar 2019, at 19:17, Mike Hammett wrote:

> I saw one article quoting Roland saying it was a route leak, but I
> haven't seen any other sources that aren't just quoting Roland.

That was the result of a miscommunication; a clarification has been
issued, FYI.


Roland Dobbins mailto:roland.dobb...@netscout.com>>

Re: Apple devices spoofing default gateway?

[Mel Beckman]

Can you post some packet captures? 

I was a network engineer on the WiFi network at SFO, for both passengers and 
baggage scanners, with several hundred APs. Several times we were misled by 
packet captures that seemed to show client traffic causing network problems, 
such as packet storms, but which ultimately always had some more mundane cause, 
like a failed DHCP server or flapping switch interface. 

The particular SFO network I worked on has Juniper switching and Aruba APs, so 
it’s not directly applicable to your ecosystem. But the complexities of 
interpreting packet captures may apply.

 -mel beckman

> On Mar 14, 2019, at 5:28 AM, Simon Lockhart  wrote:
> 
> All,
> 
> We're seeing a bit of a weird one on our network at the moment, and wondering
> if anyone else has seen it.
> 
> Since Friday we're seeing Apple devices (we believe it's both laptops and 
> iPhones) responding to ARP requests for the default gateway IP with their own
> MAC address (i.e. ARP spoofing / MITM type attack). We're only seeing it on
> Apple devices, but what's more strange is that we're only seeing it where 
> those Apple devices are connected to Cisco 1810 and 1815 APs, and where those
> APs are connected to a Cisco WLC running v8.5 software. If we downgrade the
> WLC to v8.2 the problem goes away (but v8.2 doesn't support 1815 APs, so we 
> can't roll that out globally). We're engaged with Cisco TAC, but they're 
> trying to deny it's their problem. Apple support are investigating, but aren't
> admitting to having seen it before.
> 
> Has anyone else seen or heard of similar issues over the last few days?
> 
> Many thanks,
> 
> Simon

Re: FB?

[Mike Hammett]

Do you have a link to the clarification? With the high jitter of news, all I'm 
finding is people parroting the original statement. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Roland Dobbins"  
To: nanog@nanog.org 
Sent: Thursday, March 14, 2019 7:23:00 AM 
Subject: Re: FB? 

On 14 Mar 2019, at 19:17, Mike Hammett wrote: 

> I saw one article quoting Roland saying it was a route leak, but I 
> haven't seen any other sources that aren't just quoting Roland. 

That was the result of a miscommunication; a clarification has been 
issued, FYI. 

 
Roland Dobbins  

Apple devices spoofing default gateway?

[Simon Lockhart]

All,

We're seeing a bit of a weird one on our network at the moment, and wondering
if anyone else has seen it.

Since Friday we're seeing Apple devices (we believe it's both laptops and 
iPhones) responding to ARP requests for the default gateway IP with their own
MAC address (i.e. ARP spoofing / MITM type attack). We're only seeing it on
Apple devices, but what's more strange is that we're only seeing it where 
those Apple devices are connected to Cisco 1810 and 1815 APs, and where those
APs are connected to a Cisco WLC running v8.5 software. If we downgrade the
WLC to v8.2 the problem goes away (but v8.2 doesn't support 1815 APs, so we 
can't roll that out globally). We're engaged with Cisco TAC, but they're 
trying to deny it's their problem. Apple support are investigating, but aren't
admitting to having seen it before.

Has anyone else seen or heard of similar issues over the last few days?

Many thanks,

Simon

Re: FB?

[Dobbins, Roland]

On 14 Mar 2019, at 19:17, Mike Hammett wrote:

> I saw one article quoting Roland saying it was a route leak, but I 
> haven't seen any other sources that aren't just quoting Roland.

That was the result of a miscommunication; a clarification has been 
issued, FYI.


Roland Dobbins 

FB?

[Mike Hammett]

So what happened at Facebook today ? I saw one article quoting Roland saying it 
was a route leak, but I haven't seen any other sources that aren't just quoting 
Roland. Usually there are a few independent posts out there by now. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

Re: Oracle DBA

[Mike Meredith]

On Wed, 13 Mar 2019 17:44:13 -0700, Randy Bush  may have
written:
> ya.  none of us run oracle

Yeah but some of us walk it.

-- 
Mike Meredith, University of Portsmouth
Chief Systems Engineer, Hostmaster, Security, and Timelord!
 


pgp19LcFRTpOk.pgp
Description: OpenPGP digital signature

Re: Oracle DBA

[Alain Hebert]

    Run away from...

-
Alain Hebertaheb...@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 3/13/19 8:44 PM, Randy Bush wrote:

This is totally off-topic.

ya.  none of us run oracle

RE: AS701/Verizon

[Phil Lavin]

> We're seeing consistent +100ms latency increases to Verizon customers in 
> Pennsylvania, during peak business hours for the past couple of weeks.

Verizon reached out shortly after my e-mail to say they had resolved the issue 
- latency has been within normal bounds since. Many thanks :)