Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115

[John Todd]

Since it’s come up on the list and we haven’t given a public update 
recently, I thought I’d just write a quick note on the state of 
INOC-DBA.  For those who aren’t familiar with it, INOC-DBA is a 
SIP-based hotline communications system between NOCs and CERTs:


https://www.pch.net/services/INOC_DBA

https://en.wikipedia.org/wiki/INOC-DBA

PCH has been the secretariat for INOC-DBA for the past thirteen years as 
a function of our not-for-profit purpose, serving network operators.  
During that time, the INOC-DBA back-end and self-provisioning systems 
have been completely replaced three times, and we’re currently at work 
on moving from the SER-driven 3.0 series of releases to a more modern 
BE7k-driven 4.0 system.  Because INOC-DBA has only been intermittently 
directly grant-funded, sometimes, like now, it is funded entirely out of 
our overhead budget, so progress can be slow.  The consequence is that, 
in order to make headway on the 4.0 transition, we’ve had to move 
people off of active support of the old 3.0 self-provisioning system.  
So, it’s fine for people who are already using it, but there’s not 
currently a way to create a new user within the 3.0 system, nor for 
existing users to make significant changes to call routing.


ASNs have proven to be a good identifier, allowing network operators to 
communicate with each other in a way that’s vetted, while avoiding 
putting PCH in the position of judging who qualifies to join and who 
doesn't.  Whether you know the name of a network, or where it’s 
located, or even what timezone they’re in, you know them by their ASN. 
 And a hotline system that bypasses directories and receptionists and 
escalation chains is a quick and low-friction way of reaching someone 
who has the authority and access to resolve a problem.


While email is the most venerable and well-known communication method it 
is often filtered, missed, or funneled through helpdesks that don’t 
have sufficient clue, or are stymied by dealing with someone who isn’t 
one of their own customers.  Facebook and general-purpose chat systems 
are less than ideal as well, as they’re un-vetted and quickly suffer 
the same fate as email: if they’re paid attention to at all, filters 
or automated systems are put in place to block the noise.  So, a closed 
network for voice, video, presence and chat has proven to be an 
immediate, low-noise way for those network operators who choose to use 
it, to communicate with each other.  In the 4.0 system, XMPP chat using 
the same identifiers in the same closed network is a natural extension 
and the new feature that, though hardly revolutionary, we’re most 
looking forward to releasing.


The technical issues that were discussed in this thread about NAT/PAT 
problems are certainly valid, but can be circumvented in a number of 
different ways, some of which are addressed in our documentation. SIP 
and RTP can work through NAT if correctly configured in simple 
circumstances, or in the presence of a NAT-traversal server, such as is 
included in INOC-DBA.  An organization may have multiple INOC-DBA users 
and opt to have a SIP-capable system at the border of their network with 
one side facing the public Internet, and one side facing their private 
network, and which manages call flow and media handling (Asterisk, 
Freeswitch, or any one of a number of free or commercial SIP PBX-like 
systems will do this fairly easily; again, there are tutorials in our 
documentation).  This also allows after-hours routing to PSTN lines or 
to call groups as needed, controlled by a local administrator.  We also 
have considered keeping the media path through our servers, which aids 
the NAT traversal issue while not precluding local SIP enclaves as 
described above.


One of the things that we struggle with is maintaining an appropriate 
balance between, on the one hand, keeping the network operations 
community informed of the status of the system, so they don’t feel 
compelled to ask on NANOG, versus not pro-actively over-sharing on lists 
and making a nuisance of ourselves.  Admittedly, if the 4.0 transition 
were going faster, this would be less of an issue.


So, we’re glad of the continued interest (particularly in the NANOG 
community, where INOC-DBA is not as widely used as in, for instance, the 
LACNIC community), and we apologize for the slow transition to the new 
4.0 back-end and self-provisioning system.  As always, you can contact 
us directly about INOC-DBA related stuff on opera...@pch.net


JT


---
John Todd - jt...@pch.net - +1-415-831-3123



On 29 Sep 2015, at 8:05, Bob Evans wrote:

Nice of you to check Jim. This brings up the old idea - A long time 
ago I
had an INOC phone by PCH.NET - It never rang, as we filter our 
outbound
with detail everywhere we announce. ISPs need to provide us their 
address

list.

And the few times I needed to use it , no one ever answered. ( It was 
a
decade ago before NANOG membership.) So after a while I too ig

Re: Do you have INOC-DBA set up? (was: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115)

[Pete Mundy]

On 30/09/2015, at 6:19 AM, Matthew Walster  wrote:

> ​"lolz" as the kids say.​

Current stats indicate it's actually only the old-timers that say lolz now 
days! ;)

http://www.huffingtonpost.com/entry/facebook-study-laughter_55c8b148e4b0f1cbf1e5857e

Pete

Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115

[Aaron]

We have a big, red rotary phone that sits in our NOC that we have 
attached to a VoIP box just to use for that. :)


On 9/29/2015 10:05 AM, Bob Evans wrote:

Nice of you to check Jim. This brings up the old idea - A long time ago I
had an INOC phone by PCH.NET - It never rang, as we filter our outbound
with detail everywhere we announce. ISPs need to provide us their address
list.

And the few times I needed to use it , no one ever answered. ( It was a
decade ago before NANOG membership.) So after a while I too ignored it.
Maybe this was an idea ahead of it's time ? From this painful mishap, it
could have been a great solution for NOC Engineers to help each. I find
peeringdb often outdated as companies change around and sluggish return
call if at all.  Most are like a sales line number post.

I see now a long list of registered networks in the PCH directory. Are
networks actually paying attention and using it. Is it time to take
another look ?  At midnight in your organization could you get a NOC
person with " proper BGP skills and access " to answer and care about a
bad announcement ?

https://inoc-dba-web.pch.net/inoc-dba/console.cgi?op=show_pubdir&list=org
  Link above shows lots more networks listed on the
  INOC-DBA Public Directory: Organizations

But have you used it? Did it work for you when you needed it ?
Any further comments are appreciated.

This seems like a very good proper civil approach - maybe this or
something like it ARIN might help promote and endorse as a benefit to the
community ? Be nice if with the cash they did something simple like this
and got all of us to use it? Special line forwarding ? A Emergency Only
NOC App for our phones for just this kind of situation - one that
registers a specific ASN and pin code we set on the registration page ?

Thank You
Bob Evans
CTO






On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
 wrote:


On 9/28/15 20:19, Martin Hannigan wrote:

Is this related to 104.73.161.0/24? That's ours. :-)

We'll take a look and get back to you.  Thanks for caring!



Yep, that's one of the affected prefixes.

~Seth

Hi Seth, which market was this occurring?  Was this already removed?  I'm
not seeing it this morning.  I would like to figure out what went wrong
here.  We shouldn't be nailing up any static configuration to have caused
a situation like this.







--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: Prefix hijacking by AS20115

[Christopher Morrow]

On Tue, Sep 29, 2015 at 1:29 PM, N M  wrote:
> If this is anything like what I deal with the aging timer for the bgp
> session is set to 180s by default.  After 2 years I've been unable to get
> the charter noc to enable bfd on my links to address this issue

because bfd brings it's own special sort of pain...

Re: Prefix hijacking by AS20115

[N M]

If this is anything like what I deal with the aging timer for the bgp
session is set to 180s by default.  After 2 years I've been unable to get
the charter noc to enable bfd on my links to address this issue
On Sep 29, 2015 10:59 AM, "Seth Mattinen"  wrote:

> On 9/29/15 8:18 AM, Rampley Jr, Jim F wrote:
> >
>
>> This issue was caused by a hung BGP process which was resolved last night.
>>   Nothing nefarious.  No static configuration nailed up, no BGP
>> highjacking
>> purposely done. ;)
>>
>>
>
> Is there a Cisco bug ID?
>
> ~Seth
>

Re: Do you have INOC-DBA set up? (was: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115)

[Matthew Walster]

On 29 September 2015 at 17:13, Bob Evans 
wrote:

> Neils, do you actually work at in a NOC operation with BGP operations and
> policies you can change - a backbone with customers?


​"lolz" as the kids say.​



> SayAn email/ text might work well or even better than SIP - if we had
> an APP that noticed a specific key or coded line plus your ASN to then
> ring my phone with an urgent ring tone.hence, the idea of an NOC APP
> for that.
>

​This isn't an iPhone developers conference, the answer is very rarely
"there's an app for that". The chance of that being integrated with ISP
phone systems is slim to none.

Email works. When it doesn't IRC works. It has done for a decade, it will
for the next decade. Yes, even when the 200 people post to Outages saying
"XYZ is down for me, anyone else" or the far more annoying "can someone
from XYZ contact me offlist" posts to NANOG.

M​

Re: Do you have INOC-DBA set up? (was: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115)

[Bob Evans]

Neils, do you actually work at in a NOC operation with BGP operations and
policies you can change - a backbone with customers? If not - I would
understand why email is fast enough for you.

Maybe SIP iNOC phone isn't the right answer - but it seems to work fine
everywhere I go. There just has to be a better way of communicating other
than posting an email to a board - which isn't focused on a live network
emergency. Something that's self filtered by all of us for a specific use.

SayAn email/ text might work well or even better than SIP - if we had
an APP that noticed a specific key or coded line plus your ASN to then
ring my phone with an urgent ring tone.hence, the idea of an NOC APP
for that.

Something other than "No I won't do anything different" - an idea or
concept something you would embrace for such a moment. The iNOC phone
wasn't embraced. Maybe a APP is a better idea than a phone.

Thank You
Bob Evans
CTO




> * j...@baylink.com (Jay Ashworth) [Tue 29 Sep 2015, 17:31 CEST]:
>>The idea of a private tieline network that is connected, by SIP, to a
>> line
>>appearance in the NOC of each AS, and no one else is on it, seems like a
>>fine idea to me.
>
> Until you take into account that SIP doesn't work through many
> firewalls, that people generally don't give a second thought to
> timezones, that network engineers generally dislike having to mess
> with voice systems, etc. etc.
>
> 2 out of 3 INOC-DBA calls I ever received were silent on their end
> (presumably) due to firewalls; the third call was a test.
>
>
>>And that was INOC-DBA's original goal, as I understand it:
>>
>>You're having a problem?  It's coming from some specific AS?
>>
>>Pick up the phone, mash the red INOC line button, dial the AS
>>number, and you're talking to their NOC.
>>
>>And that's *authenticated*: since it's low enough churn to set up
>>by hand, it's authenticated by humans.
>
> In other words, it wasn't secure, it wouldn't scale and churn killed it.
>
>
>>Show of hands: who has it set up, correctly, right now?
>
> No.  There is nothing I'd do after receiving a phone call that I
> wouldn't do via email anyway.
>
>
>   -- Niels.
>

Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115

[Jay Ashworth]

Well, there *is* outa...@outages.org... :-)

- Original Message -
> From: "Royce Williams" 
> To: nanog@nanog.org
> Sent: Tuesday, September 29, 2015 11:31:54 AM
> Subject: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115
> On Tue, Sep 29, 2015 at 7:12 AM, Job Snijders 
> wrote:
> >
> > Hi Bob,
> >
> > On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
> > > This seems like a very good proper civil approach - maybe this or
> > > something like it ARIN might help promote and endorse as a benefit
> > > to
> > > the community ? Be nice if with the cash they did something simple
> > > like this and got all of us to use it? Special line forwarding ? A
> > > Emergency Only NOC App for our phones for just this kind of
> > > situation
> > > - one that registers a specific ASN and pin code we set on the
> > > registration page ?
> >
> > In this day and age people use IRC or Facebook to quickly get to a
> > friend of a friend of a friend to get to a good contact. Get on with
> > the
> > times :-)
> 
> This seems lossy and unscriptable to me. There are maxint different
> flavors of $social, so it's not suitable for escalation, IMO. Also,
> many people opt out of half of them when they're not on the clock.
> And, many of them have "I don't know you so I'll bury your message"
> options, which makes being tickled by a stranger for emergency
> purposes hard. And their "APIs", so to speak, are constantly
> shifting.
> 
> But we already have a reliable, widespread, high-SNR channel: this
> list. It's the place that people go when they can't get an answer any
> other way. Email works when many other things are broken.
> 
> What if all NOCs used their NOC email distro/alias to subscribe,
> filter for posts containing their own ASes/admin-domains/prefixes,
> plus the string "problem|issue|etc", and flag them as higher priority.
> A junior NOCling could check it manually every couple of hours, and
> maybe a public web archive of the list, in case of filter failures.
> 
> I would expect most NOCs worth their salt to be monitoring nanog
> anyway. Why not leverage it?
> 
> A sibling list could be spun off -- nanog-panic-button? ;) -- if that
> would be preferable.
> 
> Royce

-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: Prefix hijacking by AS20115

[Seth Mattinen]

On 9/29/15 8:18 AM, Rampley Jr, Jim F wrote:
>

This issue was caused by a hung BGP process which was resolved last night.
  Nothing nefarious.  No static configuration nailed up, no BGP highjacking
purposely done. ;)




Is there a Cisco bug ID?

~Seth

Re: Do you have INOC-DBA set up? (was: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115)

[Niels Bakker]

* j...@baylink.com (Jay Ashworth) [Tue 29 Sep 2015, 17:31 CEST]:

The idea of a private tieline network that is connected, by SIP, to a line
appearance in the NOC of each AS, and no one else is on it, seems like a
fine idea to me.


Until you take into account that SIP doesn't work through many
firewalls, that people generally don't give a second thought to
timezones, that network engineers generally dislike having to mess
with voice systems, etc. etc.

2 out of 3 INOC-DBA calls I ever received were silent on their end
(presumably) due to firewalls; the third call was a test.



And that was INOC-DBA's original goal, as I understand it:

You're having a problem?  It's coming from some specific AS?

Pick up the phone, mash the red INOC line button, dial the AS
number, and you're talking to their NOC.

And that's *authenticated*: since it's low enough churn to set up
by hand, it's authenticated by humans.


In other words, it wasn't secure, it wouldn't scale and churn killed it.



Show of hands: who has it set up, correctly, right now?


No.  There is nothing I'd do after receiving a phone call that I
wouldn't do via email anyway.


-- Niels.

Re: Prefix hijacking by AS20115

[Sandra Murphy]

On Sep 28, 2015, at 11:59 PM, Bob Evans  wrote:

> 
> Would be nice if our membership organization ARIN ( that we all pay to
> keep us somewhat organized) had an ability to do something for you I
> never looked into it...i don't knowmaybe it does ?
> 

No one else has said this, so…

RPKI.  Which ARIN does do.

—Sandy

P.S.  The following has numerous points of weirdness.

about 104.73.161.0/24, RADB says:

route: 104.73.161.0/24
descr: Proxy for Akamai (AS20940) and Roller Networks (AS11170)
origin: AS20115
mnt-by: MAINT-CHTR-WD
changed: tim.we...@charter.com 20150312 #20:32:27Z
source: RADB

route: 104.73.161.0/24
descr: Akamai Technologies
origin: AS20940
mnt-by: AKAM1-RIPE-MNT
changed: unr...@ripe.net 2101
source: RIPE
remarks: 
remarks: * THIS OBJECT IS MODIFIED
remarks: * Please note that all data that is generally regarded as personal
remarks: * data has been removed from this object.
remarks: * To view the original object, please query the RIPE Database at:
remarks: * http://www.ripe.net/whois
remarks: 

route: 104.64.0.0/10
descr: Akamai
origin: AS35994
mnt-by: AKAM1-ALTDB-MNT
changed: abl...@akamai.com 20140518
source: ALTDB




signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Do you have INOC-DBA set up? (was: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115)

[Hugo Slabbert]

On Tue 2015-Sep-29 11:19:57 -0400, Jay Ashworth  wrote:

:


Show of hands: who has it set up, correctly, right now?


I had this in my to-do, and this thread poked me again to get on with it.  
Sadly, https://inoc-dba-web.pch.net/inoc-dba/console.cgi?op=new_account 
gives me:


Account sign up is disabled.

Please wait for the new system! 


:'(

--
Hugo


signature.asc
Description: Digital signature

Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115

[Royce Williams]

On Tue, Sep 29, 2015 at 7:12 AM, Job Snijders  wrote:
>
> Hi Bob,
>
> On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
> > This seems like a very good proper civil approach - maybe this or
> > something like it ARIN might help promote and endorse as a benefit to
> > the community ? Be nice if with the cash they did something simple
> > like this and got all of us to use it? Special line forwarding ? A
> > Emergency Only NOC App for our phones for just this kind of situation
> > - one that registers a specific ASN and pin code we set on the
> > registration page ?
>
> In this day and age people use IRC or Facebook to quickly get to a
> friend of a friend of a friend to get to a good contact. Get on with the
> times :-)

This seems lossy and unscriptable to me.  There are maxint different
flavors of $social, so it's not suitable for escalation, IMO.  Also,
many people opt out of half of them when they're not on the clock.
And, many of them have "I don't know you so I'll bury your message"
options, which makes being tickled by a stranger for emergency
purposes hard.  And their "APIs", so to speak, are constantly
shifting.

But we already have a reliable, widespread, high-SNR channel: this
list.  It's the place that people go when they can't get an answer any
other way.  Email works when many other things are broken.

What if all NOCs used their NOC email distro/alias to subscribe,
filter for posts containing their own ASes/admin-domains/prefixes,
plus the string "problem|issue|etc", and flag them as higher priority.
A junior NOCling could check it manually every couple of hours, and
maybe a public web archive of the list, in case of filter failures.

I would expect most NOCs worth their salt to be monitoring nanog
anyway.  Why not leverage it?

A sibling list could be spun off -- nanog-panic-button? ;) -- if that
would be preferable.

Royce

Do you have INOC-DBA set up? (was: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115)

[Jay Ashworth]

I entirely disagree, Job.

The idea of a private tieline network that is connected, by SIP, to a line 
appearance in the NOC of each AS, and no one else is on it, seems like a
fine idea to me.

And that was INOC-DBA's original goal, as I understand it:

You're having a problem?  It's coming from some specific AS?

Pick up the phone, mash the red INOC line button, dial the AS 
number, and you're talking to their NOC.

And that's *authenticated*: since it's low enough churn to set up
by hand, it's authenticated by humans.

Show of hands: who has it set up, correctly, right now?

- Original Message -
> From: "Job Snijders" 
> To: "Bob Evans" 
> Cc: nanog@nanog.org
> Sent: Tuesday, September 29, 2015 11:12:43 AM
> Subject: Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115
> Hi Bob,
> 
> On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
> > This seems like a very good proper civil approach - maybe this or
> > something like it ARIN might help promote and endorse as a benefit
> > to
> > the community ? Be nice if with the cash they did something simple
> > like this and got all of us to use it? Special line forwarding ? A
> > Emergency Only NOC App for our phones for just this kind of
> > situation
> > - one that registers a specific ASN and pin code we set on the
> > registration page ?
> 
> In this day and age people use IRC or Facebook to quickly get to a
> friend of a friend of a friend to get to a good contact. Get on with
> the
> times :-)
> 
> Kind regards,
> 
> Job

-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: Prefix hijacking by AS20115

[Rampley Jr, Jim F]

On 9/29/15, 9:49 AM, "Seth Mattinen"  wrote:


>On 9/29/15 7:26 AM, Rampley Jr, Jim F wrote:
>>
>>
>> On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
>>  wrote:
>>
>>> On 9/28/15 20:19, Martin Hannigan wrote:

 Is this related to 104.73.161.0/24? That's ours. :-)

 We'll take a look and get back to you.  Thanks for caring!

>>>
>>>
>>> Yep, that's one of the affected prefixes.
>>>
>>> ~Seth
>> Hi Seth, which market was this occurring?  Was this already removed?
>>I'm
>> not seeing it this morning.  I would like to figure out what went wrong
>> here.  We shouldn't be nailing up any static configuration to have
>>caused
>> a situation like this.
>>
>
>
>Reno, NV. I do believe they've finally withdrawn this morning (I just
>woke up, it was a long night).
>
>~Seth
This issue was caused by a hung BGP process which was resolved last night.
 Nothing nefarious.  No static configuration nailed up, no BGP highjacking
purposely done. ;)

Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115

[Bob Evans]

I have actually found this NANOG email to be more effective than a chat or
mombook public service. We need something more private like that.

Thank You
Bob Evans
CTO




> A friend is not someone that allows their company to hijack your prefixes.
> A friend is one that can get it to stop. Dude - wake up and drink some
> coffee.
>
> Thank You
> Bob Evans
> CTO
>
>
>
>
>> Hi Bob,
>>
>> On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
>>> This seems like a very good proper civil approach - maybe this or
>>> something like it ARIN might help promote and endorse as a benefit to
>>> the community ? Be nice if with the cash they did something simple
>>> like this and got all of us to use it? Special line forwarding ? A
>>> Emergency Only NOC App for our phones for just this kind of situation
>>> - one that registers a specific ASN and pin code we set on the
>>> registration page ?
>>
>> In this day and age people use IRC or Facebook to quickly get to a
>> friend of a friend of a friend to get to a good contact. Get on with the
>> times :-)
>>
>> Kind regards,
>>
>> Job
>>
>
>
>

Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115

[Bob Evans]

A friend is not someone that allows their company to hijack your prefixes.
A friend is one that can get it to stop. Dude - wake up and drink some
coffee.

Thank You
Bob Evans
CTO




> Hi Bob,
>
> On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
>> This seems like a very good proper civil approach - maybe this or
>> something like it ARIN might help promote and endorse as a benefit to
>> the community ? Be nice if with the cash they did something simple
>> like this and got all of us to use it? Special line forwarding ? A
>> Emergency Only NOC App for our phones for just this kind of situation
>> - one that registers a specific ASN and pin code we set on the
>> registration page ?
>
> In this day and age people use IRC or Facebook to quickly get to a
> friend of a friend of a friend to get to a good contact. Get on with the
> times :-)
>
> Kind regards,
>
> Job
>

Re: PCH.net questions and thoughts - Re: Prefix hijacking by AS20115

[Job Snijders]

Hi Bob,

On Tue, Sep 29, 2015 at 08:05:45AM -0700, Bob Evans wrote:
> This seems like a very good proper civil approach - maybe this or
> something like it ARIN might help promote and endorse as a benefit to
> the community ? Be nice if with the cash they did something simple
> like this and got all of us to use it? Special line forwarding ? A
> Emergency Only NOC App for our phones for just this kind of situation
> - one that registers a specific ASN and pin code we set on the
> registration page ?

In this day and age people use IRC or Facebook to quickly get to a
friend of a friend of a friend to get to a good contact. Get on with the
times :-)

Kind regards,

Job

PCH.net questions and thoughts - Re: Prefix hijacking by AS20115

[Bob Evans]

Nice of you to check Jim. This brings up the old idea - A long time ago I
had an INOC phone by PCH.NET - It never rang, as we filter our outbound
with detail everywhere we announce. ISPs need to provide us their address
list.

And the few times I needed to use it , no one ever answered. ( It was a
decade ago before NANOG membership.) So after a while I too ignored it.
Maybe this was an idea ahead of it's time ? From this painful mishap, it
could have been a great solution for NOC Engineers to help each. I find
peeringdb often outdated as companies change around and sluggish return
call if at all.  Most are like a sales line number post.

I see now a long list of registered networks in the PCH directory. Are
networks actually paying attention and using it. Is it time to take
another look ?  At midnight in your organization could you get a NOC
person with " proper BGP skills and access " to answer and care about a
bad announcement ?

https://inoc-dba-web.pch.net/inoc-dba/console.cgi?op=show_pubdir&list=org
 Link above shows lots more networks listed on the
 INOC-DBA Public Directory: Organizations

But have you used it? Did it work for you when you needed it ?
Any further comments are appreciated.

This seems like a very good proper civil approach - maybe this or
something like it ARIN might help promote and endorse as a benefit to the
community ? Be nice if with the cash they did something simple like this
and got all of us to use it? Special line forwarding ? A Emergency Only
NOC App for our phones for just this kind of situation - one that
registers a specific ASN and pin code we set on the registration page ?

Thank You
Bob Evans
CTO




>
>
> On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
>  wrote:
>
>>On 9/28/15 20:19, Martin Hannigan wrote:
>>>
>>>Is this related to 104.73.161.0/24? That's ours. :-)
>>>
>>>We'll take a look and get back to you.  Thanks for caring!
>>>
>>
>>
>>Yep, that's one of the affected prefixes.
>>
>>~Seth
> Hi Seth, which market was this occurring?  Was this already removed?  I'm
> not seeing it this morning.  I would like to figure out what went wrong
> here.  We shouldn't be nailing up any static configuration to have caused
> a situation like this.
>
>

Re: Prefix hijacking by AS20115

[Seth Mattinen]

On 9/29/15 7:26 AM, Rampley Jr, Jim F wrote:



On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
 wrote:


On 9/28/15 20:19, Martin Hannigan wrote:


Is this related to 104.73.161.0/24? That's ours. :-)

We'll take a look and get back to you.  Thanks for caring!




Yep, that's one of the affected prefixes.

~Seth

Hi Seth, which market was this occurring?  Was this already removed?  I'm
not seeing it this morning.  I would like to figure out what went wrong
here.  We shouldn't be nailing up any static configuration to have caused
a situation like this.




Reno, NV. I do believe they've finally withdrawn this morning (I just 
woke up, it was a long night).


~Seth

Re: Prefix hijacking by AS20115

[Mark Tinka]

On 29/Sep/15 16:26, Rampley Jr, Jim F wrote:

>
> Hi Seth, which market was this occurring?  Was this already removed?  I'm
> not seeing it this morning.  I would like to figure out what went wrong
> here.  We shouldn't be nailing up any static configuration to have caused
> a situation like this.

You'd be surprised how often this happens, especially on the back of a
conference rocking into a city/country and the local provider having
minimal BGP experience. Once the conference is done, folk leave, and the
provider forgets about things - which is not a problem since the
conference would have come with its own IP address space.

The issue goes unnoticed for 12x months when the conference is trying to
route their usual block in some other city/country, and things just seem
"strange". Someone remembers the previous year's event, calls up the
previous provider, and finds out that the tech. who worked the
activation has since left.

It's not easy...

Many other situations closer to home (i.e., paying customers) where
things like this happen, especially if the customer has IP address space
but does not do BGP (until they want to or leave to the competition).

Blackholing operations that go wrong that folk forget about as well, not
to mention other networks that cut themselves off by using public IP
address space for their enterprise network.

It's not easy at all...

Mark.

Re: Prefix hijacking by AS20115

[Rampley Jr, Jim F]

On 9/28/15, 10:24 PM, "NANOG on behalf of Seth Mattinen"
 wrote:

>On 9/28/15 20:19, Martin Hannigan wrote:
>>
>>Is this related to 104.73.161.0/24? That's ours. :-)
>>
>>We'll take a look and get back to you.  Thanks for caring!
>>
>
>
>Yep, that's one of the affected prefixes.
>
>~Seth
Hi Seth, which market was this occurring?  Was this already removed?  I'm
not seeing it this morning.  I would like to figure out what went wrong
here.  We shouldn't be nailing up any static configuration to have caused
a situation like this.

Re: Prefix hijacking by AS20115

[Christopher Morrow]

On Tue, Sep 29, 2015 at 2:04 AM, Bob Evans  wrote:
>
>
>> On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans 
>> wrote:
>>> That's something I would do. Announce announce and keep adding ports
>>> until
>>> I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in
>>> a
>>> blackhole route for the prefixes. Try to pick blocks that are as
>>> geographically located to your peering routers as possible ...IE in Reno
>>> pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
>>> . when that batch of customers makes their phones ring all night
>>> someone will listen.
>>>
>>
>> that seems like a pretty poor strategy... guaranteed to get you into
>> some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
>> the same thing as the customer-service-center. There's likely little
>> to link the 2 things together there :(
>
> You are right - probably creates more problems than good.
>
>>
>>> Would be nice if our membership organization ARIN ( that we all pay to
>>> keep us somewhat organized) had an ability to do something for you I
>>> never looked into it...i don't knowmaybe it does ?
>>
>> arin does not guarantee 'routability' of netblocks assigned to your org.
>
> Yep, I was pretty sure of that - but wouldn't it be nice if arin could
> have some communication line or at least try. Yes, never any guarantees
> really.

I'm fairly sure that the arin (or ripe or apnic or...) answer to your
question is: "read the contact info in whois... call the stated
numbers."

pretty sure that's also not going to be super helpful, email the poc's
in the peering-db.

> bob
>
>>
>>> But, in the mean time I am pretty sure you can document this well and
>>> prove your announcements of theirs was due to the fact you couldn't get
>>> proper technical attention and needed to desperately before your
>>> customers
>>> cancel after 8 hours of this. Tomorrow call your lawyers and begin to
>>> sue
>>> that cable company (did I recognize that ASN as cable TV ? ) for damages
>>> this must be causing you in ill-will amongst your customer base.
>>>
>>> I wonder just how you prove the damage...some equation based on customer
>>> calls and complaints together with how many years you have been in
>>> business as well as the number of contracts that are coming up for
>>> renewal. etc etc. Now that would be interesting to see a formula for
>>> that
>>> if anyone has been through it.
>>>
>>
>> you COULD find a charter person on-list...there are nine names on the
>> attendees list for the upcoming meeting... I imagine peeringdb likely
>> has folk listed... gosh it sure does:
>>
>> 
>>
>> what with their emails and everything.
>>
>>> Thank You
>>> Bob Evans
>>> CTO
>>>
>>>
>>>
>>>
 Start announcing their prefixes?

 Josh Luthman
 Office: 937-552-2340
 Direct: 937-552-2343
 1100 Wayne St
 Suite 1337
 Troy, OH 45373
 On Sep 28, 2015 11:09 PM, "Seth Mattinen"  wrote:

> On 9/28/15 18:30, William Herrin wrote:
>
>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen 
>> wrote:
>>
>>> I've got a problem where AS20115 continues to announce prefixes
>>> after
>>> BGP
>>> neighbors were shutdown. They claim it's a wedged BGP process but
>>> aren't
>>> in
>>> any hurry to fix it outside of a maintenance window.
>>>
>>
>> If they weren't lying to you, they'd fix it now. That's not the kind
>> of problem that waits.
>>
>> Thing is: they lied to you. Long ago they "helpfully" programmed
>> their
>> router to announce your route regardless of whether you sent a route
>> to them. They want to wait for a maintenance window to remove that
>> configuration.
>>
>>
>> I'm at a loss of what else I can do. They admit the problem but won't
>> take
>>> action saying it needs to wait for a maintenance window. Am I out of
>>> line
>>> insisting that's an unacceptable response to a problem that results
>>> in
>>> prefix/traffic hijacking?
>>>
>>
>> Try dropping the link entirely. If they still announce your
>> addresses,
>> bring it back up but report it as emergency down, escalate, and call
>> back every 10 minutes until the junior tech understands that it's
>> time
>> to call and wake up the guy who makes the decision to fix it now.
>>
>>
>
> I'm at the tail end here almost 8 hours later since the hijacking
> started.
> Their NOC is just blowing me off now and they're happy to continue the
> hijacking until it's convenient for them to have a maintenance window.
> And
> that's apparently the final decision.
>
> ~Seth
>

>>>
>>>
>>
>
>

Re: Prefix hijacking by AS20115

[Bob Evans]

> On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans 
> wrote:
>> That's something I would do. Announce announce and keep adding ports
>> until
>> I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in
>> a
>> blackhole route for the prefixes. Try to pick blocks that are as
>> geographically located to your peering routers as possible ...IE in Reno
>> pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
>> . when that batch of customers makes their phones ring all night
>> someone will listen.
>>
>
> that seems like a pretty poor strategy... guaranteed to get you into
> some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
> the same thing as the customer-service-center. There's likely little
> to link the 2 things together there :(

You are right - probably creates more problems than good.

>
>> Would be nice if our membership organization ARIN ( that we all pay to
>> keep us somewhat organized) had an ability to do something for you I
>> never looked into it...i don't knowmaybe it does ?
>
> arin does not guarantee 'routability' of netblocks assigned to your org.

Yep, I was pretty sure of that - but wouldn't it be nice if arin could
have some communication line or at least try. Yes, never any guarantees
really.

bob

>
>> But, in the mean time I am pretty sure you can document this well and
>> prove your announcements of theirs was due to the fact you couldn't get
>> proper technical attention and needed to desperately before your
>> customers
>> cancel after 8 hours of this. Tomorrow call your lawyers and begin to
>> sue
>> that cable company (did I recognize that ASN as cable TV ? ) for damages
>> this must be causing you in ill-will amongst your customer base.
>>
>> I wonder just how you prove the damage...some equation based on customer
>> calls and complaints together with how many years you have been in
>> business as well as the number of contracts that are coming up for
>> renewal. etc etc. Now that would be interesting to see a formula for
>> that
>> if anyone has been through it.
>>
>
> you COULD find a charter person on-list...there are nine names on the
> attendees list for the upcoming meeting... I imagine peeringdb likely
> has folk listed... gosh it sure does:
>
> 
>
> what with their emails and everything.
>
>> Thank You
>> Bob Evans
>> CTO
>>
>>
>>
>>
>>> Start announcing their prefixes?
>>>
>>> Josh Luthman
>>> Office: 937-552-2340
>>> Direct: 937-552-2343
>>> 1100 Wayne St
>>> Suite 1337
>>> Troy, OH 45373
>>> On Sep 28, 2015 11:09 PM, "Seth Mattinen"  wrote:
>>>
 On 9/28/15 18:30, William Herrin wrote:

> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen 
> wrote:
>
>> I've got a problem where AS20115 continues to announce prefixes
>> after
>> BGP
>> neighbors were shutdown. They claim it's a wedged BGP process but
>> aren't
>> in
>> any hurry to fix it outside of a maintenance window.
>>
>
> If they weren't lying to you, they'd fix it now. That's not the kind
> of problem that waits.
>
> Thing is: they lied to you. Long ago they "helpfully" programmed
> their
> router to announce your route regardless of whether you sent a route
> to them. They want to wait for a maintenance window to remove that
> configuration.
>
>
> I'm at a loss of what else I can do. They admit the problem but won't
> take
>> action saying it needs to wait for a maintenance window. Am I out of
>> line
>> insisting that's an unacceptable response to a problem that results
>> in
>> prefix/traffic hijacking?
>>
>
> Try dropping the link entirely. If they still announce your
> addresses,
> bring it back up but report it as emergency down, escalate, and call
> back every 10 minutes until the junior tech understands that it's
> time
> to call and wake up the guy who makes the decision to fix it now.
>
>

 I'm at the tail end here almost 8 hours later since the hijacking
 started.
 Their NOC is just blowing me off now and they're happy to continue the
 hijacking until it's convenient for them to have a maintenance window.
 And
 that's apparently the final decision.

 ~Seth

>>>
>>
>>
>

Re: Prefix hijacking by AS20115

[goemon]

On Mon, 28 Sep 2015, Seth Mattinen wrote:
I'm at the tail end here almost 8 hours later since the hijacking started. 
Their NOC is just blowing me off now and they're happy to continue the 
hijacking until it's convenient for them to have a maintenance window. And 
that's apparently the final decision.


Willful negligence. Will only be in your favor when it comes to collect 
damages.


-Dan

RE: Prefix hijacking by AS20115

[Jürgen Jaritsch]

Cogent and Level3 will tell you that you are not their customer ...HE and XO 
will react.


Jürgen Jaritsch
Head of Network & Infrastructure

ANEXIA Internetdienstleistungs GmbH

Telefon: +43-5-0556-300
Telefax: +43-5-0556-500

E-Mail: j...@anexia.at
Web: http://www.anexia.at

Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt
Geschäftsführer: Alexander Windbichler
Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601


-Original Message-
From: Paul S. [cont...@winterei.se]
Received: Dienstag, 29 Sep. 2015, 6:57
To: nanog@nanog.org [nanog@nanog.org]
Subject: Re: Prefix hijacking by AS20115

+1, this is the only sensible advice here.

NSPs actually do seem to care about not letting things like these happen.

On 2015/09/29 01:24 PM, Hank Nussbacher wrote:
> At 23:11 28/09/2015 -0400, Josh Luthman wrote:
>
>> Start announcing their prefixes?
>
> Contact the upstreams of AS20115 - Cogent, Level3, HE and XO.
>
> -Hank
>
>
>> Josh Luthman
>> Office: 937-552-2340
>> Direct: 937-552-2343
>> 1100 Wayne St
>> Suite 1337
>> Troy, OH 45373
>> On Sep 28, 2015 11:09 PM, "Seth Mattinen"  wrote:
>>
>> > On 9/28/15 18:30, William Herrin wrote:
>> >
>> >> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen 
>> >> wrote:
>> >>
>> >>> I've got a problem where AS20115 continues to announce prefixes
>> after BGP
>> >>> neighbors were shutdown. They claim it's a wedged BGP process but
>> aren't
>> >>> in
>> >>> any hurry to fix it outside of a maintenance window.
>> >>>
>> >>
>> >> If they weren't lying to you, they'd fix it now. That's not the kind
>> >> of problem that waits.
>> >>
>> >> Thing is: they lied to you. Long ago they "helpfully" programmed
>> their
>> >> router to announce your route regardless of whether you sent a route
>> >> to them. They want to wait for a maintenance window to remove that
>> >> configuration.
>> >>
>> >>
>> >> I'm at a loss of what else I can do. They admit the problem but
>> won't take
>> >>> action saying it needs to wait for a maintenance window. Am I out
>> of line
>> >>> insisting that's an unacceptable response to a problem that
>> results in
>> >>> prefix/traffic hijacking?
>> >>>
>> >>
>> >> Try dropping the link entirely. If they still announce your
>> addresses,
>> >> bring it back up but report it as emergency down, escalate, and call
>> >> back every 10 minutes until the junior tech understands that it's
>> time
>> >> to call and wake up the guy who makes the decision to fix it now.
>> >>
>> >>
>> >
>> > I'm at the tail end here almost 8 hours later since the hijacking
>> started.
>> > Their NOC is just blowing me off now and they're happy to continue the
>> > hijacking until it's convenient for them to have a maintenance
>> window. And
>> > that's apparently the final decision.
>> >
>> > ~Seth
>> >
>

Re: Prefix hijacking by AS20115

[Paul S.]

+1, this is the only sensible advice here.

NSPs actually do seem to care about not letting things like these happen.

On 2015/09/29 01:24 PM, Hank Nussbacher wrote:

At 23:11 28/09/2015 -0400, Josh Luthman wrote:


Start announcing their prefixes?


Contact the upstreams of AS20115 - Cogent, Level3, HE and XO.

-Hank



Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373
On Sep 28, 2015 11:09 PM, "Seth Mattinen"  wrote:

> On 9/28/15 18:30, William Herrin wrote:
>
>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen 
>> wrote:
>>
>>> I've got a problem where AS20115 continues to announce prefixes 
after BGP
>>> neighbors were shutdown. They claim it's a wedged BGP process but 
aren't

>>> in
>>> any hurry to fix it outside of a maintenance window.
>>>
>>
>> If they weren't lying to you, they'd fix it now. That's not the kind
>> of problem that waits.
>>
>> Thing is: they lied to you. Long ago they "helpfully" programmed 
their

>> router to announce your route regardless of whether you sent a route
>> to them. They want to wait for a maintenance window to remove that
>> configuration.
>>
>>
>> I'm at a loss of what else I can do. They admit the problem but 
won't take
>>> action saying it needs to wait for a maintenance window. Am I out 
of line
>>> insisting that's an unacceptable response to a problem that 
results in

>>> prefix/traffic hijacking?
>>>
>>
>> Try dropping the link entirely. If they still announce your 
addresses,

>> bring it back up but report it as emergency down, escalate, and call
>> back every 10 minutes until the junior tech understands that it's 
time

>> to call and wake up the guy who makes the decision to fix it now.
>>
>>
>
> I'm at the tail end here almost 8 hours later since the hijacking 
started.

> Their NOC is just blowing me off now and they're happy to continue the
> hijacking until it's convenient for them to have a maintenance 
window. And

> that's apparently the final decision.
>
> ~Seth
>

Re: Prefix hijacking by AS20115

[Christopher Morrow]

On Mon, Sep 28, 2015 at 11:59 PM, Bob Evans  
wrote:
> That's something I would do. Announce announce and keep adding ports until
> I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a
> blackhole route for the prefixes. Try to pick blocks that are as
> geographically located to your peering routers as possible ...IE in Reno
> pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
> . when that batch of customers makes their phones ring all night
> someone will listen.
>

that seems like a pretty poor strategy... guaranteed to get you into
some hot water, I suspect. Keep in mind that the 'noc' at 20115 isn't
the same thing as the customer-service-center. There's likely little
to link the 2 things together there :(

> Would be nice if our membership organization ARIN ( that we all pay to
> keep us somewhat organized) had an ability to do something for you I
> never looked into it...i don't knowmaybe it does ?

arin does not guarantee 'routability' of netblocks assigned to your org.

> But, in the mean time I am pretty sure you can document this well and
> prove your announcements of theirs was due to the fact you couldn't get
> proper technical attention and needed to desperately before your customers
> cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue
> that cable company (did I recognize that ASN as cable TV ? ) for damages
> this must be causing you in ill-will amongst your customer base.
>
> I wonder just how you prove the damage...some equation based on customer
> calls and complaints together with how many years you have been in
> business as well as the number of contracts that are coming up for
> renewal. etc etc. Now that would be interesting to see a formula for that
> if anyone has been through it.
>

you COULD find a charter person on-list...there are nine names on the
attendees list for the upcoming meeting... I imagine peeringdb likely
has folk listed... gosh it sure does:



what with their emails and everything.

> Thank You
> Bob Evans
> CTO
>
>
>
>
>> Start announcing their prefixes?
>>
>> Josh Luthman
>> Office: 937-552-2340
>> Direct: 937-552-2343
>> 1100 Wayne St
>> Suite 1337
>> Troy, OH 45373
>> On Sep 28, 2015 11:09 PM, "Seth Mattinen"  wrote:
>>
>>> On 9/28/15 18:30, William Herrin wrote:
>>>
 On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen 
 wrote:

> I've got a problem where AS20115 continues to announce prefixes after
> BGP
> neighbors were shutdown. They claim it's a wedged BGP process but
> aren't
> in
> any hurry to fix it outside of a maintenance window.
>

 If they weren't lying to you, they'd fix it now. That's not the kind
 of problem that waits.

 Thing is: they lied to you. Long ago they "helpfully" programmed their
 router to announce your route regardless of whether you sent a route
 to them. They want to wait for a maintenance window to remove that
 configuration.


 I'm at a loss of what else I can do. They admit the problem but won't
 take
> action saying it needs to wait for a maintenance window. Am I out of
> line
> insisting that's an unacceptable response to a problem that results in
> prefix/traffic hijacking?
>

 Try dropping the link entirely. If they still announce your addresses,
 bring it back up but report it as emergency down, escalate, and call
 back every 10 minutes until the junior tech understands that it's time
 to call and wake up the guy who makes the decision to fix it now.


>>>
>>> I'm at the tail end here almost 8 hours later since the hijacking
>>> started.
>>> Their NOC is just blowing me off now and they're happy to continue the
>>> hijacking until it's convenient for them to have a maintenance window.
>>> And
>>> that's apparently the final decision.
>>>
>>> ~Seth
>>>
>>
>
>

Re: Prefix hijacking by AS20115

[Hank Nussbacher]

At 23:11 28/09/2015 -0400, Josh Luthman wrote:


Start announcing their prefixes?


Contact the upstreams of AS20115 - Cogent, Level3, HE and XO.

-Hank



Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373
On Sep 28, 2015 11:09 PM, "Seth Mattinen"  wrote:

> On 9/28/15 18:30, William Herrin wrote:
>
>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen 
>> wrote:
>>
>>> I've got a problem where AS20115 continues to announce prefixes after BGP
>>> neighbors were shutdown. They claim it's a wedged BGP process but aren't
>>> in
>>> any hurry to fix it outside of a maintenance window.
>>>
>>
>> If they weren't lying to you, they'd fix it now. That's not the kind
>> of problem that waits.
>>
>> Thing is: they lied to you. Long ago they "helpfully" programmed their
>> router to announce your route regardless of whether you sent a route
>> to them. They want to wait for a maintenance window to remove that
>> configuration.
>>
>>
>> I'm at a loss of what else I can do. They admit the problem but won't take
>>> action saying it needs to wait for a maintenance window. Am I out of line
>>> insisting that's an unacceptable response to a problem that results in
>>> prefix/traffic hijacking?
>>>
>>
>> Try dropping the link entirely. If they still announce your addresses,
>> bring it back up but report it as emergency down, escalate, and call
>> back every 10 minutes until the junior tech understands that it's time
>> to call and wake up the guy who makes the decision to fix it now.
>>
>>
>
> I'm at the tail end here almost 8 hours later since the hijacking started.
> Their NOC is just blowing me off now and they're happy to continue the
> hijacking until it's convenient for them to have a maintenance window. And
> that's apparently the final decision.
>
> ~Seth
>

Re: Prefix hijacking by AS20115

[Bob Evans]

That's something I would do. Announce announce and keep adding ports until
I hit a 10 Gig port worth of traffic or saw it fixed. Be sure to put in a
blackhole route for the prefixes. Try to pick blocks that are as
geographically located to your peering routers as possible ...IE in Reno
pick the blocks that seem to be near by - like Reno, Tahoe, Sacramento
. when that batch of customers makes their phones ring all night
someone will listen.

Would be nice if our membership organization ARIN ( that we all pay to
keep us somewhat organized) had an ability to do something for you I
never looked into it...i don't knowmaybe it does ?

But, in the mean time I am pretty sure you can document this well and
prove your announcements of theirs was due to the fact you couldn't get
proper technical attention and needed to desperately before your customers
cancel after 8 hours of this. Tomorrow call your lawyers and begin to sue
that cable company (did I recognize that ASN as cable TV ? ) for damages
this must be causing you in ill-will amongst your customer base.

I wonder just how you prove the damage...some equation based on customer
calls and complaints together with how many years you have been in
business as well as the number of contracts that are coming up for
renewal. etc etc. Now that would be interesting to see a formula for that
if anyone has been through it.

Thank You
Bob Evans
CTO




> Start announcing their prefixes?
>
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> On Sep 28, 2015 11:09 PM, "Seth Mattinen"  wrote:
>
>> On 9/28/15 18:30, William Herrin wrote:
>>
>>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen 
>>> wrote:
>>>
 I've got a problem where AS20115 continues to announce prefixes after
 BGP
 neighbors were shutdown. They claim it's a wedged BGP process but
 aren't
 in
 any hurry to fix it outside of a maintenance window.

>>>
>>> If they weren't lying to you, they'd fix it now. That's not the kind
>>> of problem that waits.
>>>
>>> Thing is: they lied to you. Long ago they "helpfully" programmed their
>>> router to announce your route regardless of whether you sent a route
>>> to them. They want to wait for a maintenance window to remove that
>>> configuration.
>>>
>>>
>>> I'm at a loss of what else I can do. They admit the problem but won't
>>> take
 action saying it needs to wait for a maintenance window. Am I out of
 line
 insisting that's an unacceptable response to a problem that results in
 prefix/traffic hijacking?

>>>
>>> Try dropping the link entirely. If they still announce your addresses,
>>> bring it back up but report it as emergency down, escalate, and call
>>> back every 10 minutes until the junior tech understands that it's time
>>> to call and wake up the guy who makes the decision to fix it now.
>>>
>>>
>>
>> I'm at the tail end here almost 8 hours later since the hijacking
>> started.
>> Their NOC is just blowing me off now and they're happy to continue the
>> hijacking until it's convenient for them to have a maintenance window.
>> And
>> that's apparently the final decision.
>>
>> ~Seth
>>
>

Re: Prefix hijacking by AS20115

[Seth Mattinen]

On 9/28/15 20:19, Martin Hannigan wrote:


Is this related to 104.73.161.0/24? That's ours. :-)

We'll take a look and get back to you.  Thanks for caring!




Yep, that's one of the affected prefixes.

~Seth

Re: Prefix hijacking by AS20115

[Martin Hannigan]

Is this related to 104.73.161.0/24? That's ours. :-) 

We'll take a look and get back to you.  Thanks for caring! 

Best, 

Marty

> On Sep 28, 2015, at 23:08, Seth Mattinen  wrote:
> 
>> On 9/28/15 18:30, William Herrin wrote:
>>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen  wrote:
>>> I've got a problem where AS20115 continues to announce prefixes after BGP
>>> neighbors were shutdown. They claim it's a wedged BGP process but aren't in
>>> any hurry to fix it outside of a maintenance window.
>> 
>> If they weren't lying to you, they'd fix it now. That's not the kind
>> of problem that waits.
>> 
>> Thing is: they lied to you. Long ago they "helpfully" programmed their
>> router to announce your route regardless of whether you sent a route
>> to them. They want to wait for a maintenance window to remove that
>> configuration.
>> 
>> 
>>> I'm at a loss of what else I can do. They admit the problem but won't take
>>> action saying it needs to wait for a maintenance window. Am I out of line
>>> insisting that's an unacceptable response to a problem that results in
>>> prefix/traffic hijacking?
>> 
>> Try dropping the link entirely. If they still announce your addresses,
>> bring it back up but report it as emergency down, escalate, and call
>> back every 10 minutes until the junior tech understands that it's time
>> to call and wake up the guy who makes the decision to fix it now.
> 
> 
> I'm at the tail end here almost 8 hours later since the hijacking started. 
> Their NOC is just blowing me off now and they're happy to continue the 
> hijacking until it's convenient for them to have a maintenance window. And 
> that's apparently the final decision.
> 
> ~Seth

Re: Prefix hijacking by AS20115

[Josh Luthman]

Start announcing their prefixes?

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373
On Sep 28, 2015 11:09 PM, "Seth Mattinen"  wrote:

> On 9/28/15 18:30, William Herrin wrote:
>
>> On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen 
>> wrote:
>>
>>> I've got a problem where AS20115 continues to announce prefixes after BGP
>>> neighbors were shutdown. They claim it's a wedged BGP process but aren't
>>> in
>>> any hurry to fix it outside of a maintenance window.
>>>
>>
>> If they weren't lying to you, they'd fix it now. That's not the kind
>> of problem that waits.
>>
>> Thing is: they lied to you. Long ago they "helpfully" programmed their
>> router to announce your route regardless of whether you sent a route
>> to them. They want to wait for a maintenance window to remove that
>> configuration.
>>
>>
>> I'm at a loss of what else I can do. They admit the problem but won't take
>>> action saying it needs to wait for a maintenance window. Am I out of line
>>> insisting that's an unacceptable response to a problem that results in
>>> prefix/traffic hijacking?
>>>
>>
>> Try dropping the link entirely. If they still announce your addresses,
>> bring it back up but report it as emergency down, escalate, and call
>> back every 10 minutes until the junior tech understands that it's time
>> to call and wake up the guy who makes the decision to fix it now.
>>
>>
>
> I'm at the tail end here almost 8 hours later since the hijacking started.
> Their NOC is just blowing me off now and they're happy to continue the
> hijacking until it's convenient for them to have a maintenance window. And
> that's apparently the final decision.
>
> ~Seth
>

Re: Prefix hijacking by AS20115

[Seth Mattinen]

On 9/28/15 18:30, William Herrin wrote:

On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen  wrote:

I've got a problem where AS20115 continues to announce prefixes after BGP
neighbors were shutdown. They claim it's a wedged BGP process but aren't in
any hurry to fix it outside of a maintenance window.


If they weren't lying to you, they'd fix it now. That's not the kind
of problem that waits.

Thing is: they lied to you. Long ago they "helpfully" programmed their
router to announce your route regardless of whether you sent a route
to them. They want to wait for a maintenance window to remove that
configuration.



I'm at a loss of what else I can do. They admit the problem but won't take
action saying it needs to wait for a maintenance window. Am I out of line
insisting that's an unacceptable response to a problem that results in
prefix/traffic hijacking?


Try dropping the link entirely. If they still announce your addresses,
bring it back up but report it as emergency down, escalate, and call
back every 10 minutes until the junior tech understands that it's time
to call and wake up the guy who makes the decision to fix it now.




I'm at the tail end here almost 8 hours later since the hijacking 
started. Their NOC is just blowing me off now and they're happy to 
continue the hijacking until it's convenient for them to have a 
maintenance window. And that's apparently the final decision.


~Seth

Re: Prefix hijacking by AS20115

[William Herrin]

On Mon, Sep 28, 2015 at 9:01 PM, Seth Mattinen  wrote:
> I've got a problem where AS20115 continues to announce prefixes after BGP
> neighbors were shutdown. They claim it's a wedged BGP process but aren't in
> any hurry to fix it outside of a maintenance window.

If they weren't lying to you, they'd fix it now. That's not the kind
of problem that waits.

Thing is: they lied to you. Long ago they "helpfully" programmed their
router to announce your route regardless of whether you sent a route
to them. They want to wait for a maintenance window to remove that
configuration.


> I'm at a loss of what else I can do. They admit the problem but won't take
> action saying it needs to wait for a maintenance window. Am I out of line
> insisting that's an unacceptable response to a problem that results in
> prefix/traffic hijacking?

Try dropping the link entirely. If they still announce your addresses,
bring it back up but report it as emergency down, escalate, and call
back every 10 minutes until the junior tech understands that it's time
to call and wake up the guy who makes the decision to fix it now.

Regards,
Bill Herrin



-- 
William Herrin  her...@dirtside.com  b...@herrin.us
Owner, Dirtside Systems . Web: 

Prefix hijacking by AS20115

[Seth Mattinen]

I've got a problem where AS20115 continues to announce prefixes after 
BGP neighbors were shutdown. They claim it's a wedged BGP process but 
aren't in any hurry to fix it outside of a maintenance window.


I'm at a loss of what else I can do. They admit the problem but won't 
take action saying it needs to wait for a maintenance window. Am I out 
of line insisting that's an unacceptable response to a problem that 
results in prefix/traffic hijacking?


~Seth