My best parsing of that ticket, with some guesses :
- Infinera management card goes Really Bad, knocks out local waves, and
starts spewing garbage out onto the management network
- Management network propagates the garbage , other Infinera management
cards get it and fall into the same state,
On 12/31/18 3:31 PM, Keith Medcalf wrote:
It could have been worse:
https://www.cio.com.au/article/65115/all_systems_down/
"Make network changes only between 2am and 5am on weekends."
Wow. Just wow. I suppose the IT types are considerably different than Process
Operations. Our rule is
On 12/31/18, Keith Medcalf wrote:
>> It could have been worse:
>> https://www.cio.com.au/article/65115/all_systems_down/
>
> "Make network changes only between 2am and 5am on weekends."
>
> Wow. Just wow.
yeah. out of all the possible lessons they could have learned..
> I suppose the IT
On Mon, Dec 31, 2018 at 7:24 AM Naslund, Steve wrote:
> Bad design if that’s the case, that would be a huge subnet.
According to the notes at the URL Saku shared, they suffered a cascade
failure from which they needed the equipment vendor's help to recover.
That indicates at least two grave
On Mon, Dec 31, 2018 at 12:31 PM Keith Medcalf wrote:
> > It could have been worse:
> > https://www.cio.com.au/article/65115/all_systems_down/
>
> "Make network changes only between 2am and 5am on weekends."
>
> Wow. Just wow. I suppose the IT types are considerably different
> than Process
> It could have been worse:
> https://www.cio.com.au/article/65115/all_systems_down/
"Make network changes only between 2am and 5am on weekends."
Wow. Just wow. I suppose the IT types are considerably different than Process
Operations. Our rule is to only make changes scheduled at 09:00
A note for the guys hanging on to those POTS lines…It won’t really help. One
of our sites in Dubuque Iowa had ten CenturyLink PRIs (they are the LEC there)
homed off of a 5ESS switch. These all were unable to process calls during the
CenturyLink problem. The ISDN messaging returned indicated
This seems entirely plausible given that DWDM amplifiers and lasers being a
complex analog system, they need OOB to align.
--
Eric
> On 31 Dec 2018, at 16:06, Saku Ytti wrote:
>
> Hey Steve,
>
> I will continue to speculate, as that's all we have.
>
>> 1. Are you telling me that several
with this stuff.
> >
> >
> >
> > -
> > Mike Hammett
> > Intelligent Computing Solutions
> > http://www.ics-il.com
> >
> > Midwest-IX
> > http://www.midwest-ix.com
> >
> >
> > From: "Saku Ytti&quo
mputing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"Saku Ytti"
> *To: *"nanog list"
> *Sent: *Sunday, December 30, 2018 7:42:49 AM
> *Subject: *CenturyLink RCA?
>
On 12/31/18, Aaron1 wrote:
> Yeah, could have been one of those...gone from bad to worse things like Dave
> mentioned... initial problem and course of action perhaps led to a worse
> problem.
>
> I’ve had DWDM issues that have taken down multiple locations far apart from
> each other due to how
Yeah, could have been one of those...gone from bad to worse things like Dave
mentioned... initial problem and course of action perhaps led to a worse
problem.
I’ve had DWDM issues that have taken down multiple locations far apart from
each other due to how the transport guys hauled stuff
A
On Mon, Dec 31, 2018 at 11:33 AM Naslund, Steve
wrote:
> They shouldn’t need OOB to operate existing lambdas just to configure new
> ones. One possibility is that the management interface also handles master
> timing which would be a really bad idea but possible (should be redundant
> and it
They shouldn’t need OOB to operate existing lambdas just to configure new ones.
One possibility is that the management interface also handles master timing
which would be a really bad idea but possible (should be redundant and it
should be able to free run for a reasonable amount of time).
I agree 100%. Now they need to figure out why bricking the management network
stopped forwarding on the optical side. > (Forgive my top posting, not on my
desktop as I’m out of town)
Steven Naslund
Chicago IL
>
>Wild guess, based on my own experience as a NOC admin/head of operations at a
See my comments in line.
Steve
>Hey Steve,
>I will continue to speculate, as that's all we have.
> 1. Are you telling me that several line cards failed in multiple cities in
> the same way at the same time? Don't think so unless the same software fault
> was propagated to all of them. If
t wrote:
>>
>> It's technical enough so that laypeople immediately lose interest, yet
>> completely useless to anyone that works with this stuff.
>>
>>
>>
>> -
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>
Hey Steve,
I will continue to speculate, as that's all we have.
> 1. Are you telling me that several line cards failed in multiple cities in
> the same way at the same time? Don't think so unless the same software fault
> was propagated to all of them. If the problem was that they needed to
.com
>
> ____
> From: "Saku Ytti"
> To: "nanog list"
> Sent: Sunday, December 30, 2018 7:42:49 AM
> Subject: CenturyLink RCA?
>
> Apologies for the URL, I do not know official source and I do not
> share the URLs sentiment.
>
uting Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>
> From: "Saku Ytti"
> To: "nanog list"
> Sent: Sunday, December 30, 2018 7:42:49 AM
> Subject: CenturyLink RCA?
>
> Apol
*From: *"Saku Ytti"
*To: *"nanog list"
*Sent: *Sunday, December 30, 2018 7:42:49 AM
*Subject: *CenturyLink RCA?
Apologies for the URL, I do not know official source and I do not
share the URLs sentiment.
https://fuckingcenturylink.com/
Can someone
Ytti"
To: "nanog list"
Sent: Sunday, December 30, 2018 7:42:49 AM
Subject: CenturyLink RCA?
Apologies for the URL, I do not know official source and I do not
share the URLs sentiment.
https://fuckingcenturylink.com/
Can someone translate this to IP engineer? What did act
Apologies for the URL, I do not know official source and I do not
share the URLs sentiment.
https://fuckingcenturylink.com/
Can someone translate this to IP engineer? What did actually happen?
>From my own history, I rarely recognise the problem I fixed from
reading the public RCA. I hope
23 matches
Mail list logo
