Re: Are the days of the showpiece NOC office display gone forever?

[Ben Cannon]

It’d be real interesting to open-source this somehow, produce a useable open or 
quasi open (maybe curated somehow) reputation score for email. 

Ms. Lady Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net
"The only fully end-to-end encrypted global telecommunications company in the 
world.”

FCC License KJ6FJJ

Sent from my iPhone via RFC1149.

> On Dec 30, 2020, at 3:04 PM, Rich Kulawiec  wrote:
> 
> On Tue, Dec 22, 2020 at 10:41:43PM -0700, Wayne Bouchard wrote:
>> And if the last 15 years has shown us anything, it is that when you
>> can't get past the auto-attendant and talk to a real human, and if
>> that person can't talk to you like a person instead of reading scripts
>> at you, your stress levels go way up as does your desire to break
>> things. Automation in customer service (or excessive emphasis on
>> procedures) is a really nice way of taking a five minute problem and
>> turning it into an hour long ordeal.
> 
> There are some easy methods for service/support organizations to decrease
> the pain that this inflicts on people reporting problems.
> 
> For example, one thing that I've taught people to do is to make liberal
> use of procmail in order to sort incoming traffic to role accounts.
> It requires diligence, but that diligence is repaid many times over by
> how it expedites dealing with problems.  A simple example of this is
> that when a problem report is received at the RFC 2142 security@ role
> address, and it's clueful, well-written, and important, a procmail rule
> gets created for the sending address so that all future messages from
> that address are prioritized...because it obviously came from someone who
> knows what the heck they're doing and did us a favor by telling us that
> we have a problem.  Chances are that any future messages from them will
> be similarly helpful and that if we respond to those quickly we may be
> able to forestall a lot more messages that aren't going to be as clueful.
> 
> The opposite thing is done with clueless/misdirected/etc. reports:
> they're not discarded, but they go into the low-priority queue.
> 
> Everything else goes somewhere in the middle.
> 
> Repeated hundreds or thousands of times over many years, this builds a
> ruleset that pre-sorts messages rather well.  It's not perfect, it's not
> foolproof, but it helps us *and* it helps lower the frustration level of
> people sending clueful messages, because it better positions us to read,
> act on, and respond to those.  Those people are catching our mistakes,
> the least we can do is try to pay attention.
> 
> (Hint: a useful way to begin building such a ruleset is to grab all the
> addresses from NANOG, dnsops, outages, etc. and pre-load the ruleset
> with them...because traffic received at role accounts from participants
> in these mailing lists is probably useful.)
> 
> ---rsk

Re: Are the days of the showpiece NOC office display gone forever?

[Rich Kulawiec]

On Tue, Dec 22, 2020 at 10:41:43PM -0700, Wayne Bouchard wrote:
> And if the last 15 years has shown us anything, it is that when you
> can't get past the auto-attendant and talk to a real human, and if
> that person can't talk to you like a person instead of reading scripts
> at you, your stress levels go way up as does your desire to break
> things. Automation in customer service (or excessive emphasis on
> procedures) is a really nice way of taking a five minute problem and
> turning it into an hour long ordeal.

There are some easy methods for service/support organizations to decrease
the pain that this inflicts on people reporting problems.

For example, one thing that I've taught people to do is to make liberal
use of procmail in order to sort incoming traffic to role accounts.
It requires diligence, but that diligence is repaid many times over by
how it expedites dealing with problems.  A simple example of this is
that when a problem report is received at the RFC 2142 security@ role
address, and it's clueful, well-written, and important, a procmail rule
gets created for the sending address so that all future messages from
that address are prioritized...because it obviously came from someone who
knows what the heck they're doing and did us a favor by telling us that
we have a problem.  Chances are that any future messages from them will
be similarly helpful and that if we respond to those quickly we may be
able to forestall a lot more messages that aren't going to be as clueful.

The opposite thing is done with clueless/misdirected/etc. reports:
they're not discarded, but they go into the low-priority queue.

Everything else goes somewhere in the middle.

Repeated hundreds or thousands of times over many years, this builds a
ruleset that pre-sorts messages rather well.  It's not perfect, it's not
foolproof, but it helps us *and* it helps lower the frustration level of
people sending clueful messages, because it better positions us to read,
act on, and respond to those.  Those people are catching our mistakes,
the least we can do is try to pay attention.

(Hint: a useful way to begin building such a ruleset is to grab all the
addresses from NANOG, dnsops, outages, etc. and pre-load the ruleset
with them...because traffic received at role accounts from participants
in these mailing lists is probably useful.)

---rsk

Re: Are the days of the showpiece NOC office display gone forever?

[Allen Kitchen]

Can't resist two comments:

For those who might otherwise wonder, "Flapper" has no connection to the US
Roaring Twenties, but was a concept pioneered by Jonathan Swift in
"Gulliver's Travels" particular to the land of Laputa (NOT to be confused
with Lilliput, with which many are familiar..) - where certain people are
so self-important that they are accompanied at all times by a dedicated
servant who, when in the opinion of the servant the master would benefit by
awareness of ome event or situation, would (gently) cudgel the master with
a bladder of dried peas or pebbles, either on the ears or the eyes, so that
the master's attention would be drawn from their intense interior
environment to something occurring in the mundane world. (In my opinion,
this was Swift's most trenchant critique of civil society.)

For the second comment - no discussion of a flapper bypass system is
complete without:
https://xkcd.com/806
Oh, have I often wished...

Blessings..

..Allen









Allen M. Kitchen
404 Franklin Street, Butler PA 16001
412-295-7763 (cell; text OK 24x7)
allenmckinleykitc...@gmail.com


On Wed, Dec 23, 2020 at 12:12 PM Keith Medcalf  wrote:

> On Tuesday, 22 December, 2020 22:42, Wayne Bouchard wrote:
> >On Wed, Dec 23, 2020 at 02:58:32PM +1000, Robert Brockway wrote:
> >> On Thu, 17 Dec 2020, Tom Beecher wrote:
>
> >> If the last 50 years has shown us anything it is that humans and
> >> computers working together can achieve far more than either in
> >> isolation.
>
> > And if the last 15 years has shown us anything, it is that when you
> > can't get past the auto-attendant and talk to a real human, and if
> > that person can't talk to you like a person instead of reading scripts
> > at you, your stress levels go way up as does your desire to break
> > things. Automation in customer service (or excessive emphasis on
> > procedures) is a really nice way of taking a five minute problem and
> > turning it into an hour long ordeal.
>
> The correct term of art for these humans that execute "scripts" when
> answering inquiries is "flapper".  Often there are multiple layers of
> "flapper" before one can communicate with someone who has any clue what
> they (or you) are speaking about.  This is deliberate in an attempt to
> cull the wheat from the chaff of the support calls because 99.999%
> of callers are "chaff" and do not have a problem that is worthy of
> attention by someone who knows what they are doing.
>
> Many organizations which employ "flappers" have "flapper bypass systems"
> in place in which there are either "magical incantations" or perhaps an
> entry in the CRM system that identifies the probability that a caller is
> "wheat" vs "chaff" so that the multi-level flapper system can be
> bypassed.
>
> There are even a few organizations which do not employ flappers at all
> -- though they are few and far between.
>
> If an organization does not have a functional "flapper bypass system"
> then usually the most effective system to bypass multiple layers of
> flappers is what is called the "shit principle".  The "shit principle"
> states that shit works best when flowing downhill and therefore the most
> effective "flapper bypass" is to direct the problem to the higest level
> executive in the organization with the least probability of being able
> to address the issue.  That person will simply direct an under-thing to
> "take care of this and have them stop bothering me" which will result in
> you immediately having the issue resolved by a competent individual.
>
> Organizations without other functional "flapper bypass" procedures
> usually have a huge organization dedicated to addressing issues raised
> through the "shit principle" since this is, in reality, their chosen
> "flapper bypass" system.
>
> --
> Be decisive.  Make a decision, right or wrong.  The road of life is
> paved with flat squirrels who could not make a decision.
>
>
>
>

RE: Are the days of the showpiece NOC office display gone forever?

[Keith Medcalf]

On Tuesday, 22 December, 2020 22:42, Wayne Bouchard wrote:
>On Wed, Dec 23, 2020 at 02:58:32PM +1000, Robert Brockway wrote:
>> On Thu, 17 Dec 2020, Tom Beecher wrote:

>> If the last 50 years has shown us anything it is that humans and
>> computers working together can achieve far more than either in
>> isolation.

> And if the last 15 years has shown us anything, it is that when you
> can't get past the auto-attendant and talk to a real human, and if
> that person can't talk to you like a person instead of reading scripts
> at you, your stress levels go way up as does your desire to break
> things. Automation in customer service (or excessive emphasis on
> procedures) is a really nice way of taking a five minute problem and
> turning it into an hour long ordeal.

The correct term of art for these humans that execute "scripts" when
answering inquiries is "flapper".  Often there are multiple layers of
"flapper" before one can communicate with someone who has any clue what
they (or you) are speaking about.  This is deliberate in an attempt to
cull the wheat from the chaff of the support calls because 99.999%
of callers are "chaff" and do not have a problem that is worthy of
attention by someone who knows what they are doing.

Many organizations which employ "flappers" have "flapper bypass systems"
in place in which there are either "magical incantations" or perhaps an
entry in the CRM system that identifies the probability that a caller is
"wheat" vs "chaff" so that the multi-level flapper system can be
bypassed.

There are even a few organizations which do not employ flappers at all
-- though they are few and far between.

If an organization does not have a functional "flapper bypass system"
then usually the most effective system to bypass multiple layers of
flappers is what is called the "shit principle".  The "shit principle"
states that shit works best when flowing downhill and therefore the most
effective "flapper bypass" is to direct the problem to the higest level
executive in the organization with the least probability of being able
to address the issue.  That person will simply direct an under-thing to
"take care of this and have them stop bothering me" which will result in
you immediately having the issue resolved by a competent individual.

Organizations without other functional "flapper bypass" procedures
usually have a huge organization dedicated to addressing issues raised
through the "shit principle" since this is, in reality, their chosen
"flapper bypass" system.

--
Be decisive.  Make a decision, right or wrong.  The road of life is
paved with flat squirrels who could not make a decision.

Re: Are the days of the showpiece NOC office display gone forever?

[Mark Tinka]

On 12/23/20 07:41, Wayne Bouchard wrote:


And if the last 15 years has shown us anything, it is that when you
can't get past the auto-attendant and talk to a real human, and if
that person can't talk to you like a person instead of reading scripts
at you, your stress levels go way up as does your desire to break
things. Automation in customer service (or excessive emphasis on
procedures) is a really nice way of taking a five minute problem and
turning it into an hour long ordeal.

(pet peeve)


The good news is that choice deals with this problem.

The level of patience we've had to allow this type of customer 
interaction has been drastically reduced by our experiences with free or 
paid services we experience with apps on our phones. Without realizing 
it, our basic expectations rise from how we experience one app that has 
nothing to do with the other. It put more stock in choice.


Either we are deleting an app 5 seconds after it doesn't meet our 
re-trained expectations, or we are cancelling a contract and moving on 
to another provider.


I, like you, refuse to call a call centre and ask for support. The same 
goes for online services whose support is limited to bots or FAQ's. The 
moment I can't get a fix via an e-mail and I have to speak to someone 
waiting with a script, I cancel and move on.


Mark.

Re: Are the days of the showpiece NOC office display gone forever?

[Wayne Bouchard]

On Wed, Dec 23, 2020 at 02:58:32PM +1000, Robert Brockway wrote:
> On Thu, 17 Dec 2020, Tom Beecher wrote:
> If the last 50 years has shown us anything it is that humans and computers 
> working together can achieve far more than either in isolation.
> 
> Cheers,
> 
> Rob

And if the last 15 years has shown us anything, it is that when you
can't get past the auto-attendant and talk to a real human, and if
that person can't talk to you like a person instead of reading scripts
at you, your stress levels go way up as does your desire to break
things. Automation in customer service (or excessive emphasis on
procedures) is a really nice way of taking a five minute problem and
turning it into an hour long ordeal.

(pet peeve)

-Wayne

---
Wayne Bouchard
w...@typo.org
Network Dude
http://www.typo.org/~web/

Re: Are the days of the showpiece NOC office display gone forever?

[Robert Brockway]

On Thu, 17 Dec 2020, Tom Beecher wrote:



I'm sure when the automation is perfect and widespread to the point that
it catches and alerts on every network event, the monitoring rooms will
disappear.



The chances of this happening are exactly 0%.


Indeed.  More broadly, a lot of people have tried to get rid of operations 
staff and suffered the consequences.



Contrary to what salespeople will say, the answer is not 100% automation,
or 100% humans. The proper answer is an often changing combination of the
two.


Exactly.  There is an argument to be said that human operators are 
actually part of the computer system.  This is implied in terms like 
'wetware' but not often explicitely stated.


If the last 50 years has shown us anything it is that humans and computers 
working together can achieve far more than either in isolation.


Cheers,

Rob

Re: Are the days of the showpiece NOC office display gone forever?

[Ben Cannon]

I’ve learned that the secret is automation + intelligent trained and empowered 
staff.

Ms. Lady Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net
"The only fully end-to-end encrypted global telecommunications company in the 
world.”

FCC License KJ6FJJ

Sent from my iPhone via RFC1149.

> On Dec 17, 2020, at 3:17 PM, Peter E. Fry  wrote:
> 
> 
> 
>> Subject: Re: Are the days of the showpiece NOC office display gone forever? 
>> From: Tom Beecher  
>> To: Matt Erculiani 
>> Cc: NANOG Operators' Group  
>> Date: Thursday, 12/17/2020 11:59:37
> [...] 
>> Contrary to what salespeople will say, the answer is not 100% automation, or 
>> 100% humans. The proper answer is an often changing combination of the two. 
> I believe the desired combination is automation + button-pushing monkeys.
> 
>> ML is not the magical unicorn solution that solves everything, contrary to 
>> what many papers and salespeople tell you. [...]
> But which will write Shakespeare first?  I'd bet on the monkeys, although 
> both come up with some unique and unanticipated ways to fail.
> 
> Oh well.  Back to pushing buttons.  Someday my masterpiece will be complete...
> 
> Peter E. Fry
> 
> 

Re: Are the days of the showpiece NOC office display gone forever?

[Peter E . Fry]

Subject: Re: Are the days of the showpiece NOC office display gone 
forever?

From: Tom Beecher 
To: Matt Erculiani 
Cc: NANOG Operators' Group 
Date: Thursday, 12/17/2020 11:59:37


[...]



Contrary to what salespeople will say, the answer is not 100% 
automation, or 100% humans. The proper answer is an often changing 
combination of the two.


I believe the desired combination is automation + button-pushing 
monkeys.






ML is not the magical unicorn solution that solves everything, 
contrary to what many papers and salespeople tell you. [...]


But which will write Shakespeare first?  I'd bet on the monkeys, 
although both come up with some unique and unanticipated ways to fail.


Oh well.  Back to pushing buttons.  Someday my masterpiece will be 
complete...


Peter E. Fry

Re: Are the days of the showpiece NOC office display gone forever?

[Tom Beecher]

>
> I'm sure when the automation is perfect and widespread to the point that
> it catches and alerts on every network event, the monitoring rooms will
> disappear.
>

The chances of this happening are exactly 0%.


> But unless you have an entire organization dedicated to automation
> development or pay an incredibly large sum of money for pre-built packages,
> the business decision may still be made to actively monitor the network
> with eyeballs.
>

Contrary to what salespeople will say, the answer is not 100% automation,
or 100% humans. The proper answer is an often changing combination of the
two.

 Every failure mode is known until a new one pops up. Automation without
> any kind of ML secret sauce relies on known failure-modes.


ML is not the magical unicorn solution that solves everything, contrary to
what many papers and salespeople tell you. Let's take a network interface
that is randomly shitting on packets. What is more important operationally,
identifying that the packets are being shat on, or having ML predict when
the next shatting will occur? Clearly the first right? You want to find it
and fix it as fast as possible. There could be a place for ML when it comes
to diagnosing the reason for the shitting , but that's different.

There may be some interesting applications for ML in tracking down really
complicated operational anomalies, but it will never be a primary mode of
detection for the same reason you said ; Every failure mode is known until
a new one pops up. You can't train an ML model for a failure condition that
you don't know exists.

On Wed, Dec 16, 2020 at 5:51 PM Matt Erculiani  wrote:

> >  That is not to say that large monitoring rooms are a better choice over
> automation (which they are not).
>
> I'm sure when the automation is perfect and widespread to the point that
> it catches and alerts on every network event, the monitoring rooms will
> disappear.
>
> But unless you have an entire organization dedicated to automation
> development or pay an incredibly large sum of money for pre-built packages,
> the business decision may still be made to actively monitor the network
> with eyeballs.
>
> Every failure mode is known until a new one pops up. Automation without
> any kind of ML secret sauce relies on known failure-modes.
>
> Not advocating one or the other, just playing Devil's advocate for the
> Devil's advocate.
>
> -Matt
>
> On Wed, Dec 16, 2020 at 2:28 PM Töma Gavrichenkov 
> wrote:
>
>> Peace,
>>
>> On Thu, Dec 17, 2020, 12:21 AM Lady Benjamin PD Cannon 
>> wrote:
>>
>>> We are still operating ours - 27 1080P projectors - but with a skeleton
>>> crew of just 3.  Given the air volume, it’s almost like outside.
>>>
>>
>> A devil advocate here,
>>
>> First of all, COVID-19 is really serious.
>>
>> With that in mind, with all the necessary precautions office space *may*
>> be managed safely to prevent the spread.
>>
>> Production plants had security measures preventing workforce injuries for
>> a century already.  Just a bit of that, with constant trainings, would get
>> your monitoring room safe, especially with all the bars closed and
>> everything operating on delivery.
>>
>> That is not to say that large monitoring rooms are a better choice over
>> automation (which they are not).
>>
>> --
>> Töma
>>
>>>
>
> --
> Matt Erculiani
> ERCUL-ARIN
>

Re: Are the days of the showpiece NOC office display gone forever?

[Bryan Holloway]

"I'd piss on a spark plug if I thought it'd do any good ..."

(Ed.: I'd take that "NOC" any day.)


On 12/17/20 4:33 PM, Joe Provo wrote:

On Wed, Dec 16, 2020 at 12:49:52PM -0800, Eric Kuhnke wrote:
[snip]

Are the days of such an environment gone forever?


We can only hope so.

Re: Are the days of the showpiece NOC office display gone forever?

[Joe Provo]

On Wed, Dec 16, 2020 at 12:49:52PM -0800, Eric Kuhnke wrote:
[snip]
> Are the days of such an environment gone forever?

We can only hope so.

-- 
Posted from my personal account - see X-Disclaimer header.
Joe Provo / Gweep / Earthling 

Re: Are the days of the showpiece NOC office display gone forever?

[Michael Perkins via NANOG]

 I remember working in the showpiece "Uncle Bernie" Ebbers had built in 
Ashburn, VA, for UUNET. You can even catch a glimpse of me in the American 
Greed episode dedicated to WorldCom's downfall. I wonder just what that place 
looks like now. Since then, I have seen NOCs with multiple displays for 
multiple customers, but only designed to be useful to NOC staff, not 
prospective customers or executives looking for something pretty to watch. 
 
-Original Message-
From: Eric Kuhnke 
To: nanog@nanog.org list 
Sent: Wed, Dec 16, 2020 3:49 pm
Subject: Are the days of the showpiece NOC office display gone forever?

With the covid19 situation, obviously lots of ISPs have their NOC personnel 
working from home, with VPN (or remote desktop) access to all the internal 
tools, VoIP at home, etc.
In the traditional sense, by "showpiece NOC" I mean a room designed for the 
purpose of having large situational awareness displays on a wall, network 
weathermaps and charts, alerting systems, composed of four or more big flat 
panel displays. Ideally configured to be actually useful for NOC purposes and 
also something impressive looking for customer tours.
To what extent potential customers find that sort of thing to be a signifier of 
seriousness on the part of an ISP, I suppose depends on what sort of customers 
they are, and their relative degree of technical sophistication.

Are the days of such an environment gone forever? 

Re: Are the days of the showpiece NOC office display gone forever?

[Mark Tinka]

On 12/17/20 00:50, Matt Erculiani wrote:

But unless you have an entire organization dedicated to automation 
development or pay an incredibly large sum of money for pre-built 
packages, the business decision may still be made to actively monitor 
the network with eyeballs.


Solutions do exist, or so the salesmen/women that peddle them tell me.

I'm not convinced. Also, I'm a bit ancient.

Mark.

Re: Are the days of the showpiece NOC office display gone forever?

[Mark Tinka]

On 12/16/20 23:48, Max Harmony via NANOG wrote:


It seems like the "something impressive looking" might just change. Add a bit of 
marketing copy about how "our monitoring is distributed, just like our servers" (or some 
more punchy variant) and show customers what the smaller monitoring stations WFH employees are 
using look like.


Nowadays, customers tend to care about you watching their circuit, and 
service.


Most still get surprised when they don't understand why the NOC never 
knew their circuit went down, and need help understanding the difference 
between monitoring the core, and monitoring customers who choose to 
purchase that service.


Mark.

Re: Are the days of the showpiece NOC office display gone forever?

[Mark Tinka]

On 12/16/20 22:49, Eric Kuhnke wrote:

With the covid19 situation, obviously lots of ISPs have their NOC 
personnel working from home, with VPN (or remote desktop) access to 
all the internal tools, VoIP at home, etc.


In the traditional sense, by "showpiece NOC" I mean a room designed 
for the purpose of having large situational awareness displays on a 
wall, network weathermaps and charts, alerting systems, composed of 
four or more big flat panel displays. Ideally configured to be 
actually useful for NOC purposes and also something impressive looking 
for customer tours.


To what extent potential customers find that sort of thing to be a 
signifier of seriousness on the part of an ISP, I suppose depends on 
what sort of customers they are, and their relative degree of 
technical sophistication.


Are the days of such an environment gone forever?


The customers we've been dealing with, particularly this year, fancy 
being listened and spoken to more than what our brand of NOC coffee is.


Mark.

Re: Are the days of the showpiece NOC office display gone forever?

[Eric Kuhnke]

Perhaps I should have clarified: "from the perspective of persons who have
the word "Sales" in their job titles, considered to be impressive looking
for customer tours"


On Wed, Dec 16, 2020 at 4:25 PM Randy Bush  wrote:

> > In the traditional sense, by "showpiece NOC" I mean a room designed for
> the
> > purpose of having large situational awareness displays on a wall, network
> > weathermaps and charts, alerting systems, composed of four or more big
> flat
> > panel displays. Ideally configured to be actually useful for NOC purposes
> > and also something impressive looking for customer tours.
>
> though your message has a current date, its content seems to be at least
> 15 years old
>
> randy
>

Re: Are the days of the showpiece NOC office display gone forever?

[James R Cutler]

> On Dec 16, 2020, at 7:25 PM, Randy Bush  wrote:
> 
>> In the traditional sense, by "showpiece NOC" I mean a room designed for the
>> purpose of having large situational awareness displays on a wall, network
>> weathermaps and charts, alerting systems, composed of four or more big flat
>> panel displays. Ideally configured to be actually useful for NOC purposes
>> and also something impressive looking for customer tours.
> 
> though your message has a current date, its content seems to be at least
> 15 years old
> 
> randy

The EDS Network Management Center, or IMS, had multiple large screens, 
shadow-free lighting, and lots of console positions and a visitor gallery with 
curtained window overlooking the operations room. We finished it right around 
1990. Right when the suburban sprawl ws starting to hit Plano. After so much 
there during construction, I was taken aback when I realized I did not 
recognize the inside the building with it’s finished walls.

That implies “at least 15 years” could well be “30 years”
.

James R. Cutler
james.cut...@consultant.com
GPG keys: hkps://hkps.pool.sks-keyservers.net

Re: Are the days of the showpiece NOC office display gone forever?

[Randy Bush]

> In the traditional sense, by "showpiece NOC" I mean a room designed for the
> purpose of having large situational awareness displays on a wall, network
> weathermaps and charts, alerting systems, composed of four or more big flat
> panel displays. Ideally configured to be actually useful for NOC purposes
> and also something impressive looking for customer tours.

though your message has a current date, its content seems to be at least
15 years old

randy

Re: Are the days of the showpiece NOC office display gone forever?

[Töma Gavrichenkov]

Peace,

On Thu, Dec 17, 2020, 1:50 AM Matt Erculiani  wrote:

> I'm sure when the automation is perfect and widespread to the point that
> it catches and alerts on every network event, the monitoring rooms will
> disappear.
>

Which is never, but:

With a proper RCA after each incident, not blaming customers or peers
(which is common) but accepting the blame, and not personally, but as a
fault of failover, security and notification mechanisms to be refined, and
by employing third party tools trained on big chunks of data of yours and
others

— you can reduce the number of screens required for daily routine to a
sustainable amount that a single onshift engineer can carry home.

And that is around where the investment returns to you.

--
Töma

Re: Are the days of the showpiece NOC office display gone forever?

[Matt Erculiani]

>  That is not to say that large monitoring rooms are a better choice over
automation (which they are not).

I'm sure when the automation is perfect and widespread to the point that it
catches and alerts on every network event, the monitoring rooms will
disappear.

But unless you have an entire organization dedicated to automation
development or pay an incredibly large sum of money for pre-built packages,
the business decision may still be made to actively monitor the network
with eyeballs.

Every failure mode is known until a new one pops up. Automation without any
kind of ML secret sauce relies on known failure-modes.

Not advocating one or the other, just playing Devil's advocate for the
Devil's advocate.

-Matt

On Wed, Dec 16, 2020 at 2:28 PM Töma Gavrichenkov  wrote:

> Peace,
>
> On Thu, Dec 17, 2020, 12:21 AM Lady Benjamin PD Cannon 
> wrote:
>
>> We are still operating ours - 27 1080P projectors - but with a skeleton
>> crew of just 3.  Given the air volume, it’s almost like outside.
>>
>
> A devil advocate here,
>
> First of all, COVID-19 is really serious.
>
> With that in mind, with all the necessary precautions office space *may*
> be managed safely to prevent the spread.
>
> Production plants had security measures preventing workforce injuries for
> a century already.  Just a bit of that, with constant trainings, would get
> your monitoring room safe, especially with all the bars closed and
> everything operating on delivery.
>
> That is not to say that large monitoring rooms are a better choice over
> automation (which they are not).
>
> --
> Töma
>
>>

-- 
Matt Erculiani
ERCUL-ARIN

Re: Are the days of the showpiece NOC office display gone forever?

[Max Harmony via NANOG]

On 16 Dec 2020, at 15.49, Eric Kuhnke  wrote:
> 
> With the covid19 situation, obviously lots of ISPs have their NOC personnel 
> working from home, with VPN (or remote desktop) access to all the internal 
> tools, VoIP at home, etc.
> 
> In the traditional sense, by "showpiece NOC" I mean a room designed for the 
> purpose of having large situational awareness displays on a wall, network 
> weathermaps and charts, alerting systems, composed of four or more big flat 
> panel displays. Ideally configured to be actually useful for NOC purposes and 
> also something impressive looking for customer tours.
> 
> To what extent potential customers find that sort of thing to be a signifier 
> of seriousness on the part of an ISP, I suppose depends on what sort of 
> customers they are, and their relative degree of technical sophistication.
> 
> Are the days of such an environment gone forever?

It seems like the "something impressive looking" might just change. Add a bit 
of marketing copy about how "our monitoring is distributed, just like our 
servers" (or some more punchy variant) and show customers what the smaller 
monitoring stations WFH employees are using look like.


signature.asc
Description: Message signed with OpenPGP

Re: Are the days of the showpiece NOC office display gone forever?

[Miles Fidelman]

Dave Cohen wrote:
Frankly, I think the days of the "showpiece NOC" being relevant ended 
a while before COVID. I worked at a Tier 1 for >10 years in a 
customer-facing capacity, largely dealing with "serious" enterprise 
customers. The number of customers who toured the NOC in that time was 
less than 5 (i.e much less than 1%), despite generally offering it up 
to new and quickly-scaling customers. I actually spent more time 
visiting customer NOCs over that time than I did my own with customers 
(and some of theirs were more impressive anyway).
I just remember that, back in my BBN days (a LONG time ago), the various 
Defense Data Network NOCs, around the world, used to keep their 
big-screen monitors tuned to CNN.


Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is.   Yogi Berra

Theory is when you know everything but nothing works.
Practice is when everything works but no one knows why.
In our lab, theory and practice are combined:
nothing works and no one knows why.  ... unknown

Re: Are the days of the showpiece NOC office display gone forever?

[Tom Beecher]

>
>  Ideally configured to be actually useful for NOC purposes and also
> something impressive looking for customer tours.
>

Call me crazy, but I have never cared about the second half of that.

On Wed, Dec 16, 2020 at 3:51 PM Eric Kuhnke  wrote:

> With the covid19 situation, obviously lots of ISPs have their NOC
> personnel working from home, with VPN (or remote desktop) access to all the
> internal tools, VoIP at home, etc.
>
> In the traditional sense, by "showpiece NOC" I mean a room designed for
> the purpose of having large situational awareness displays on a wall,
> network weathermaps and charts, alerting systems, composed of four or more
> big flat panel displays. Ideally configured to be actually useful for NOC
> purposes and also something impressive looking for customer tours.
>
> To what extent potential customers find that sort of thing to be a
> signifier of seriousness on the part of an ISP, I suppose depends on what
> sort of customers they are, and their relative degree of technical
> sophistication.
>
> Are the days of such an environment gone forever?
>
>
>
>

Re: Are the days of the showpiece NOC office display gone forever?

[Töma Gavrichenkov]

Peace,

On Thu, Dec 17, 2020, 12:21 AM Lady Benjamin PD Cannon  wrote:

> We are still operating ours - 27 1080P projectors - but with a skeleton
> crew of just 3.  Given the air volume, it’s almost like outside.
>

A devil advocate here,

First of all, COVID-19 is really serious.

With that in mind, with all the necessary precautions office space *may* be
managed safely to prevent the spread.

Production plants had security measures preventing workforce injuries for a
century already.  Just a bit of that, with constant trainings, would get
your monitoring room safe, especially with all the bars closed and
everything operating on delivery.

That is not to say that large monitoring rooms are a better choice over
automation (which they are not).

--
Töma

>

Re: Are the days of the showpiece NOC office display gone forever?

[Lady Benjamin PD Cannon]

We are still operating ours - 27 1080P projectors - but with a skeleton crew of 
just 3.  Given the air volume, it’s almost like outside.   Everyone else is 
WFH.   
—L.B.

Lady Benjamin PD Cannon, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
b...@6by7.net 
"The only fully end-to-end encrypted global telecommunications company in the 
world.”
FCC License KJ6FJJ



> On Dec 16, 2020, at 12:49 PM, Eric Kuhnke  wrote:
> 
> With the covid19 situation, obviously lots of ISPs have their NOC personnel 
> working from home, with VPN (or remote desktop) access to all the internal 
> tools, VoIP at home, etc.
> 
> In the traditional sense, by "showpiece NOC" I mean a room designed for the 
> purpose of having large situational awareness displays on a wall, network 
> weathermaps and charts, alerting systems, composed of four or more big flat 
> panel displays. Ideally configured to be actually useful for NOC purposes and 
> also something impressive looking for customer tours.
> 
> To what extent potential customers find that sort of thing to be a signifier 
> of seriousness on the part of an ISP, I suppose depends on what sort of 
> customers they are, and their relative degree of technical sophistication.
> 
> Are the days of such an environment gone forever? 
> 
> 
> 

Re: Are the days of the showpiece NOC office display gone forever?

[Eric Kuhnke]

I would not be surprised to see Cisco CTC (the alarm control
panel/monitoring software for 15454s carrying TDM, SDH/SONET circuits)
still being used in the year 2030 in some places.


On Wed, Dec 16, 2020 at 1:10 PM Paul Amaral  wrote:

> We used to have some CRTs with MRTG running in the late 90’s 
>
>
>
> P
>
>
>
> *From:* NANOG  *On Behalf Of *Eric
> Kuhnke
> *Sent:* Wednesday, December 16, 2020 3:50 PM
> *To:* nanog@nanog.org list 
> *Subject:* Are the days of the showpiece NOC office display gone forever?
>
>
>
> With the covid19 situation, obviously lots of ISPs have their NOC
> personnel working from home, with VPN (or remote desktop) access to all the
> internal tools, VoIP at home, etc.
>
>
>
> In the traditional sense, by "showpiece NOC" I mean a room designed for
> the purpose of having large situational awareness displays on a wall,
> network weathermaps and charts, alerting systems, composed of four or more
> big flat panel displays. Ideally configured to be actually useful for NOC
> purposes and also something impressive looking for customer tours.
>
>
>
> To what extent potential customers find that sort of thing to be a
> signifier of seriousness on the part of an ISP, I suppose depends on what
> sort of customers they are, and their relative degree of technical
> sophistication.
>
>
>
> Are the days of such an environment gone forever?
>
>
>
>
>
>
>

RE: Are the days of the showpiece NOC office display gone forever?

[Paul Amaral via NANOG]

We used to have some CRTs with MRTG running in the late 90’s 

 

P

 

From: NANOG  On Behalf Of Eric Kuhnke
Sent: Wednesday, December 16, 2020 3:50 PM
To: nanog@nanog.org list 
Subject: Are the days of the showpiece NOC office display gone forever?

 

With the covid19 situation, obviously lots of ISPs have their NOC personnel 
working from home, with VPN (or remote desktop) access to all the internal 
tools, VoIP at home, etc.

 

In the traditional sense, by "showpiece NOC" I mean a room designed for the 
purpose of having large situational awareness displays on a wall, network 
weathermaps and charts, alerting systems, composed of four or more big flat 
panel displays. Ideally configured to be actually useful for NOC purposes and 
also something impressive looking for customer tours.

 

To what extent potential customers find that sort of thing to be a signifier of 
seriousness on the part of an ISP, I suppose depends on what sort of customers 
they are, and their relative degree of technical sophistication.

 

Are the days of such an environment gone forever? 

 

 

 

Re: Are the days of the showpiece NOC office display gone forever?

[Töma Gavrichenkov]

Peace,

On Wed, Dec 16, 2020, 11:50 PM Eric Kuhnke  wrote:

> In the traditional sense, by "showpiece NOC" I mean a room designed for
> the purpose of having large situational awareness displays on a wall,
> network weathermaps and charts, alerting systems, composed of four or more
> big flat panel displays.
>

a) Without a change in processes you need to replicate some of that at home
of every onshift engineer of your NOC.  This might sound expensive but a
single failure might cost you more.

b) But to be honest, most of the monitoring done by "a hundred of blinking
red eyes in a monitoring room" should be replaced with data analysis,
correlation tools, and automated alerts.  Those don't look that awesome on
the Instagram accounts of your marketing team but work better.

--
Töma

>

Re: Are the days of the showpiece NOC office display gone forever?

[Dave Cohen]

Frankly, I think the days of the "showpiece NOC" being relevant ended a
while before COVID. I worked at a Tier 1 for >10 years in a customer-facing
capacity, largely dealing with "serious" enterprise customers. The number
of customers who toured the NOC in that time was less than 5 (i.e much less
than 1%), despite generally offering it up to new and quickly-scaling
customers. I actually spent more time visiting customer NOCs over that time
than I did my own with customers (and some of theirs were more impressive
anyway).

Now, I did spend a lot of time coordinating calls between the NOC and
customers, some of which was in the break-fix, mea culpa sort of vain, but
also a lot of "spend time getting comfortable" types of conversations too,
especially with customers considering their first service. So it wasn't
that folks didn't care about what was going on there inasmuch as they
recognized (quite rightfully IMO) that they weren't going to get any value
being there that they couldn't get meeting the folks and learning about
operational procedures, etc. over a conference call, so why waste the
time/money to travel for that sort of thing?

I don't know if I have a biased sample or if this is reflective of the norm
these days, but the "NOC Tour" was something that didn't get executed on
very often.

On Wed, Dec 16, 2020 at 3:51 PM Eric Kuhnke  wrote:

> With the covid19 situation, obviously lots of ISPs have their NOC
> personnel working from home, with VPN (or remote desktop) access to all the
> internal tools, VoIP at home, etc.
>
> In the traditional sense, by "showpiece NOC" I mean a room designed for
> the purpose of having large situational awareness displays on a wall,
> network weathermaps and charts, alerting systems, composed of four or more
> big flat panel displays. Ideally configured to be actually useful for NOC
> purposes and also something impressive looking for customer tours.
>
> To what extent potential customers find that sort of thing to be a
> signifier of seriousness on the part of an ISP, I suppose depends on what
> sort of customers they are, and their relative degree of technical
> sophistication.
>
> Are the days of such an environment gone forever?
>
>
>
>

-- 
- Dave Cohen
craetd...@gmail.com
@dCoSays
www.venicesunlight.com