Re: [c-nsp] ISIS/BFD Monitoring

[Alex K.]

The real issue with IP SLA probes, in my humble opinion, is their sheer
number.

We really tried to use those, thank god that customer NMS supports doing it
centrally (from NMS console), but still - we find those really
unmanageable. Manual configuration for the sake of monitoring, is a bit
outdated, in my humble opinion. But still, we do use them. Syslog messages
aren't easier either. Especially when it comes to translate those into an
alarm.

Anyhow, we're going give RouteExplorer a try. Hopefully, it all can by tied
down to one central NOC console (by regular SNMP magic and usual rain
dancing).

Maybe one day, ISIS down SNMP trap, will include additional information
(such as the interface in question, ifIndex), in addition to the mysterious
ISIS session ID.


בתאריך 18 בספט' 2017 9:14 AM,‏ "Graham Beneke"  כתב:

We have a similar challenge monitoring OSPF over layer-2 providers who
don't drop the physical port in a link failure.

We run syslog but find that it doesn't provide useful information of
state for the NOC guys. Most of the guys logging faults with vendors
don't understand the states and what they mean.

We have a set of threshold alerts on our NMS which flag when traffic
levels drop extremely low. This works well for major links that always
have traffic.

We also have setup "ip sla" on some links. The status of the ip sla can
then be polled by the NMS over SNMP to detect the state of the link.


On 16/09/2017 09:24, Alex K. wrote:
> Thank you Aaron.
>
> In my humble opinion, syslog is a bit inapplicable. The way I know it, it
> will have an error message like "ISIS session to  hostname> changed to down" (maybe it do has interface name, I don't recall
> the exact message right now).
>
> Anyhow, not the neighboring router hostname nor interface name, have the
> exact circuit (that went down) information. In our case, it encoded into
> interface description.
>
> To your knowledge, the tools you mentioned, are able to append that
> information to the original syslog message?
>
> בתאריך 16 בספט' 2017 4:57 AM,‏ "Aaron Gould"  כתב:
>
>> Kiwi syslogd or maybe splunk
>>
>> -Aaron
>>
>>
> ___
> cisco-nsp mailing list  cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Łukasz Bromirski]

> On 18 Sep 2017, at 11:36, Nick Hilliard  wrote:
> 
> Graham Beneke wrote:
>> We also have setup "ip sla" on some links. The status of the ip sla can
>> then be polled by the NMS over SNMP to detect the state of the link.
> 
> My favourite failure mode is when a provider drops the link MTU without
> notice.

During internal reroute event. Once every n-th months during random
time of the day. When Mars and Moon have this specific setting in
the sky.

But event then.. not always of course.

— 
;)
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Nick Hilliard]

Graham Beneke wrote:
> We also have setup "ip sla" on some links. The status of the ip sla can
> then be polled by the NMS over SNMP to detect the state of the link.

My favourite failure mode is when a provider drops the link MTU without
notice.

Nick
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Graham Beneke]

We have a similar challenge monitoring OSPF over layer-2 providers who
don't drop the physical port in a link failure.

We run syslog but find that it doesn't provide useful information of
state for the NOC guys. Most of the guys logging faults with vendors
don't understand the states and what they mean.

We have a set of threshold alerts on our NMS which flag when traffic
levels drop extremely low. This works well for major links that always
have traffic.

We also have setup "ip sla" on some links. The status of the ip sla can
then be polled by the NMS over SNMP to detect the state of the link.


On 16/09/2017 09:24, Alex K. wrote:
> Thank you Aaron.
>
> In my humble opinion, syslog is a bit inapplicable. The way I know it, it
> will have an error message like "ISIS session to  hostname> changed to down" (maybe it do has interface name, I don't recall
> the exact message right now).
>
> Anyhow, not the neighboring router hostname nor interface name, have the
> exact circuit (that went down) information. In our case, it encoded into
> interface description.
>
> To your knowledge, the tools you mentioned, are able to append that
> information to the original syslog message?
>
> בתאריך 16 בספט' 2017 4:57 AM,‏ "Aaron Gould"  כתב:
>
>> Kiwi syslogd or maybe splunk
>>
>> -Aaron
>>
>>
> ___
> cisco-nsp mailing list  cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/


___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Clinton Work]

I have used RouteExplorer since 2007.  RouteExplorer listens to ISIS,
OSPF, or BGP routing updates and keeps a searchable history.  The tool
works great in a multi-vendor environment (Cisco, Juniper, Nokia, ...). 
 I have setup alerts to monitor specific ISIS links or when an ISIS
router is isolated.   I find the topology views (called layouts) very
useful where you have a large network with frequent changes.  The
topology views can be exported as a JPG, PDF, or SVG file.   When links
or routers are down they will turn red in the topology view and you can
step back in the routing history to see what the topology looked.   

--
Clinton Work
Airdrie, AB

On Sat, Sep 16, 2017, at 01:11 AM, Alex K. wrote:
> Thank you Alan.
> 
> RouteExplorer indeed seems to be appropriate. Anyone here have been using
> it and can share their experience?
>
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Pierre Emeriaud]

2017-09-16 9:11 GMT+02:00 Alex K. :
> RouteExplorer indeed seems to be appropriate. Anyone here have been using
> it and can share their experience?

We're using Packet Design Route Explorer on several of our networks
and it has been a very, very useful tool, for live monitoring /
alerting, planned outages impact calculation, a posteriori RCA and
some light traffic engineering.

Having all ISIS and bgp updates makes it very convenient for
troubleshooting complex incidents, and honestly I don't know how we
could have done the same without this tool.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Alex K.]

Thank you Aaron.

In my humble opinion, syslog is a bit inapplicable. The way I know it, it
will have an error message like "ISIS session to  changed to down" (maybe it do has interface name, I don't recall
the exact message right now).

Anyhow, not the neighboring router hostname nor interface name, have the
exact circuit (that went down) information. In our case, it encoded into
interface description.

To your knowledge, the tools you mentioned, are able to append that
information to the original syslog message?

בתאריך 16 בספט' 2017 4:57 AM,‏ "Aaron Gould"  כתב:

> Kiwi syslogd or maybe splunk
>
> -Aaron
>
>
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Alex K.]

Thank you Alan.

RouteExplorer indeed seems to be appropriate. Anyone here have been using
it and can share their experience?

בתאריך 15 בספט' 2017 2:23 PM,‏ "Alan Buxey"  כתב:

> RouteExplorer is a nice tool
>
> (Commercial, from Packet Design)
>
>
>
> On 15 Sep 2017 10:50 am, "Alex K."  wrote:
>
>> Hello everyone,
>>
>> A customer of mine, ran into interesting problem - his monitoring software
>> unable to provide him with a meaningful alert, in case a link goes down.
>>
>> As an ISP, they have lots of links, they run ISIS/BFD on all of them but,
>> as it regularly happens with carriers, layer 2 never actually goes down
>> (apart from SDH and dark fiber links, but those are few).
>>
>> I tested their equipment (mainly Cisco gear) and all it generates, is a
>> trap wich basically say - "ISIS sission 657843347853325524854 went down".
>>
>> What they looking for, is a monitoring system, which on the scenario
>> above,
>> is able to provide the NOC team with a meaningful alert, such as: "ISIS
>> status on interface  changed to down. Please notify
>> the network team".
>>
>> Any suggestions and sharing of personal experience will be appreciated.
>>
>> Thank you.
>> ___
>> cisco-nsp mailing list  cisco-nsp@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Aaron Gould]

Kiwi syslogd or maybe splunk

-Aaron

___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: [c-nsp] ISIS/BFD Monitoring

[Alan Buxey]

RouteExplorer is a nice tool

(Commercial, from Packet Design)



On 15 Sep 2017 10:50 am, "Alex K."  wrote:

> Hello everyone,
>
> A customer of mine, ran into interesting problem - his monitoring software
> unable to provide him with a meaningful alert, in case a link goes down.
>
> As an ISP, they have lots of links, they run ISIS/BFD on all of them but,
> as it regularly happens with carriers, layer 2 never actually goes down
> (apart from SDH and dark fiber links, but those are few).
>
> I tested their equipment (mainly Cisco gear) and all it generates, is a
> trap wich basically say - "ISIS sission 657843347853325524854 went down".
>
> What they looking for, is a monitoring system, which on the scenario above,
> is able to provide the NOC team with a meaningful alert, such as: "ISIS
> status on interface  changed to down. Please notify
> the network team".
>
> Any suggestions and sharing of personal experience will be appreciated.
>
> Thank you.
> ___
> cisco-nsp mailing list  cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[c-nsp] ISIS/BFD Monitoring

[Alex K.]

Hello everyone,

A customer of mine, ran into interesting problem - his monitoring software
unable to provide him with a meaningful alert, in case a link goes down.

As an ISP, they have lots of links, they run ISIS/BFD on all of them but,
as it regularly happens with carriers, layer 2 never actually goes down
(apart from SDH and dark fiber links, but those are few).

I tested their equipment (mainly Cisco gear) and all it generates, is a
trap wich basically say - "ISIS sission 657843347853325524854 went down".

What they looking for, is a monitoring system, which on the scenario above,
is able to provide the NOC team with a meaningful alert, such as: "ISIS
status on interface  changed to down. Please notify
the network team".

Any suggestions and sharing of personal experience will be appreciated.

Thank you.
___
cisco-nsp mailing list  cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/