Re: Service Provider NetFlow Collectors

[Aaron1]

I’m still using nfsen/nfdump

Been looking at manageengine netflow analyzer lately and liking it, we might be 
buying some time on Calix flowanalyze which might be an improved version of 
xangati 

Aaron

> On Dec 30, 2018, at 10:44 PM, Michael Gehrmann  
> wrote:
> 
> 
> Add Flowtraq to your list.
> 
> Cheers
> Mike
> 
> 
>> On Mon, 31 Dec 2018 at 14:30, Erik Sundberg  wrote:
>> Hi Nanog….
>> 
>>  
>> 
>> We are looking at replacing our Netflow collector. I am wonder what other 
>> service providers are using to collect netflow data off their Core and Edge 
>> Routers. Pros/Cons… What to watch out for any info would help.
>> 
>>  
>> 
>> We are mainly looking to analyze the netflow data. Bonus if it does ddos 
>> detection and mitigation.
>> 
>>  
>> 
>> We are looking at
>> 
>> ManageEngine Netflow Analyzer
>> 
>> PRTG
>> 
>> Plixer – Scrutinizer
>> 
>> PeakFlow
>> 
>> Kentik
>> 
>> Solarwinds NTA
>> 
>>  
>> 
>>  
>> 
>> Thanks in advance…
>> 
>>  
>> 
>> Erik
>> 
>>  
>> 
>> 
>> 
>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files 
>> or previous e-mail messages attached to it may contain confidential 
>> information that is legally privileged. If you are not the intended 
>> recipient, or a person responsible for delivering it to the intended 
>> recipient, you are hereby notified that any disclosure, copying, 
>> distribution or use of any of the information contained in or attached to 
>> this transmission is STRICTLY PROHIBITED. If you have received this 
>> transmission in error please notify the sender immediately by replying to 
>> this e-mail. You must destroy the original transmission and its attachments 
>> without reading or saving in any manner. Thank you.

Re: Service Provider NetFlow Collectors

[Michael Gehrmann]

Add Flowtraq to your list.

Cheers
Mike


On Mon, 31 Dec 2018 at 14:30, Erik Sundberg  wrote:

> Hi Nanog….
>
>
>
> We are looking at replacing our Netflow collector. I am wonder what other
> service providers are using to collect netflow data off their Core and Edge
> Routers. Pros/Cons… What to watch out for any info would help.
>
>
>
> We are mainly looking to analyze the netflow data. Bonus if it does ddos
> detection and mitigation.
>
>
>
> We are looking at
>
> ManageEngine Netflow Analyzer
>
> PRTG
>
> Plixer – Scrutinizer
>
> PeakFlow
>
> Kentik
>
> Solarwinds NTA
>
>
>
>
>
> Thanks in advance…
>
>
>
> Erik
>
>
>
> --
>
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
> or previous e-mail messages attached to it may contain confidential
> information that is legally privileged. If you are not the intended
> recipient, or a person responsible for delivering it to the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of any of the information contained in or attached to
> this transmission is STRICTLY PROHIBITED. If you have received this
> transmission in error please notify the sender immediately by replying to
> this e-mail. You must destroy the original transmission and its attachments
> without reading or saving in any manner. Thank you.
>

Service Provider NetFlow Collectors

[Erik Sundberg]

Hi Nanog

We are looking at replacing our Netflow collector. I am wonder what other 
service providers are using to collect netflow data off their Core and Edge 
Routers. Pros/Cons... What to watch out for any info would help.

We are mainly looking to analyze the netflow data. Bonus if it does ddos 
detection and mitigation.

We are looking at
ManageEngine Netflow Analyzer
PRTG
Plixer - Scrutinizer
PeakFlow
Kentik
Solarwinds NTA


Thanks in advance...

Erik




CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or 
previous e-mail messages attached to it may contain confidential information 
that is legally privileged. If you are not the intended recipient, or a person 
responsible for delivering it to the intended recipient, you are hereby 
notified that any disclosure, copying, distribution or use of any of the 
information contained in or attached to this transmission is STRICTLY 
PROHIBITED. If you have received this transmission in error please notify the 
sender immediately by replying to this e-mail. You must destroy the original 
transmission and its attachments without reading or saving in any manner. Thank 
you.

Re: CenturyLink

[Gary E. Miller]

Yo Saku!

On Sun, 30 Dec 2018 19:29:37 +0200
Saku Ytti  wrote:

> I've always wondered what
> niche rubidium covers that isn't handled by much cheaper crystal ovens
> or actually precise oscillators.

The Rb frequency reference will be two or three orders of magnitude
more stable than an expensive ovenized crystal.

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
g...@rellim.com  Tel:+1 541 382 8588

Veritas liberabit vos. -- Quid est veritas?
"If you can’t measure it, you can’t improve it." - Lord Kelvin


pgpUj_KSvGwaK.pgp
Description: OpenPGP digital signature

Re: CenturyLink

[Saku Ytti]

On Sun, 30 Dec 2018 at 20:04, Matthew Huff  wrote:

> Regulatory.

> If we were to lose the GPS signal (antenna failure, etc...) then our stratum 
> 1 time sources wouldn't drift as much and as quickly. For telco and general 
> usage, the cost may not be worthwhile, but when you have auditors looking 
> over your shoulder

Do you happen to recall specifics? What type of accuracy do you have
to retain GPS free and for how long and who does fall under the
regulatory requirements?

Thanks!
-- 
  ++ytti

RE: CenturyLink

[Matthew Huff]

Regulatory.

If we were to lose the GPS signal (antenna failure, etc...) then our stratum 1 
time sources wouldn't drift as much and as quickly. For telco and general 
usage, the cost may not be worthwhile, but when you have auditors looking over 
your shoulder


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: Saku Ytti [mailto:s...@ytti.fi] 
Sent: Sunday, December 30, 2018 12:30 PM
To: Matthew Huff 
Cc: Shawn L ; nanog@nanog.org
Subject: Re: CenturyLink

Hey Matthew,

> We use an older model of 
> https://www.microsemi.com/product-directory/enterprise-network-time-servers/4117-syncserver-s600
>  with rubidium oscillator. Not cheap, but hardened and extremely accurate.

Out of interest why rubidium? For short term stability oscillator choice isn't 
much of a matter for free running even rubidium will be off good +-1us per day 
or +-30us per week. I've always wondered what niche rubidium covers that isn't 
handled by much cheaper crystal ovens or actually precise oscillators.

--
  ++ytti

Re: CenturyLink

[Luke Guillory]

We have a few s600’s deployed as well, rock solid and van handle 10k+ queries a 
second.


ns



Sent from my iPhone

On Dec 30, 2018, at 11:22 AM, Matthew Huff mailto:mh...@ox.com>> 
wrote:

We use an older model of 
https://www.microsemi.com/product-directory/enterprise-network-time-servers/4117-syncserver-s600
 with rubidium oscillator. Not cheap, but hardened and extremely accurate.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

From: Shawn L [mailto:sha...@up.net]
Sent: Sunday, December 30, 2018 9:40 AM
To: nanog@nanog.org
Cc: Matthew Huff mailto:mh...@ox.com>>; 
l...@satchell.net
Subject: Re: CenturyLink


Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are 
using for this.



thanks






-Original Message-
From: "Raymond Burkholder" mailto:r...@oneunified.net>>
Sent: Saturday, December 29, 2018 12:01pm
To: "Matthew Huff" mailto:mh...@ox.com>>, 
"l...@satchell.net" 
mailto:l...@satchell.net>>, 
"nanog@nanog.org" 
mailto:nanog@nanog.org>>
Subject: Re: CenturyLink

On 2018-12-29 7:51 a.m., Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.
>
On one occasion, due to bad firmware or a configuration issue, I have
seen GPS stratum 1 diverge from NTP.  It was somewhat eye brow raising
to the company.  My NTP monitored servers were shown to be diverging
their GPS/NTP, but after looking at twice or thrice, it was the other
way around.

Re: CenturyLink

[Saku Ytti]

Hey Matthew,

> We use an older model of 
> https://www.microsemi.com/product-directory/enterprise-network-time-servers/4117-syncserver-s600
>  with rubidium oscillator. Not cheap, but hardened and extremely accurate.

Out of interest why rubidium? For short term stability oscillator
choice isn't much of a matter for free running even rubidium will be
off good +-1us per day or +-30us per week. I've always wondered what
niche rubidium covers that isn't handled by much cheaper crystal ovens
or actually precise oscillators.

-- 
  ++ytti

RE: CenturyLink

[Matthew Huff]

We use an older model of 
https://www.microsemi.com/product-directory/enterprise-network-time-servers/4117-syncserver-s600
 with rubidium oscillator. Not cheap, but hardened and extremely accurate.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

From: Shawn L [mailto:sha...@up.net]
Sent: Sunday, December 30, 2018 9:40 AM
To: nanog@nanog.org
Cc: Matthew Huff ; l...@satchell.net
Subject: Re: CenturyLink


Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are 
using for this.



thanks



-Original Message-
From: "Raymond Burkholder" mailto:r...@oneunified.net>>
Sent: Saturday, December 29, 2018 12:01pm
To: "Matthew Huff" mailto:mh...@ox.com>>, 
"l...@satchell.net" 
mailto:l...@satchell.net>>, 
"nanog@nanog.org" 
mailto:nanog@nanog.org>>
Subject: Re: CenturyLink

On 2018-12-29 7:51 a.m., Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.
>
On one occasion, due to bad firmware or a configuration issue, I have
seen GPS stratum 1 diverge from NTP.  It was somewhat eye brow raising
to the company.  My NTP monitored servers were shown to be diverging
their GPS/NTP, but after looking at twice or thrice, it was the other
way around.

Re: CenturyLink RCA?

[Saku Ytti]

Hey John,

Your criticism is warranted, but would also be addressed by
explanation DCN/OOB being the source of the problem.

At any rate, I am looking forward to stop speculating and start
reading post-mortem written by someone who knows how networks work.

On Sun, 30 Dec 2018 at 18:28, John Von Essen  wrote:
>
> One thing that is troubling when reading that URL is that it appears several 
> steps of restoration required teams to go onsite for local login, etc.,. 
> Granted, to troubleshoot hardware you need to be physically present to pop a 
> line card in and out, but CTL/LVL3 should have full out-of-band console and 
> power control to all core devices, we shouldn't be waiting for someone to 
> drive to a location to get console or do power cycling. And I would imagine 
> the first step to alot of the troubleshooting was power cycling and local 
> console logs.
>
>
> -John
>
>
>
> On 12/30/18 10:42 AM, Mike Hammett wrote:
>
> It's technical enough so that laypeople immediately lose interest, yet 
> completely useless to anyone that works with this stuff.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> 
> From: "Saku Ytti" 
> To: "nanog list" 
> Sent: Sunday, December 30, 2018 7:42:49 AM
> Subject: CenturyLink RCA?
>
> Apologies for the URL, I do not know official source and I do not
> share the URLs sentiment.
> https://fuckingcenturylink.com/
>
> Can someone translate this to IP engineer? What did actually happen?
> From my own history, I rarely recognise the problem I fixed from
> reading the public RCA. I hope CenturyLink will do better.
>
> Best guess so far that I've heard is
>
> a) CenturyLink runs global L2 DCN/OOB
> b) there was HW fault which caused L2 loop (perhaps HW dropped BPDU,
> I've had this failure mode)
> c) DCN had direct access to control-plane, and L2 congested
> control-plane resources causing it to deprovision waves
>
> Now of course this is entirely speculation, but intended to show what
> type of explanation is acceptable and can be used to fix things.
> Hopefully CenturyLink does come out with IP-engineering readable
> explanation, so that we may use it as leverage to support work in our
> own domains to remove such risks.
>
> a) do not run L2 DCN/OOB
> b) do not connect MGMT ETH (it is unprotected access to control-plane,
> it  cannot be protected by CoPP/lo0 filter/LPTS ec)
> c) do add in your RFP scoring item for proper OOB port (Like Cisco CMP)
> d) do fail optical network up
>
> --
>   ++ytti
>


-- 
  ++ytti

Re: CenturyLink

[Mel Beckman]

We have had great success with the TimeMachines TM1001A. Simple, robust, and 
over several years we’ve had zero outages on more than 40 installations. And 
unlike most of the competition, not ridiculously overpriced.

https://timemachinescorp.com/product/gps-time-server-tm1000a

 -mel beckman

On Dec 30, 2018, at 6:40 AM, Shawn L via NANOG 
mailto:nanog@nanog.org>> wrote:


Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are 
using for this.



thanks



-Original Message-
From: "Raymond Burkholder" mailto:r...@oneunified.net>>
Sent: Saturday, December 29, 2018 12:01pm
To: "Matthew Huff" mailto:mh...@ox.com>>, 
"l...@satchell.net" 
mailto:l...@satchell.net>>, 
"nanog@nanog.org" 
mailto:nanog@nanog.org>>
Subject: Re: CenturyLink


On 2018-12-29 7:51 a.m., Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.
>
On one occasion, due to bad firmware or a configuration issue, I have
seen GPS stratum 1 diverge from NTP.  It was somewhat eye brow raising
to the company.  My NTP monitored servers were shown to be diverging
their GPS/NTP, but after looking at twice or thrice, it was the other
way around.

Re: CenturyLink RCA?

[John Von Essen]

One thing that is troubling when reading that URL is that it appears 
several steps of restoration required teams to go onsite for local 
login, etc.,. Granted, to troubleshoot hardware you need to be 
physically present to pop a line card in and out, but CTL/LVL3 should 
have full out-of-band console and power control to all core devices, we 
shouldn't be waiting for someone to drive to a location to get console 
or do power cycling. And I would imagine the first step to alot of the 
troubleshooting was power cycling and local console logs.



-John



On 12/30/18 10:42 AM, Mike Hammett wrote:
It's technical enough so that laypeople immediately lose interest, yet 
completely useless to anyone that works with this stuff.




-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


*From: *"Saku Ytti" 
*To: *"nanog list" 
*Sent: *Sunday, December 30, 2018 7:42:49 AM
*Subject: *CenturyLink RCA?

Apologies for the URL, I do not know official source and I do not
share the URLs sentiment.
https://fuckingcenturylink.com/

Can someone translate this to IP engineer? What did actually happen?
From my own history, I rarely recognise the problem I fixed from
reading the public RCA. I hope CenturyLink will do better.

Best guess so far that I've heard is

a) CenturyLink runs global L2 DCN/OOB
b) there was HW fault which caused L2 loop (perhaps HW dropped BPDU,
I've had this failure mode)
c) DCN had direct access to control-plane, and L2 congested
control-plane resources causing it to deprovision waves

Now of course this is entirely speculation, but intended to show what
type of explanation is acceptable and can be used to fix things.
Hopefully CenturyLink does come out with IP-engineering readable
explanation, so that we may use it as leverage to support work in our
own domains to remove such risks.

a) do not run L2 DCN/OOB
b) do not connect MGMT ETH (it is unprotected access to control-plane,
it  cannot be protected by CoPP/lo0 filter/LPTS ec)
c) do add in your RFP scoring item for proper OOB port (Like Cisco CMP)
d) do fail optical network up

--
  ++ytti

Re: CenturyLink RCA?

[Mike Hammett]

It's technical enough so that laypeople immediately lose interest, yet 
completely useless to anyone that works with this stuff. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Saku Ytti"  
To: "nanog list"  
Sent: Sunday, December 30, 2018 7:42:49 AM 
Subject: CenturyLink RCA? 

Apologies for the URL, I do not know official source and I do not 
share the URLs sentiment. 
https://fuckingcenturylink.com/ 

Can someone translate this to IP engineer? What did actually happen? 
>From my own history, I rarely recognise the problem I fixed from 
reading the public RCA. I hope CenturyLink will do better. 

Best guess so far that I've heard is 

a) CenturyLink runs global L2 DCN/OOB 
b) there was HW fault which caused L2 loop (perhaps HW dropped BPDU, 
I've had this failure mode) 
c) DCN had direct access to control-plane, and L2 congested 
control-plane resources causing it to deprovision waves 

Now of course this is entirely speculation, but intended to show what 
type of explanation is acceptable and can be used to fix things. 
Hopefully CenturyLink does come out with IP-engineering readable 
explanation, so that we may use it as leverage to support work in our 
own domains to remove such risks. 

a) do not run L2 DCN/OOB 
b) do not connect MGMT ETH (it is unprotected access to control-plane, 
it cannot be protected by CoPP/lo0 filter/LPTS ec) 
c) do add in your RFP scoring item for proper OOB port (Like Cisco CMP) 
d) do fail optical network up 

-- 
++ytti 

Re: CenturyLink

[Shawn L via NANOG]

Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are 
using for this.
 
thanks
 

-Original Message-
From: "Raymond Burkholder" 
Sent: Saturday, December 29, 2018 12:01pm
To: "Matthew Huff" , "l...@satchell.net" , 
"nanog@nanog.org" 
Subject: Re: CenturyLink



On 2018-12-29 7:51 a.m., Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.
>
On one occasion, due to bad firmware or a configuration issue, I have 
seen GPS stratum 1 diverge from NTP.  It was somewhat eye brow raising 
to the company.  My NTP monitored servers were shown to be diverging 
their GPS/NTP, but after looking at twice or thrice, it was the other 
way around.

CenturyLink RCA?

[Saku Ytti]

Apologies for the URL, I do not know official source and I do not
share the URLs sentiment.
https://fuckingcenturylink.com/

Can someone translate this to IP engineer? What did actually happen?
>From my own history, I rarely recognise the problem I fixed from
reading the public RCA. I hope CenturyLink will do better.

Best guess so far that I've heard is

a) CenturyLink runs global L2 DCN/OOB
b) there was HW fault which caused L2 loop (perhaps HW dropped BPDU,
I've had this failure mode)
c) DCN had direct access to control-plane, and L2 congested
control-plane resources causing it to deprovision waves

Now of course this is entirely speculation, but intended to show what
type of explanation is acceptable and can be used to fix things.
Hopefully CenturyLink does come out with IP-engineering readable
explanation, so that we may use it as leverage to support work in our
own domains to remove such risks.

a) do not run L2 DCN/OOB
b) do not connect MGMT ETH (it is unprotected access to control-plane,
it  cannot be protected by CoPP/lo0 filter/LPTS ec)
c) do add in your RFP scoring item for proper OOB port (Like Cisco CMP)
d) do fail optical network up

-- 
  ++ytti

RE: CenturyLink

[Matthew Huff]

Actually, on all our trading systems, our times are synced via PTP instead of 
ntpd for at least 50 microsecond accuracy. The stratum 1 clocks as well as NIST 
time are only used as comparison to verify compliance and reality. We use ntpq 
to determine the offset  from NIST for reporting.


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Stephen Satchell
Sent: Saturday, December 29, 2018 10:01 AM
To: nanog@nanog.org
Subject: Re: CenturyLink

On 12/29/18 6:51 AM, Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.

Having been a participant on Standards Working Groups, I understand completely. 
 Regulations, like Standards, need to be written to apply to as wide a 
population as possible.  Do those regulations dictate how you monitor drift?  
For example, can you have a system (I would use CentOS) that syncs to the 
appliance as well as NIST and your inside NTP sources, and use ntpq(8) to read 
the drift directly?