Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 0:18, Niels Bakker wrote:


You're just wrong here.


Sorry, I'm not.  I've seen what happens when flow telemetry is 'squeezed 
out' by pipe-filling DDoS attacks, interrupted by fat-fingers, etc.


It'll happen to you, one day.  And then you'll understand.

---
Roland Dobbins 

Re: Zero rating implentation strategies

[Christian Kuhtz]

Inline..

On Sep 1, 2015, at 10:13 AM, Jean-Francois Mezei 
> wrote:

On 15-09-01 12:36, Christian Kuhtz wrote:

Zero rating is not a new concept. It has existed in the mobile world since the 
days of the dumb phone.

Yes, but is now in a different scale and when you start to zero rate
content that comes from CDN (aka: many IPs which may also serve non zero
rated content).

I don't agree with that assertion.  Not a new concept. I worked on stuff like 
that over a decade ago, with multiple CDN providers.   Wasn't even mobile, it 
was for DSL.

There are scaling issues if the idea was out of touch with how interwebs 
actually work.  Only in those cases might it have technically worked but priced 
itself out of reality.  But there wasn't anything fundamentally to say it 
couldn't be done.

CDN's have for a very long time had ways to implement custom mappings and tweak 
their algorithms to source specific content to a specific subscriber in a very 
specific way or place source nodes in particular locations (logically or 
physically).  They only varied in their ability or willingness to do so on 
terms that might seem reasonable to a given partner (which generally means you 
either do or don't come to terms on contracts).  It wasn't a fundamental 
technical barrier even 10-15 yrs ago.

Got a reference to why this was killed by the regulator in Canada?

Bell Mobility zero rating its own MobileTV offering was decided in
January 2015 by CRTC:

Thank you. The key here appears to be this:

Subsection 27(2) prohibits Canadian carriers from
conferring an undue disadvantage to others, or an undue preference to
itself or others.

So this appears to be about distorting the competition between their own 
services and third parties rather than about zero rating per se.

Mobile networks typically use their packet core (and prior iterations of the 
same termination, rating, billing, management gear) to rate and bill specific 
to each subscriber.  It is done with voice minutes, text, data.  Whether or not 
you consider the solutions scalable is up to one's individual judgement. But 
this zero rating billing model has and is very widely deployed and at massive 
scale. In fact, there are specific interfaces defined for just these purposes 
in the applicable standards.  There are many many conceivable ways in which 
billing mediation and associated infrastructure for zero rating can and is 
implemented. This is not new and is very well understood in the mobile industry.

Not sure what you're after here.

What I am after is what sort of logic is used to determine whether a
packet is to be counted towards total monthly usage or not.  Is it based
on sourse IP address ?  DPI equipment looking at content ? And more
importantly, if it is possible to zero rate any flow that is below a
certain speed. (whether from a radio station or some heart monitor).

All of the above and more.  Depends on content. For example, if you obtain the 
knowledge of origin and subscriber, you have what is needed. Origin should not 
be constrained to origin in terms of IP address space or port numbers. It can 
be marked in the protocol header, content, sourced from specific interfaces etc 
etc. The ways are pretty endless and the gear is readily available. There is an 
entire cottage industry peddling this stuff from sourcing, detection through 
rating and billing in addition to the major equipment vendors.

There are no inherent limits. You can do it with or without DPI. You can do it 
with or without subscriber management systems (like BRAS). You can do it with 
or without knowledge of source or destination IP addresses. You can do it with 
any kind of content. You can do it with or without flow awareness.  You can do 
it with or without special protocol semantics.

What is capable is (as always) up to the engineering clue, business savvy and 
business model supporting the implementation.  And I really mean anything can, 
has, and will be brought to life at scale given a suitable business cause.

I don't mean to be obnoxious. The question is a bit like asking if interwebs 
have limits ;-).. Yes, there are stupid ways to screw things up, but there is a 
huge variety of ways to make it work and make money doing it. And without 
pissing off regulators.  There is no red flag from a regulatory perspective 
based on the technology itself in my opinion.

Hope this helps.

Best regards,
Christian

Re: NetFlow - path from Routers to Collector

[George, Wes]

On 9/1/15, 1:36 PM, "NANOG on behalf of Roland Dobbins"
 wrote:

>It should've already been spent for an OOB/DCN network, which should've
>been provisioned with flow telemetry in mind.

I'm going to interpret that "should" in the same way as the MUST in
RFC6919. :-)
Yes, it's a good practice, but like most other proactive security
measures, is extremely hard to justify spending money on it to avoid the
risk that it breaks fantastically when it is needed most.
Though you could provide a little insurance against the problem you're
highlighting here via a QoS policy that prioritizes flow data over
customer traffic.


Several of the OOB networks/designs I'm familiar with significantly
predate the entire concept of flow telemetry, as well as my own networking
career, and are still rocking the same set of Cisco 2500 routers with
async cards (many with uptimes measured in years) and 64k leased lines or
dialup on demand they've been using for literally almost 2 decades. When
one of those ancient devices dies of old age, you scrounge for the
cheapest equivalent you can find to replace it to maintain your oob access
to the 9600/8/1/none console ports for when things have gone truly
pear-shaped.
Often there is a separate management network that can deal with ethernet
speeds, but it's separate for security reasons and not always as rigidly
independent from the in band network for connectivity, i.e. It might be a
VPN riding over the regular network and thus not completely protected from
the problem you're concerned about.

Thanks,

Wes

Anything below this line has been added by my company’s mail server, I
have no control over it.
---








>


This E-mail and any of its attachments may contain Time Warner Cable 
proprietary information, which is privileged, confidential, or subject to 
copyright belonging to Time Warner Cable. This E-mail is intended solely for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient of this E-mail, you are hereby notified that any 
dissemination, distribution, copying, or action taken in relation to the 
contents of and attachments to this E-mail is strictly prohibited and may be 
unlawful. If you have received this E-mail in error, please notify the sender 
immediately and permanently delete the original and any copy of this E-mail and 
any printout.

Re: NetFlow - path from Routers to Collector

[Shane Ronan]

Roland,

While your way may be best practice, sometimes real life gets in the way 
of best practice.


Shane

On 9/1/15 1:12 PM, Roland Dobbins wrote:


On 2 Sep 2015, at 0:08, Steve Meuse wrote:


Your advice is not "one size fits all".


Actually, it is.

Large backbone networks have DCNs/OOBs, and that's where they export 
their NDE.



I've done netflow over production links for two very large backbone
networks.
Did you manage your routers and switches and hosts and so forth 
in-band, too?



Over the combined 17(?) years, never saw a problem.


Until you do.

Running flow telemetry in-band is penny-wise and pound-foolish, for 
networks of any size, in any circumstances.  All management-plane 
traffic (and that's what flow telemetry is) should be segregated from 
the production network data plane.



---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Avi Freedman]

Looking at probably 100 networks' flow paths over the last year,
I'd say 1 or 2 have OOB for flow.

Maybe another 10-20 have interest in taking simpler time series
data of top talkers over their OOB networks, but not the flow
itself.

Agree w Roland that it can cause problems with telemetry if
there are big network misconfigs.  But for folks seeing DDoS,
we implement rate-limiting of the flows/sec via local proxies
to avoid overwhelming network capacity with the flow data...

Avi

>   I think the key here is that Roland isn't often constrained by
> these financial considerations.
> 
>   I would respectfully disagree with Roland here and agree with
> Job, Niels, etc.
> 
>   A few networks have robust out of band networks, but most
> I've seen have an interesting mixture of things and inband is usually
> the best method.
> 
>   Those that do have "seperate" networks may actually be CoC
> services from another deparment in the same company riding the same
> P/PE devices (sometimes routers).
> 
>   I've seen oob networks on DSL, datacenter wifi or cable swaps
> through the fence to an adjacent rack.
> 
>   An oob network need not be high bandwidth enough to do netflow
> sampling, this is well regarded as a waste of money by many as the costs
> for these can often be orders of magnitude more compared to a pure-IP
> or internet service.
> 
>   I'll say this ranks up there with people who think
> MPLS VPN == Encryption.  It's not unless you think a few byte
> label is going to confuse people.
> 
>   - Jared

Re: NetFlow - path from Routers to Collector

[Steve Meuse]

On Tue, Sep 1, 2015 at 12:14 PM, Roland Dobbins  wrote:

> On 1 Sep 2015, at 23:10, Job Snijders wrote:
>
> > To answer your first question: i see no issue in transporting flow
> > export traffic over the same backbone used to serve customer traffic.
>
> This is not good advice, for the reasons I stated previously in this
> thread.


Your advice is not "one size fits all".

I've done netflow over production links for two very large backbone
networks. Over the combined 17(?) years, never saw a problem. If your
network is likely to be partitioned by a small number of failures, that
might be a different case, but you can't make blanket statements.

-Steve

Re: NetFlow - path from Routers to Collector

[Niels Bakker]

* rdobb...@arbor.net (Roland Dobbins) [Tue 01 Sep 2015, 19:13 CEST]:
Running flow telemetry in-band is penny-wise and pound-foolish, for 
networks of any size, in any circumstances.


You're just wrong here.


-- Niels.

Re: NetFlow - path from Routers to Collector

[Jared Mauch]

I think the key here is that Roland isn't often constrained by
these financial considerations.

I would respectfully disagree with Roland here and agree with
Job, Niels, etc.

A few networks have robust out of band networks, but most
I've seen have an interesting mixture of things and inband is usually
the best method.

Those that do have "seperate" networks may actually be CoC
services from another deparment in the same company riding the same
P/PE devices (sometimes routers).

I've seen oob networks on DSL, datacenter wifi or cable swaps
through the fence to an adjacent rack.

An oob network need not be high bandwidth enough to do netflow
sampling, this is well regarded as a waste of money by many as the costs
for these can often be orders of magnitude more compared to a pure-IP
or internet service.

I'll say this ranks up there with people who think
MPLS VPN == Encryption.  It's not unless you think a few byte
label is going to confuse people.

- Jared

On Tue, Sep 01, 2015 at 01:32:04PM -0400, Shane Ronan wrote:
> So in your world, the money always exists for a separate flow telemetry
> network?
> 
> On 9/1/15 1:29 PM, Roland Dobbins wrote:
> > On 2 Sep 2015, at 0:18, Niels Bakker wrote:
> >
> >> You're just wrong here.
> >
> > Sorry, I'm not.  I've seen what happens when flow telemetry is
> > 'squeezed out' by pipe-filling DDoS attacks, interrupted by
> > fat-fingers, etc.
> >
> > It'll happen to you, one day.  And then you'll understand.
> >
> > ---
> > Roland Dobbins 

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Zero rating implentation strategies

[Jean-Francois Mezei]

On 15-09-01 13:35, Christian Kuhtz wrote:
>  There is no red flag from a regulatory perspective based on the technology 
> itself in my opinion.

There are other sections in the Canadian  Telecommunicatiosn Act about
controlling content. And there was a reference to the supreme court on
whether ISPs were liable for content that their deliver to customers,
which found that an ISP that blindly delivers packets doesn't control
content and therefore is not responsible for it.

So depending on how the zero rating is implemented, this may (or may
not) have implications.

What I am trying to do is to build a list of realistic techniques (aka:
what industry actually uses) which should force Vidéotron to fees up to
which one it uses.

Re: Zero rating implentation strategies

[Jean-Francois Mezei]

On 15-09-01 12:36, Christian Kuhtz wrote:

> Zero rating is not a new concept. It has existed in the mobile world since 
> the days of the dumb phone.

Yes, but is now in a different scale and when you start to zero rate
content that comes from CDN (aka: many IPs which may also serve non zero
rated content).


>  Got a reference to why this was killed by the regulator in Canada?

Bell Mobility zero rating its own MobileTV offering was decided in
January 2015 by CRTC:
http://www.crtc.gc.ca/eng/archive/2015/2015-26.htm

##
The Commission finds that Bell Mobility Inc. (Bell Mobility) and
Quebecor Media Inc., Videotron Ltd. and Videotron G.P. (collectively,
Videotron), violated subsection 27(2) of the Telecommunications Act by
exempting their mobile TV services Bell Mobile TV and illico.tv from
data charges. Subsection 27(2) prohibits Canadian carriers from
conferring an undue disadvantage to others, or an undue preference to
itself or others.
##




> Mobile networks typically use their packet core (and prior iterations of the 
> same termination, rating, billing, management gear) to rate and bill specific 
> to each subscriber.  It is done with voice minutes, text, data.  Whether or 
> not you consider the solutions scalable is up to one's individual judgement. 
> But this zero rating billing model has and is very widely deployed and at 
> massive scale. In fact, there are specific interfaces defined for just these 
> purposes in the applicable standards.  There are many many conceivable ways 
> in which billing mediation and associated infrastructure for zero rating can 
> and is implemented. This is not new and is very well understood in the mobile 
> industry.  
>
>Not sure what you're after here.

What I am after is what sort of logic is used to determine whether a
packet is to be counted towards total monthly usage or not.  Is it based
on sourse IP address ?  DPI equipment looking at content ? And more
importantly, if it is possible to zero rate any flow that is below a
certain speed. (whether from a radio station or some heart monitor).

Re: NetFlow - path from Routers to Collector

[Niels Bakker]

* rdobb...@arbor.net (Roland Dobbins) [Tue 01 Sep 2015, 19:30 CEST]:

On 2 Sep 2015, at 0:18, Niels Bakker wrote:


You're just wrong here.


Sorry, I'm not.  I've seen what happens when flow telemetry is 
'squeezed out' by pipe-filling DDoS attacks, interrupted by 
fat-fingers, etc.


This is the dumbest thing I've read on this mailing list in a while.
(On the other hand, I don't read most threads.)



It'll happen to you, one day.  And then you'll understand.


Who is saying I haven't done all of the above already?


-- Niels.

Re: Zero rating implentation strategies

[Jean-Francois Mezei]

On 15-09-01 14:19, Owen DeLong wrote:
> The regulatory killing of that was probably unrelated to implementation.

Demonstrating that the Mobile TV packets traveled in exactly the same
way as any other packets over the Bell Mobility network was a large part
of the decision. When packets travel undifferentiated on the same pipe,
then it is not justified that one packet be treated differently than the
other one.

This is quite different from Bell Canada's wireline TV service where
multicast is used and on a separate VLAN with dedicated capacity which
means that TV packets don't cause congestion on data packets and vice versa.

This is why understanding of how some marketing tactic is implemented at
the network level is important.

Re: NetFlow - path from Routers to Collector

[Tarko Tikan]

hey,


It should've already been spent for an OOB/DCN network, which should've
been provisioned with flow telemetry in mind.


Bad advice. No amount of money will fix major platforms that are not 
happy to export flow telemetry via router management ports. Sometimes it 
can be done via nasty vrf leaking hacks, sometimes it cannot be done at 
all. Management ports are typically directly connected to routing 
engines while netflow data is generated in hardware in PFE.


In-band netflow works on all platforms without such issues.

--
tarko

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 0:08, Steve Meuse wrote:


Your advice is not "one size fits all".


Actually, it is.

Large backbone networks have DCNs/OOBs, and that's where they export 
their NDE.



I've done netflow over production links for two very large backbone
networks.
Did you manage your routers and switches and hosts and so forth in-band, 
too?



Over the combined 17(?) years, never saw a problem.


Until you do.

Running flow telemetry in-band is penny-wise and pound-foolish, for 
networks of any size, in any circumstances.  All management-plane 
traffic (and that's what flow telemetry is) should be segregated from 
the production network data plane.



---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Shane Ronan]

So in your world, the money always exists for a separate flow telemetry 
network?


On 9/1/15 1:29 PM, Roland Dobbins wrote:

On 2 Sep 2015, at 0:18, Niels Bakker wrote:


You're just wrong here.


Sorry, I'm not.  I've seen what happens when flow telemetry is 
'squeezed out' by pipe-filling DDoS attacks, interrupted by 
fat-fingers, etc.


It'll happen to you, one day.  And then you'll understand.

---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 0:32, Shane Ronan wrote:

So in your world, the money always exists for a separate flow 
telemetry network?


It should've already been spent for an OOB/DCN network, which should've 
been provisioned with flow telemetry in mind.


---
Roland Dobbins 

Re: Zero rating implentation strategies

[Owen DeLong]

The regulatory killing of that was probably unrelated to implementation.

The regulators probably objected to the mobile provider creating an advantage 
(no data charges) for their own service against competing video services that 
would incur data charges.

Perhaps that will help you understand what you need for a regulatory challenge.

Owen

> On Sep 1, 2015, at 09:36 , Christian Kuhtz  wrote:
> 
> 
> Zero rating is not a new concept. It has existed in the mobile world since 
> the days of the dumb phone.  Got a reference to why this was killed by the 
> regulator in Canada?
> 
> Mobile networks typically use their packet core (and prior iterations of the 
> same termination, rating, billing, management gear) to rate and bill specific 
> to each subscriber.  It is done with voice minutes, text, data.  Whether or 
> not you consider the solutions scalable is up to one's individual judgement. 
> But this zero rating billing model has and is very widely deployed and at 
> massive scale. In fact, there are specific interfaces defined for just these 
> purposes in the applicable standards.  There are many many conceivable ways 
> in which billing mediation and associated infrastructure for zero rating can 
> and is implemented. This is not new and is very well understood in the mobile 
> industry.  Not sure what you're after here.
> 
> Best regards,
> Christian
> 
> 
>> On Aug 31, 2015, at 6:01 PM, Jean-Francois Mezei 
>>  wrote:
>> 
>> 
>> Last year, one large mobile operator in Canada started to zero-rate its
>> own mobile TV offering. It appears that routers kept counting all the
>> data, but that the company then subtracted usage generated by its video
>> servers to come up with billable Gigabytes for each user.
>> 
>> (This was quashed by the regulator in Canada)
>> 
>> In the last week, another mobile operator announced it was zero rating
>> approved music streaming services (Spotify, Google Play and a few others).
>> 
>> If you are dealing with "foreign" content that comes from servers you
>> don't control, what are the "best practices" to zero-rate content from
>> multiple outside sources ?
>> 
>> To make matters more interesting, the FAQ for that service indicates
>> that if you listen to a music stream that exceeds 128kbps, you MAY be
>> charged for the data, and that you will be charged to listen to videos
>> that could be offered by that service, and for non streaming data such
>> as album covers, list of songs etc.
>> 
>> Would this point to specific IPs (streaming servers for low quality
>> 128kbps sound) ? How scalable is this when you start to have a whole
>> bunch of source IPs whose traffic is to be zero rated ?
>> 
>> Or would another way of doing this to setup private routes into the
>> ISP's network for each approved service, so the data would enter through
>> a different interface and be counted separately ?
>> 
>> Or, and this is my most important question: Is it possible with current
>> networking software to zero rate any data flow that is less than a
>> certain value (eg: 128kbs) ?
>> 
>> Or would current software require network operator to get 5 minute usage
>> for each user and only bill if average data rate during last 5 minutes
>> exceeded 128kbps ? (which means that your music is billed if you also
>> listen to netflix at same time since total data flows are greater than
>> 128kbps)
>> 
>> 
>> Of note: not all customers get this treatment, only those with higher
>> end packages. Those with lower end packages are charged for usage by
>> those very same services.
>> 
>> And for the record, this isn't to setup a similar system, it is to
>> better understand the issue for a regulatory challenge.

Re: NetFlow - path from Routers to Collector

[Avi Freedman]

(Jared wrote):



> Most people I've seen have little data or insight into their 
> networks, or don't have the level that they would desire as 
> tools are expensive or impossible to justify due to capital costs.  
> Tossing in a recurring opex cost of DC XC fee  + transport + XC fee + 
> redundant aggregation often doesn't have the ROI you infer here. 
> I've put together some models in this area.  It seems to me the 
> DC/real estate companies involved could make a lot (more) money by 
> offering an OOB service that is 10Mb/s flat-rate for the same as an XC 
> fee and compete with their customers.

Equinix does have a very aggressively priced 10Mb/s flat-rate OOB (single 
IP only but that's not that hard to work around) for essentially XC
pricing.  It's been stable but not something you'd rely on for 100%
packet delivery to some other point on the Internet (so more for
reaching a per-pop OOB than for making a coherent OOB network with
a bunch of monitoring running 24x7).

Still, it's a good value for what it is.



> - Jared

Avi Freedman
CEO, Kentik
avi at kentik dot com

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 3:45, Chuck Church wrote:

Most OOB is lacking redundancy too, so a single failure can really 
take the shine off an OOB deployment.


Even if you're using old, grandfathered equipment for it, there's no 
reason why your OOB/DCN can't have a reasonable degree of redundancy.  
Since, it's like, *what you use to control your entire network*.


Underinvesting in management capabilities and capacities has always been 
a problem, of course.  Some organizations just won't learn until they've 
gone through a disaster or three.


---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Jared Mauch]

> On Sep 1, 2015, at 6:51 PM, Roland Dobbins  wrote:
> 
> 
> On 2 Sep 2015, at 5:38, Jared Mauch wrote:
> 
>> Please stop digging,
> 
> Since I'm not digging, I've no reason to stop.  I see and deal with the 
> various quirks of more different platforms exporting flow telemetry than most 
> folks, all day, every day, so I know just a little bit about this topic.

You are, Avi has said that the number of people with a network is outnumbered 
about 50:1 using his most favorable numbers.  This means for your one example 
there are 50 people not doing this and the world hasn’t ended for them.  If you 
aren’t listening to Avi, please
trust me, you don’t need your own OOB network for flow, nor is putting your 
flow there going to provide you some magical value.  If you
can’t provision enough bandwidth for your telemetry data, you will obviously 
need to prune it back.  1:10k sampling works and you don’t
need much more than that unless you’re at extremely low bitrates.  Most attacks 
last under 1 hour and even the small ones shout out
in netflow data doing a simple hash sort algorithm with the proper keys.  You 
can even use QoS to mitigate if your goal is attack
traffic as they’re mostly UDP based attacks, see: 
https://tools.ietf.org/html/draft-byrne-opsec-udp-advisory-00 for some 
advice/input.

I’ve shared my own input at recent NANOG meetings, including policers to keep 
the attacks under control.

>> Sounds like you haven’t used Cisco recently.
> 
> I use Cisco all the time, thanks.  They aren't perfect - no vendor is.  They 
> have various issues with their NetFlow implementations on various platforms - 
> for example, bursts of wildly inaccurate flow statistics from CRS boxes when 
> a linecard is rebooted, a problem which has persisted for years and is just 
> now being addressed. Odd stuff with EARL8 on Sup2T/DFC4 in certain 
> configurations, and so forth.

I’m not talking about datacenter class equipment that you seem so focused on 
like the Earl7 with the TICO etc that did software sampling out of the hardware 
tcam and would be overrun.

> But Niels is grossly exaggerating. I get very usable flow telemetry from them 
> in many, many networks.  I deal with  flow telemetry from many, many 
> vendors/platforms, and I can confidently assert that Cisco are nowhere near 
> the bottom of the heap when it comes to the verisimilitude and functionality 
> of their flow telemetry export.  Quite the opposite

What people often don’t see is true “scale”[1] of netflow.  When you have 
enough attributes or want to actually look at your IPv6 there have been 
significant shortcomings.  We had to remind the patent holder for netflow how 
to implement it for their own hardware.

- Jared

aside: will you be in Yokohama?  We should get lunch/dinner.


[1] - I hate this word, vendors use it as an excuse to hardcode limits and to 
not properly respond to valid use cases

Re: NetFlow - path from Routers to Collector

[Avi Freedman]

(Said Roland:)

> Again, to clarify - I count VLANs/VRFs as being sufficiently out-of-band 
> to handle flow telemetry on a reasonable basis without mixing it in with 
> customer traffic.
> 
> That changes the ratio.



> I agree with you, Avi, and others that a dedicated OOB network *just for 
> flow telemetry* doesn't make economic sense in most (any?) scenarios.
> 
> What I'm saying is that it oughtn't to be mixed in with customer 
> data-plane traffic.  Ideally, all management-plane traffic would 
> traverse a separate physical infrastructure.  Since we don't live in an 
> ideal world, virtual separation is generally Good Enough.

We see well under 20% doing logical separation but definitely folks
doing it...  For the definition of OOB as "separate routers and 
switches", we don't see anyone really sending flow over that kind
of OOB network.

> ---
> Roland Dobbins 

Avi Freedman
CEO, Kentik
avi at kentik dot com

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 7:27, Avi Freedman wrote:

We see well under 20% doing logical separation but definitely folks 
doing it...


~20% matches our subjective observations, as well.

We're doing our best to increase that number.

---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Scott Weeks]

--- rdobb...@arbor.net wrote:

Again, to clarify - I count VLANs/VRFs as being sufficiently out-of-band 
to handle flow telemetry on a reasonable basis without mixing it in with 
customer traffic.
---

Glad you clarified that.  Now, I understand your stance.  Like the 
others are saying, in 18 years of netgeeking I never had OOB with 
flow.  It has always been in a VLAN or VRF.




--- freed...@freedman.net wrote:

We see well under 20% doing logical separation but definitely folks
doing it...  
--

Now that's a scary number.  VLANS/VRFs are too easy not to do that.


scott

Re: NetFlow - path from Routers to Collector

[jim deleskie]

I've not read every reply, but let me add my voice as some who has worked
on and ran SEVERAL large networks, in no case in the last long number of
years have I had access to an OOB network that was sized to carry anything
in large volume, and in fact like many others replied on a robust number of
path at that us many the networks inband.  These networks survived many
"large" DDoS attacks and far more fat finger incidents then I like to think
about.  I don't think I've even worked with a client network as far as I
can remember that had a nailed up / robust OOB network for data collection
or other purposes.

-jim

On Tue, Sep 1, 2015 at 8:30 PM, Avi Freedman  wrote:

> (Jared wrote):
>
> 
>
> > Most people I've seen have little data or insight into their
> > networks, or don't have the level that they would desire as
> > tools are expensive or impossible to justify due to capital costs.
> > Tossing in a recurring opex cost of DC XC fee  + transport + XC fee +
> > redundant aggregation often doesn't have the ROI you infer here.
> > I've put together some models in this area.  It seems to me the
> > DC/real estate companies involved could make a lot (more) money by
> > offering an OOB service that is 10Mb/s flat-rate for the same as an XC
> > fee and compete with their customers.
>
> Equinix does have a very aggressively priced 10Mb/s flat-rate OOB (single
> IP only but that's not that hard to work around) for essentially XC
> pricing.  It's been stable but not something you'd rely on for 100%
> packet delivery to some other point on the Internet (so more for
> reaching a per-pop OOB than for making a coherent OOB network with
> a bunch of monitoring running 24x7).
>
> Still, it's a good value for what it is.
>
> 
>
> > - Jared
>
> Avi Freedman
> CEO, Kentik
> avi at kentik dot com
>
>

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 0:44, Jared Mauch wrote:

	I think the key here is that Roland isn't often constrained by these 
financial considerations.


That's entirely true.  I deal every day with customers who are, though.

	I would respectfully disagree with Roland here and agree with Job, 
Niels, etc.


I understand where you and they are coming from, in this regard.  I just 
disagree, as well.


	A few networks have robust out of band networks, but most I've seen 
have an interesting mixture of things


Concur 100%.


and inband is usually the best method.


Let me be clear - OOB for flow telemetry can be actually provisioned on 
the same boxes which are handling the production network traffic.  It 
isn't ideal, but it's better than running it truly inband in the 
production network, mixed in with customer traffic.  VLANs, VRFs, 
whatever are a reasonable compromise, and a lot of folks do this.


Inband is a huge risk, especially in a world of multi-hundred gb/sec 
reflection/amplification attacks (not to mention the other catastrophic 
failure scenarios).  I know you sink a lot of UDP at the edges of your 
network to ameliorate this problem, but not all operators do that or 
agree with it either in principle or as a matter of optimal utility.  I 
understand that this sort of thing is a decision that all network 
operators must make for themselves based upon their knowledge of their 
own networks and customer needs.


	Those that do have "seperate" networks may actually be CoC services 
from another deparment in the same company riding the same P/PE 
devices (sometimes routers).


Yes, that's what I'm getting at above.  It isn't ideal, but there's no 
reason to make the perfect the enemy of the merely good, agreed.


	I've seen oob networks on DSL, datacenter wifi or cable swaps through 
the fence to an adjacent rack.


Absolutely.  All kinds of creative lashups to get console access in 
difficult situations (and, as you noted previously, an increasing number 
of devices don't support serial console at all, which is highly 
annoying).


---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 5:49, Jared Mauch wrote:

Other platforms (e.g.: IOS-XR based) have issues with the MgmtEther 
interfaces which make them inoperable for many use-cases.


I'm agreeing with you.  Dedicated management ports on many boxes don't 
actually support important management-plane functions, like flow 
telemetry - which is nuts, but that's what happens.


There are many technical details that are easily overlooked by those 
not using the routers to their abilities, so a small network (as Wes 
mentioned before with 2500s/T1s) still as OOB is unlikely to see
data rates comparable to what is seen from a large router exporting 
data from hundreds of

gigs of flows.


That's true.  I understand that even on large networks, the OOB/DCN is 
built from old, grandfathered equipment.  I spend a lot of time helping 
network operators calculate optimal flow sampling rates, flow cache 
sizes, etc., and an important consideration in making optimal 
configuration choices is what the OOB/DCN network can handle.


Often net flow vendors tell customers things that create more flow 
records which equals slightly higher data resolution but no actual net 
difference in results except for the lowest of bitrates.


Concur 100%.  I spend a non-trivial amount of time talking folks down 
from the assumption that unnecessarily-low flow sampling ratios are 
required (these are mainly 'security' folks, not network engineers).


---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 0:46, Niels Bakker wrote:


This is the dumbest thing I've read on this mailing list in a while.


It happens.  You can deny it all you like, but I've seen it happen, and 
the resultant confusion and additional time to resolve problems it 
causes.


---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 6:06, Jared Mauch wrote:

You are, Avi has said that the number of people with a network is 
outnumbered about 50:1 using his most favorable numbers.


Again, to clarify - I count VLANs/VRFs as being sufficiently out-of-band 
to handle flow telemetry on a reasonable basis without mixing it in with 
customer traffic.


That changes the ratio.

This means for your one example there are 50 people not doing this and 
the world hasn’t ended for them.  If you aren’t listening to Avi, 
please trust me, you don’t need your own OOB network for flow, nor 
is putting your flow there going to provide you some magical value.


I agree with you, Avi, and others that a dedicated OOB network *just for 
flow telemetry* doesn't make economic sense in most (any?) scenarios.


What I'm saying is that it oughtn't to be mixed in with customer 
data-plane traffic.  Ideally, all management-plane traffic would 
traverse a separate physical infrastructure.  Since we don't live in an 
ideal world, virtual separation is generally Good Enough.


1:10k  sampling works and you don’t need much more than that unless 
you’re at extremely low bitrates.  Most attacks last under 1 hour 
and even the  small ones shout out in netflow data doing a simple hash 
sort algorithm with the proper keys


Concur 100%.  I spend a lot of time explaining to customers that no, 
they don't need/want 1:1 even if they could get it, and that the 'wake' 
left by attack traffic stands out very well even at relatively high 
sampling ratios.


Most of the network-oriented folks seem to grasp this pretty quickly.  
It's generally the 'security' types who often seem 
conceptually/attitudinally incapable of understanding these principles.



.  You can even use QoS to mitigate if your goal is attack
traffic as they’re mostly UDP based attacks, see: 
https://tools.ietf.org/html/draft-byrne-opsec-udp-advisory-00 for some 
advice/input.


I know you do this, and I understand why.  Not everyone agrees with this 
and does it, and I also understand why (not).


ntp is easy, because there's the timesync packet-size classification 
hook.  It gets a little dicier with other things.


I’ve shared my own input at recent NANOG meetings, including 
policers to keep the attacks under control.


And it's valuable experience to share, nobody disputes that.

I’m not talking about datacenter class equipment that you seem so 
focused on like the Earl7 with the TICO etc that did software sampling 
out of the hardware tcam and would be overrun.


I'm pretty sure the CRSes I referred to with the linecard-reboot issue 
in my example aren't datacenter-class equipment.


;>

What people often don’t see is true “scale”[1] of netflow.  When 
you have enough attributes or want to actually look at your IPv6 there 
have been significant shortcomings.  We had to remind the patent 
holder for netflow how to implement it for their own hardware.


This is very true.  IPv6 flow telemetry is another area in which 
IPv4/IPv6 feature parity lags.  Because of your focus on large-scale 
IPv6 deployment over the course of many years, you see and experience a 
lot more IPv6-related deficiencies than most folks.



aside: will you be in Yokohama?  We should get lunch/dinner.


Yes, and yes.

;>

[1] - I hate this word, vendors use it as an excuse to hardcode limits 
and to not properly respond to valid use cases


Concur 100%.

Another annoying vendor trait is use-case obsession.  In many contexts, 
the right answer is to understand that there is a baseline plateau of 
vitally necessary scaling (that word, again) capacity and required 
functionality which is universally applicable, irrespective of 
variations in particular use cases.


I recently had a discussion with someone who was asking me how many 
attack sources one typically sees in a given DDoS attack.  My response 
was that there is no 'typical'; and that for IPv4, the theoretical 
potential is 2^32 sources, while in IPv6, the theoretical potential is 
for 2^128 sources.


It was a light-bulb moment.

;>

---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 7:38, jim deleskie wrote:

These networks survived many "large" DDoS attacks and far more fat 
finger incidents then I like to think

about.


What I'm saying is that keeping flow telemetry and other 
management-plane traffic from mixing with customer data-plane traffic is 
important in order to ensure visibility and control during a significant 
network-impacting event.


I've personally been involved in assisting multiple operators in 
multiple incidents in which either DDoS attack traffic or inadvertent 
routing redistribution excursions led to loss of visibility and control, 
resulting in unnecessarily-long times to resolution.


Virtual separation is generally Good Enough, and what we see with 
customers who run it all in-band is an increasing number who're taking 
steps to achieve at least virtual separation (~20%, as Avi noted, is 
about what we see implemented, currently).  It isn't nearly as many as 
we would like to see, and it isn't happening as fast as we'd like to see 
it, but we encourage it wherever we can.


The OP on this thread was essentially asking about the best approach.  
OOB, whether virtual or physical, is the best approach.  Economic 
factors may militate against this, at least initially, but a disaster or 
two can change that economic analysis.


I also suspect that increasing use of 'SDN'-type (apologies for using 
that overused acronym!) orchestration across the entire network topology 
(e.g., not just within the IDC) is going to lead to more separation of 
management-plane traffic from customer data-plane traffic, as the 
implications sink in.


---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 5:38, Jared Mauch wrote:


Please stop digging,


Since I'm not digging, I've no reason to stop.  I see and deal with the 
various quirks of more different platforms exporting flow telemetry than 
most folks, all day, every day, so I know just a little bit about this 
topic.



Sounds like you haven’t used Cisco recently.


I use Cisco all the time, thanks.  They aren't perfect - no vendor is.  
They have various issues with their NetFlow implementations on various 
platforms - for example, bursts of wildly inaccurate flow statistics 
from CRS boxes when a linecard is rebooted, a problem which has 
persisted for years and is just now being addressed. Odd stuff with 
EARL8 on Sup2T/DFC4 in certain configurations, and so forth.


But Niels is grossly exaggerating. I get very usable flow telemetry from 
them in many, many networks.  I deal with  flow telemetry from many, 
many vendors/platforms, and I can confidently assert that Cisco are 
nowhere near the bottom of the heap when it comes to the verisimilitude 
and functionality of their flow telemetry export.  Quite the opposite


---
Roland Dobbins 

RE: NetFlow - path from Routers to Collector

[Chuck Church]

Agree.  Most OOB is lacking redundancy too, so a single failure can really take 
the shine off an OOB deployment.  Especially when you've put your management 
traffic on it, including radius traffic, and you're using 802.1X.  Found that 
out the hard way a few years ago.  

Chuck

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Tarko Tikan
Sent: Tuesday, September 01, 2015 3:47 PM
To: nanog@nanog.org
Subject: Re: NetFlow - path from Routers to Collector

hey,

> It should've already been spent for an OOB/DCN network, which 
> should've been provisioned with flow telemetry in mind.

Bad advice. No amount of money will fix major platforms that are not happy to 
export flow telemetry via router management ports. Sometimes it can be done via 
nasty vrf leaking hacks, sometimes it cannot be done at all. Management ports 
are typically directly connected to routing engines while netflow data is 
generated in hardware in PFE.

In-band netflow works on all platforms without such issues.

--
tarko

Re: FTTP Advice, Michigan and other areas

[Fletcher Kittredge]

Jared;

What you are trying to do is quite achievable, but a huge topic worthy of a
book, not an email post. Also, situations vary significantly between states
due to incumbents, regulatory regimes, and level of state support. NANOG is
a bad place to get advice about this topic. There are many other venues
with literally thousands of other organizations/groups/companies on the
same path as you. I would start from the FTTH Council, Next Century Cities,
Institute for Local Self-Reliance and work out from there.


On Tue, Sep 1, 2015 at 4:27 PM, Jared Mauch  wrote:

> I’m looking for some advice/input from people either public or private
> about woes building fiber to reach people outside the footprints of the
> existing incumbents.
>
> There is a group of people looking to organize private fiber to reach
> areas that are unserved.
>
> There’s been recent local people doing this like Lightspeed (Lansing) and
> the Vergennes Broadband folks.
>
> When it come to private right of way, public right of way use, swaps, pole
> attach and other things, any best practices people can share either in
> public or private?
>
> TL;DR background for those interested:
>
> Many wireless ISPs are finding it harder to locate equipment or
> utilize frequencies based on interference or congestion.  Advanced
> encodings like 16-QAM that are seen in 802.11ac hardware also introduce
> latencies that are not ideal.  The FCC is also making it harder for
> equipment to be qualified in this space, in some cases rightly so due to
> out of band emissions or just adjacent frequency noise.  The revisions of
> rules in 5Ghz are helpful, but the cellular industry is also looking to
> exploit these frequencies to solve indoor coverage.
>
> There are two groups I’m trying to assist, a local cooperative
> which is trying to just own the fiber and let providers gain access and
> some WISPs that are looking to improve service due to increase customer
> demand.
>
> Getting service on the fiber is “easy” once it’s there, but
> gaining access or building it is the part I’m looking for insights in.
>
> - Jared
>
>


-- 
Fletcher Kittredge
GWI
8 Pomerleau Street
Biddeford, ME 04005-9457
207-602-1134

Re: NetFlow - path from Routers to Collector

[Leo Bicknell]

In a message written on Wed, Sep 02, 2015 at 12:29:25AM +0700, Roland Dobbins 
wrote:
> On 2 Sep 2015, at 0:18, Niels Bakker wrote:
> > You're just wrong here.
> 
> Sorry, I'm not.  I've seen what happens when flow telemetry is 'squeezed 
> out' by pipe-filling DDoS attacks, interrupted by fat-fingers, etc.
> 
> It'll happen to you, one day.  And then you'll understand.

Ah, I see your mistake, you're thinking everyone cares about that
problem.

They don't.  

Good, fast, cheap, pick two.  You've selected good.  Some people
pick fast and cheap.  They are not wrong, you are not right.  Just a
different lifestyle choice.

-- 
   Leo Bicknell - bickn...@ufp.org - CCIE 3440
PGP keys at http://www.ufp.org/~bicknell/


pgpjRA813DA9_.pgp
Description: PGP signature

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 3:34, Nick Hilliard wrote:

> If you want to handle netflow data export for large amounts of traffic, it
> would be pretty dumb to push it through the management plane of the router.

Concur 100%. You must use a port capable of doing so.

---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Jared Mauch]

> On Sep 1, 2015, at 6:36 PM, Roland Dobbins  wrote:
> 
> On 2 Sep 2015, at 3:24, Niels Bakker wrote:
> 
>> Because variety of flow telemetry delivery options isn't the #1 ranked 
>> purchasing decider.
> 
> Actually, it is more often than you think.  No use routing packets if you 
> can't see what they do.
> 
>> Otherwise no Cisco would ever have been sold.
> 
> Which is utter nonsense, of course, since Cisco a) invented flow telemetry 
> and b) has been the consistent leader in innovating flow telemetry (FNF, 
> IPFIX, anyone?).  The EARL6/EARL7 problems are the only stumbles Cisco has 
> made in this regard.

Roland,

Please stop digging, Sounds like you haven’t used Cisco recently.  I’m happy to 
elaborate privately.

- Jared

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 0:55, Avi Freedman wrote:

Looking at probably 100 networks' flow paths over the last year, I'd 
say 1 or 2 have OOB for flow.


Far fewer have it than should, agreed.  A reasonable compromise is 
VLANs, VRFs, and so on to at least keep it out of the data-plane of the 
production network.


But for folks seeing DDoS, we implement rate-limiting of the flows/sec 
via local proxies

to avoid overwhelming network capacity with the flow data...


A lot of networks do that - they collect the flow telemetry relatively 
topologically near their edge routers which are exporting it, do 
distributed analysis (depending upon what tools they're using for 
collection/analysis), and then the analysis results are what's 
long-hauled - and this is much less than the raw flow telemetry volume.


---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Nick Hilliard]

On 01/09/2015 21:13, valdis.kletni...@vt.edu wrote:
> On Tue, 01 Sep 2015 22:47:09 +0300, Tarko Tikan said:
>> Bad advice. No amount of money will fix major platforms that are not
>> happy to export flow telemetry via router management ports.
> 
> And that box ended up in your rack, why exactly?
> 
>  have this functionality because we were OK with sending flow telemetry over
> the inband network">

this approach is fine for bitty boxes handling small quantities of traffic.
 If you want to handle netflow data export for large amounts of traffic, it
would be pretty dumb to push it through the management plane of the router.

Nick

Re: FTTP Advice, Michigan and other areas

[William Herrin]

On Tue, Sep 1, 2015 at 4:27 PM, Jared Mauch  wrote:
> I’m looking for some advice/input from people either public or private
> about woes building fiber to reach people outside the footprints of the
> existing incumbents.

I worked on one such project, indirectly, years ago. Fiber in a small
town. My three takeaways were:

At the time it cost about $5 per year per pole to rent attachments on
the phone pole. An attachment is a particular distance up the pole
where you are allowed to attach your cables. Rights of way from pole
to pole included in the deal. The power company usually owns the poles
and is usually required by regulators to rent attachments.

"Help" the local schools with a "partnership" to bring them fiber
interconnects. Your part of the partnership is interconnecting the
underfunded schools well below your cost. Their part is clearing the
bureaucratic hurdles to stringing your fiber all over town. 'Cause
tech is a source of tax revenue... unless it's all about the kids.

Resist the urge to be a cable and phone operator at the same time.
Yes, there's a temptingly large bucket of money over there, but it's
not for you. The incumbents aren't going to take your entry in to the
market sitting down. If you would beat them, you need the folks who
would battle the incumbents for the buckets of "content" money to all
be pulling for your success.

Regards,
Bill Herrin





-- 
William Herrin  her...@dirtside.com  b...@herrin.us
Owner, Dirtside Systems . Web: 

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 3:24, Niels Bakker wrote:

Because variety of flow telemetry delivery options isn't the #1 ranked 
purchasing decider.


Actually, it is more often than you think.  No use routing packets if 
you can't see what they do.



Otherwise no Cisco would ever have been sold.


Which is utter nonsense, of course, since Cisco a) invented flow 
telemetry and b) has been the consistent leader in innovating flow 
telemetry (FNF, IPFIX, anyone?).  The EARL6/EARL7 problems are the only 
stumbles Cisco has made in this regard.


---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 2:38, George, Wes wrote:

Often there is a separate management network that can deal with 
ethernet
speeds, but it's separate for security reasons and not always as 
rigidly
independent from the in band network for connectivity, i.e. It might 
be a
VPN riding over the regular network and thus not completely protected 
from

the problem you're concerned about.


Sure, or a VRF, or whatever.

While that's not ideal, it's far better than doing management-plane 
stuff inband in the production network, though.


And those 2500 console concentrator connections are a great resource to 
have when everything goes haywire and you need something that lets you 
get to and actually type on the console.  I'm not knocking them, and I 
understand that old, grandfathered equipment is used for these 
applications, and understand that in many cases they're underprovisioned 
for flow telemetry.


Which is why using VLANs, VRFs, whatever on the production network gear 
is completely understandable, and a lot of folks do just as you say.


---
Roland Dobbins 

FTTP Advice, Michigan and other areas

[Jared Mauch]

I’m looking for some advice/input from people either public or private about 
woes building fiber to reach people outside the footprints of the existing 
incumbents.

There is a group of people looking to organize private fiber to reach areas 
that are unserved.

There’s been recent local people doing this like Lightspeed (Lansing) and the 
Vergennes Broadband folks.

When it come to private right of way, public right of way use, swaps, pole 
attach and other things, any best practices people can share either in public 
or private?

TL;DR background for those interested:

Many wireless ISPs are finding it harder to locate equipment or utilize 
frequencies based on interference or congestion.  Advanced encodings like 
16-QAM that are seen in 802.11ac hardware also introduce latencies that are not 
ideal.  The FCC is also making it harder for equipment to be qualified in this 
space, in some cases rightly so due to out of band emissions or just adjacent 
frequency noise.  The revisions of rules in 5Ghz are helpful, but the cellular 
industry is also looking to exploit these frequencies to solve indoor coverage.

There are two groups I’m trying to assist, a local cooperative which is 
trying to just own the fiber and let providers gain access and some WISPs that 
are looking to improve service due to increase customer demand.

Getting service on the fiber is “easy” once it’s there, but gaining 
access or building it is the part I’m looking for insights in.

- Jared

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 2 Sep 2015, at 2:47, Tarko Tikan wrote:


In-band netflow works on all platforms without such issues.


There's no law that says that you must only plug designated management 
ports into OOB/DCN management networks.


---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Valdis . Kletnieks]

On Tue, 01 Sep 2015 22:47:09 +0300, Tarko Tikan said:
> hey,
>
> > It should've already been spent for an OOB/DCN network, which should've
> > been provisioned with flow telemetry in mind.
>
> Bad advice. No amount of money will fix major platforms that are not
> happy to export flow telemetry via router management ports.

And that box ended up in your rack, why exactly?




pgpHDnaQXv1N9.pgp
Description: PGP signature

Re: NetFlow - path from Routers to Collector

[Niels Bakker]

On Tue, 01 Sep 2015 22:47:09 +0300, Tarko Tikan said:
Bad advice. No amount of money will fix major platforms that are 
not happy to export flow telemetry via router management ports.


Correct.  And for a proper network you may not wish to have those 
connections from in-band ports to your OOB/management network everywhere.



* valdis.kletni...@vt.edu (valdis.kletni...@vt.edu) [Tue 01 Sep 2015, 22:13 
CEST]:

And that box ended up in your rack, why exactly?


Because variety of flow telemetry delivery options isn't the #1 ranked 
purchasing decider.  Otherwise no Cisco would ever have been sold.



that didn't have this functionality because we were OK with sending 
flow telemetry over the inband network">


Everything is a tradeoff.  Welcome to the real world, where we have to 
make things work rather than pose on mailing lists about what we think 
other people should have.



-- Niels.

Re: FTTP Advice, Michigan and other areas

[Jared Mauch]

> On Sep 1, 2015, at 5:38 PM, William Herrin  wrote:
> 
> On Tue, Sep 1, 2015 at 4:27 PM, Jared Mauch  wrote:
>> I’m looking for some advice/input from people either public or private
>> about woes building fiber to reach people outside the footprints of the
>> existing incumbents.
> 
> I worked on one such project, indirectly, years ago. Fiber in a small
> town. My three takeaways were:
> 
> At the time it cost about $5 per year per pole to rent attachments on
> the phone pole. An attachment is a particular distance up the pole
> where you are allowed to attach your cables. Rights of way from pole
> to pole included in the deal. The power company usually owns the poles
> and is usually required by regulators to rent attachments.

I’ve heard that auditing the bills is quite important as they often don’t
know who is attached to the poles.  Labeling requirements are stricter
as well and I can often pick out the yellow comcast labels here, the blue
university/merit labels and others without slowing down.

> "Help" the local schools with a "partnership" to bring them fiber
> interconnects. Your part of the partnership is interconnecting the
> underfunded schools well below your cost. Their part is clearing the
> bureaucratic hurdles to stringing your fiber all over town. 'Cause
> tech is a source of tax revenue... unless it's all about the kids.

We have this place called Merit around here that some people may know
that tries to cover the schools.  They also got either NTIA or RUS
monies (i don’t recall which) and are controlled by various state
universities.  I’ve not yet met their new president but recall having
many conversations with people there off and on for ~20 years.


> Resist the urge to be a cable and phone operator at the same time.
> Yes, there's a temptingly large bucket of money over there, but it's
> not for you. The incumbents aren't going to take your entry in to the
> market sitting down. If you would beat them, you need the folks who
> would battle the incumbents for the buckets of "content" money to all
> be pulling for your success.

The incumbents aren’t battling for these areas, they either have only
cellular data or no service.  I am served by a fixed wireless customer 
myself and even with Michigan Bell/Ameritech/SBC/AT fiber 1200 feet away
there is no broadband here except fixed wireless, dial or cellular data.

Content is clearly the largest data source and IMHO the easiest to solve,
most of the content people are willing to either place equipment in a network
permise or something else.

The wireless ISP wants to expand with fiber, mostly burial.  The local
cooperative wants to just build fiber on these mostly rural routes and
have someone like the WISP expand into the FTTP business and lease the
strands back.  The incumbents are unlikely to buy fiber from this model
in my mind, but I’m willing to accept counter data points.  I recall
some of the traditional bell company people being upset using cable
fiber due to the perceived poor quality to recover their networks post
Katrina, likely because they had more splices and lower OTDR results.

- jared

Re: NetFlow - path from Routers to Collector

[Jared Mauch]

> On Sep 1, 2015, at 6:37 PM, Roland Dobbins  wrote:
> 
> On 2 Sep 2015, at 3:34, Nick Hilliard wrote:
> 
>> If you want to handle netflow data export for large amounts of traffic, it
>> would be pretty dumb to push it through the management plane of the router.
> 
> Concur 100%. You must use a port capable of doing so.

My experience in running large networks is these ports often can’t handle the 
traffic
involved.

The packet path in a juniper (for example to go from the PFE -> RE -> Ethernet) 
is very
sensitive to the jitter introduced by increased traffic loads and may result in 
the box
becoming unstable.

Other platforms (e.g.: IOS-XR based) have issues with the MgmtEther interfaces
which make them inoperable for many use-cases.  There are many technical details
that are easily overlooked by those not using the routers to their abilities, so
a small network (as Wes mentioned before with 2500s/T1s) still as OOB is 
unlikely to see
data rates comparable to what is seen from a large router exporting data from 
hundreds of
gigs of flows.

Often net flow vendors tell customers things that create more flow records
which equals slightly higher data resolution but no actual net difference 
in results except for the lowest of bitrates.  

Making sure your flow implementation is optimized (ingress only, relevant links 
only)
is one part of having it scale.  I’ve seen many a solution that scales poorly
or requires dozens of boxes for datasets that don’t require it.  It’s
easy to say over specify for an attack because of the “Think of the 
Children^WDDoS”
mentality that exists, but when you are on the receiving end of a large attack
there are better tools to use.

- Jared

Re: NetFlow - path from Routers to Collector

[Jared Mauch]

> On Sep 1, 2015, at 6:39 PM, Roland Dobbins  wrote:
> 
> On 2 Sep 2015, at 3:45, Chuck Church wrote:
> 
>> Most OOB is lacking redundancy too, so a single failure can really take the 
>> shine off an OOB deployment.
> 
> Even if you're using old, grandfathered equipment for it, there's no reason 
> why your OOB/DCN can't have a reasonable degree of redundancy.  Since, it's 
> like, *what you use to control your entire network*.

Most networks use inband to manage them.

> Underinvesting in management capabilities and capacities has always been a 
> problem, of course.  Some organizations just won't learn until they've gone 
> through a disaster or three.

Yes.  let me know when the vendors catch up in this area.  I often see people 
say to create a new network as job security vs making the inband network 
survive attacks or be provisioned properly.

Most people I’ve seen have little data or insight into their networks, or don’t 
have the level that they would desire as tools are expensive or impossible to 
justify due to capital costs.  Tossing in a recurring opex cost of DC XC fee  + 
transport + XC fee + redundant aggregation often doesn’t have the ROI you infer 
here.  I’ve put together some models in this area.  It seems to me the DC/real 
estate companies involved could make a lot (more) money by offering an OOB 
service that is 10Mb/s flat-rate for the same as an XC fee and compete with 
their customers.

Things continue to be a challenge as less equipment works with a serial console 
and the expectation of developers of these embedded solutions don’t take into 
account low bitrate connections that are often used in last-resort situations.

We have a well oiled set of processes and checklists to monitor and test our 
management network.  Patrick Gilmore has personally mocked me because of its 
method and technique, but the reality is it works.

- Jared

Re: BGP advertise-best-external on RR

[Mohamed Kamal]

Hi,

Diverse-path will only send the second best path, and in my case I have 
three routes not two. In addition to that, every PE will have to peer 
with the RR via a second session (on the same RR, as I will not deploy a 
new standalone shadow RR) and this will increase the BGP sessions to the 
double.


Add-path will have a network-wide IOS upgrade for this BGP capability to 
be supported which is not viable now.


So, is there any other recommendation other than the internet VRF with 
different RDs solution?


Regards,

Mohamed Kamal
Core Network Sr. Engineer

On 8/25/2015 11:37 AM, Jeff Tantsura wrote:

Hi,

In your case I¹d recommend to use diverse path, due to its simplicity and
non disruptive deployment characteristics.
As you know - diverse path requires additional BGP session per additional
(second, next, etc) path, in most cases not a problem, however mileage
might vary.

To my memory, in Cisco land - it has only been implemented in IOS, not XR,
please check.

Cheers,
Jeff




-Original Message-
From: Diptanshu Singh 
Date: Monday, August 24, 2015 at 10:53 PM
To: Mohamed Kamal 
Cc: "nanog@nanog.org" 
Subject: Re: BGP advertise-best-external on RR


Yes . In the case of diverse path , shadow route reflector will be the
one wherever  you enable commands to trigger diverse path computation.

Good thing with diverse path is that the RR-Clients don't have to have
any support but bad thing is that it can only reflect One additional
best-path( second best path ) .

Sent from my iPhone


On Aug 24, 2015, at 2:31 PM, Mohamed Kamal  wrote:

It's only supported on the 15.2(4)S and later not the SRE train. I
might consider an upgrade.

One more question regarding this, can you configure the RR to be the
main and shadow RR?

Mohamed Kamal
Core Network Sr. Engineer


On 8/24/2015 9:16 PM, Diptanshu Singh wrote:
BGP Add-Path might be your friend . You can look at diverse-path as
well .

Re: DDoS appliances reviews needed

[Ramy Hashish]

And at the end of the day it's commercial, vendors pay, I don't trust much.

I am looking for hands-on experience

Ramy


>
> Message: 8
> Date: Thu, 27 Aug 2015 16:00:10 +
> From: Steve Mikulasik 
> To: "nanog@nanog.org" 
> Subject: RE: DDoS appliances reviews needed
> Message-ID:
> <
> by1pr0701mb1781d116ed99f67358e79877fa...@by1pr0701mb1781.namprd07.prod.outlook.com
> >
>
> Content-Type: text/plain; charset=UTF-8
>
> I assume they don't list their testing methodology right? It always feels
> like they just read vendor spec sheets.
>
> Steve Mikulasik
>
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joe Chisolm
> Sent: Thursday, August 27, 2015 9:53 AM
> To: nanog@nanog.org
> Subject: Re: DDoS appliances reviews needed
>
> Gartner did a report about a year ago.  Not free.
> https://www.gartner.com/doc/2910217/ddos-comparison-defense-approaches
>
> On 08/26/2015 07:40 AM, Ramy Hashish wrote:
> > Good day all,
> >
> > Anybody here has experienced a PoC for any anti DDoS appliance, or
> > already using a anti DDoS appliance in production and able to share
> > his user experience/review?
> >
> > We need to collect good reviews from people whom got their hands dirty
> > with the configuration/attack mitigation, real experience.
> >
> > Thanks,
> >
> > Ramy
> >
>
> --
> Joe Chisolm
> Computer Translations, Inc.
> Network and Datacenter Consulting
> Marble Falls, Tx.
> 830-265-8018
>
> Public Key Available at http://www.sks-keyservers.net
>
>
>
>

Re: NetFlow - path from Routers to Collector

[Avi Freedman]

Agreed, we are as well :)

VLAN, VRF, whatever.

+ optimal tweaks include local flow proxy that can also rate 
limit / re-sample, and send topk talkers over 'true' OOB.

Avi Freedman
CEO, Kentik
avi at kentik dot com

> On 2 Sep 2015, at 7:27, Avi Freedman wrote:
> 
> > We see well under 20% doing logical separation but definitely folks 
> > doing it...
> 
> ~20% matches our subjective observations, as well.
> 
> We're doing our best to increase that number.
> 
> ---
> Roland Dobbins 

Re: BGP advertise-best-external on RR

[dip]

There's no such thing as free lunch right :). If BGP Add-Path or
Diverse-Path isn't option for you then as you mentioned Internet VRF with
different RD is the third option but you have to understand the
implications here as well around increase in % of memory. Another option
could be to bypass the RR's and just fully mesh it.

But if none of the above is an option, then you may have to do some
unnatural act to solve it.

On Tue, Sep 1, 2015 at 7:51 AM, Mohamed Kamal  wrote:

> Hi,
>
> Diverse-path will only send the second best path, and in my case I have
> three routes not two. In addition to that, every PE will have to peer with
> the RR via a second session (on the same RR, as I will not deploy a new
> standalone shadow RR) and this will increase the BGP sessions to the double.
>
> Add-path will have a network-wide IOS upgrade for this BGP capability to
> be supported which is not viable now.
>
> So, is there any other recommendation other than the internet VRF with
> different RDs solution?
>
> Regards,
>
> Mohamed Kamal
> Core Network Sr. Engineer
>
> On 8/25/2015 11:37 AM, Jeff Tantsura wrote:
>
>> Hi,
>>
>> In your case I¹d recommend to use diverse path, due to its simplicity and
>> non disruptive deployment characteristics.
>> As you know - diverse path requires additional BGP session per additional
>> (second, next, etc) path, in most cases not a problem, however mileage
>> might vary.
>>
>> To my memory, in Cisco land - it has only been implemented in IOS, not XR,
>> please check.
>>
>> Cheers,
>> Jeff
>>
>>
>>
>>
>> -Original Message-
>> From: Diptanshu Singh 
>> Date: Monday, August 24, 2015 at 10:53 PM
>> To: Mohamed Kamal 
>> Cc: "nanog@nanog.org" 
>> Subject: Re: BGP advertise-best-external on RR
>>
>> Yes . In the case of diverse path , shadow route reflector will be the
>>> one wherever  you enable commands to trigger diverse path computation.
>>>
>>> Good thing with diverse path is that the RR-Clients don't have to have
>>> any support but bad thing is that it can only reflect One additional
>>> best-path( second best path ) .
>>>
>>> Sent from my iPhone
>>>
>>> On Aug 24, 2015, at 2:31 PM, Mohamed Kamal  wrote:

 It's only supported on the 15.2(4)S and later not the SRE train. I
 might consider an upgrade.

 One more question regarding this, can you configure the RR to be the
 main and shadow RR?

 Mohamed Kamal
 Core Network Sr. Engineer

 On 8/24/2015 9:16 PM, Diptanshu Singh wrote:
> BGP Add-Path might be your friend . You can look at diverse-path as
> well .
>

>>
>

Re: Zero rating implentation strategies

[Christian Kuhtz]

Zero rating is not a new concept. It has existed in the mobile world since the 
days of the dumb phone.  Got a reference to why this was killed by the 
regulator in Canada?

Mobile networks typically use their packet core (and prior iterations of the 
same termination, rating, billing, management gear) to rate and bill specific 
to each subscriber.  It is done with voice minutes, text, data.  Whether or not 
you consider the solutions scalable is up to one's individual judgement. But 
this zero rating billing model has and is very widely deployed and at massive 
scale. In fact, there are specific interfaces defined for just these purposes 
in the applicable standards.  There are many many conceivable ways in which 
billing mediation and associated infrastructure for zero rating can and is 
implemented. This is not new and is very well understood in the mobile 
industry.  Not sure what you're after here.

Best regards,
Christian


> On Aug 31, 2015, at 6:01 PM, Jean-Francois Mezei 
>  wrote:
> 
> 
> Last year, one large mobile operator in Canada started to zero-rate its
> own mobile TV offering. It appears that routers kept counting all the
> data, but that the company then subtracted usage generated by its video
> servers to come up with billable Gigabytes for each user.
> 
> (This was quashed by the regulator in Canada)
> 
> In the last week, another mobile operator announced it was zero rating
> approved music streaming services (Spotify, Google Play and a few others).
> 
> If you are dealing with "foreign" content that comes from servers you
> don't control, what are the "best practices" to zero-rate content from
> multiple outside sources ?
> 
> To make matters more interesting, the FAQ for that service indicates
> that if you listen to a music stream that exceeds 128kbps, you MAY be
> charged for the data, and that you will be charged to listen to videos
> that could be offered by that service, and for non streaming data such
> as album covers, list of songs etc.
> 
> Would this point to specific IPs (streaming servers for low quality
> 128kbps sound) ? How scalable is this when you start to have a whole
> bunch of source IPs whose traffic is to be zero rated ?
> 
> Or would another way of doing this to setup private routes into the
> ISP's network for each approved service, so the data would enter through
> a different interface and be counted separately ?
> 
> Or, and this is my most important question: Is it possible with current
> networking software to zero rate any data flow that is less than a
> certain value (eg: 128kbs) ?
> 
> Or would current software require network operator to get 5 minute usage
> for each user and only bill if average data rate during last 5 minutes
> exceeded 128kbps ? (which means that your music is billed if you also
> listen to netflix at same time since total data flows are greater than
> 128kbps)
> 
> 
> Of note: not all customers get this treatment, only those with higher
> end packages. Those with lower end packages are charged for usage by
> those very same services.
> 
> And for the record, this isn't to setup a similar system, it is to
> better understand the issue for a regulatory challenge.

Re: NetFlow - path from Routers to Collector

[Mark Tinka]

On 1/Sep/15 17:33, Serge Vautour wrote:

> Hello,
>
> For those than run Internet connected routers, how do you get your NetFlow 
> data from the routers to your collectors? Do you let the flow export traffic 
> use the same links as your customer traffic to route back to central 
> collectors? Or do you send this traffic over private network management type 
> path? If you send this traffic over the "Internet" (within your AS), are you 
> worried about security?

We forward it in-band.

Have been doing so for years, no major drama.

Mark.

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 1 Sep 2015, at 22:39, Mark Tinka wrote:

> Have been doing so for years, no major drama.

Until there is.

;>

---
Roland Dobbins 

Re: NetFlow - path from Routers to Collector

[Shane Ronan]

It's usually not laziness, it's most often related to cost.
On Sep 1, 2015 12:00 PM, "Rod Beck"  wrote:

> Roland is correct. With the caveat that your Internet customer traffic may
> flow over the fibers as your separate management circuits. You should aim
> for end to end physical diversity. This is all common sense, but laziness
> some times takes precedence.
>
>
>
> Roderick Beck
> Sales - Europe and the Americas
> Hibernia Networks
> http://www.hibernianetworks.com
> Budapest and New York
> 36-30-859-5144
> rod.b...@hibernianetworks.com
>
> 
> From: NANOG  on behalf of Roland Dobbins <
> rdobb...@arbor.net>
> Sent: Tuesday, September 1, 2015 5:43 PM
> To: nanog@nanog.org
> Subject: Re: NetFlow - path from Routers to Collector
>
> On 1 Sep 2015, at 22:39, Mark Tinka wrote:
>
> > Have been doing so for years, no major drama.
>
> Until there is.
>
> ;>
>
> ---
> Roland Dobbins 
> This e-mail and any attachments thereto is intended only for use by the
> addressee(s) named herein and may be proprietary and/or legally privileged.
> If you are not the intended recipient of this e-mail, you are hereby
> notified that any dissemination, distribution or copying of this email, and
> any attachments thereto, without the prior written permission of the sender
> is strictly prohibited. If you receive this e-mail in error, please
> immediately telephone or e-mail the sender and permanently delete the
> original copy and any copy of this e-mail, and any printout thereof. All
> documents, contracts or agreements referred or attached to this e-mail are
> SUBJECT TO CONTRACT. The contents of an attachment to this e-mail may
> contain software viruses that could damage your own computer system. While
> Hibernia Networks has taken every reasonable precaution to minimize this
> risk, we cannot accept liability for any damage that you sustain as a
> result of software viruses. You should carry out your own virus checks
> before opening any attachment.
>

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 1 Sep 2015, at 23:10, Job Snijders wrote:

> To answer your first question: i see no issue in transporting flow
> export traffic over the same backbone used to serve customer traffic.

This is not good advice, for the reasons I stated previously in this thread.

---
Roland Dobbins 

NetFlow - path from Routers to Collector

[Serge Vautour]

Hello,

For those than run Internet connected routers, how do you get your NetFlow data 
from the routers to your collectors? Do you let the flow export traffic use the 
same links as your customer traffic to route back to central collectors? Or do 
you send this traffic over private network management type path? If you send 
this traffic over the "Internet" (within your AS), are you worried about 
security?

Thanks,
Serge

Re: NetFlow - path from Routers to Collector

[Rod Beck]

Roland is correct. With the caveat that your Internet customer traffic may flow 
over the fibers as your separate management circuits. You should aim for end to 
end physical diversity. This is all common sense, but laziness some times takes 
precedence.



Roderick Beck
Sales - Europe and the Americas
Hibernia Networks
http://www.hibernianetworks.com
Budapest and New York
36-30-859-5144
rod.b...@hibernianetworks.com


From: NANOG  on behalf of Roland Dobbins 

Sent: Tuesday, September 1, 2015 5:43 PM
To: nanog@nanog.org
Subject: Re: NetFlow - path from Routers to Collector

On 1 Sep 2015, at 22:39, Mark Tinka wrote:

> Have been doing so for years, no major drama.

Until there is.

;>

---
Roland Dobbins 
This e-mail and any attachments thereto is intended only for use by the 
addressee(s) named herein and may be proprietary and/or legally privileged. If 
you are not the intended recipient of this e-mail, you are hereby notified that 
any dissemination, distribution or copying of this email, and any attachments 
thereto, without the prior written permission of the sender is strictly 
prohibited. If you receive this e-mail in error, please immediately telephone 
or e-mail the sender and permanently delete the original copy and any copy of 
this e-mail, and any printout thereof. All documents, contracts or agreements 
referred or attached to this e-mail are SUBJECT TO CONTRACT. The contents of an 
attachment to this e-mail may contain software viruses that could damage your 
own computer system. While Hibernia Networks has taken every reasonable 
precaution to minimize this risk, we cannot accept liability for any damage 
that you sustain as a result of software viruses. You should carry out your own 
virus checks before opening any attachment.

Re: NetFlow - path from Routers to Collector

[Job Snijders]

On Tue, Sep 01, 2015 at 08:33:42AM -0700, Serge Vautour wrote:
> For those than run Internet connected routers, how do you get your
> NetFlow data from the routers to your collectors? Do you let the flow
> export traffic use the same links as your customer traffic to route
> back to central collectors? Or do you send this traffic over private
> network management type path? If you send this traffic over the
> "Internet" (within your AS), are you worried about security?

To answer your first question: i see no issue in transporting flow
export traffic over the same backbone used to serve customer traffic.

Not entirely security related, but a neat trick is to use a tool like
'samplicator' to distribute the UDP packets to all applications of
interest. You'll find that on many router platforms you can only
configure a limited amount of netflow/sflow collectors, often less then
the amount of applications that need the data for dissemination.

Especially if you have multiple independent instances of the application
for redundancy purposes!

And, keep in mind, you can anycast the instances of 'samplicator' in
your network :-)

https://github.com/sleinen/samplicator

Kind regards,

Job

Re: NetFlow - path from Routers to Collector

[Roland Dobbins]

On 1 Sep 2015, at 22:33, Serge Vautour wrote:


Or do you send this traffic over private network management type path?


This is how to do it.

So in case of a network partition event, you don't end up losing 
visibility into your network traffic when you need it the most.


You should already have an OOB/DCN network for managing your 
routers/switches/hosts, no?   Use that, after ensuring its scaled 
adequately.


---
Roland Dobbins