subject:"DDOS, IDS, RTBH, and Rate limiting"


On 2014-11-21 03:12, Roland Dobbins wrote:

On 21 Nov 2014, at 6:22, Denys Fedoryshchenko wrote:


Netflow is stateful stuff,


This is factually incorrect; NetFlow flows are unidirectional in
nature, and in any event have no effect on processing of data-plane
traffic.
Word stateful has nothing common with stateful firewall.Stateful 
protocol. a protocol which requires keeping of the internal state on 
the server is known as a stateful protocol. And sure 
unidirectional/bidirectional is totally unrelated.




and just to run it on wirespeed, on hardware, you need to utilise 
significant part of TCAM,


Again, this is factually incorrect.

http://en.wikipedia.org/wiki/NetFlow#NetFlow_support
Proof, that majority of solutions runs *flow not in software.

Cisco 65xx (yes, they are obsolete, but they run stuff wirespeed)
Aug 24 12:30:53: %EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM threshold 
exceeded, TCAM Utilization [97%]
This is best example. Also on many Cisco's if you use UBRL, then you 
cannot use NetFlow, just because they use same part of TCAM resources. 
Others, for example Juniper, are using sampling (read - missing data), 
just to not overflow resources, and has various limitations, such as 
RE-DPC communication pps limit, licensing limit.
For example MS-DPC is pretty good one, few million flows in hardware, 
7-8Gbps of traffic, and... cost $12.


i am not talking that on some hardware it is just impossible to run 
it.


This is also factually incorrect.  Some platforms/linecards do not in
fact support NetFlow (or other varieties of flow telemetry) due to
hardware limitations.

But still they can run fine mirroring, and fastnetmon will do it's job.



And last thing, from one of public papers, netflow delaying factors:
1. Flow record expiration


This is tunable.
In certain limits. You can't set flow-active-timeout less than 60 
seconds in Junos 14 for example.
On some platforms even if you can, you just run in the limits of 
platforms again (forwarding - management communications).



• Typical delay: 15-60 sec.


This is an entirely subjective assessment, and does not reflect
operational realities.  These are typically *maximum values* - and
they are well within operationally-useful timeframes.  Also, the
effect of NetFlow cache size and resultant FIFOing of flow records is
not taken into account, nor is the effect on flow termination and
flow-record export of TCP FIN or RST flags denoting TCP traffic taken
into account.

So for a small hosting(up to 10G), i believe, FastNetMon is best 
solution.


This is a gross over-generalization unsupported by facts.  Many years
of operational experience with NetFlow and other forms of flow
telemetry by large numbers of network operators of all sizes and
varieties contract this over-generalization.

Fastnetmon and similar tools popularity says for itself.


It is generally unwise to make sweeping statements regarding
operational impact which are not borne out by significant operational
experience in production networks.
What can be asserted without evidence can be dismissed without 
evidence.





Faster, and no significant investments to equipment.


This statement indicates a lack of understanding of opex costs,
irrespective of capex costs.
Sweet marketing buzzwords, that is used together with some unclear 
calculations,
to sell suffering hosting providers various expensive tools, that is not 
necessary for them.
OPEX of fastnetmon is a small fee for qualified sysadmin, and often not 
required,

because already hosting operator should have him.



Bigger hosting providers might reuse their existing servers, segment 
the network, and implement inexpensive monitoring on aggregation 
switches without any additional cost again.


This statement indicates a lack of operational experience in networks
of even minimal scale.

Ah, and there is one more huge problem with netflow vs FastNetMon - 
netflow just by design cannot be adapted to run pattern matching, 
while it is trivial to patch FastNetMon for that, turning it to 
mini-IDS for free.


This statement betrays a lack of understanding of NetFlow-based (and
other flow telemetry-based) detection and classification, as well as
the undesirability and negative operational impact of stateful
IDS/'IPS' deployments in production networks.

You should also note that FastNetMon is far from unique; there are
multiple other open-source tools which provide the same type of
functionality, and none of them have replaced flow telemetry, either.
Thats a power of opensource. Since FastNetMon is not only tool, worth to 
mention others,
people here will benefit from using it, for free. And i'm sure, author 
of FastNetMon will

not feel offended at all.



Tools such as FastNetMon supplement flow telemetry, in situations in
which such tools can be deployed.  They do not begin to replace flow
telemetry, and they are not inherently superior to flow telemetry.

Again, I'm sure FastNetMon is a useful tool in many circumstances.

Re: DDOS, IDS, RTBH, and Rate limiting


On 2014-11-21 06:45, freed...@freedman.net wrote:
Netflow is stateful stuff, and just to run it on wirespeed, on 
hardware,

you need to utilise significant part of TCAM,


Cisco ASRs and MXs with inline jflow can do hundreds of K flows/second
without affecting packet forwarding.
Yes, i agree,those are good for netflow, but when they already exist in 
network.
Does it worth to buy ASR, if L3 switch already doing the job 
(BGP/ACL/rate-limit/routing)?


i am not talking that on some hardware it is just impossible to run 
it.
So everything about netflow are built on assumption that hosting or 
ISP

can run it. And based on some observations, majority of small/middle
hosting providers are using minimal,just BGP capable L3 switch as 
core,
and cheapest but reliable L2/L3 on aggregation, and both are capable 
in

best case to run sampled sFlow.


Actually, sFlow from many vendors is pretty good (per your points about 
flow

burstiness and delays), and is good enough for dDoS detection.  Not for
security forensics, or billing at 99.99% accuracy, but good enough for
traffic visibility, peering analytics, and (d)DoS detection.
Well, if it is available, except hardware limitations, there is second 
obstacle,
software licensing cost. On latest JunOS, for example on EX2200, you 
need
to purchase license (EFL), and if am not wrong it is $3000 for 48port 
units.
So if only sFlow feature is on stake, it worth to think, to purchase 
license,
or to purchase server. Prices for JFlow license on MX, just for 5/10G is 
way above cost

of very decent server.



snip


So for a small hosting(up to 10G), i believe, FastNetMon is best
solution. Faster, and no significant investments to equipment. Bigger
hosting providers might reuse their existing servers, segment the
network, and implement inexpensive monitoring on aggregation switches
without any additional cost again.


It can be useful to have a 10G network monitoring box of course...

And with the right setup you can run FastNetMon or other tools in
addition to generating flow that can be of use for other purposes
as well...
Technically there is ipt_NETFLOW, that can generate netflow on same box, 
for
statistical/telemetry purposes. But i am not sure it is possible to run 
them

together.




Ah, and there is one more huge problem with netflow vs FastNetMon -
netflow just by design cannot be adapted to run pattern matching, 
while

it is trivial to patch FastNetMon for that, turning it to mini-IDS for
free.


It's true, having a network tap can be useful for doing PCAP-y stuff.

But taps can be difficult or at least time consuming for people to
put in at scale.  Even, we've seen, for folks with 10G networks.
Often because they can get 90% of what they need for 4 different
business purposes from just flow :)
About scaling, i guess it depends on proper deployment strategy and 
sysadmins/developers
capabilities. For example to deploy new ruleset for my pcap-based 
homemade

analyser to 150 probes across the country - is just one click.


---
Best regards,
Denys

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-21 Thread Roland Dobbins



On 21 Nov 2014, at 15:17, Denys Fedoryshchenko wrote:

Word stateful has nothing common with stateful firewall.Stateful 
protocol. a protocol which requires keeping of the internal state on 
the server is known as a stateful protocol.


Correct - and NetFlow is not stateful, by this definition.


And sure unidirectional/bidirectional is totally unrelated.


On the contrary, it is quite relevant.


Cisco 65xx (yes, they are obsolete, but they run stuff wirespeed)


They are not obsolete - they perform very well with Sup2T and 
EARL8-based linecards.


Aug 24 12:30:53: %EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM threshold 
exceeded, TCAM Utilization [97%]


This is from a 6500 with either an EARL6 or EARL7 ASIC, which had many 
caveats with regards to NetFlow, including a lack of packet-sampled 
control of flow creation - i.e., sampled NetFlow.  As part of the 
extended team which defined requirements for the EARL8 ASIC, which is 
utilized in the Sup2T and DFC-4 enabled linecards, I can assure you that 
this is no longer an issue with 6500s running EARL8-based Sups and 
linecards.


Also on many Cisco's if you use UBRL, then you cannot use NetFlow, 
just because they use same part of TCAM resources.


This is where TCAM carving comes into play.  Also, it is not so much an 
issue with newer hardware, per the above.  Also, URBL is not commonly 
used in ISP networks.



Others, for example Juniper, are using sampling (read - missing data),


The largest networks in the world use sampled NetFlow every hour of 
every day for many purposes, including DDoS 
detection/classification/traceback.  It works quite well for all those 
purposes.


just to not overflow resources, and has various limitations, such as 
RE-DPC communication pps limit, licensing limit.
For example MS-DPC is pretty good one, few million flows in hardware, 
7-8Gbps of traffic, and... cost $12.


You get what you pay for.

But still they can run fine mirroring, and fastnetmon will do it's 
job.


On the contrary - SPAN nee port mirroring cuts into the 
frames-per-second budget of linecards, as the traffic is in essence 
being duplicated.  It is not 'free', and it has a profound impact on the 
the switch's data-plane traffic forwarding capacity.


Unlike NetFlow.

In certain limits. You can't set flow-active-timeout less than 60 
seconds in Junos 14 for example.


Platforms vary, this is true.  However, I have never run into an issue 
with an active flow timer of 60s, nor have I ever run into anyone who 
has done so.


On some platforms even if you can, you just run in the limits of 
platforms again (forwarding - management communications).


This is incorrect.


Fastnetmon and similar tools popularity says for itself.


Yes, it does - they are far less popular that NetFlow, because they do 
not scale on networks of any size, nor do they provide traceback (given 
your lack of comments on traceback elsewhere in this thread, it appears 
that you aren't familiar with this concept).



What can be asserted without evidence can be dismissed without 
evidence.


You make my point very well, thank you.  There is overwhelming evidence 
that NetFlow and similar forms of flow telemetry scale well and provide 
real, measurable, actionable operational value on networks of all types 
and sizes.  The reason for the popularity of flow telemetry is that it 
is low-opex (no probes to deply); low-capex (no probes to deploy); 
scales to tb/sec speeds; is practicable for large networks (no probes to 
deploy); provides instantaneous traceback (probes can't do this); and 
provides statistics on dropped traffic (probes can't do this, either).



Sweet marketing buzzwords,


It's pretty obvious which half of this 'conversation' is focused on 
marketing; and it isn't mine.



that is used together with some unclear calculations,


No calculations have been discussed during the course of this 
'conversation'.



to sell suffering hosting providers various expensive tools,


I'm uninterested in selling anyone anything.  What I'm interested in 
doing is correcting the misinformation you are promulgating regarding 
the utility of flow telemetry coupled with open-source flow analysis 
systems.  There has been no mention of any commercial systems or 
products in my half of this 'conversation'.



that is not necessary for them.


Again, the benefits of flow telemetry are quite clear for networks of 
any size.


OPEX of fastnetmon is a small fee for qualified sysadmin, and often 
not required, because already hosting operator should have him.


You obviously do not know what the term opex actually means, nor what it 
encompasses.



I can agree only that arguing about this subject is waste of time.


Yes, it isn't a profitable use of time to argue with someone who does 
not have the degree of operational expertise nor experience to back his 
demonstrably incorrect assertions.



where netflow just by design cannot outperform it


Again, this is a completely unsupported

Re: DDOS, IDS, RTBH, and Rate limiting


On 2014-11-21 14:50, Roland Dobbins wrote:

On 21 Nov 2014, at 15:17, Denys Fedoryshchenko wrote:

Word stateful has nothing common with stateful firewall.Stateful 
protocol. a protocol which requires keeping of the internal state on 
the server is known as a stateful protocol.


Correct - and NetFlow is not stateful, by this definition.

Not stateful, if you pick on server word.
To be able to make bytes/packets accounting for a flow, you need to keep 
this specific flow previous state. To be able to differentiate between 
flows with same src/dst ip+ports (if one is ended, next is started with 
same data) you need to track it's state, again. And just to keep track 
of _flows_ in packet switched network you need states. Surprising lack 
of knowledge.





And sure unidirectional/bidirectional is totally unrelated.


On the contrary, it is quite relevant.


Cisco 65xx (yes, they are obsolete, but they run stuff wirespeed)


They are not obsolete - they perform very well with Sup2T and
EARL8-based linecards.
Seems yes, i'm wrong on that point, i was not successful to run netflow 
reliable way , but it was before CSCul90377 and CSCui17732 fixed.





Others, for example Juniper, are using sampling (read - missing data),


The largest networks in the world use sampled NetFlow every hour of
every day for many purposes, including DDoS
detection/classification/traceback.  It works quite well for all those
purposes.
Use case of fastnetmon is not largest networks. Sampled netflow is 
useless for per-traffic billing purpose for example.




just to not overflow resources, and has various limitations, such as 
RE-DPC communication pps limit, licensing limit.
For example MS-DPC is pretty good one, few million flows in hardware, 
7-8Gbps of traffic, and... cost $12.


You get what you pay for.
While i can pay $1500 for a server, and get netflow and ~3second BGP 
blackholing with fastnetmon.




But still they can run fine mirroring, and fastnetmon will do it's 
job.


On the contrary - SPAN nee port mirroring cuts into the
frames-per-second budget of linecards, as the traffic is in essence
being duplicated.  It is not 'free', and it has a profound impact on
the the switch's data-plane traffic forwarding capacity.

Unlike NetFlow.
In hosting case mirroring usually done for uplink port, but i have to 
agree, it might be a problem.



Yes, it does - they are far less popular that NetFlow, because they do
not scale on networks of any size, nor do they provide traceback
(given your lack of comments on traceback elsewhere in this thread, it
appears that you aren't familiar with this concept).
You make my point very well, thank you.  There is overwhelming
evidence that NetFlow and similar forms of flow telemetry scale well
and provide real, measurable, actionable operational value on networks
of all types and sizes.  The reason for the popularity of flow
telemetry is that it is low-opex (no probes to deply); low-capex (no
probes to deploy); scales to tb/sec speeds; is practicable for large
networks (no probes to deploy); provides instantaneous traceback
(probes can't do this); and provides statistics on dropped traffic
(probes can't do this, either).
And again and again we are going to tb/s. I don't need TB/s, i dont need 
traceback,nor on relatively small ISP nor on VDS provider i dont need 
all that above. I just need inexpensive way to block attacked ip and/or 
announce it from different location within minimal timeframe, to 
minimize impact on other customers.
You might be highly professional with large scale operators, but small 
guys needs and capabilities are very different.
I had developed tool similar to fastnetmon for almost same purpose, 
detecting attacks and switching affected network by BGP to protected 
backbone. After calculating OPEX/CAPEX, capable server turned to be 
much cheaper alternative in short and long term than buying netflow 
capable hardware (and support for it) just for netflow purposes, and 
buying hardware for netflow collector.

Let's talk numbers.
My case is small hosting, 4G, C4948-10G, one 10G uplink, one 10G port is 
free. Switch is not capable to run sFlow or Netflow.
Decent server is available already, since it is hosting company, so the 
only expenses are 10G 82599 card, which is around $500. Even in case 
server is not available, based on data from fastnetmon author still 
total cost is within $1500. Deployment time - hours from installing 
hardware, without distrupting existing traffic.
Major expenses - tuning server according author recommendations, and 
writing shell script that will send to 4948 command to blackhope IP. For 
qualified sysadmin it is 2 hours of work, and $500 max as a labor 
cost. Thats it. What can be cheaper than $2000 in this case? I guess i 
wont get answer.



I'm uninterested in selling anyone anything.  What I'm interested in
doing is correcting the misinformation you are promulgating regarding
the utility of flow telemetry coupled with open-source flow

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-21 Thread Peter Phaal

Actually, sFlow from many vendors is pretty good (per your points about
flow
burstiness and delays), and is good enough for dDoS detection. Not for
security forensics, or billing at 99.99% accuracy, but good enough for
traffic visibility, peering analytics, and (d)DoS detection.

Well, if it is available, except hardware limitations, there is second
obstacle,
software licensing cost. On latest JunOS, for example on EX2200, you need
to purchase license (EFL), and if am not wrong it is $3000 for 48port units.
So if only sFlow feature is on stake, it worth to think, to purchase
license,
or to purchase server.

Juniper no longer charges for sFlow on the EX2200 (as of Junos 11.2):

http://www.juniper.net/techpubs/en_US/junos11.2/information-products/topic-collections/release-notes/11.2/junos-release-notes-11.2.pdf

I am not aware of any vendor requiring an additional license to enable sFlow.

sFlow (packet sampling) works extremely well for the DDoS flood
detection / mitigation use case. The measurements are build into low
cost commodity switch hardware and can be enabled operationally
without adversely impacting switch performance. A flood attack
generates high packet rates and sampling a 10G port at 1-in-10,000
will reliably detect flood attacks within seconds.

For most use cases, it is much less expensive to use switches to
perform measurement than to attach taps / mirror port probes. If your
switches don't already support sFlow, you can buy a 10G capable white
box switch for a few thousand dollars that will let you monitor 1.2
Terabits/sec. If you go with an open platform such as Cumulus Linux,
you could even run your DDoS mitigation software on the switch and
dispense with the external server. Embedded instrumentation is simple
to deploy and reduces operational complexity and cost when compared to
add on probe solutions.

Peter Phaal
InMon Corp.

Re: DDOS, IDS, RTBH, and Rate limiting

On 2014-11-21 18:41, Peter Phaal wrote:
Actually, sFlow from many vendors is pretty good (per your points
about

flow
burstiness and delays), and is good enough for dDoS detection. Not
for
security forensics, or billing at 99.99% accuracy, but good enough
for

traffic visibility, peering analytics, and (d)DoS detection.

So if only sFlow feature is on stake, it worth to think, to purchase
license,
or to purchase server.

Juniper no longer charges for sFlow on the EX2200 (as of Junos 11.2):

http://www.juniper.net/techpubs/en_US/junos11.2/information-products/topic-collections/release-notes/11.2/junos-release-notes-11.2.pdf

I am not aware of any vendor requiring an additional license to enable
sFlow.

Peter Phaal
InMon Corp.
Wow, that's great news then, i'm using mostly Cisco gear now, but seems
will have to take a look to Juniper, thanks for information.
If it is free, then if EX2200 available, it is much easier to run sFlow
and write custom collector for it, than installing custom probe(in most
common cases).

---
Best regards,
Denys

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-21 Thread Tim Jackson

pmacct includes sfacctd which is an sflow collector.. Accessible via
the same methods as it's nfacctd collector or pcap based collector..

--
Tim

On Fri, Nov 21, 2014 at 9:06 AM, Denys Fedoryshchenko de...@visp.net.lb wrote:
On 2014-11-21 18:41, Peter Phaal wrote:

Well, if it is available, except hardware limitations, there is second
obstacle,
software licensing cost. On latest JunOS, for example on EX2200, you need
to purchase license (EFL), and if am not wrong it is $3000 for 48port
units.
So if only sFlow feature is on stake, it worth to think, to purchase
license,
or to purchase server.

Juniper no longer charges for sFlow on the EX2200 (as of Junos 11.2):

http://www.juniper.net/techpubs/en_US/junos11.2/information-products/topic-collections/release-notes/11.2/junos-release-notes-11.2.pdf

I am not aware of any vendor requiring an additional license to enable
sFlow.

Peter Phaal
InMon Corp.

Wow, that's great news then, i'm using mostly Cisco gear now, but seems will
have to take a look to Juniper, thanks for information.
If it is free, then if EX2200 available, it is much easier to run sFlow and
write custom collector for it, than installing custom probe(in most common
cases).

---
Best regards,
Denys

Re: DDOS, IDS, RTBH, and Rate limiting

Thanks! Most important there is plugin API,so it is easy to write custom
code to do some analysis and on events - actions.

On 2014-11-21 20:32, Tim Jackson wrote:

pmacct includes sfacctd which is an sflow collector.. Accessible via
the same methods as it's nfacctd collector or pcap based collector..

--
Tim

On Fri, Nov 21, 2014 at 9:06 AM, Denys Fedoryshchenko
de...@visp.net.lb wrote:

On 2014-11-21 18:41, Peter Phaal wrote:

Actually, sFlow from many vendors is pretty good (per your points
about

flow
burstiness and delays), and is good enough for dDoS detection. Not
for
security forensics, or billing at 99.99% accuracy, but good enough
for

traffic visibility, peering analytics, and (d)DoS detection.

Well, if it is available, except hardware limitations, there is
second

obstacle,
software licensing cost. On latest JunOS, for example on EX2200, you
need
to purchase license (EFL), and if am not wrong it is $3000 for
48port

units.
So if only sFlow feature is on stake, it worth to think, to purchase
license,
or to purchase server.

Juniper no longer charges for sFlow on the EX2200 (as of Junos 11.2):

http://www.juniper.net/techpubs/en_US/junos11.2/information-products/topic-collections/release-notes/11.2/junos-release-notes-11.2.pdf

I am not aware of any vendor requiring an additional license to
enable

sFlow.

add on probe solutions.

Peter Phaal
InMon Corp.

Wow, that's great news then, i'm using mostly Cisco gear now, but
seems will

have to take a look to Juniper, thanks for information.
If it is free, then if EX2200 available, it is much easier to run
sFlow and
write custom collector for it, than installing custom probe(in most
common

cases).

---
Best regards,
Denys

DDOS, IDS, RTBH, and Rate limiting

2014-11-20 Thread Pavel Odintsov

Hello, folks!

I'm author of fastnetmon, thank you for some PR for my toolkit :)

I use this tool for similar type of attacks and we do analyze all
traffic from uplinks ports using port mirroring. You can look at this
network diagram:
https://raw.githubusercontent.com/FastVPSEestiOu/fastnetmon/master/network_map.png

I tried to use netflow many years ago but it's not accurate enough and
not so fast enough and produce big overhead on middle class network
routers. It's because I wrote this tool and do every packet analyze.
It can detect attack in 2 seconds max and call BGP blackhole as quick
as thought.

It can detect three types of attacks:
1) Speed attack for certain IP (we ban every IP which exceed 1 Gbps)
2) Packet per second attack for certain IP (we ban every IP which
exceed 100 000 ppps)
3) And flow flood (very useful mode in networks with big bandwidth/pps
per client)

FastNetMon can handle 2-3 million of packets per second and ~20Gbps on
standard i7 2600 Linux box with Intel 82599 NIC.

If you need any help or suggestions you can email me directly or ask via GitHub.

Thank you!

-- 
Sincerely yours, Pavel Odintsov

Re: DDOS, IDS, RTBH, and Rate limiting



On 21 Nov 2014, at 4:36, Pavel Odintsov wrote:


I tried to use netflow many years ago but it's not accurate enough and
not so fast enough and produce big overhead on middle class network
routers.


These statements are not supported by the facts.  NetFlow (and other 
varieties of flow telemetry) has been used for many years for traffic 
engineering-related analysis, capacity planning, and security purposes.  
I've never seen the CPU utilization on even a modest mid-range router 
rise above single-digits, except once due to a bug (which was fixed 
quickly).


Flow telemetry scales and provides invaluable edge-to-edge traceback 
information.  NetFlow telemetry is accurate enough to be used for all 
the purposes noted above by network operators across the world, from the 
smallest to the largest networks in the world.


There are several excellent open-source NetFlow analysis tools which 
allow folks to benefit from NetFlow analysis without spending a lot of 
money. Some of these projects have been maintained and enhanced for many 
years; their authors would not do that if NetFlow analytics weren't 
sufficient to needs.


Packet-based analysis is certainly useful, but does not scale and does 
not provide traceback information.


FastNetMon can handle 2-3 million of packets per second and ~20Gbps on 
standard i7 2600 Linux box with Intel 82599 NIC.


See the comments above with regards to scale.  This is inadequate for a 
network of any size, it does not provide traceback information, and it 
does not lend itself to broad deployment across a network of any size.


I'm sure FastNetMon is a fine tool, and it's very good of you to spend 
the time and effort to develop it and to make it available.  However, 
making demonstrably-inaccurate statements about other technologies which 
are in wide use by network operators and which have a proven track 
record in the field is probably not the best way to encourage folks to 
try FastNetMon.


---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-20 Thread Denys Fedoryshchenko


On 2014-11-20 23:59, Roland Dobbins wrote:

On 21 Nov 2014, at 4:36, Pavel Odintsov wrote:


I tried to use netflow many years ago but it's not accurate enough and
not so fast enough and produce big overhead on middle class network
routers.


These statements are not supported by the facts.  NetFlow (and other
varieties of flow telemetry) has been used for many years for traffic
engineering-related analysis, capacity planning, and security
purposes.  I've never seen the CPU utilization on even a modest
mid-range router rise above single-digits, except once due to a bug
(which was fixed quickly).

Flow telemetry scales and provides invaluable edge-to-edge traceback
information.  NetFlow telemetry is accurate enough to be used for all
the purposes noted above by network operators across the world, from
the smallest to the largest networks in the world.

There are several excellent open-source NetFlow analysis tools which
allow folks to benefit from NetFlow analysis without spending a lot of
money. Some of these projects have been maintained and enhanced for
many years; their authors would not do that if NetFlow analytics
weren't sufficient to needs.

Packet-based analysis is certainly useful, but does not scale and does
not provide traceback information.

FastNetMon can handle 2-3 million of packets per second and ~20Gbps on 
standard i7 2600 Linux box with Intel 82599 NIC.


See the comments above with regards to scale.  This is inadequate for
a network of any size, it does not provide traceback information, and
it does not lend itself to broad deployment across a network of any
size.

I'm sure FastNetMon is a fine tool, and it's very good of you to spend
the time and effort to develop it and to make it available.  However,
making demonstrably-inaccurate statements about other technologies
which are in wide use by network operators and which have a proven
track record in the field is probably not the best way to encourage
folks to try FastNetMon.


Netflow is stateful stuff, and just to run it on wirespeed, on hardware, 
you need to utilise significant part of TCAM,

i am not talking that on some hardware it is just impossible to run it.
So everything about netflow are built on assumption that hosting or ISP 
can run it. And based on some observations, majority of small/middle 
hosting providers are using minimal,just BGP capable L3 switch as core, 
and cheapest but reliable L2/L3 on aggregation, and both are capable in 
best case to run sampled sFlow.

And last thing, from one of public papers, netflow delaying factors:
1. Flow record expiration
2. Exporting process
• Typical delay: 15-60 sec.
So for a small hosting(up to 10G), i believe, FastNetMon is best 
solution. Faster, and no significant investments to equipment. Bigger 
hosting providers might reuse their existing servers, segment the 
network, and implement inexpensive monitoring on aggregation switches 
without any additional cost again.
Ah, and there is one more huge problem with netflow vs FastNetMon - 
netflow just by design cannot be adapted to run pattern matching, while 
it is trivial to patch FastNetMon for that, turning it to mini-IDS for 
free.


---
Best regards,
Denys

Re: DDOS, IDS, RTBH, and Rate limiting



On 21 Nov 2014, at 6:22, Denys Fedoryshchenko wrote:


Netflow is stateful stuff,


This is factually incorrect; NetFlow flows are unidirectional in nature, 
and in any event have no effect on processing of data-plane traffic.


and just to run it on wirespeed, on hardware, you need to utilise 
significant part of TCAM,


Again, this is factually incorrect.

i am not talking that on some hardware it is just impossible to run 
it.


This is also factually incorrect.  Some platforms/linecards do not in 
fact support NetFlow (or other varieties of flow telemetry) due to 
hardware limitations.



And last thing, from one of public papers, netflow delaying factors:
1. Flow record expiration


This is tunable.


• Typical delay: 15-60 sec.


This is an entirely subjective assessment, and does not reflect 
operational realities.  These are typically *maximum values* - and they 
are well within operationally-useful timeframes.  Also, the effect of 
NetFlow cache size and resultant FIFOing of flow records is not taken 
into account, nor is the effect on flow termination and flow-record 
export of TCP FIN or RST flags denoting TCP traffic taken into account.


So for a small hosting(up to 10G), i believe, FastNetMon is best 
solution.


This is a gross over-generalization unsupported by facts.  Many years of 
operational experience with NetFlow and other forms of flow telemetry by 
large numbers of network operators of all sizes and varieties contract 
this over-generalization.


It is generally unwise to make sweeping statements regarding operational 
impact which are not borne out by significant operational experience in 
production networks.



Faster, and no significant investments to equipment.


This statement indicates a lack of understanding of opex costs, 
irrespective of capex costs.


Bigger hosting providers might reuse their existing servers, segment 
the network, and implement inexpensive monitoring on aggregation 
switches without any additional cost again.


This statement indicates a lack of operational experience in networks of 
even minimal scale.


Ah, and there is one more huge problem with netflow vs FastNetMon - 
netflow just by design cannot be adapted to run pattern matching, 
while it is trivial to patch FastNetMon for that, turning it to 
mini-IDS for free.


This statement betrays a lack of understanding of NetFlow-based (and 
other flow telemetry-based) detection and classification, as well as the 
undesirability and negative operational impact of stateful IDS/'IPS' 
deployments in production networks.


You should also note that FastNetMon is far from unique; there are 
multiple other open-source tools which provide the same type of 
functionality, and none of them have replaced flow telemetry, either.


Tools such as FastNetMon supplement flow telemetry, in situations in 
which such tools can be deployed.  They do not begin to replace flow 
telemetry, and they are not inherently superior to flow telemetry.


Again, I'm sure FastNetMon is a useful tool in many circumstances.  But 
it would be a much better idea to define FastNetMon positively in terms 
of its own inherent value, rather than attempting to define it based 
upon factually incorrect negative 'comparisons' to other 
well-established, widely-used technologies which have demonstrable track 
records within the global operational community.


---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting



On 21 Nov 2014, at 9:19, Robert Duffy wrote:

What open-source NetFlow analysis tools would you recommend for 
quickly

detecting a DDoS attack?


I generally recommend that folks get started with something like 
nfdump/nfsen or ntop.  There are other, more sophisticated tools out 
there, but these allow one to get up and running quickly, and to gain 
valuable operational experience with which to evaluate more 
sophisticated tools, if they're needed.


---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-20 Thread Tim Jackson

I highly recommend pmacct and it's in-memory tables. Lightweight, easy to
query and super fast.

You can also easily run multiple aggregates of traffic to find what you are
interested in, tag common interface types to easily filter traffic..

Or you can use pmacct to insert this into whatever database you want, AMQP
or MongoDB..

My current favorite is using an IMT table for DoS detection and another for
aggregates for interesting traffic types and querying this every X minutes
and inserting it into ElasticSearch. Kibana makes the most powerful netflow
dashboard ever.

--
Tim
On Nov 20, 2014 6:39 PM, Roland Dobbins rdobb...@arbor.net wrote:


 On 21 Nov 2014, at 9:19, Robert Duffy wrote:

  What open-source NetFlow analysis tools would you recommend for quickly
 detecting a DDoS attack?


 I generally recommend that folks get started with something like
 nfdump/nfsen or ntop.  There are other, more sophisticated tools out there,
 but these allow one to get up and running quickly, and to gain valuable
 operational experience with which to evaluate more sophisticated tools, if
 they're needed.

 ---
 Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-20 Thread Data Zone

What happens when someone spoofs legitimate hosts that your customers use?

On Thu, Nov 20, 2014 at 3:36 PM, Pavel Odintsov pavel.odint...@gmail.com
wrote:

 Hello, folks!

 I'm author of fastnetmon, thank you for some PR for my toolkit :)

 I use this tool for similar type of attacks and we do analyze all
 traffic from uplinks ports using port mirroring. You can look at this
 network diagram:

 https://raw.githubusercontent.com/FastVPSEestiOu/fastnetmon/master/network_map.png

 I tried to use netflow many years ago but it's not accurate enough and
 not so fast enough and produce big overhead on middle class network
 routers. It's because I wrote this tool and do every packet analyze.
 It can detect attack in 2 seconds max and call BGP blackhole as quick
 as thought.

 It can detect three types of attacks:
 1) Speed attack for certain IP (we ban every IP which exceed 1 Gbps)
 2) Packet per second attack for certain IP (we ban every IP which
 exceed 100 000 ppps)
 3) And flow flood (very useful mode in networks with big bandwidth/pps
 per client)

 FastNetMon can handle 2-3 million of packets per second and ~20Gbps on
 standard i7 2600 Linux box with Intel 82599 NIC.

 If you need any help or suggestions you can email me directly or ask via
 GitHub.

 Thank you!

 --
 Sincerely yours, Pavel Odintsov

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-20 Thread Robert Duffy

Roland, you seem to have a lot of experience with these kinds of tools.
What open-source NetFlow analysis tools would you recommend for quickly
detecting a DDoS attack?

On Thu, Nov 20, 2014 at 5:12 PM, Roland Dobbins rdobb...@arbor.net wrote:


 On 21 Nov 2014, at 6:22, Denys Fedoryshchenko wrote:

  Netflow is stateful stuff,


 This is factually incorrect; NetFlow flows are unidirectional in nature,
 and in any event have no effect on processing of data-plane traffic.

  and just to run it on wirespeed, on hardware, you need to utilise
 significant part of TCAM,


 Again, this is factually incorrect.

  i am not talking that on some hardware it is just impossible to run it.


 This is also factually incorrect.  Some platforms/linecards do not in fact
 support NetFlow (or other varieties of flow telemetry) due to hardware
 limitations.

  And last thing, from one of public papers, netflow delaying factors:
 1. Flow record expiration


 This is tunable.

  • Typical delay: 15-60 sec.


 This is an entirely subjective assessment, and does not reflect
 operational realities.  These are typically *maximum values* - and they are
 well within operationally-useful timeframes.  Also, the effect of NetFlow
 cache size and resultant FIFOing of flow records is not taken into account,
 nor is the effect on flow termination and flow-record export of TCP FIN or
 RST flags denoting TCP traffic taken into account.

  So for a small hosting(up to 10G), i believe, FastNetMon is best solution.


 This is a gross over-generalization unsupported by facts.  Many years of
 operational experience with NetFlow and other forms of flow telemetry by
 large numbers of network operators of all sizes and varieties contract this
 over-generalization.

 It is generally unwise to make sweeping statements regarding operational
 impact which are not borne out by significant operational experience in
 production networks.

  Faster, and no significant investments to equipment.


 This statement indicates a lack of understanding of opex costs,
 irrespective of capex costs.

  Bigger hosting providers might reuse their existing servers, segment the
 network, and implement inexpensive monitoring on aggregation switches
 without any additional cost again.


 This statement indicates a lack of operational experience in networks of
 even minimal scale.

  Ah, and there is one more huge problem with netflow vs FastNetMon -
 netflow just by design cannot be adapted to run pattern matching, while it
 is trivial to patch FastNetMon for that, turning it to mini-IDS for free.


 This statement betrays a lack of understanding of NetFlow-based (and other
 flow telemetry-based) detection and classification, as well as the
 undesirability and negative operational impact of stateful IDS/'IPS'
 deployments in production networks.

 You should also note that FastNetMon is far from unique; there are
 multiple other open-source tools which provide the same type of
 functionality, and none of them have replaced flow telemetry, either.

 Tools such as FastNetMon supplement flow telemetry, in situations in which
 such tools can be deployed.  They do not begin to replace flow telemetry,
 and they are not inherently superior to flow telemetry.

 Again, I'm sure FastNetMon is a useful tool in many circumstances.  But it
 would be a much better idea to define FastNetMon positively in terms of its
 own inherent value, rather than attempting to define it based upon
 factually incorrect negative 'comparisons' to other well-established,
 widely-used technologies which have demonstrable track records within the
 global operational community.

 ---
 Roland Dobbins rdobb...@arbor.net




-- 
Regards,
Rob


Robert Duffy
eSecureData.com Inc.
1478 Hartley Ave.
Coquitlam, BC V3K 7A1
T: (800) 620-1985
F: (800) 620-1986

This communication is intended for the use of the recipient to which it is
addressed, and may contain confidential, personal and or privileged
information. Please contact the sender immediately if you are not the
intended recipient of this communication, and do not copy, distribute, or
take action relying on the contents herein. Any communication received in
error, or subsequent reply, should be deleted or destroyed.

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-20 Thread Robert Duffy

I've been using NTOP for couple of years.  I'm mostly looking for something
that can quickly detect DDoS attacks in a datacenter environment.  Thanks
for the suggestions.  Ill check them out.

On Thu, Nov 20, 2014 at 6:50 PM, Tim Jackson jackson@gmail.com wrote:

 I highly recommend pmacct and it's in-memory tables. Lightweight, easy to
 query and super fast.

 You can also easily run multiple aggregates of traffic to find what you are
 interested in, tag common interface types to easily filter traffic..

 Or you can use pmacct to insert this into whatever database you want, AMQP
 or MongoDB..

 My current favorite is using an IMT table for DoS detection and another for
 aggregates for interesting traffic types and querying this every X minutes
 and inserting it into ElasticSearch. Kibana makes the most powerful netflow
 dashboard ever.

 --
 Tim
 On Nov 20, 2014 6:39 PM, Roland Dobbins rdobb...@arbor.net wrote:

 
  On 21 Nov 2014, at 9:19, Robert Duffy wrote:
 
   What open-source NetFlow analysis tools would you recommend for quickly
  detecting a DDoS attack?
 
 
  I generally recommend that folks get started with something like
  nfdump/nfsen or ntop.  There are other, more sophisticated tools out
 there,
  but these allow one to get up and running quickly, and to gain valuable
  operational experience with which to evaluate more sophisticated tools,
 if
  they're needed.
 
  ---
  Roland Dobbins rdobb...@arbor.net
 




-- 
Regards,
Rob


Robert Duffy
eSecureData.com Inc.
1478 Hartley Ave.
Coquitlam, BC V3K 7A1
T: (800) 620-1985
F: (800) 620-1986

This communication is intended for the use of the recipient to which it is
addressed, and may contain confidential, personal and or privileged
information. Please contact the sender immediately if you are not the
intended recipient of this communication, and do not copy, distribute, or
take action relying on the contents herein. Any communication received in
error, or subsequent reply, should be deleted or destroyed.

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-20 Thread Avi Freedman


 Netflow is stateful stuff, and just to run it on wirespeed, on hardware, 
 you need to utilise significant part of TCAM,

Cisco ASRs and MXs with inline jflow can do hundreds of K flows/second
without affecting packet forwarding.

 i am not talking that on some hardware it is just impossible to run it.
 So everything about netflow are built on assumption that hosting or ISP 
 can run it. And based on some observations, majority of small/middle 
 hosting providers are using minimal,just BGP capable L3 switch as core, 
 and cheapest but reliable L2/L3 on aggregation, and both are capable in 
 best case to run sampled sFlow.

Actually, sFlow from many vendors is pretty good (per your points about flow 
burstiness and delays), and is good enough for dDoS detection.  Not for 
security forensics, or billing at 99.99% accuracy, but good enough for
traffic visibility, peering analytics, and (d)DoS detection.

snip

 So for a small hosting(up to 10G), i believe, FastNetMon is best 
 solution. Faster, and no significant investments to equipment. Bigger 
 hosting providers might reuse their existing servers, segment the 
 network, and implement inexpensive monitoring on aggregation switches 
 without any additional cost again.

It can be useful to have a 10G network monitoring box of course...

And with the right setup you can run FastNetMon or other tools in
addition to generating flow that can be of use for other purposes
as well...

 Ah, and there is one more huge problem with netflow vs FastNetMon - 
 netflow just by design cannot be adapted to run pattern matching, while 
 it is trivial to patch FastNetMon for that, turning it to mini-IDS for 
 free.

It's true, having a network tap can be useful for doing PCAP-y stuff.

But taps can be difficult or at least time consuming for people to
put in at scale.  Even, we've seen, for folks with 10G networks.
Often because they can get 90% of what they need for 4 different
business purposes from just flow :)

 Best regards,
 Denys

Avi Freedman| Your flow has something to show you; can you see it?|
CEO, CloudHelix | (avi at cloudhelix dot com) | my name one word on skype |

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-20 Thread Paul S.


WANguard from andrisoft has worked well on this for us.

It supports flow telemetry and mirrored ports both (We use flows 
strictly), and does what it says it does.


No complaints.

On 11/21/2014 午後 12:00, Robert Duffy wrote:

I've been using NTOP for couple of years.  I'm mostly looking for something
that can quickly detect DDoS attacks in a datacenter environment.  Thanks
for the suggestions.  Ill check them out.

On Thu, Nov 20, 2014 at 6:50 PM, Tim Jackson jackson@gmail.com wrote:


I highly recommend pmacct and it's in-memory tables. Lightweight, easy to
query and super fast.

You can also easily run multiple aggregates of traffic to find what you are
interested in, tag common interface types to easily filter traffic..

Or you can use pmacct to insert this into whatever database you want, AMQP
or MongoDB..

My current favorite is using an IMT table for DoS detection and another for
aggregates for interesting traffic types and querying this every X minutes
and inserting it into ElasticSearch. Kibana makes the most powerful netflow
dashboard ever.

--
Tim
On Nov 20, 2014 6:39 PM, Roland Dobbins rdobb...@arbor.net wrote:


On 21 Nov 2014, at 9:19, Robert Duffy wrote:

  What open-source NetFlow analysis tools would you recommend for quickly

detecting a DDoS attack?


I generally recommend that folks get started with something like
nfdump/nfsen or ntop.  There are other, more sophisticated tools out

there,

but these allow one to get up and running quickly, and to gain valuable
operational experience with which to evaluate more sophisticated tools,

if

they're needed.

---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting


On 21 Nov 2014, at 12:08, Paul S. wrote:

 WANguard from andrisoft has worked well on this for us.

I believe the thread was focusing on open-source tools.

---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-09 Thread Paul S.


I've used the first one, and hacked on the second.

WANGuard, when deployed properly, works amazingly well.

ddosmon is only useful if you have netflow v5 flows (or sflow that can 
get converted to nfv5), but also works well when coupled with exabgp / 
openbgpd.


I added some per ip limiting / exemption features to it (which may or 
may not work, I no longer use it. We've moved to something in house) -- 
available on this fork (https://github.com/Wintereise/ddosmon-mod)


The atheme framework it's built on is fairly easy to extend as well.

But yeah, automated rtbh is really easy (and cheap!) to do these days.

On 11/9/2014 午前 11:56, srn.na...@prgmr.com wrote:

http://www.andrisoft.com/software/wanguard/ddos-mitigation-protection

https://bitbucket.org/tortoiselabs/ddosmon

https://github.com/FastVPSEestiOu/fastnetmon

I have no idea if any of them actually work.

On 11/08/2014 05:10 PM, Eric C. Miller wrote:

Today, we experienced (3) separate DDoS attacks from Eastern Asia, all generating 
 2Gbps towards a single IP address in our network. All 3 attacks targeted 
different IP addresses with dst UDP 19, and the attacks lasted for about 5 minutes 
and stopped as fast as they started.

Does anyone have any suggestions for mitigating these type of attacks?

A couple of things that we've done already...

We set up BGP communities with our upstreams, and tested that RTBH can be set 
and it does work. However, by the time that we are able to trigger the black 
hole, the attack is almost always over.

For now, we've blocked UDP 19 incoming at our edge, so that if future, similar 
attacks occur, it doesn't affect our internal links.

What I think that I need is an IDS that can watch our edge traffic and 
automatically trigger a block hole advertisement for any internal IP beginning to 
receive  100Mbps of traffic. A few searches are initially coming up dry...



Eric Miller, CCNP
Network Engineering Consultant
(407) 257-5115

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-09 Thread Miles Fidelman


Roland Dobbins wrote:


On 9 Nov 2014, at 10:37, Jon Lewis wrote:

I'm sure it's not always the case, but in my experience as a SP, the 
victim virtually always did something to instigate the attack, and is 
usually someone you don't want as a customer.


This may be a reflection of your experience and customer base, but it 
isn't a valid generalization.  Legitimate customers are attacked all 
the time, for various reasons - including unknowingly having their 
servers compromised and used as CCs by miscreants, who're then 
attacked by other miscreants.


But to say that attacks are 'virtually always' provoked by customers 
themselves simply isn't true.  DDoS extortion, ideologically-motivated 
DDoS attacks, maskirovkas intended as a distraction away from other 
activities, simple nihilism, et. al. are, unfortunately, quite common.


When I worked for a cloud hosting provider, the DDoS victims tended 
to be fraudulent signups who were doing malicious or anti-social 
things on the net and were not paying customers anyway.


Many DDoS attacks are miscreant-vs.-miscreant, that's certainly true.  
Compromised machines are 'attractive nuisances', which is yet another 
reason it's important to have visibility into your network traffic 
(it's easy to get started with NetFlow and open-source tools).





Granted, a sample size of 1 - but the most recent event where we were 
the vector for a reflection attack, the target was a game hosting 
system.  Based on some interaction with their sysadmin, it became pretty 
clear that this is fairly common for them, and the motivations had more 
to do with hacking gameplay than anything else.


Miles Fidelman





--
In theory, there is no difference between theory and practice.
In practice, there is.    Yogi Berra

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-09 Thread Joe Chisolm

Look at the products from RioRey (www.riorey.com).  IMHO I think their 
technology is much better than some of the other players out here.

On 11/08/2014 07:10 PM, Eric C. Miller wrote:
 Today, we experienced (3) separate DDoS attacks from Eastern Asia, all 
 generating  2Gbps towards a single IP address in our network. All 3 attacks 
 targeted different IP addresses with dst UDP 19, and the attacks lasted for 
 about 5 minutes and stopped as fast as they started.

 Does anyone have any suggestions for mitigating these type of attacks?

 A couple of things that we've done already...

 We set up BGP communities with our upstreams, and tested that RTBH can be set 
 and it does work. However, by the time that we are able to trigger the black 
 hole, the attack is almost always over.

 For now, we've blocked UDP 19 incoming at our edge, so that if future, 
 similar attacks occur, it doesn't affect our internal links.

 What I think that I need is an IDS that can watch our edge traffic and 
 automatically trigger a block hole advertisement for any internal IP 
 beginning to receive  100Mbps of traffic. A few searches are initially 
 coming up dry...



 Eric Miller, CCNP
 Network Engineering Consultant
 (407) 257-5115





-- 
Joe Chisolm
Computer Translations, Inc.
Marble Falls, Tx.
830-265-8018

Public Key Available at www.sks-keyservers.net

DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread Eric C. Miller

Today, we experienced (3) separate DDoS attacks from Eastern Asia, all 
generating  2Gbps towards a single IP address in our network. All 3 attacks 
targeted different IP addresses with dst UDP 19, and the attacks lasted for 
about 5 minutes and stopped as fast as they started.

Does anyone have any suggestions for mitigating these type of attacks?

A couple of things that we've done already...

We set up BGP communities with our upstreams, and tested that RTBH can be set 
and it does work. However, by the time that we are able to trigger the black 
hole, the attack is almost always over.

For now, we've blocked UDP 19 incoming at our edge, so that if future, similar 
attacks occur, it doesn't affect our internal links.

What I think that I need is an IDS that can watch our edge traffic and 
automatically trigger a block hole advertisement for any internal IP beginning 
to receive  100Mbps of traffic. A few searches are initially coming up dry...



Eric Miller, CCNP
Network Engineering Consultant
(407) 257-5115

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread Miles Fidelman


Eric C. Miller wrote:

Today, we experienced (3) separate DDoS attacks from Eastern Asia, all generating 
 2Gbps towards a single IP address in our network. All 3 attacks targeted 
different IP addresses with dst UDP 19, and the attacks lasted for about 5 minutes 
and stopped as fast as they started.

Does anyone have any suggestions for mitigating these type of attacks?



The phrase automated offensive cyber counter-attack has been coming to 
mind rather frequently, of late.  I wonder if DARPA might fund some work 
in this area. :-)





--
In theory, there is no difference between theory and practice.
In practice, there is.    Yogi Berra

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread Tim Raphael

Check out Arbour Networks, they produce a range of DDoS scrubbing appliances 
that do pretty much what you want.

Regards,

Tim Raphael

 On 9 Nov 2014, at 9:10 am, Eric C. Miller e...@ericheather.com wrote:
 
 Today, we experienced (3) separate DDoS attacks from Eastern Asia, all 
 generating  2Gbps towards a single IP address in our network. All 3 attacks 
 targeted different IP addresses with dst UDP 19, and the attacks lasted for 
 about 5 minutes and stopped as fast as they started.
 
 Does anyone have any suggestions for mitigating these type of attacks?
 
 A couple of things that we've done already...
 
 We set up BGP communities with our upstreams, and tested that RTBH can be set 
 and it does work. However, by the time that we are able to trigger the black 
 hole, the attack is almost always over.
 
 For now, we've blocked UDP 19 incoming at our edge, so that if future, 
 similar attacks occur, it doesn't affect our internal links.
 
 What I think that I need is an IDS that can watch our edge traffic and 
 automatically trigger a block hole advertisement for any internal IP 
 beginning to receive  100Mbps of traffic. A few searches are initially 
 coming up dry...
 
 
 
 Eric Miller, CCNP
 Network Engineering Consultant
 (407) 257-5115

RE: DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread Frank Bulk

Here's a thought-provoking video on what Brocade has done with its SDN
software load on the MLX:
http://vimeo.com/87476840 (demo at ~15 minute mark)

I've written it before: if there was a software feature in routers where I
could specify the maximum rate any prefix size (up to /32) could receive,
that would be very helpful.  If my fastest speed (residential) customer was
100 Mbps and I specified that 200 Mbps was the highest, I would never see
high-rate attacks enter our network.

Frank

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Eric C. Miller
Sent: Saturday, November 08, 2014 7:10 PM
To: NANOG (nanog@nanog.org)
Subject: DDOS, IDS, RTBH, and Rate limiting

Today, we experienced (3) separate DDoS attacks from Eastern Asia, all
generating  2Gbps towards a single IP address in our network. All 3 attacks
targeted different IP addresses with dst UDP 19, and the attacks lasted for
about 5 minutes and stopped as fast as they started.

Does anyone have any suggestions for mitigating these type of attacks?

A couple of things that we've done already...

We set up BGP communities with our upstreams, and tested that RTBH can be
set and it does work. However, by the time that we are able to trigger the
black hole, the attack is almost always over.

For now, we've blocked UDP 19 incoming at our edge, so that if future,
similar attacks occur, it doesn't affect our internal links.

What I think that I need is an IDS that can watch our edge traffic and
automatically trigger a block hole advertisement for any internal IP
beginning to receive  100Mbps of traffic. A few searches are initially
coming up dry...



Eric Miller, CCNP
Network Engineering Consultant
(407) 257-5115

Re: DDOS, IDS, RTBH, and Rate limiting



On 9 Nov 2014, at 8:10, Eric C. Miller wrote:


Does anyone have any suggestions for mitigating these type of attacks?


You can start with S/RTBH (or flowspec, if your platform supports it):

http://tools.ietf.org/html/rfc5635

http://tools.ietf.org/html/rfc5575

https://app.box.com/s/xznjloitly2apixr5xge

Here's a preso which discusses reflection/amplification attacks, 
including chargen reflection/amplification attacks such as the one you 
describe:


https://app.box.com/s/r7an1moswtc7ce58f8gg

---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting



On 9 Nov 2014, at 8:59, Frank Bulk wrote:

I've written it before: if there was a software feature in routers 
where I
could specify the maximum rate any prefix size (up to /32) could 
receive,

that would be very helpful.


QoS generally isn't a suitable mechanism for DDoS mitigation, as the 
programmatically-generated attack traffic ends up 'crowding out' 
legitimate traffic.


S/RTBH, flowspec, and other methods tend to produce better results.

---
Roland Dobbins rdobb...@arbor.net

RE: DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread Frank Bulk

There's no doubt, rate-limiting is a poor-man's way of getting the job done,
but for small operators who aren't as well instrumented (whether that due to
staff or resources), a simple rule such as:
access-list 100 ip host 0.0.0.0 0.0.0.0 rate-limit 20
access-list 100 ip host 0.0.0.0 0.0.0.255 rate-limit 500
int vlan 10
description Internet uplink
 ip access-group 100 in
!
would be great.

Yes, the /32 under attack would essentially be out of service, but at least
the downstream network doesn't get congested and more customers affected.

Frank

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Roland Dobbins
Sent: Saturday, November 08, 2014 8:28 PM
To: NANOG
Subject: Re: DDOS, IDS, RTBH, and Rate limiting


On 9 Nov 2014, at 8:59, Frank Bulk wrote:

 I've written it before: if there was a software feature in routers 
 where I
 could specify the maximum rate any prefix size (up to /32) could 
 receive,
 that would be very helpful.

QoS generally isn't a suitable mechanism for DDoS mitigation, as the 
programmatically-generated attack traffic ends up 'crowding out' 
legitimate traffic.

S/RTBH, flowspec, and other methods tend to produce better results.

---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting



On 9 Nov 2014, at 9:42, Frank Bulk wrote:

There's no doubt, rate-limiting is a poor-man's way of getting the job 
done,
but for small operators who aren't as well instrumented (whether that 
due to

staff or resources)


You might as well just D/RTBH the victim and have done with it, heh.

This is applicable:

http://mailman.nanog.org/pipermail/nanog/2010-January/016747.html

---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread srn . nanog

http://www.andrisoft.com/software/wanguard/ddos-mitigation-protection

https://bitbucket.org/tortoiselabs/ddosmon

https://github.com/FastVPSEestiOu/fastnetmon

I have no idea if any of them actually work.

On 11/08/2014 05:10 PM, Eric C. Miller wrote:
 Today, we experienced (3) separate DDoS attacks from Eastern Asia, all 
 generating  2Gbps towards a single IP address in our network. All 3 attacks 
 targeted different IP addresses with dst UDP 19, and the attacks lasted for 
 about 5 minutes and stopped as fast as they started.
 
 Does anyone have any suggestions for mitigating these type of attacks?
 
 A couple of things that we've done already...
 
 We set up BGP communities with our upstreams, and tested that RTBH can be set 
 and it does work. However, by the time that we are able to trigger the black 
 hole, the attack is almost always over.
 
 For now, we've blocked UDP 19 incoming at our edge, so that if future, 
 similar attacks occur, it doesn't affect our internal links.
 
 What I think that I need is an IDS that can watch our edge traffic and 
 automatically trigger a block hole advertisement for any internal IP 
 beginning to receive  100Mbps of traffic. A few searches are initially 
 coming up dry...
 
 
 
 Eric Miller, CCNP
 Network Engineering Consultant
 (407) 257-5115

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread Jon Lewis


On Sat, 8 Nov 2014, Miles Fidelman wrote:


Does anyone have any suggestions for mitigating these type of attacks?


The phrase automated offensive cyber counter-attack has been coming to mind 
rather frequently, of late.  I wonder if DARPA might fund some work in this 
area. :-)


When you're being hit with one of the UDP reflection DDoS's, attacking the 
world in response isn't likely to work too well.


In theory, you could write something that takes flow data from your 
transit routers, and in either near or real time, looks at that data and 
triggers an RTBH route for any IP that is responsible for more than a 
certain defined threshold of inbound traffic.  In practice, it gets a 
little more complicated than that, as you'll likely want to whitelist some 
IPs and/or maybe be able to set different thresholds for different IPs. 
But it's not that complicated a problem to solve.  Have a default 
threshold, and a table of networks and thresholds.  Once a minute, look at 
the top X local destinations over the past minute.  For each one, check to 
see if it has a custom threshold.  If it doesn't, it gets the default. 
Then see if it's over its threshold.  If it is, generate an RTBH route and 
email your NOC.


The tricky part is when to remove the route...since you can't tell if the 
attack has ended while the target is black holed by your upstreams.


--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: DDOS, IDS, RTBH, and Rate limiting



On 9 Nov 2014, at 10:12, Jon Lewis wrote:

The tricky part is when to remove the route...since you can't tell if 
the attack has ended while the target is black holed by your 
upstreams.


You can with NetFlow, if you've D/RTBHed the IP in question on your own 
infrastructure.  NetFlow reports statistics on dropped traffic (except 
on a few platforms with implementation deficiencies).


But this kind of thing punishes the victim.  It's far better to do 
everything possible to *protect* the target(s) of an attack, and only 
use D/RTBH as a last resort.


---
Roland Dobbins rdobb...@arbor.net

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread Trent Farrell

A quick and dirty win is to get your upstream(s) to kill anything UDP 19 to
your prefixes at their ingress points if it becomes a common thing. You
lose visibility as to when you're getting targeted by that type of attack
again though, which could matter depending on your network.


On Saturday, November 8, 2014, Jon Lewis jle...@lewis.org wrote:

 On Sat, 8 Nov 2014, Miles Fidelman wrote:

  Does anyone have any suggestions for mitigating these type of attacks?


 The phrase automated offensive cyber counter-attack has been coming to
 mind rather frequently, of late.  I wonder if DARPA might fund some work in
 this area. :-)


 When you're being hit with one of the UDP reflection DDoS's, attacking the
 world in response isn't likely to work too well.

 In theory, you could write something that takes flow data from your
 transit routers, and in either near or real time, looks at that data and
 triggers an RTBH route for any IP that is responsible for more than a
 certain defined threshold of inbound traffic.  In practice, it gets a
 little more complicated than that, as you'll likely want to whitelist some
 IPs and/or maybe be able to set different thresholds for different IPs. But
 it's not that complicated a problem to solve.  Have a default threshold,
 and a table of networks and thresholds.  Once a minute, look at the top X
 local destinations over the past minute.  For each one, check to see if it
 has a custom threshold.  If it doesn't, it gets the default. Then see if
 it's over its threshold.  If it is, generate an RTBH route and email your
 NOC.

 The tricky part is when to remove the route...since you can't tell if the
 attack has ended while the target is black holed by your upstreams.

 --
  Jon Lewis, MCP :)   |  I route
  |  therefore you are
 _ http://www.lewis.org/~jlewis/pgp for PGP public key_



-- 

*Trent Farrell*

*Riot Games*

*IP Network Engineer*

E: tfarr...@riotgames.com | IE:  +353 83 446 6809 | US: +1 424 285 9825

Summoner name: Foro

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-08 Thread Jon Lewis

How many holes are you going to stick fingers in to stop the flows?  Good 
luck getting your provider to put in such a filter and make it anything 
more than temporary...and then there's still DNS, NTP, SNMP, and other 
protocols an attacker can easily utilize when they find that chargen isn't 
getting the job done.


On Sat, 8 Nov 2014, Trent Farrell wrote:


A quick and dirty win is to get your upstream(s) to kill anything UDP 19 to
your prefixes at their ingress points if it becomes a common thing. You
lose visibility as to when you're getting targeted by that type of attack
again though, which could matter depending on your network.


On Saturday, November 8, 2014, Jon Lewis jle...@lewis.org wrote:


On Sat, 8 Nov 2014, Miles Fidelman wrote:

 Does anyone have any suggestions for mitigating these type of attacks?




The phrase automated offensive cyber counter-attack has been coming to
mind rather frequently, of late.  I wonder if DARPA might fund some work in
this area. :-)



When you're being hit with one of the UDP reflection DDoS's, attacking the
world in response isn't likely to work too well.

In theory, you could write something that takes flow data from your
transit routers, and in either near or real time, looks at that data and
triggers an RTBH route for any IP that is responsible for more than a
certain defined threshold of inbound traffic.  In practice, it gets a
little more complicated than that, as you'll likely want to whitelist some
IPs and/or maybe be able to set different thresholds for different IPs. But
it's not that complicated a problem to solve.  Have a default threshold,
and a table of networks and thresholds.  Once a minute, look at the top X
local destinations over the past minute.  For each one, check to see if it
has a custom threshold.  If it doesn't, it gets the default. Then see if
it's over its threshold.  If it is, generate an RTBH route and email your
NOC.

The tricky part is when to remove the route...since you can't tell if the
attack has ended while the target is black holed by your upstreams.

--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_




--

*Trent Farrell*

*Riot Games*

*IP Network Engineer*

E: tfarr...@riotgames.com | IE:  +353 83 446 6809 | US: +1 424 285 9825

Summoner name: Foro



--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: DDOS, IDS, RTBH, and Rate limiting