from:"Denys Fedoryshchenko"


On 2020-05-13 22:53, Töma Gavrichenkov wrote:

Peace,

On Wed, May 13, 2020 at 10:43 PM Elad Cohen  wrote:

For you nothing will work.


Is it a personal attack?

IPv6 is working good for me so far ;-)

--
Töma

It works for Elad as well.
He is pushing others for IPv4+ suffering, while he is happily using 
IPv6.

(Anybody can check his x-originating-ip header)
ROTFL

Re: An appeal for more bandwidth to the Internet Archive


On 2020-05-13 13:10, Bill Woodcock wrote:

On 2020-05-13 11:00, Mark Delany wrote:

On 13May20, Denys Fedoryshchenko allegedly wrote:
What about introducing some cache offloading, like CDN doing? 
(Google,

Facebook, Netflix, Akamai, etc)
Maybe some opensource communities can help as well

Surely someone has already thought thru the idea of a community CDN?
Perhaps along the lines of pool.ntp.org? What became of that
discussion?


Yes, Jeff Ubois and I have been discussing it with Brewster.

There was significant effort put into this some eighteen or twenty
years ago, backed mostly by the New Zealand government…  Called the
“Internet Capacity Development Group.”  It had a NOC and racks full of
servers in a bunch of datacenters, mostly around the Pacific Rim, but
in Amsterdam and Frankfurt as well, I think.  PCH put quite a lot of
effort into supporting it, because it’s a win for ISPs and IXPs to
have community caches with local or valuable content that they can
peer with.  There’s also a much higher hit-rate (and thus efficiency)
to caching things the community actually cares about, rather than
whatever random thing a startup is paying Akamai or Cloudflare or
whatever to push, which may never get viewed at all.  It ran well
enough for about ten years, but over the long term it was just too
complex a project to survive at scale on community support alone.  It
was trending toward more and more of the hard costs being met by PCH’s
donors, and less and less by the donors who were supporting the
content publishers, which was the goal.

The newer conversation is centered around using DAFs to support it on
behalf of non-profit content like the Archive, Wikipedia, etc., and
that conversation seems to be gaining some traction.  Unfortunately
because there are now a smaller number of really wealthy people who
need places to shove all their extra money.  Not how I’d have liked to
get here.

I think this is a simple equation.

1) The minimum cost of implementation and technical support efforts
I think earlier this was the main problem, 10 years ago there was no 
such level

of software automation as it is available today.
2) Win for operators.
Before it was more trivial by running squid and trivial cache, now, with 
HTTPS it is

not possible.
3) Proud badge of non-profit projects supporter and charity activities.
(Whether it is possible to write off tax/etc as donations - depends on 
the laws of your country)

Re: An appeal for more bandwidth to the Internet Archive

On 2020-05-13 11:00, Mark Delany wrote:

On 13May20, Denys Fedoryshchenko allegedly wrote:

What about introducing some cache offloading, like CDN doing? (Google,
Facebook, Netflix, Akamai, etc)

Maybe some opensource communities can help as well

Surely someone has already thought thru the idea of a community CDN?
Perhaps along the lines of pool.ntp.org? What became of that
discussion?

Maybe a TOR network could be repurposed to cover the same ground.

Mark.
I believe tor is not efficient at all for this purposes. Privacy have
very high overhead.

Several schemes exist:
1)ISP announce in some way subnets he want to be served from his cache.
1.A)Apple cache way - just HTTP(S) request will turn specific IP to ISP
cache. Not secure at all.
1.B)BGP + DNS, most common way. ISP does peering with CDN, CDN will
return ISP cache nodes IP's to DNS requests.
It means for example content.archive.org will have local node A/
records (btw where is IPv6 for archive?) for

customers of ISP with this node, or anybody who is peering with it.
Huge drawback - archive.org will need to provide TLS certificates for
web.archive.org each local node, this is bad and probably no-go.
Yes, i know some schemes exist, that certificate is not present on local
node, but some "precalculated" result used, but it is too complex.
1.C)BGP + HTTP redirect. If ISP has peering with archive.org, to all
subnets announced users will get 302 or some HTTP redirect.
Next is almost same and much better, but will require small
modifications of content engine or frontend balancers.
1.D)BGP + HTTP rewrite. If ISP <*same as before*> URL is rewritten
within content
e.g.
http://web.archive.org/web/20200511193226/https://git.kernel.org/torvalds/t/linux-5.7-rc5.tar.gz
will appear as

http://emu.st.node.archive.org/web/20200511193226/https://git.kernel.org/torvalds/t/linux-5.7-rc5.tar.gz
or
http://archive-org.proxy.emu.st/web/20200511193226/https://git.kernel.org/torvalds/t/linux-5.7-rc5.tar.gz
In second option ISP can handle SSL certificate by himself.
2)BGP announce of archive.org subnets locally. Prone to leaks, require
TLS certificates and etc, no-go.

You can still modify some schemes, and make other options that no one
has yet implemented.
For example, to do everything through javascript (CDNs cannot afford it,
because of way they work),
and for example, website generate content links dynamically, for that
client request some /config.json file
(which is dynamically generated and cached for a while), so we give it
to IPs that have a local node - URL of the local node, for the rest -

default url.

Re: An appeal for more bandwidth to the Internet Archive

What about introducing some cache offloading, like CDN doing? (Google, 
Facebook, Netflix, Akamai, etc)
I think it can be rolled pretty quickly, with minimum labor efforts, at 
least for heavy content.
Maybe some opensource communities can help as well, and same scheme can 
be applied then to other non-profits.
But sure something more smooth like nginx caching, not bunch of 
rsync/ssh scripts, as many Linux mirrors have.

On 2020-05-13 08:25, Tim Požár wrote:

Internet Archive primary office is located at 300 Funston in San
Francisco.  This was a Christian Science church so it has the roman
columns you would expect for a church / library.  You can see it on
Google Street Views at:

https://www.google.com/maps/place/300+Funston+Ave,+San+Francisco,+CA+94118

Although they serve content out of this site, their primary site for
bandwidth is at 2512 Florida Ave, Richmond, CA.

IA does have satellite offices around the world for scanning, etc.,
the public facing servers are location in these two locations.

Tim

On 5/12/20 9:24 PM, Terrence Koeman wrote:
Aren't they in a former church or something? I vaguely remember their 
location to be significant for some reason or another. So location may 
weigh heavily.

-- Regards,
    Terrence Koeman, PhD/MTh/BPsy
      Darkness Reigns (Holding) B.V.

Please quote relevant replies.
Spelling errors courtesy of my 'smart'phone.

*From:* David Hubbard 
*Sent:* Wednesday, 13 May 2020 06:02
*To:* nanog@nanog.org
*Subject:* Re: An appeal for more bandwidth to the Internet Archive

Could the operation be moved out of California to achieve
dramatically reduced operating costs and perhaps solve some 
problems
via cost savings vs increased donation?  I have to imagine with 
the

storage and processing requirements that the footprint and power
usage in SFO is quite costly.  I have equipment in a few 
California

colo's and it's easily 3x what I pay for similar in Nevada, before
even getting into tax abatement advantages.

On 5/12/20, 1:33 PM, "NANOG on behalf of colin johnston"

wrote:

     Is the increased usage due to more users or more existing 
users

having higher bandwidth at home to request faster ?
     Would be interested if IPS configured firewall used to block
out invalid traffic/spam traffic and if such traffic increased 
when

back end network capacity increased ?
     What countries are requesting the most data and does this
analysis throw up questions as to why ?
     Are there high network usage hitters which raise question as 
to

why asking for so much data time and time again and is this valid
traffic use ?

     Colin

     > On 12 May 2020, at 17:33, Tim Požár  wrote:
     >
     > Jared...
     >
     > Thanks for sharing this.  I was the first Director of
Operations from '96 to '98, at was was then Internet Archive/Alex. 
I was the network architect back then got them their ASN and
original address space. Folks may also know, I help start SFMIX 
with

Matt Peterson.
     >
     > A bit more detail in this...  Some of this I got from Jonah
Edwards who is the current Network Architect at IA.  Yes, the 
bottle
neck was the line cards.  They have upgraded and that has 
certainly

helped the bandwidth of late.
     >
     > Peering would be a big help for IA. At this point they have
two 10Gb LAG interfaces that show up on SFMIX that was turned up
last February. Looking at the last couple of weeks the 95th
percentile on this 20Gb LAG is 3 Gb.  As they just turned up on
SFMIX, they are just starting to get peers turned up there. 
Eyeball

networks that show up on SFMIX are highly encouraged to start
peering with them.  Alas, they are v4 only at this point.
     >
     > Additionally, if folks do have some fat pipes that can 
donate

bandwidth at 200 Paul, I am sure Jonah won't turn it down.
     >
     > Tim
     >
     > On 5/12/20 4:45 AM, Jared Brown wrote:
     >> Hello all!
     >> Last week the Internet Archive upgraded their bandwidth 
30%
from 47 Gbps to 62 Gbps. It was all gobbled up immediately. 
There's

a lovely solid green graph showing how usage grows vertically as
each interface comes online until it too is 100% saturated. 
Looking

at the graph legend you can see that their usage for the past 24
hours averages 49.76G on their 50G of transport.
     >> To see the pretty pictures follow the below link:
     >>

https://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/

     >> Relevant parts from the blog post:
     >> "A year ago, usage was 30Gbits/sec. At the beginning of 
this

year, we were at 40Gbits/sec, and we were handling it. ...
     >> Then Covid-19 hit and demand rocketed to

Paid abuse desks idea? Was: Urgently need contact at Facebook of Instagram and also Omegle

2020-05-02 Thread Denys Fedoryshchenko


On 2020-05-03 01:10, Anne P. Mitchell, Esq. wrote:

There is a woman torturing animals on Omegle, she is advertising it on
her Instagram account.  Need to get this in front of the right people
to have her traced and shut down.

Please let me know if you can provide a contact for either org.

Anne

---
Anne P. Mitchell, Attorney at Law
Dean of Cyberlaw & Cybersecurity, Lincoln Law School
Advisor, Colorado Innovation Response Team Task Force
CEO/President, SuretyMail Email Reputation Certification
Policy Drafting and Review for Businesses
Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal anti-spam 
law)
Legislative Consultant, GDPR, CCPA (CA) & CCDPA (CO) Compliance 
Consultant

Board of Directors, Denver Internet Exchange
Chair Emeritus, Asilomar Microcomputer Workshop
Legal Counsel: The CyberGreen Institute
Former Counsel: Mail Abuse Prevention System (MAPS)
Location: Boulder, Colorado


Many times I came across the fact that large content operators do not 
have adequate support.
I understand that increasing the support is not reasonable, its too 
expensive.
Why they don't make a separate "route" with payment and a list of urgent 
complaints criteria,

where the complainant pays a deposit of say $ 100-$200/hr,
and if he writes nonsense or not urgent case, qualified for free 
support, he simply loses money

and pays the operator for the time spent.

If case qualifies for urgent support, the money is returned to person 
seeking support.

But how things are going on now, it is totally wrong.
Large content networks are turning into trash bin and a hotbed of crime 
due to the inability to

quickly control illegal content.

I wouldn’t refuse such $100 option for ISP, if someone wants to send me 
complaints about torrents for $100,

i don’t mind looking at it carefully for compliance with local law.
And at the same time I could respond more quickly to complaints on DDOS 
and SPAM, hire dedicated staff, its for good

too. Also he can handle additionally regular abuse complaints channels.

Re: mail admins? Proposal of solution

2020-05-01 Thread Denys Fedoryshchenko


On 2020-04-30 02:43, Mark Andrews wrote:

And it is still on going.  Just got 4 of these.

Mark


Technical proposal how to solve that.

At 1st of month send monthly reminder manually, to each subscriber, but 
encode recipient address in Reply-To: a bit special way.

First, you need catch-all alias on special domain.
Then, you calculate reply-to, for example if destination recipient:
hash("secret-keyword" + u...@domain.com)=aabbccdd
So, it becomes:
Reply-To: aabbc...@monthly.nanog.org

Then just check all bounces received to *@monthly.nanog.org, and 
unsubscribe relevant users.

Re: Abuse Desks

2020-04-29 Thread Denys Fedoryshchenko


On 2020-04-28 18:57, Mike Hammett wrote:

I noticed over the weekend that a Fail2Ban instance's complain
function wasn't working. I fixed it. I've noticed a few things:

1) Abusix likes to return RIR abuse contact information. The vast
majority are LACNIC, but it also has kicked back a couple for APNIC
and ARIN. When I look up the compromised IP address in Abusix via the
CLI, the APNIC and ARIN ones return both ISP contact information and
RIR information. When I look them up on the RIR's whois, it just shows
the ISP abuse information. Weird, but so rare it's probably just an
anomaly. However, almost everything I see in LACNIC's region is
returned with only the LACNIC abuse information when the ones I've
checked on LACNIC's whois list valid abuse information for that
prefix. Can anyone confirm they've seen similar behavior out of
Abusix? I reached out to them, but haven't heard back.
2) Digital Ocean hits my radar far more than any other entity.
3) Azure shows up a lot less than GCP or AWS, which are about similar
to each other.
4) Around 5% respond saying it's been addressed (or why it's not in
the event of security researchers) within a couple hours. The rest I
don't know. I've had a mix of small and large entities in that
response.
5) HostGator seems to have an autoresponder (due to a 1 minute
response) that just indicates that you sent nothing actionable,
despite the report including the relevant log file entries.
6) Charter seems to have someone actually looking at it as it took
them 16 - 17 hours to respond, but they say they don't have enough
information to act on, requesting relevant log file entries...  which
were provided in the initial report and are even included in their
response. They request relevant log file entries with the date, time,
timezone, etc. all in the body in plain text, which was delivered.
7) The LACNIC region has about 1/3 of my reports.

Do these mirror others' observations with security issues and how
abuse desks respond?


Although many people write here - no need to worry about such minor 
things, i strongly disagree.


If someone littering server ssh logs for an hour, most likely on the 
other side:
1) A botnet-infected computer that needs to be fixed. Today ssh 
bruteforce,
tomorrow spam and hosting scam and very real financial losses for some 
people.
2) A hacker who is looking for an easy target. If he succeed, law 
enforcement
will come to you tomorrow and might waste lot of your time. And 
sometimes it’s
some kid who, possibly will get an early warning, will not break his 
life by getting

a criminal term.

And how to fight with lazy operators who start differentiate on abuse, 
which is worth their

majestic attention.
I send proper abuse reports if there is no reaction to them - I make a 
null route of incoming SYN
requests on all my servers, and sometimes i share an IP list with other 
operators who want to live

in a "clean" internet, and not in a garbage dump.
I have several resources hosted, so at the end techies of those 
"majestic ISPs" come with tears,
when their customers start to torture their support and sales, and beg 
to be unlocked and

most start to read abuse mailbox.
Or they just lose customers.

Re: FlowSpec

2020-04-23 Thread Denys Fedoryshchenko


On 2020-04-23 19:12, Roland Dobbins wrote:

On 23 Apr 2020, at 22:57, Denys Fedoryshchenko wrote:


In general operators don't like flowspec


Its increasing popularity tens to belie this assertion.

Yes, you're right that avoiding overflowing the TCAM is very
important.  But as Rich notes, a growing number of operators are in
fact using flowspec within their own networks, when it's appropriate.

One of operators told me why they dont provide flowspec anymore:
customers are installing rules by scripts, stacking them,
and not removing then when they dont need them anymore.
RETN solved that by limiting number of rules customer can install.



Smart network operators tend to do quite a bit of lab testing,
prototyping, PoCs, et. al. against the very specific combinations of
platforms/linecards/ASICs/OSes/trains/revisions before generally
deploying new features and functionality; this helps ameliorate many
concerns.
Definitely, and i know some hosting operators who provide Flowspec 
functionality
different way - over their own web interface/API. This way they can do 
unit tests,

and do additional verifications.

But if there is direct BGP, things like 
https://dyn.com/blog/longer-is-not-better/
might happen, if customer is using some exotic, "nightly-build" BGP 
implementation.

Re: FlowSpec

2020-04-23 Thread Denys Fedoryshchenko


On 2020-04-23 18:13, Colton Conor wrote:

Do any of the large transit providers support FlowSpec to transit
customers / other carriers, or is that not a thing since they want to
sell DDoS protection services? FlowSpec sounds much better than RTBH
(remotely triggered blackhole), but I am not sure if  FlowSpec is
widely implemented. I see the large router manufacturers support it.


RETN

They have extended blackholing, and FlowSpec, sure its all have costs.
I'm using both services from them and quite satisfied.

In general operators don't like flowspec, because it is not easy to 
implement it right,

there is bugs and most important its "eating" TCAM.
For example: 
https://blog.cloudflare.com/todays-outage-post-mortem-82515/

Re: FlowSpec

2020-04-23 Thread Denys Fedoryshchenko


On 2020-04-23 18:13, Colton Conor wrote:

Do any of the large transit providers support FlowSpec to transit
customers / other carriers, or is that not a thing since they want to
sell DDoS protection services? FlowSpec sounds much better than RTBH
(remotely triggered blackhole), but I am not sure if  FlowSpec is
widely implemented. I see the large router manufacturers support it.

RETN considered Tier-2?
They offer it, but it is more expensive than

Re: "Is BGP safe yet?" test

2020-04-20 Thread Denys Fedoryshchenko


On 2020-04-20 22:01, Rubens Kuhl wrote:

On Mon, Apr 20, 2020 at 3:37 PM Denys Fedoryshchenko
 wrote:


There is simple use case that will prove this page is giving false
positive
for their "name" strategy.
Any AS owner with default route only (yes it happens a lot) users
will
get:
"YOUR ISP TERRIBLE, HIS BGP NOT SAFE!".
But he have nothing to validate! His BGP is implemented safely,
its just his upstream is not validating routes.


So, that same ISP who is not validating because it has a default route
could push its providers to do validation and then be as safe as other
validating themselves ?

Rubens
Typically, those who have "default route only" are too small to be 
heard,

and their "wishes" doesn't go beyond the first line of support.
Not to mention that it does not work at all if upstream is a monopoly,
especially a state monopoly, who wont move a finger for "optional 
features".


And most important, the most common answer:
All Tier-1 implemented it? No.
Major hosting operators, such as AWS, gcloud, etc? - No.
So...

Re: "Is BGP safe yet?" test

2020-04-20 Thread Denys Fedoryshchenko


On 2020-04-20 19:24, Tom Beecher wrote:

Technical people need to make the business case to management for RKPI
by laying out what it would cost to implement (equipment, resources,
ongoing opex), and what the savings are to the company from protecting
themselves against hijacks. By taking this step, I believe RPKI will
become viewed by non-technical decision makers as a 'Cloudflare
initiative' instead of a 'good of the internet' initiative, especially
by some companies who compete with Cloudflare in the CDN space.

I believe that will change the calculus and make it a more difficult
sell for technical people to get resources approved to make it happen.

If i am not wrong, for most routers implementing RPKI means spinning up 
VM

with RPKI cache that need significant tinkering?
I guess it is a blocker for many, unless some "ready made" solutions 
offered

by vendors.
Also, if ISP configure his router and it did crashed because he 
installed
some "no warranty whatsoever" software from cloudflare github, what is 
next?

I guess this might be not welcome in support contracts.

P.S. Sorry for previous post top-posted. Just by mistake hit "Send" 
before i finished it

Re: "Is BGP safe yet?" test

2020-04-20 Thread Denys Fedoryshchenko

There is simple use case that will prove this page is giving false 
positive

for their "name" strategy.
Any AS owner with default route only (yes it happens a lot) users will 
get:

"YOUR ISP TERRIBLE, HIS BGP NOT SAFE!".
But he have nothing to validate! His BGP is implemented safely,
its just his upstream is not validating routes.

On 2020-04-20 21:21, Andrey Kostin wrote:

Mark Tinka писал 2020-04-20 12:57:

On 20/Apr/20 18:50, Tom Beecher wrote:


I (and Ben, and a few others) are all too familiar with the ARIN 
madness

around their TAL.

Simple - we just don't accept it, which means our networks will be
unsafe against North American resources. Highly doubtful my 
organization

is that interested in how the ARIN region may or may not impact our
interest in deploying RPKI on this side of the planet, when the rest 
of

the world are less mad about it :-).


So this means that there is no single source of truth for PRKI
implementation all around the world and there are different shades,
right? As a logical conclusion, the information provided on that page
may be considered incorrect in terms of proclaiming particular network
safe or not safe, but when it's claimed (sometimes blatantly) we now
have to prove to our clients that we are not bad guys.

Kind regards,
Andrey

Re: Constant Abuse Reports / Borderline Spamming from RiskIQ

2020-04-13 Thread Denys Fedoryshchenko


On 2020-04-13 17:25, Kushal R. wrote:

From the past few months we have been receiving a constant stream of
abuse reports from a company that calls themselves RiskIQ
(RiskIQ.com).

The problem isn’t the abuse reports themselves but the way they send
them. We receive copies of the report, on our sales, billing,
TECH-POCs and almost everything other email address of ours that is
available publicly. It doesn’t end there, they even online on our
website and start using our support live chat and as recently as
tomorrow they I see that they have now started using Twitter
(@riskiq_irt) to do the same.

We understand these reports and deal with them as per our policies and
timelines but this constant spamming by them from various channels is
not appreciated.

Does anyone have a similar experience with them?


If the problem of abuse legit and arises with enviable constancy, maybe 
it is time to take fundamental measures to combat abuse?
I had to block port 25 by default on some operators and create a 
self-care web page for removing it,
 with the requirement to read legal agreement where consequences stated, 
if the client start spamming.
For those who are bruteforcing other people's servers / credentials, 
soft-throttling ACL had to be implemented.
And as they wrote earlier, it’s better to kick out exceptionally bad 
customers than to destroy your reputation.

Re: South Africa On Lockdown - Coronavirus - Update!

2020-03-24 Thread Denys Fedoryshchenko


On 2020-03-24 18:59, Randy Bush wrote:
He's a network operator. From North America, on the North American 
Network
Operators mailing list. Something you are not, so please stop spouting 
your

drivel on a list that has nothing to do with you.


this is not how we should act in under pressure

+1

NANOG very often inspired me and my friends as the best example of the 
discussion, process of solving various problems related to internet in 
general. And now everybody around the world got major, similar problem. 
And someone now says, go away from here, is this a maillist for North 
American operators? If this is really decision supported by the 
majority(which i doubt), i would like to know that.


For example, a reminder that the VPN is very poorly balanced over LAG - 
was very useful.


And like this short message about South Africa was highly useful for my 
country of residence, Lebanon, too. We are under full, quite severe 
lock-down, with fines, checkpoints and etc.
Several ISPs here who kept NOC support (single person) near server room, 
in office - got fined too, as usual businesses, because there is no such 
regulation to allow them to run. This is terrible, because in such 
circumstances ISP cannot continue to provide reliable service, and their 
customers without internet might not keep isolation.
Therefore, I carefully read this group, how everyone solves similar 
problems.
And this is why, how is it solved in North America or other countries or 
regions can be an example for other countries.

Re: TCP-AMP DDoS Attack - Fake abuse reports problem

2020-02-21 Thread Denys Fedoryshchenko

Good luck responding to such SYN/ACK, when you get 10+Gbps of them (real 
case happened while ago with colleague).
Sure those SYN/ACK are not from single location, and attackers might use 
whole /24 for SYN spoofing.


On 2020-02-21 03:34, Amir Herzberg wrote:

If I read your description correctly:

- Attacker sends spoofed TCP SYN from your IP address(es) and
different src ports, to some TCP servers (e.g. port 80)
- TCP servers respond with SYN/ACK  ; many servers resend the SYN/ACK
hence amplification .
- *** your system does not respond ***
- Servers may think you're doing SYN-Flood against them, since
connection remains in SYN_RCVD, and hence complain. In fact, we don't
really know what is the goal of the attackers; they may in fact be
trying to do SYN-Flood against these servers, and you're just a
secondary victim and not the even the target, that's also possible.

Anyway, is this the case?

If it is... may I ask, do you (or why don't you) respond to the
unsolicited SYN/ACK with RST as per the RFC?

I suspect you don't, maybe due to these packets being dropped by
FW/NAT, that's quite common. But as you should understand by now from
my text, this (non-standard) behavior is NOT recommended. The problem
may disappear if you reconfigure your FW/NAT (or host) to respond with
RST to unsolicited SYN/ACK.

As I explained above, if my conjectures are true, then OVH as well as
the remote servers may have a valid reason to consider you either as
the attacker or as an (unknowning, perhaps) accomplice.

I may be wrong - sorry if so - and would appreciate, in any case, if
you can confirm or clarify, thanks.

--
Amir Herzberg

Comcast professor of Security Innovations, University of Connecticut

Homepage: https://sites.google.com/site/amirherzberg/home

Foundations of Cyber-Security (part I: applied crypto, part II:
network-security):
https://www.researchgate.net/project/Foundations-of-Cyber-Security

On Thu, Feb 20, 2020 at 5:23 PM Octolus Development
 wrote:


A very old attack method called TCP-AMP (
https://pastebin.com/jYhWdgHn ) has been getting really popular
recently.

I've been a victim of it multiple times on many of my IP's and every
time it happens - My IP's end up getting blacklisted in major big
databases. We also receive tons of abuse reports for "Port
Scanning".

Example of the reports we're getting:

tcp: 51.81.XX.XX:19342 -> 209.208.XX.XX:80 (SYN_RECV)
tcp: 51.81.XX.XX:14066 -> 209.208.XX.XX:80 (SYN_RECV)

OVH are threatening to kick us off their network, because we are
victims of this attack. And requesting us to do something about it,
despite the fact that there is nothing you can do when you are being
victim of an DDoS Attack.

Anyone else had any problems with these kind of attacks?

The attack basically works like this;
- The attacker scans the internet for TCP Services, i.e port 80.
- The attacker then sends spoofed requests from our IP to these TCP
Services, which makes the remote service attempt to connect to us to
initiate the handshake.. This clearly fails.
... Which ends up with hundreds of request to these services,
reporting us for "port flood".

Re: akamai yesterday - what in the world was that

2020-02-12 Thread Denys Fedoryshchenko


It would be really nice if the major CDNs had virtual machines small
network operators with very expensive regional transport costs could
spin up.  Hit rate would be very low, of course, but the ability to
grab some of these mass-market huge updates and serve them on the
other end of the regional transport at essentially no extra cost would
be great. I'm sure legal arrangements make that difficult, though.

+1

I think primary reason is that many major CDN offload nodes implemented 
such way that they require significant amount of maintenance and 
support. And doesnt matter, small or big ISP - they will have problems, 
and when the company that installed this CDN node is huge, like Facebook 
or Google, to crank all the bureaucratic wheels to change silly power 
supply or HDD - it comes at a huge cost for them. Also add that small 
ISP often dont have 24/7 support shifts, less qualified for complex 
issues, more likely to have poor infrastructure 
(temperature/power/reliability), that means more support expenses.
And they don’t give a damn that because of their "behemothness", they 
increase the digital inequality gap. When a large ISP or ISP cartel 
member enter some regional market, local providers will not be able to 
compete with him, since they cannot afford CDN nodes due traffic volume.


Many of CDN also do questionable "BGP as signalling only" setups with 
proprietary TCP probing/loss, that often doesn't work reliably. Each of 
them is trying to reinvent the wheel, "this time not round, but 
dodecahedral". And when it fails, ISP will waste time of support, until 
it reach someone who understand issue. In most cases, this is a blackbox 
setup, and when problem happens ISP are endlessly trying to explain 
problem to outsourced support, who have very limited access as well, and 
responding like robot according to the his "support workflow", with zero 
feedback to common problems.


Honestly, it's time to develop an open standard for caching content on 
open CDN nodes, which should be easy to use for both content providers 
and ISPs.
For example, at one time existed a special hardcoded "retracker.local" 
server in many torrent clients, which optionally(if resolved on ISP, 
static entry in recursor) was used for the discovery of nearest seeders 
inside network of a local provider.

http://retracker.local/
Maybe it is possible to make a similar scheme, if the content provider 
wants "open" CDN to work, it will set some alternative scheme 
cdn://content.provider.com/path/file or other kind of hint, with content 
validity/authenticity mechanism. After that, the browser will attempt to 
do CDN discovery, for example: "content.provider.com.reservedtld" and 
will push request through it.

I'm sure someone will have a better idea how to do that.

As a result, the installation of such "offloading node" will be just 
installing container/vm and, if the load is increased, increasing the 
number of servers/vm instances.

Re: Recommended DDoS mitigation appliance?

2019-11-18 Thread Denys Fedoryshchenko


On 2019-11-18 04:23, Richard wrote:

I would say you are making some assumptions that are not fact based.
The OP is very knowledgeable and would not mince words or waste
bandwidth. Let us see what he has to say in regards to your remarks.
He will be able to make this more clear once he has read what people
have stated in other responses.

Respectfully, of course, Richard Golodner
On 11/17/19 8:12 PM, Töma Gavrichenkov wrote:


Peace,

On Mon, Nov 18, 2019, 1:49 AM Rabbi Rob Thomas 
wrote:


I am going to assume you want it to spit out 10G clean, what

size

dirty traffic are you expecting it to handle?


Great question!  Let's say between 6Gbps and 8Gbps dirty.


As someone making a living as a DDoS mitigation engineer for the
last 10 years (minus 1 month) I should say your threat model is sort
of unusual.  Potential miscreants today should be assumed to have
much more to show you even on a daily basis.

Is it like you also have something filtering upstream for you, e.g.
flowspec-enabled peers?

--
Töma





AFAIK new threats (SYN+ACK amplification) can't be mitigated over 
flowspec and they can reach 40+Gbps easily.

Re: SFP oraganizers / storage recommendations

2019-10-30 Thread Denys Fedoryshchenko


On 2019-10-30 15:35, Matthew Huff wrote:

Any recommendations to keep track of different SFP and keep them
organized? Any storage boxes / trays designed for SFPs?

3D printed some, but i have small amounts.
Like this one: https://www.thingiverse.com/thing:2855165
There is many more designs, for example whole tray for 50 pieces.

Re: Mx204 alternative

2019-09-02 Thread Denys Fedoryshchenko


On 2019-09-02 17:16, Saku Ytti wrote:

On Mon, 2 Sep 2019 at 16:26, Denys Fedoryshchenko
 wrote:


or some QFX, for example, Broadcom Tomahawk 32x100G switches only do
line-rate with >= 250B packets according to datasheets.


Only is peculiar term here. 100Gbps is 148Mpps, give or take 100PPM,
at 250B it's still some 50Mpps. Times 32 that's 1600Mpps, or 1.6Gpps.
Only implies it's modest compared to some other solution, what is that
solution? XEON doing ~nothing (not proper lookup even) is some couple
hundred Mpps, far cry from 1.6Gpps with ACL, QoS and L3 lookup.
I don't care about wire rate on chip with lot of ports, because
statistics. 250B average size on 32x100GE on a chip is fine to me.
250B average size on 32x100GE with 32 chips, would be horrifying.

I'm not saying XEON does not have application, I'm just saying XEON is
bps and pps expensive chip compared to almost anything out there,
however there are some application with very deep touch where it is
marketable.
Btw. technically Tomahawk and Trio are very different, Trio has tens
or hundreds of cores executing software, cores happen to have domain
specific instruction set, but still software box with lot of cores.
Tomahawk is pipeline box, having domain specific hardware and largely
not running a software (but all pipelines today are somewhat
programmable anyhow). On Trio you are mostly just time limited on what
you can do, on Tomahawk you have physical hardware restrictions on
what you can do.
Of course, they are much stronger (and cheaper in $/bps or $/pps) when 
it comes to L2/L3 lookup, basic stateless filters, simple QoS.
But can Trio perform stateful firewall filtering for millions of flows+ 
lot of mpps that Xeon easily handle? Thats the case of recent DDoS 
attacks.

Re: Mx204 alternative

2019-09-02 Thread Denys Fedoryshchenko


On 2019-09-02 15:52, Baldur Norddahl wrote:


Maturity is such a subjective word. But yes there are plenty of
options for routing protocols on a Linux. Every internet exchange is
running BGP on Linux for the route server after all.

I am not recommending a server over MX204. I think MX204 is brilliant.
It is one of the cheapest options and if that is not cheap enough,
THEN the server solution is probably what you may be looking for.

You can move a lot of traffic even with an old leftover server.
Especially if you are not concerned with moving 64 bytes DDoS at line
speed, because likely you would be down anyway in that case.

As to the OPEX I would claim there are small shops that would have an
easier time with a server, because they know how to do that. They
would have only one or two routers and learning how to run JUNOS just
for that might never happen. It all depends on what workforce you
have. Network people or server guys?

Regards

Baldur





I think that such types of DDoS are much easier to solve on a server 
with XDP/eBPF than on MX.
And much cheaper if we are talking about the new SYN+ACK DDoS and it is 
exactly 64b ddos case. I used multiple 82599.


From snabbco discussion, issue #1013, "If you read Intel datasheets then 
the minimum packet rate they are guaranteeing is 64B for 10G (82599), 
128B for 40G (XL710), and 256B for 100G (FM10K)."


But "hardware", ASIC enabled routers such as MX might be not better and 
even need some tuning.

https://kb.juniper.net/InfoCenter/index?page=content=KB33477=METADATA
"On summit MX204 and MX10003 platforms, the line rate frame size is 119 
byte for 10/40GbE port and 95 byte for 100GbE port."
or some QFX, for example, Broadcom Tomahawk 32x100G switches only do 
line-rate with >= 250B packets according to datasheets.

Re: Reflection DDoS last week

2019-08-28 Thread Denys Fedoryshchenko


On 2019-08-28 02:23, Damian Menscher via NANOG wrote:

On Wed, Aug 21, 2019 at 3:21 PM Töma Gavrichenkov 
wrote:


On Thu, Aug 22, 2019 at 12:17 AM Damian Menscher 
wrote:

Some additional questions, if you're able to answer them (off-list

is fine if there are things that can't be shared broadly):

- Was the attack referred to law enforcement?


It is being referred to now.  This would most probably get going
under
the jurisdiction of the Netherlands.


Deeper analysis and discussion indicates there were several victims:
we saw brief attacks targeting some of our cloud customers with
syn-ack peaks above 125 Mpps; another provider reported seeing 275Mpps
sustained.  So presumably there are a few law enforcement
investigations under way, in various jurisdictions.


- Were any transit providers asked to trace the
source of the spoofing to either stop the attack
or facilitate the law enforcement investigation?


No tracing the source was not deemed a high priority task.


Fair enough.  I just didn't want to duplicate effort.

The source of the spoofing has been traced.  The responsible hosting
provider has kicked off their problem customer, and is exploring the
necessary filtering to prevent a recurrence.

If anyone sees more of this style of attack please send up a flare so
the community knows to track down the new source.

Damian


One of my clients suffered from such attacks.
And you know what the secondary harm is? Typical false flag issue.
Even if you have decent DDoS protection setup, it is highly likely that 
involuntary reflectors administrators will not puzzle what to do with 
this, they will simply block your subnet/ASN.
For example attacker spoof hosting operator subnets, did SYN flood to 
all credit card processing gateways, and sure legit hosting gets 
SYN+ACK.
And this hosting after suffering to block this SYN+ACK reflection will 
find an unpleasant thing - not a single credit card processing gateway 
is available from his subnets.
Good example is EAGames, Rockstar, fs.com of those, who just set static 
ACL

Re: Reflection DDoS last week

2019-08-24 Thread Denys Fedoryshchenko

Hi,

Same happened in Lebanon(country). Similar pattern: carpet bombing for
multiple prefixes of specific ASN.
I suspect it is a new trend in DDoS-for-hire, and ISP who did not
install data scrubbing appliances will feel severe pain from such
attacks, since they use SYN + ACK from legit servers.

On 2019-08-21 22:44, Töma Gavrichenkov wrote:

Peace,

Here's to confirm that the pattern reported before in NANOG was indeed
a reflection DDoS attack. On Sunday, it also hit our customer, here's
the report:

https://www.prnewswire.com/news-releases/root-cause-analysis-and-incident-report-on-the-august-ddos-attack-300905405.html

tl;dr: basically that was a rather massive reflected SYN/ACK carpet
bombing against several datacenter prefixes (no particular target was
identified).

--
Töma

On Sat, Aug 17, 2019, 1:06 AM Jim Shankland
wrote:

Greetings,

I'm seeing slow-motion (a few per second, per IP/port pair) syn
flood
attacks ostensibly originating from 3 NL-based IP blocks:
88.208.0.0/18 [1]
, 5.11.80.0/21 [2], and 78.140.128.0/18 [3] ("ostensibly" because
... syn flood,
and BCP 38 not yet fully adopted).

Why is this syn flood different from all other syn floods? Well ...

1. Rate seems too slow to do any actual damage (is anybody really
bothered by a few bad SYN packets per second per service, at this
point?); but

2. IPs/port combinations with actual open services are being
targeted
(I'm seeing ports 22, 443, and 53, just at a glance, to specific IPs

with those services running), implying somebody checked for open
services first;

3. I'm seeing this in at least 2 locations, to addresses in
different,
completely unrelated ASes, implying it may be pretty widespread.

Is anybody else seeing the same thing? Any thoughts on what's going
on?
Or should I just be ignoring this and getting on with the weekend?

Jim

Links:
--
[1] http://88.208.0.0/18
[2] http://5.11.80.0/21
[3] http://78.140.128.0/18

Re: really amazon?

2019-07-31 Thread Denys Fedoryshchenko


On 2019-07-31 23:13, Scott Christopher wrote:

Valdis Klētnieks wrote:


On Wed, 31 Jul 2019 16:36:08 -, Richard Williams via NANOG said:

>  To contact AWS SES about spam or abuse the correct email address is 
ab...@amazonaws.com

You know that, and I know that, but why doesn't the person at AWS 
whose job it

is to keep the ARIN info correct and up to date know that?


Because it will get spammed if publicly listed in WHOIS.


They can send autoreply with correct address (even as picture, but yes, 
From: can be spoofed, so might be bad idea), make error message with 
link to captcha, custom error

in reject (e.g. web url to submit report), and etc.
So many ways to be more helpful in such critical matters.

But at least not "User not found".

Re: Colo in Africa

2019-07-18 Thread Denys Fedoryshchenko


Africa, Russia...

You can take as example Lebanon.
Capital and major city in tiny country, ~40km away from each other, and 
only way you can get 2 points connected over microwaves(due mountains - 
several hops), over "licensed" providers, DSP, who hook this points for 
$10-$30/mbps/month. And many of them don't have support at evenings and 
weekend. Of course, due crappy electricity in country and economical 
situation, discharged batteries and outages at evening/night at 
"licensed" DSP sites - common case.
The laws of the country are so cool, that it is even forbidden to lay 
optics from the building standing next to other building, unless you are 
government monopoly (and they don't sell fiber connectivity).


In Africa, many people do not have electricity at all and cook on open 
fire, i imagine what difficulties they have with connectivity.
The last time when I worked with a team on study to invest in telecom in 
Africa - results discouraged even trying to engage in telecom subject 
there.
I think the only ones who are interested in decent connectivity there - 
mobile operators. Maybe worth to find connections and talk to them.



On 2019-07-17 20:16, Mark Tinka wrote:

On 17/Jul/19 17:04, Rod Beck wrote:


The cross continent connectivity is not going to be particularly
reliable. Prone to cuts due to wars and regional turmoil. And
imagine how it takes to repair problems at the physical layer.


I think that view is too myopic... you make it sound like Namibia,
Botswana, Zimbabwe and Zambia are at war. Just like all other
continents, unrest exists in some states, not all of them.

For the regions the OP is interested in, there isn't any conflict
there that would prevent him from deploying network.

Terrestrial connectivity is not a viable solution because:

* It costs too much.
* Different countries (even direct neighbors) do not share social,
economic or political values.
* Most of the available network is in the hands of incumbents,
typically controlled by the gubbermint.
* It costs too much.
* There isn't sufficient capacity to drive prices down when crossing
2 or more countries.
* It costs too much.
* Many markets are closed off and it's impossible to obtain licenses
to compete.
* It costs too much.
* Much of the network is old and has barely been upgraded.
* It costs too much.
* For those bold enough to build, the terrain in some parts is not a
walkover.

* It costs too much.

Mark.

Re: Cost effective time servers

2019-06-21 Thread Denys Fedoryshchenko


On 2019-06-21 14:19, Niels Bakker wrote:

* j...@west.net (Jay Hennigan) [Fri 21 Jun 2019, 05:19 CEST]:

On 6/20/19 07:39, David Bass wrote:
What are folks using these days for smaller organizations, that need 
to dole out time from an internal source?


If you want to go really cheap and don't value your time, but do value 
knowing the correct time, a GPS receiver with a USB interface and a 
Raspberry Pi would do the trick.


Have you tried this?  Because I have, and it's absolutely terrible.
GPS doesn't give you the correct time, it's supposed to give you a
good 1pps clock discipline against which you can measure your device's
internal clock and adjust accordingly for drift due to it not being
Cesium-based, influenced by room temperature etc.

You're unlikely to get the 1pps signal across USB, and even then
there'll likely be significant latencies in the USB stack compared to
the serial interface that these setups traditionally use.


I think it depends on recipe you are using.
Raspberry have low latency GPIO, and some receivers have 1pps output.
https://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html

Re: Free Program to take netflow

2019-05-17 Thread Denys Fedoryshchenko

Fastnetmon have that: 
https://fastnetmon.com/fastnetmon-advanced-traffic-persistency/

I used it for such purposes.

On 2019-05-17 17:26, Dennis Burgess via NANOG wrote:

I am looking for a free program to take netflow and output what the
top traffic ASes to and from my AS are.   Something that we can look
at every once in a while, and/or spin up and get data then shutdown..
Just have two ports need netflow from currently.

Thanks in advance.

DENNIS BURGESS, MIKROTIK CERTIFIED TRAINER

Author of "Learn RouterOS- Second Edition"

LINK TECHNOLOGIES, INC -- Mikrotik & WISP Support Services

OFFICE: 314-735-0270  Website: http://www.linktechs.net [1]

Create Wireless Coverage's with www.towercoverage.com [2]



Links:
--
[1] http://www.linktechs.net/
[2] http://germany.nuclearcat.com/www.towercoverage.com

Re: BGP prefix filter list / BGP hijacks, different type

2019-05-17 Thread Denys Fedoryshchenko

I wanted to mention one additional important point in all these 
monitoring discussion.

Right now, for one of my subnets Google services stopped working.
Why? Because it seems like someone from Russia did BGP hijack, BUT, 
exclusively for google services (most likely some kind of peering).
Quite by chance, I noticed that the traceroute from the google cloud to 
this subnet goes through Russia, although my country has nothing to do 
with Russia at all, not even transit traffic through them.
Sure i mailed noc@google, but reaching someone in big companies is not 
easiest job, you need to search for some contact that answers. And good 
luck for realtime communications.
And, all large CDNs have their own "internet", although they have BGP, 
they often interpret it in their own way, which no one but them can 
monitor and keep history. No looking glass for sure, as well.
If your network is announced by a malicious party from another country, 
you will not even know about it, but your requests(actually answers from 
service) will go through this party.

Re: QFX5k question

2019-03-24 Thread Denys Fedoryshchenko


On 2019-03-24 00:32, Thomas Bellman wrote:

They do have limited feature set, though.  E.g, they only look at
the first 64 octets of each packet (and that includes L2 and L2.5
headers) when deciding what to do with a packet, and can't chase
the IPv6 header chain; thus, if there is an extension header before
the TCP/UDP header, they won't know what TCP/UDP ports are used,
or even if it is TCP, UDP or something else.  Dealing with packets
exiting tunnels (MPLS, VXLAN, et.c) is also limited.

Some declared features - do not work.
For example, IPIP termination through filters is claimed, but does not 
work.

https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/ipip-tunnel-services-filter-qfx-series.html
Perhaps "not implemented yet", possibly errata, nevertheless it is very 
unpleasant when you buy equipment and this is a key necessary function.
Therefore, if any more or less complex (uncommon) features are used, it 
is better to test them first.

Facebook dropping MSS on congestion

2019-03-20 Thread Denys Fedoryshchenko


Good day,

I am writing here, as in technical support ticket I will most likely end 
up to the outsourcing guys, who will try to write some formal reply and 
close the ticket quickly to keep KPI high:)
I have a faint hope that someone will read and listen. It may also be 
useful to colleagues.
I noticed at last few month, if some congestion occurs on the network 
(specific subnet), facebook reduces the maximum segment size (MSS), even 
down to 256 bytes. Purely academically, on paper - this will reduce 
latency.

In reality - it will cause avalanche effect.
If ISP have CGNAT, or some other appliances - with great probability 
they will encounter the fact that pps will increase 4-5 times, and might 
hit pps limit on hardware. Additionally, overhead on IP headers will 
increase significantly, especially on ipv6, and this will further 
aggravate the congestion.
Facebook don't do that, please. And thank you, if you listen to 
suggestions.


Denys

Re: Webzilla

2019-03-19 Thread Denys Fedoryshchenko


On 2019-03-18 23:24, Ronald F. Guilmette wrote:
In message 
,

Eric Kuhnke  wrote:

Looking at the AS adjacencies for Webzilla, what would prevent them 
from
disconnecting all of their US/Western Euro based peers and transits, 
and
remaining online behind a mixed selection of the largest Russian ASes? 
I do
not think that any amount of well-researched papers and appeals to 
ethical

ISPs on the NANOG mailing list will bring down those relationships.


In the early years of the 20th century, Vladimir Lenin, leader of the
Bolshevik, revolution, famously quipped to his communist collegues that
"The capitalists will sell us the rope to hang them with."  His 
prescient

words have endured even the fall of the empire he founded because they
clarify a simple and fundamental truth -- in capitalist systems, short
term greed often overrides both rationality and simple common sense.
My hope is that it will not be so on this occasion, and that enligtened
long-term self interest will prevail, at least among those companies 
that

are peering with any of Webzilla's ASNs.

Your speech is very reminiscent of this very Lenin, who climbed on an 
armored car and broadcasted speech to the "worker class" and told how 
bad are rich and how to restore justice.
Only instead of rich people you have "those pesky Russians", and instead 
of the working class - "Western democracies". But let's not get into 
politics too deep.
What prevents those who consider the activities of this hosting to be so 
harmful that they are worth blocking - to filter and add to the ACL 
lists of networks, where Webzilla AS is origin?
Or make some easy to use lists, API, BGP feed, and those who decide to 
participate will null-route offenders, and you will see how many people 
will support you.
If this list is compiled carefully, then I am sure it will interest 
many(including me). If it turns into a political tool or a tool for 
extortion ... then of course not.


And generally speaking, all these speeches from an armored cars end with 
a witch hunt, and almost always entire nations or categories of people 
are appointed as witches, depending on the trends.
Who will be next? Cloudflare? Their attempt to maintain neutrality 
annoys many.

Amazon? They react very slowly to abuse.
OVH? It seems they do not care about abuse at all.
Or maybe it will go into fashion to make the guilty - legal arms 
sellers? Or internet-stores who sell alcohol?
Just create a cause for a depeering, and a lot of people with their 
special views will demand a depeering at every opportunity.


P.S. North Korea, as far as I know, is very limited in connectivity 
choice, and this does not prevent them from creating a bunch of 
problems.
As Max Tulyev said, and they are good example, just sprayed through 
countless proxies.

Re: bloomberg on supermicro: sky is falling

2018-10-04 Thread Denys Fedoryshchenko


On 2018-10-04 23:37, Naslund, Steve wrote:

I was wondering about where this chip tapped into all of the data and
timing lines it would need to have access to.  It would seem that
being really small creates even more problems making those
connections.  I am a little doubtful about the article.  It would seem
to me better to create a corrupted copy of something like a front side
bus chipset, memory controller or some other component that handles
data lines than create a new component that would then require a
motherboard redesign to integrate correctly.  It would seem that as
soon as the motherboard design was changed someone would wonder "hey,
where are all those data lines going?"  It would also require less
people in on the plan to corrupt or replace a device already in the
design.  All you need is a way to intercept the original chip supply
and insert your rogue devices.

On the opposite side of the argument, does anyone think it is strange
that all of the companies mentioned in the article along with the PRC
managed to get a simultaneous response back to Bloomberg.  Seems
pretty pre-calculated to me.  Or did some agency somewhere tell
everyone they better shut up about the whole thing?

Steven Naslund
Chicago IL


Just theory - tapping on same lines as SPI flash (let's assume it is not 
QSPI), so we are "in parallel", as "snooper" chip.

First - it can easily snoop by listening MISO/MOSI/CS/CLK.
When required data pattern and block detected during snooping, it can 
remember offset(s) of required data.
When, later, BMC send over MOSI request for this "offset", we override 
BMC and force CS high (inactive), so main flash chip will not answer, 
and answer instead of him our, different data from "snooper".

Voila... instead of root:password we get root:nihao

Re: bloomberg on supermicro: sky is falling

2018-10-04 Thread Denys Fedoryshchenko


On 2018-10-04 21:52, Scott Weeks wrote:

--- matlock...@gmail.com wrote:
From: Ken Matlock 

Would be remiss in our duties if we didn't also link
AWS' blog, in response to the Bloomberg article.
--


Every company and the Chinese gov't is saying "no,
Bloomberg is wrong":

https://www.bloomberg.com/news/articles/2018-10-04/the-big-hack-amazon-apple-supermicro-and-beijing-respond

Can't wait to see how this evolves...

scott
It would be better for them(AMZN, SMCI, AAPL)  to prove that these 
events did not take place - in court.
In the opposite case, even if this article is full of inaccuracies, 
judging by the discussions of security specialists, the scenario 
indicated in the article is quite possible.
Unpopulated SOIC-8 near populated SOIC-16 flash for BMC firmware is 
sweet spot for custom MCU - snooping on flash SPI(most likely) bus and 
probably altering some data.
At the same time there will be a good precedent, if this article is 
fabricated - such journalists need to be taught a lesson.

And if they wont go to the court, there is something to think about.

Re: OpenDNS CGNAT Issues

2018-09-12 Thread Denys Fedoryshchenko




On 2018-09-12 19:40, Lee Howard wrote:

On 09/11/2018 09:31 AM, Matt Hoppes wrote:

So don't CGNat?  Buy IPv4 addresses at auction?


Buy IPv4 addresses until CGN is cheaper. If a customer has to call,
and you have to assign an IPv4 address, you have to recover the cost
of that call and address.
While ((CostOfCall + CostOfAddress)*NumberOfCalls) >
(CostOfAddress*NumberOfNewCustomers):
 BuyAddresses(NumberOfNewCustomers)

Meanwhile, deploy IPv6, and move toward IPv4aaS, probably 464xlat or
MAP, but your religion may vary. That way your "CGN" is an IPv6-IPv4
translator, and that's easier than managing dual-stack.

At the very least, dual-stack your web sites now, so the rest of us
can get to it without translation.



Just regarding ipv4 issue solution, this process can be somehow 
automated by detecting those who use opendns(by netflow, for example), 
to avoid "CostOfCall" part.
Also, to avoid false claiming of nat pool, he can nat DNS requests for 
OpenDNS to different ip pool, that cannot be claimed.

Re: Linux BNG

2018-07-15 Thread Denys Fedoryshchenko


On 2018-07-15 19:00, Raymond Burkholder wrote:

On 07/15/2018 09:03 AM, Denys Fedoryshchenko wrote:

On 2018-07-14 22:05, Baldur Norddahl wrote:
I have considered OpenFlow and might do that. We have OpenFlow 
capable
switches and I may be able to offload the work to the switch 
hardware.
But I also consider this solution harder to get right than the idea 
of

using Linux with tap devices. Also it appears the Openvswitch
implements a different flavour of OpenFlow than the hardware switch
(the hardware is limited to some fixed tables that Broadcom made up),
so I might not be able to start with the software and then move on to
hardware.


AFAIK openflow is suitable for datacenters, but doesn't scale well for 
users termination purposes.

You will run out from TCAM much sooner than you expect.


Denys, could you expand on this?  In a linux based solution (say with
OVS), TCAM is memory/software based, and in following their dev
threads, they have been optimizing flow caches continuously for
various types of flows: megaflow, tiny flows, flow quantity and
variety, caching, ...

When you mate OVS with something like a Mellanox Spectrum switch (via
SwitchDev) for hardware based forwarding, I could see certain hardware
limitations applying, but don't have first hand experience with that.

But I suppose you will see these TCAM issues on hardware only
specialized openflow switches.
Yes, definitely only on hardware switches and biggest issue it is 
vendor+hardware
dependent. This means if i find "right" switch, and make your solution 
depending on it,
and vendor decided to issue new revision, or even new firmware, there is 
no guarantee

"unusual" setup will keep working.
That what makes many people afraid to use it.

Openflow IMO by nature is built to do complex matching, and for example 
for
typical 12-tuple it is 750-4000 entries max in switches, but you go to 
l2 only matching
which was possible at moment i tested, on my experience, only on PF5820 
- you can do L2

entries only matching, then it can go 80k flows.
But again, sticking to specific vendor is not recommended.

About OVS, i didnt looked much at it, as i thought it is not suitable 
for BNG purposes,
like for tens of thousands users termination, i thought it is more about 
high speed

switching for tens of VM.




On edge based translations, is hardware based forwarding actually
necessary, since there are so many software functions being performed
anyway?
IMO at current moment 20-40G on single box is a boundary point when 
packet forwarding
is preferable(but still not necessary) to do in hardware, as passing 
packets

thru whole Linux stack is really not best option. But it works.
I'm trying to find an alternative solution, bypassing full stack using 
XDP,

so i can go beyond 40G.



But then, it may be conceivable that by buying a number of servers,
and load spreading across the servers will provide some resiliency and
will come in at a lower cost than putting in 'big iron' anyway.

Because then there are some additional benefits:  you can run Network
Function Virtualization at the edge and provide additional services to
customers.

+1

For IPoE/PPPoE - servers scale very well, while on "hardware" eventually 
you will
hit a limit how many line cards you can put in chassis and then you need 
to buy new chassis.
I am not talking that some chassis have countless unobvious limitations 
you might hit
inside chassis (in pretty old Cisco 6500/7600, which is not EOL, it is a 
nightmare).


If ISP have big enough chassis, he need to remember, that he need second 
one
at same place, and preferable with same amount of line cards, while with 
servers

you are more reliable even with N+M(where M for example N/4) redundancy.

Also when premium customers ask me for some unusual things, it is much 
easier
to move them to separate nodes with extended options for termination, 
where i can

implement their demands over custom vCPE.

Re: Linux BNG

2018-07-15 Thread Denys Fedoryshchenko


On 2018-07-14 22:05, Baldur Norddahl wrote:

I have considered OpenFlow and might do that. We have OpenFlow capable
switches and I may be able to offload the work to the switch hardware.
But I also consider this solution harder to get right than the idea of
using Linux with tap devices. Also it appears the Openvswitch
implements a different flavour of OpenFlow than the hardware switch
(the hardware is limited to some fixed tables that Broadcom made up),
so I might not be able to start with the software and then move on to
hardware.
AFAIK openflow is suitable for datacenters, but doesn't scale well for 
users termination purposes.

You will run out from TCAM much sooner than you expect.
Linux tap device has very high overhead, it suits no more than working 
as some hotspot gateway for 100s of users.


Regards,

Baldur

Re: Linux BNG

2018-07-15 Thread Denys Fedoryshchenko


On 2018-07-15 06:09, Jérôme Nicolle wrote:

Hi Baldur,

Le 14/07/2018 à 14:13, Baldur Norddahl a écrit :

I am investigating Linux as a BNG


As we say in France, it's like your trying to buttfuck flies (a local
saying standing for "reinventing the wheel for no practical reason").

You can say that about whole opensource ecosystem, why to bother, if
*proprietary solution name* exists. It is endless flamewar topic.



Linux' kernel networking stack is not made for this kind of job. 6WIND
or fd.io may be right on the spot, but it's still a lot of dark magic
for something that has been done over and over for the past 20 years by
most vendors.

And it just works.

Linux developers are working continuously to improve this, for example
latest feature, XDP, able to process several Mpps on <$1000 server.
Ask yourself, why Cloudflare "buttfuck flies" and doesn't buy some
proprietary vendor who 20 years does filtering in hardware?
https://blog.cloudflare.com/how-to-drop-10-million-packets/
I am doing experiments with XDP as well, to terminate PPPoE, and it is
doing that quite well over XDP.



DHCP (implying straight L2 from the CPE to the BNG) may be an option
bust most codebases are still young. PPP, on the other hand, is
field-tested for extremely large scale deployments with most vendors.

DHCP here, at least from RFC 2131 existence in March 1997.
Quite old, isn't it?
When you stick to PPPoE, you tie yourself with necessary layers of
encapsulation/decapsulation, and this is seriously degrading performance 
at
_user_ level at least. With some development experience of firmware for 
routers,

i can tell that hardware offloading of ipv4 routing (DHCP) obviousl
is much easier and cheaper, than offloading PPPoE encap/decap+ipv4 
routing.

Also Vendors keep screwing up their routers with PPP, and for example
one of them failed processing properly PADO in newest firmware revision.
Another problem, with PPPoE you subscribe to headache called reduced 
mtu, that also

will give a lot of unpleasant hours for ISP support.



If I were in you shooes, and I don't say I'd want to (my BNGs are 
scaled

to less than a few thousand of subscribers, with 1-4 concurrent session
each), I'd stick to plain old bitstream (PPP) model, with a decent
subscriber framework on my BNGs (I mostly use Juniper MXs, but I also
like Nokia's and Cisco's for some features).

I am consulting operators from few hundreds to hundreds of thousands.
It is very rare, when Linux bng doesn't suit them.



But let's say we would want to go forward and ditch legacy / 
proprietary

code to surf on the NFV bullshit-wave. What would you actually need ?

Linux does soft-recirculation at every encapsulation level by memory
copy. You can't scale anything with that. You need to streamline
decapsulation with 6wind's turborouter or fd.io frameworks. It'll cost
you a few thousand of man-hours to implement your first prototype.

6wind/fd.io is great solutions, but not suitable for mentioned task.
They are mostly created for very tailor made tasks or even as core of 
some
vendor solution. Implementing your BNG based on such frameworks, or 
DPDK, is really
reinventing the wheel, unless you will sell it or can save by that 
millions

of US$.



Let's say you got a woking framework to treat subsequent headers on the
fly (because decapsulation is not really needed, what you want is just
to forward the payload, right ?)… Well, you'd need to address
provisionning protocols on the same layers. Who would want to rebase a
DHCP server with alien packet forms incoming ? I gess no one.
accel-ppp does all that and exactly for IPoE termination, and no black 
magic

there.



Well, I could dissert on the topic for hours, because I've already 
spent

months to address such design issues in scalable ISP networks, and the
conclusion is :

- PPPoE is simple and proven. Its rigid structure alleviates most of 
the

dual-stack issues. It is well supported and largelly deployed.

PPPoE has VERY serious flaws.
1)Security of PPPoE sucks big time. Anybody who run rogue PPPoE server 
in your network
will create significant headache for you, while with DHCP you have at 
least "DHCP snooping".
DHCP snooping supported in very many vendors switches, while for PPPoE 
most of them

have nothing, except... you stick each user to his own vlan.
Why to pppox them then?
2)DHCP can send some circuit information in Option 82, this is very 
useful for

billing and very cost efficient on last stage of access switches.
3)Modern FTTX(GPON) solutions are built with QinQ in mind, so IPoE fits 
there flawlessly.


- DHCP requires hacks (in the form of undocummented options from 
several
vendors) to seemingly work on IPv4, but the multicast boundaries for 
NDP

are a PITA to handle, so no one implemented that properly yet. So it is
to avoid for now.
While you can do multicast(mostly for IPTV, yes it is not easy, and need 
some
vendor magic on "native" layer (DHCP), with PPP you can forget about 
multicast

entirely.

Re: Linux BNG

2018-07-14 Thread Denys Fedoryshchenko


On 2018-07-14 15:13, Baldur Norddahl wrote:

Hello

I am investigating Linux as a BNG. The BNG (Broadband Network Gateway)
being the thing that acts as default gateway for our customers.

The setup is one VLAN per customer. Because 4095 VLANs is not enough,
we have QinQ with double VLAN tagging on the customers. The customers
can use DHCP or static configuration. DHCP packets need to be option82
tagged and forwarded to a DHCP server. Every customer has one or more
static IP addresses.

IPv4 subnets need to be shared among multiple customers to conserve
address space. We are currently using /26 IPv4 subnets with 60
customers sharing the same default gateway and netmask. In Linux terms
this means 60 VLAN interfaces per bridge interface.

However Linux is not quite ready for the task. The primary problem
being that the system does not scale to thousands of VLAN interfaces.

We do not want customers to be able to send non routed packets
directly to each other (needs proxy arp). Also customers should not be
able to steal another customers IP address. We want to hard code the
relation between IP address and VLAN tagging. This can be implemented
using ebtables, but we are unsure that it could scale to thousands of
customers.

I am considering writing a small program or kernel module. This would
create two TAP devices (tap0 and tap1). Traffic received on tap0 with
VLAN tagging, will be stripped of VLAN tagging and delivered on tap1.
Traffic received on tap1 without VLAN tagging, will be tagged
according to a lookup table using the destination IP address and then
delivered on tap0. ARP and DHCP would need some special handling.

This would be completely stateless for the IPv4 implementation. The
IPv6 implementation would be harder, because Link Local addressing
needs to be supported and that can not be stateless. The customer CPE
will make up its own Link Local address based on its MAC address and
we do not know what that is in advance.

The goal is to support traffic of minimum of 10 Gbit/s per server.
Ideally I would have a server with 4x 10 Gbit/s interfaces combined
into two 20 Gbit/s channels using bonding (LACP). One channel each for
upstream and downstream (customer facing). The upstream would be layer
3 untagged and routed traffic to our transit routers.

I am looking for comments, ideas or alternatives. Right now I am
considering what kind of CPU would be best for this. Unless I take
steps to mitigate, the workload would probably go to one CPU core only
and be limited to things like CPU cache and PCI bus bandwidth.

accel-ppp supports IPoE termination for both IPv4 and IPv6, with radius 
and everything.
It is also done such way, that it will utilize multicore server 
efficiently (might need some tuning, depends on hardware).
It should handle 2x10G easily on decent server, about 4x10 it depends on 
your hardware and how well tuning are done.

Re: New Active Exploit: memcached on port 11211 UDP & TCP being exploited for reflection attacks

2018-02-28 Thread Denys Fedoryshchenko

I want to add one software vendor, who is major contributor to ddos 
attacks.
Mikrotik till now shipping their quite popular routers, with wide open 
DNS recursor,
that don't have even mechanism for ACL in it. Significant part of DNS 
amplification attacks

are such Mikrotik recursors.
They don't care till now.

On 2018-02-28 14:31, Job Snijders wrote:

Dear all,

Before the group takes on the pitchforks and torches and travels down 
to
the hosting providers' headquarters - let's take a step back and look 
at

the root of this issue: the memcached software has failed both the
Internet community and its own memcached users.

It is INSANE that memcached is/was[1] shipping with default settings
that make the daemon listen and respond on UDP on INADDR_ANY. Did 
nobody

take notes during the protocol wars where we were fodder for all the
CHARGEN & NTP ordnance?

The memcached software shipped with a crazy default that required no
authentication - allowing everyone to interact with the daemon. This is
an incredibly risky proposition for memcached users from a
confidentiality perspective; and on top of that the amplification 
factor

is up to 15,000x. WHAT?!

And this isn't even new information, open key/value stores have been a
security research topic for a number of years, these folks reported 
that

in the 2015/2016 time frame they observed more than 100,000 open
memcached instances: https://aperture-labs.org/pdf/safeconf16.pdf

Vendors need to ensure that a default installation of their software
does not pose an immediate liability to the user itself and those 
around

them. No software is deployed in a vacuum.

A great example of how to approach things is the behavior of the
PowerDNS DNS recursor: this recursor - out of the box - binds to only
127.0.0.1, and blocks queries from RFC 1918 space. An operator has to
consciously perform multiple steps to make it into the danger zone.
This is how things should be.

Kind regards,

Job

[1]:
https://github.com/memcached/memcached/commit/dbb7a8af90054bf4ef51f5814ef7ceb17d83d974

ps. promiscuous defaults are bad, mmkay?
Ask your BGP vendor for RFC 8212 support today! :-)

Re: Blockchain and Networking

2018-01-08 Thread Denys Fedoryshchenko

Each offsite copy of git repository will give alert then, as all
hashes in chain changed at some moment.
Same principle as blockchain.

On 2018-01-08 09:54, tglas...@earthlink.net wrote:

Uh since MITM Bill perk of custody is key.

//tsg

Sent from my HTC
- Reply message -
From: "Denys Fedoryshchenko" <de...@visp.net.lb>
To: <nanog@nanog.org>
Subject: Blockchain and Networking
Date: Mon, Jan 8, 2018 10:03

On 2018-01-08 08:59, Peter Kristolaitis wrote:

On 2018-01-08 12:52 AM, William Herrin wrote:

I'm having trouble envisioning a scenario where blockchain does

that >> any

better than plain old PKI.
>> Blockchain is great at proving chain of custody, but when do you

need >> to do

that in computer networking?
>> Regards,
Bill Herrin

> There's probably some potential in using a blockchain for things

like

configuration management.  You can authenticate who made what change
and when (granted, we can kinda-sorta do this already with the

various

authentication and logging mechanisms, but the blockchain is an
immutable, permanent record inherently required for the system to

work

at all).
> That immutable, sequenced chain of events would let you do things

like

"make my test environment look like production did last Thursday at
9AM" trivially by reading the blockchain up until that timestamp,

then

running a fork of the chain for the new test environment to track

its

own changes during testing.
> Or when you know you did something 2 months ago for client A, and

you

need your new NOC guy to now do it for client B -- the blockchain
becomes the documentation of what was done.
> We can build all of the above in other ways today, of course.  But
there's certainly something to be said for a vendor-supported

solution

that is inherent in the platform and requires no additional
infrastructure.  Whether or not that's worth the complexities of
managing a blockchain on networking devices is, perhaps, a whole

other

discussion.   :)
> - Peter

Why to reinvent git? :)
Lot of tools available also, to see diff on git commits, to see who
did commit, and what exactly he changed.
(it is possible to cryptographically sign commits, as well, and yes,
they are chain signed, as "blockchain")

Re: Blockchain and Networking

2018-01-07 Thread Denys Fedoryshchenko


On 2018-01-08 08:59, Peter Kristolaitis wrote:

On 2018-01-08 12:52 AM, William Herrin wrote:
I'm having trouble envisioning a scenario where blockchain does that 
any

better than plain old PKI.

Blockchain is great at proving chain of custody, but when do you need 
to do

that in computer networking?

Regards,
Bill Herrin


There's probably some potential in using a blockchain for things like
configuration management.  You can authenticate who made what change
and when (granted, we can kinda-sorta do this already with the various
authentication and logging mechanisms, but the blockchain is an
immutable, permanent record inherently required for the system to work
at all).

That immutable, sequenced chain of events would let you do things like
"make my test environment look like production did last Thursday at
9AM" trivially by reading the blockchain up until that timestamp, then
running a fork of the chain for the new test environment to track its
own changes during testing.

Or when you know you did something 2 months ago for client A, and you
need your new NOC guy to now do it for client B -- the blockchain
becomes the documentation of what was done.

We can build all of the above in other ways today, of course.  But
there's certainly something to be said for a vendor-supported solution
that is inherent in the platform and requires no additional
infrastructure.  Whether or not that's worth the complexities of
managing a blockchain on networking devices is, perhaps, a whole other
discussion.   :)

- Peter

Why to reinvent git? :)
Lot of tools available also, to see diff on git commits, to see who did 
commit, and what exactly he changed.
(it is possible to cryptographically sign commits, as well, and yes, 
they are chain signed, as "blockchain")

Re: Spectre/Meltdown impact on network devices

2018-01-07 Thread Denys Fedoryshchenko

AFAIK, Meltdown/Spectre require access to some proper programming 
language and ability to run attacker own code.
If underprivileged user can't spawn shell on device or run some python 
code - i guess you are safe.


I guess people need to push support of vendors, for equipment who has 
programming languages/shell, to release statement about possibility of 
vulnerability.
As fixing require significant changes in "memory" operation model, i 
doubt they will do such thing, i guess in best case they will restrict 
access to insert

code under nonprivileged users (if it is allowed now).
For example, even old Cisco IOS has TCL, but logically under level 15, 
so i assume it is safe.


On 2018-01-07 21:02, Jean | ddostest.me via NANOG wrote:

Hello,

I'm curious to hear the impact on network devices of this new hardware
flaws that everybody talk about. Yes, the Meltdown/Spectre flaws.

I know that some Arista devices seem to use AMD chips and some say that
they might be immune to one of these vulnerability. Still, it's 
possible

to spawn a bash shell in these and one with limited privileges could
maybe find some BGP/Ospf/SNMP passwords. Maybe it's also possible to
leak a full config.

I understand that one need access but still it could be possible for 
one

to social engineer a NOC user, hijack the account with limited access
and maybe run the "exploit".

I know it's a lot of "if" and "maybe", but still I'm curious what is 
the

status of big networking systems? Are they vulnerable?

Thanks

Jean

Re: Bandwidth distribution per ip


<>


Are you claiming that your bandwidth is being equally divided 1024
ways (you mentioned a /22) or just that each host (IP) is not
receiving the full bandwidth? What is the bandwidth ordered and what
is the bandwidth you're seeing per host(IP)?

Some facts from today.
Ordered capacity 3.3Gbit
Received capacity ~2.1Gbit if they apply bandwidth limit
In this example they removed limit, but you can have approximate picture 
how bandwidth is distributed (top 20 ips):


[x.x.x.14   ] 22433902b 36435p avg 615b 0.81%b 1.10%p 26596 Kbit/s
[x.x.x.13   ] 22715108b 34887p avg 651b 0.82%b 1.06%p 26929 Kbit/s
[x.x.x.10   ] 22741911b 31719p avg 716b 0.83%b 0.96%p 26961 Kbit/s
[x.x.x.11   ] 23874482b 34157p avg 698b 0.87%b 1.04%p 28304 Kbit/s
[x.x.x.15   ] 24393258b 29622p avg 823b 0.89%b 0.90%p 28919 Kbit/s
[x.x.x.12   ] 24715746b 33880p avg 729b 0.90%b 1.03%p 29301 Kbit/s
[x.x.x.9] 25720774b 36000p avg 714b 0.93%b 1.09%p 30492 Kbit/s
[x.x.x.8] 29599218b 40647p avg 728b 1.07%b 1.23%p 35090 Kbit/s
[y.y.y.122  ] 52015361b 52743p avg 986b 1.89%b 1.60%p 61666 Kbit/s
[y.y.y.116  ] 52161788b 55435p avg 940b 1.89%b 1.68%p 61839 Kbit/s
[y.y.y.114  ] 55409677b 56945p avg 973b 2.01%b 1.73%p 65690 Kbit/s
[y.y.y.120  ] 59971853b 59782p avg 1003b 2.18%b 1.81%p 71098 Kbit/s
[y.y.y.126  ] 60821991b 65184p avg 933b 2.21%b 1.98%p 72106 Kbit/s
[y.y.y.117  ] 61811624b 58374p avg 1058b 2.24%b 1.77%p 73279 Kbit/s
[y.y.y.113  ] 62492070b 63001p avg 991b 2.27%b 1.91%p 74086 Kbit/s
[y.y.y.119  ] 63128246b 63545p avg 993b 2.29%b 1.93%p 74840 Kbit/s
[y.y.y.121  ] 64392950b 66418p avg 969b 2.34%b 2.01%p 76340 Kbit/s
[y.y.y.115  ] 65723751b 64100p avg 1025b 2.39%b 1.94%p 77917 Kbit/s
[y.y.y.124  ] 66646572b 62637p avg 1064b 2.42%b 1.90%p 79011 Kbit/s
[y.y.y.123  ] 70332553b 68284p avg 1030b 2.55%b 2.07%p 83381 Kbit/s
[y.y.y.125  ] 70545386b 67441p avg 1046b 2.56%b 2.04%p 83634 Kbit/s
[y.y.y.118  ] 71393238b 69490p avg 1027b 2.59%b 2.11%p 84639 Kbit/s
[x.x.x.6] 123028709b 137530p avg 894b 4.47%b 4.17%p 145855 Kbit/s
[x.x.x.4] 124816100b 137221p avg 909b 4.53%b 4.16%p 147974 Kbit/s
[x.x.x.7] 126130939b 143443p avg 879b 4.58%b 4.35%p 149532 Kbit/s
[x.x.x.3] 128316371b 139360p avg 920b 4.66%b 4.22%p 152123 Kbit/s
[x.x.x.0] 132445418b 143143p avg 925b 4.81%b 4.34%p 157018 Kbit/s
[x.x.x.1] 133197094b 143713p avg 926b 4.84%b 4.35%p 157910 Kbit/s
[x.x.x.2] 135346483b 146510p avg 923b 4.91%b 4.44%p 160458 Kbit/s
[x.x.x.5] 135366769b 147766p avg 916b 4.92%b 4.48%p 160482 Kbit/s
Average packet size 834 (with ethernet header, max avg sz 1514)
Time 6748, total bytes 2753819139, total speed 3188235 Kbit/s

As you can see max single ip takes is 4.48% of bandwidth.
Also i cannot waste ipv4 for larger pools, just because of some deadly
flawed equipment/configuration.

Re: Bandwidth distribution per ip


On 2017-12-20 19:12, Saku Ytti wrote:
On 20 December 2017 at 19:04, Denys Fedoryshchenko <de...@visp.net.lb> 
wrote:


As person who is in love with embedded systems development, i just 
watched
today beautiful 10s of meters long 199x machine, where multi kW VFDs 
manage
huge motors(not steppers), dragging synchronously and printing on thin 
paper
with crazy speed and all they have is long ~9600 link between a bunch 
of

encoders
and PLC dinosaur managing all this beauty. If any of them will apply a 
bit

wrong
torque, stretched paper will rip apart.
In fact nothing complex there, and technology is ancient these days.
Engineers who cannot synchronize and update few virtual "subinstances"
policing ratio based on feedback, in one tiny, expensive box, with
reasonable
update ratio, having in hands modern technologies, maybe incompetent?


As appealing it is to say everyone, present company excluded, is
incompetent, I think explanation is more complex than that. Solution
has to be economic and marketable. I think elephant flow detection and
unequal mapping of hash result to physical interface is economic and
marketable solution, but it needs that extra level of abstraction,
i.e. you cannot just backport it via software if hardware is missing
that sort of capability.

Even highly incompetent in such matters person as me, know, that some of
modern architecture challenges is when NPU consists of a large number of
"processing cores", each having his own counters, and additionally it 
might

be also multiple NPU handling same customer traffic. On such conditions
updating _precise_ counters(for bitrate measurements, for example) is 
not
trivial anymore as sum = a(1) + .. + a(n), due synchronization, shared 
resources

access and etc.
But still it's solvable in most of cases, even dead wrong way of running 
script
and changing policer value on each "unit" once per second mostly solve 
problem.
And if architecturally some NPU cannot do such job, it means they are 
flawed,
and should be avoided for specific tasks, same as some BCM chipset 
switches
with claimed 32k macs, but choking from 20k macs, because of 8192 
entries tcam and
probably imperfect hash + linear probing on collision. Sure such switch 
is not

suitable for aggregation and termination.
Still, i am running some dedicated servers on colo in EU/US, some over 
10G(bonding),
and _single_ ip on server, i never faced such balancing issues, thats 
why i am asking,
if someone had such carrier, who require to balance bandwidth between 
many ips,

with quite high precision, to not lose expensive bandwidth.

Re: Bandwidth distribution per ip


On 2017-12-20 19:16, Blake Hudson wrote:

Denys Fedoryshchenko wrote on 12/20/2017 8:55 AM:
National operator here ask customers to distribute bandwidth between 
all ip's equally, e.g. if i have /22, and i have in it CDN from one of 
the big content providers, this CDN use only 3 ips for ingress 
bandwidth, so bandwidth distribution is not equal between ips and i am 
not able to use all my bandwidth.


And for me, it sounds like faulty aggregation + shaping setup, for 
example, i heard once if i do policing on some models of Cisco switch, 
on an aggregated interface, if it has 4 interfaces it will install 25% 
policer on each interface and if hashing is done by dst ip only, i 
will face such issue, but that is old and cheap model, as i recall.


Did anybody in the world face such requirements?
Is such requirements can be considered as legit?


Not being able to use all of your bandwidth is a common issue if you
are provided a bonded connection (aka Link Aggregation Group). For
example, you are provided a 4Gbps service over 4x1Gbps ethernet links.
Ethernet traffic is not typically balanced across links per frame,
because this could lead to out of order delivery or jitter, especially
in cases where the links have different physical characteristics.
Instead, a hashing algorithm is typically used to distribute traffic
based on flows. This results in each flow having consistent packet
order and latency characteristics, but does force a flow over a single
link, resulting in the flow being limited to the performance of that
link. In this context, flows can be based on src/dst MAC address, IP
address, or TCP/UDP port information, depending on the traffic type
(some IP traffic is not TCP/UDP and won't have a port) and equipment
type (layer 3 devices typically hash by layer 3 or 4 info).

Your operator may be able to choose an alternative hashing algorithm
that could work better for you (hashing based on layer 4 information
instead of layer 3 or 2, for example). This is highly dependent on
your provider's equipment and configuration - it may be a global
option on the equipment or may not be an option at all. Bottom line,
if you expected 4Gbps performance for each host on your network,
you're unlikely to get it on service delivered through 4x 1Gbps links.
10Gbps+ links between you and your ISP's peers would better serve
those needs (any 1Gbps bonds in the path between you and your
provider's edge are likely to exhibit the same characteristics).

--Blake


No bonding to me, usually it is dedicated 1G/10G/etc link.
Also i simulated this bandwidth for "hashability", and any layer4 aware 
hashing

on cisco/juniper provided perfectly balanced bandwidth distribution.
On my tests i can see that they have some balancing clearly by dst ip 
only.

Re: Bandwidth distribution per ip


On 2017-12-20 17:52, Saku Ytti wrote:
On 20 December 2017 at 16:55, Denys Fedoryshchenko <de...@visp.net.lb> 
wrote:


And for me, it sounds like faulty aggregation + shaping setup, for 
example,

i heard once if i do policing on some models of Cisco switch, on an
aggregated interface, if it has 4 interfaces it will install 25% 
policer on

each interface and if hashing is done by dst ip only, i will face such
issue, but that is old and cheap model, as i recall.


One such old and cheap model is ASR9k trident, typhoon and tomahawk.

It's actually pretty demanding problem, as technically two linecards
or even just ports sitting on two different NPU might as well be
different routers, they don't have good way to communicate to each
other on BW use. So N policer being installed as N/member_count per
link is very typical.

ECMP is fact of life, and even thought none if any provider document
that they have per-flow limitations which are lower than nominal rate
of connection you purchases, these do exist almost universally
everywhere. People who are most likely to see these limits are people
who tunnel everything, so that everything from their say 10Gbps is
single flow, from POV of the network.
In IPv6 world at least tunnel encap end could write hash to IPv6 flow
label, allowing core potentially to balance tunneled traffic, unless
tunnel itself guarantees order.

I don't think it's fair for operator to demand equal bandwidth per IP,
but you will expose yourself to more problems if you do not have
sufficient entropy. We are slowly getting solutions to this, Juniper
Trio and BRCM Tomahawk3 can detect elephant flows and dynamically
unequally map hash results to physical ports to alleviate the problem.
As person who is in love with embedded systems development, i just 
watched
today beautiful 10s of meters long 199x machine, where multi kW VFDs 
manage
huge motors(not steppers), dragging synchronously and printing on thin 
paper
with crazy speed and all they have is long ~9600 link between a bunch of 
encoders
and PLC dinosaur managing all this beauty. If any of them will apply a 
bit wrong

torque, stretched paper will rip apart.
In fact nothing complex there, and technology is ancient these days.
Engineers who cannot synchronize and update few virtual "subinstances"
policing ratio based on feedback, in one tiny, expensive box, with 
reasonable

update ratio, having in hands modern technologies, maybe incompetent?

National operator doesn't provide IPv6, that's one of the problems.
In most of cases there is no tunnels, but imperfection still exist.
When ISP pays ~$150k/month (bandwidth very expensive here), and because
CDN has 3 units & 3 ingress ips, and carrier have bonding somewhere over
4 links, it just means ~$37.5k is lost in rough estimations,
no sane person will accept that.
Sometimes one CDN unit are in maintenance, and 2 existing can perfectly
serve demand, but because of this "balancing" issues - it just go down,
as half of capacity missing.

But, tunnels in rare cases true too, but what we can do, if they don't 
have

reasonable DDoS protection tools all world have (not even blackholing).
Many DDoS-protection operators charge extra for more tunnel endpoints 
with
balancing, and this balancing is not so equal as well (same src+dst ip 
at best).

And when i did round-robin on my own solution, i noticed except
this "bandwidth distribution" issue, latency on each ip is unequal,
so RR create for me "out of order" issues.
Another problem, most popular services in region (in matters of 
bandwidth)
is facebook, whatsapp, youtube. Most of them is big fat flows running 
over

few ips. I doubt i can convince them to balance over more.

Bandwidth distribution per ip

National operator here ask customers to distribute bandwidth between all 
ip's equally, e.g. if i have /22, and i have in it CDN from one of the 
big content providers, this CDN use only 3 ips for ingress bandwidth, so 
bandwidth distribution is not equal between ips and i am not able to use 
all my bandwidth.


And for me, it sounds like faulty aggregation + shaping setup, for 
example, i heard once if i do policing on some models of Cisco switch, 
on an aggregated interface, if it has 4 interfaces it will install 25% 
policer on each interface and if hashing is done by dst ip only, i will 
face such issue, but that is old and cheap model, as i recall.


Did anybody in the world face such requirements?
Is such requirements can be considered as legit?

Broadcom chipset limitations Was: Switch/Router

2017-12-12 Thread Denys Fedoryshchenko


What are those limitations?

I started to be afraid from those, because just hit recently nasty hash 
collision issue with EX4550,
with declared 32k mac's it badly choked on 28k macs, and even magic 
"mac-lookup-length" didn't helped.


I'm considering EX4600, but afraid from it and that possibly other 
juniper models have hash collision issues too.
(As said on KB32325, affected models EX3200, EX3300, EX4200, EX4500, 
EX4550, and EX6210 have 8192 hash table FDB entries, but it might be 
outdated)
And till now vendor didn't make troubleshooting of this problem easier, 
when you upgrade from 12.x to 15.x, you get dead RE hitting load average 
30+,

and you can't even ssh to switch.

On 2017-12-12 17:37, Chuck Anderson wrote:

Juniper MX150, except only single PS.  But they are cheap enough you
could buy two.  Upside: most of the MX feature set is available
because it is vMX (software) inside.

QFX5110 is more expensive but has more ports and dual PS.  Downside:
Broadcom chipset limitations.

On Tue, Dec 12, 2017 at 09:47:17AM -0500, K MEKKAOUI wrote:

Hi



I am looking for a router preferably (or switch) with the following 
specs:


1-  Carrier grade

2-  Dual power supply

3-  1RU

4-  Gig and 10Gig interfaces.

5-  Does support protocols like BGP, etc.



Any recommendation please? Your help will be appreciated.

Re: PCIe adapters supporting long distance 10GB fiber?


On 2017-06-20 23:44, Baldur Norddahl wrote:
But what foundation do you have for asserting that switch hardware is 
any

different in this regard? I can say that we are using 80 km modules in
various hardware without any issues. I admittedly do not use any high 
power

modules in servers, but I will need better evidence than this to assume
that it would not work just fine.
For switches i guess it is same story as for PoE on them - total power 
budget matters.
So if you will pack whole EX4500 with 10G 80km SFP+ it might have 
problems as well,
but for normal use, and if few only are "long distance/high power", at 
any case 3.3V supply rail
by design in switch should handle many SFP, so if there is 48 ports, it 
should handle by specs at least 72W peak load.
It might be multiple power rails for groups of ports, but still, much 
better than just 750mA on network card.
But that's just guessing, i never seen circuit diagrams of good 
switches, or at least reference design,

as it is all NDA material.




Den 20. jun. 2017 22.24 skrev "Denys Fedoryshchenko" 
<de...@visp.net.lb>:


On 2017-06-20 22:07, Baldur Norddahl wrote:

I would expect anything mounted in a computer to have all the power 
you
could want. It is not like the ATX power supply cares about an extra 
watt

or two.

As I understand the issue it is more about cooling than power and is
primarly a concern in high density switches were you could have 48 or 
more

to power and cool.

SFP needs 3.3V, it might be supplied from regulator on the card or 
directly

PCI-Express,
can't be absolutely sure, in reference design it is just 3.3V_NIA and 
then

filter,
also reference design SFP power circuit define max 750mA/3.3V max to 
SFP,

thats only 2.475W.

FTLX1471D3BCV (10km SM) - up to 285mA
FTLX1671D3BCL (40km SM) - up to 400mA, and thumb rule in electronics it 
is

better to not exceed 50%
of max specs of designed max current, as for many parts it is stated 
for

25C & etc operating conditions.

I expect it might work, but noone knows how long, and how reliable, if 
it

is not cooled very well.
And 82599 sensitive to cooling(it is very old card after all), as soon 
as

it is not enough, it starts to glitch.





Den 20. jun. 2017 18.09 skrev "Denys Fedoryshchenko" 
<de...@visp.net.lb>:


I guess it depends on NIC, there is many spinoffs of Intel X520 with 
much

weaker power supply circuitry.
It might work with good NIC, but you can't rely on it on long term, 
IMHO.
Even 40km Finisar SFP+ has Pdiss 1.5W. Also they mention: "The 
typical

power consumption of the FTLX1672D3BTL may exceed the limit of 1.5W
specified for the Power Level II transceivers"
If we talk about 80km, Pdiss is 1.8W.
While 10GBASE-LR is <1W

On 2017-06-20 16:30, Max Tulyev wrote:

We use Intel NICs with SFP+ holes. It works good with long and short

range SFP+ modules, including CWDM/DWDM.

On 15.06.17 12:10, chiel wrote:

Hello,


We are deploying more and more server based routers (based on BSD). 
We
have now come to the point where we need to have 10GB uplinks one 
these
devices and I prefer to plug in a long range 10GB fiber straight 
into
the server without it going first into a router/switch from vendor 
x. It

seems to me that all the 10GB PCIe cards only support either copper
10GBASE-T, short range 10GBASE-SR or the 10 Km 10GBASE-LR (but only 
very
few). Are there any PCIe cards that support 10GBASE-ER and 
10GBASE-ZR? I

can't seem to find any.

Chiel

Re: PCIe adapters supporting long distance 10GB fiber?


On 2017-06-20 22:07, Baldur Norddahl wrote:

I would expect anything mounted in a computer to have all the power you
could want. It is not like the ATX power supply cares about an extra 
watt

or two.

As I understand the issue it is more about cooling than power and is
primarly a concern in high density switches were you could have 48 or 
more

to power and cool.
SFP needs 3.3V, it might be supplied from regulator on the card or 
directly PCI-Express,
can't be absolutely sure, in reference design it is just 3.3V_NIA and 
then filter,
also reference design SFP power circuit define max 750mA/3.3V max to 
SFP, thats only 2.475W.


FTLX1471D3BCV (10km SM) - up to 285mA
FTLX1671D3BCL (40km SM) - up to 400mA, and thumb rule in electronics it 
is better to not exceed 50%
of max specs of designed max current, as for many parts it is stated for 
25C & etc operating conditions.


I expect it might work, but noone knows how long, and how reliable, if 
it is not cooled very well.
And 82599 sensitive to cooling(it is very old card after all), as soon 
as it is not enough, it starts to glitch.





Den 20. jun. 2017 18.09 skrev "Denys Fedoryshchenko" 
<de...@visp.net.lb>:


I guess it depends on NIC, there is many spinoffs of Intel X520 with 
much

weaker power supply circuitry.
It might work with good NIC, but you can't rely on it on long term, 
IMHO.

Even 40km Finisar SFP+ has Pdiss 1.5W. Also they mention: "The typical
power consumption of the FTLX1672D3BTL may exceed the limit of 1.5W
specified for the Power Level II transceivers"
If we talk about 80km, Pdiss is 1.8W.
While 10GBASE-LR is <1W

On 2017-06-20 16:30, Max Tulyev wrote:


We use Intel NICs with SFP+ holes. It works good with long and short
range SFP+ modules, including CWDM/DWDM.

On 15.06.17 12:10, chiel wrote:


Hello,

We are deploying more and more server based routers (based on BSD). 
We
have now come to the point where we need to have 10GB uplinks one 
these
devices and I prefer to plug in a long range 10GB fiber straight 
into
the server without it going first into a router/switch from vendor 
x. It

seems to me that all the 10GB PCIe cards only support either copper
10GBASE-T, short range 10GBASE-SR or the 10 Km 10GBASE-LR (but only 
very
few). Are there any PCIe cards that support 10GBASE-ER and 
10GBASE-ZR? I

can't seem to find any.

Chiel

Re: PCIe adapters supporting long distance 10GB fiber?


On 2017-06-20 18:59, Hunter Fuller wrote:

On Tue, Jun 20, 2017 at 10:29 AM Chris Adams  wrote:


For Linux at least, the standard driver includes a load-time option to
disable vendor check.  Just add "options ixgbe 
allow_unsupported_sfp=1"

to your module config and it works just fine.



For anyone who may be going down this road, if you have a two-port 
Intel
NIC, I discovered you have to pass "allow_unsupported_sfp=1,1" or it 
will

only apply to the first port. Hope that helps someone.
Also it wont work with X710, you need to do NVRAM hack for it, SFP are 
checked in firmware.

Re: PCIe adapters supporting long distance 10GB fiber?

I guess it depends on NIC, there is many spinoffs of Intel X520 with 
much weaker power supply circuitry.
It might work with good NIC, but you can't rely on it on long term, 
IMHO. Even 40km Finisar SFP+ has Pdiss 1.5W. Also they mention: "The 
typical power consumption of the FTLX1672D3BTL may exceed the limit of 
1.5W specified for the Power Level II transceivers"

If we talk about 80km, Pdiss is 1.8W.
While 10GBASE-LR is <1W

On 2017-06-20 16:30, Max Tulyev wrote:

We use Intel NICs with SFP+ holes. It works good with long and short
range SFP+ modules, including CWDM/DWDM.

On 15.06.17 12:10, chiel wrote:

Hello,

We are deploying more and more server based routers (based on BSD). We
have now come to the point where we need to have 10GB uplinks one 
these

devices and I prefer to plug in a long range 10GB fiber straight into
the server without it going first into a router/switch from vendor x. 
It

seems to me that all the 10GB PCIe cards only support either copper
10GBASE-T, short range 10GBASE-SR or the 10 Km 10GBASE-LR (but only 
very
few). Are there any PCIe cards that support 10GBASE-ER and 10GBASE-ZR? 
I

can't seem to find any.

Chiel

Re: Russian diplomats lingering near fiber optic cables

2017-06-02 Thread Denys Fedoryshchenko


On 2017-06-02 12:19, Ben McGinnes wrote:

On Fri, Jun 02, 2017 at 10:28:38AM +0300, Denys Fedoryshchenko wrote:


American diplomats are doing also all sort of nasty stuff in
Russia(and not only),


Yes they have and for a very long time.


but that's a concern of the equivalent of FBI/NSA/etc, not operators
public discussion places, unless it really affect operators anyhow.
Just amazing, how NANOG slipped into pure politics.


The network(s) have been political for a very long time and will only
become more so as time passes.  Remember, the engineers wishing for
the purity of technical discussion are usually the same ones crying
that, "information wants to be free."

https://www.nanog.org/list
6. Postings of political, philosophical, and legal nature are 
prohibited.

It is quite clear.

I do not deny networks sometimes are deeply affected by political 
factors,
but current discussion is pure FUD, based on very questionable MSM 
source.
IMHO any sane person wont like to receive this trash in his mailbox in 
list,
that supposed to be politics-free, as there is enough of this garbage in 
internet.
I do discuss such things too, when i have mood for that, but in 
designated places only.




Well, no matter.  You want purely technical, okay, let's start with
authorised mail hosts.

You need to add 144.76.183.226/32 to the SPF record for visp.net.lb,
which is currently triggering softfails everywhere.  It might be wise
to explicitly state whether or not it is just 144.76.183.226/32 in the
SPF record for nuclearcat.com given the deny all instruction for that
domain.
Thanks for the hint, fixed, i use this domain only for old maillist 
subscriptions,

so i missed that, after i migrated SMTP to my private server.

Re: Russian diplomats lingering near fiber optic cables

2017-06-02 Thread Denys Fedoryshchenko


On 2017-06-02 05:42, Ben McGinnes wrote:

On Thu, Jun 01, 2017 at 07:15:12PM -0700, Joe Hamelin wrote:


The Seattle Russian Embassy is in the Westin Building just 4 floors
above the fiber meet-me-room and five floors above the NRO tap room.
They use to come ask us (an ISP) for IT help back in '96 when they
would drag an icon too far off the screen in Windows 3.11. We were
on the same floor.


So when Flynn & Friends in the Trump Transition Team were trying to
establish that back channel link to Vladimir Putin they should've just
wandered into the nearest colo facility ... okay, then.  I guess they
did it the other way because they wanted the trench coats.


Regards,
Ben
American diplomats are doing also all sort of nasty stuff in Russia(and 
not only),

but that's a concern of the equivalent of FBI/NSA/etc, not operators
public discussion places, unless it really affect operators anyhow.
Just amazing, how NANOG slipped into pure politics.

Re: difference with caching when connected to telia internet

2017-03-17 Thread Denys Fedoryshchenko


On 2017-03-17 18:04, Aaron Gould wrote:
Thanks, but James, you would not believe how rapidly the traffic to my 
local
caches drop off, *and* on the same day I brought up my new Telia 
internet
connection.  ...and furthermore, my internet inbound traffic went 
*through

the roof*

-Aaron
Most probably they silently readvertise BGP announces of customers to 
their caches.
Maybe just announce less specific prefixes to Telia, and more specific 
to caches?

Also at least GGC has BGP communities for priorities.

Re: Recent NTP pool traffic increase (update)

2016-12-21 Thread Denys Fedoryshchenko


Hello,

I'm not sure i should continue to CC nanog, if someone interested to be 
in CC for further updates this story please let me know.


TP-Link not related, it was misunderstanding or wrong customer report.

Tenda routers i believe most of cheap models are affected by this 
problem.
On ISPs i have access i see too many of them sending requests to 
133.100.9.2 (if it is unreachable, repeating each 10 seconds), this 
particular ip seems hardcoded there. I am sure as soon as your server is 
down, way they coded - it will make all this routers to ddos this 
particular ip, repeating NTP queries very often without any 
backoff/protection mechanism.
Particular model i tested - W308R / Wireless N300 Home Router, but i 
believe many models are affected.

Firmware: System Version: 5.0.7.53_en hw version : v1.0

Another possible vendor is LB-Link, but i dont have yet any info from 
customers who own them.


On 2016-12-21 11:00, FUJIMURA Sho wrote:

Hello.

I'm Sho FUJIMURA.
Thank you for your information.
I operate the public NTP Services as 133.100.9.2 and 133.100.11.8.
I'd like to reduce the traffic because I have trouble with too much
traffic recently.
So, I'm interested in the root of the the problem.
If possible, would you please tell me the model numbers of Tenda and
TP-Link??

--
Sho FUJIMURA
Information Technology Center, Fukuoka University.
8-19-1, Nanakuma, Jyonan-ku, Fukuoka, 8140180, Japan

fujim...@fukuoka-u.ac.jp

2016-12-20 5:33 GMT+09:00 Denys Fedoryshchenko <de...@visp.net.lb>:


I'm not sure if this issue relevant to discussed topic, Tenda
routers here for a while on market, and i think i noticed this issue
just now,
because NTP servers they are using supposedly for healthcheck went
down (or NTP owners blocked ISP's i support, due such routers).

At least after checking numerous users, i believe Tenda hardcoded
those NTP IPs. What worsen issue, that in Lebanon several times per
day, for example at 18pm - short electricity cutoff,
and majority of users routers will reboot and surely reconnect, so
it will look like a countrywide spike in NTP traffic.

I checked for a 10min also this NTP ips in dns responses, none of
thousands of users tried to resolve any name with them over any DNS
server, so i conclude they are hardcoded somewhere in firmware.

Here is traffic of Tenda router after reconnecting (but not full
powercycle, i dont have it in my hands). But as you can see, no DNS
resolution attempts:

20:15:59.305739 PPPoE  [ses 0x1483] CHAP, Success (0x03), id 1, Msg
S=XX M=Authentication succeeded
20:15:59.306100 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
length 12
20:15:59.317840 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 1,
length 24
20:15:59.317841 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 1,
length 12
20:15:59.317867 PPPoE  [ses 0x1483] IPCP, Conf-Nack (0x03), id 1,
length 18
20:15:59.325253 PPPoE  [ses 0x1483] IPCP, Conf-Request (0x01), id 2,
length 24
20:15:59.325273 PPPoE  [ses 0x1483] IPCP, Conf-Ack (0x02), id 2,
length 24
20:15:59.335589 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
133.100.9.2.123: NTPv3, Client, length 48
20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
192.5.41.41.123: NTPv3, Client, length 48
20:15:59.335588 PPPoE  [ses 0x1483] IP 172.17.49.245.123 >
192.5.41.40.123: NTPv3, Client, length 48

Here is example of Tenda traffic if it is unable to reach
destination, it repeats request each 10 seconds endlessly, my guess
they are using ntp to show
status of internet connection.
So, now that NTP servers getting quite significant DDoS such way.

19:57:52.162863 IP (tos 0x0, ttl 64, id 38515, offset 0, flags
[none], proto UDP (17), length 76)
172.16.31.67.123 > 192.5.41.40.123: [udp sum ok] NTPv3, length
48
Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
0 (1s), precision 0
Root Delay: 0.00, Root dispersion: 0.00,
Reference-ID: (unspec)
Reference Timestamp:  0.0
Originator Timestamp: 0.0
Receive Timestamp:0.0
Transmit Timestamp:   3691177063.0 (2016/12/19
22:57:43)
Originator - Receive Timestamp:  0.0
Originator - Transmit Timestamp: 3691177063.0
(2016/12/19 22:57:43)
19:57:52.163277 IP (tos 0x0, ttl 64, id 38516, offset 0, flags
[none], proto UDP (17), length 76)
172.16.31.67.123 > 192.5.41.41.123: [udp sum ok] NTPv3, length
48
Client, Leap indicator:  (0), Stratum 0 (unspecified), poll
0 (1s), precision 0
Root Delay: 0.00, Root dispersion: 0.00,
Reference-ID: (unspec)
Reference Timestamp:  0.0
Originator Timestamp: 0.0
Receive Timestamp:0.0
Transmit Timestamp:   3691177063.0 (2016/12/19
22:57:43)
Originator - Receive Timestamp:  0.0
Originator - Transmit Timestamp: 3691177063.0
(2016/12/19 22:57:43)
19:57:52.164435 IP (tos 0x0, ttl 64, id 38517, offset 0, flags
[none], proto UDP (17), length 76)
172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length
48
Client, Leap indicator:  (0)

Re: Recent NTP pool traffic increase (update)

2016-12-19 Thread Denys Fedoryshchenko

	Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), 
precision 0

Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 0.0
  Receive Timestamp:0.0
  Transmit Timestamp:   3691177073.0 (2016/12/19 22:57:53)
Originator - Receive Timestamp:  0.0
	Originator - Transmit Timestamp: 3691177073.0 (2016/12/19 
22:57:53)
19:58:02.165061 IP (tos 0x0, ttl 64, id 38520, offset 0, flags [none], 
proto UDP (17), length 76)

172.16.31.67.123 > 133.100.9.2.123: [udp sum ok] NTPv3, length 48
	Client, Leap indicator:  (0), Stratum 0 (unspecified), poll 0 (1s), 
precision 0

Root Delay: 0.00, Root dispersion: 0.00, Reference-ID: (unspec)
  Reference Timestamp:  0.0
  Originator Timestamp: 0.0
  Receive Timestamp:0.0
  Transmit Timestamp:   3691177073.0 (2016/12/19 22:57:53)
Originator - Receive Timestamp:  0.0
	Originator - Transmit Timestamp: 3691177073.0 (2016/12/19 
22:57:53)



On 2016-12-19 21:40, Roland Dobbins wrote:

On 20 Dec 2016, at 2:22, Denys Fedoryshchenko wrote:


If it is necessary i can investigate further.


Yes, please!

---
Roland Dobbins <rdobb...@arbor.net>

Re: Recent NTP pool traffic increase (update)

2016-12-19 Thread Denys Fedoryshchenko

Many sorry! Update, seems illiterate in english (worse than me, hehe) 
customer was not precise about model of router, while he reported issue.

I noticed now many customers using specific models of routers reported 
issues with internet connection.
Analyzing internet traffic, i noticed that this routers seems 
excessively requesting ntp from those ip addresses, and not trying 
others:

 > 192.5.41.40.123: NTPv3, Client, length 48
 > 192.5.41.41.123: NTPv3, Client, length 48
 > 133.100.9.2.123: NTPv3, Client, length 48

I'm asking customer to make photo of device, to retrieve model and 
revision, and checking other customers as well, if they are abusing same 
servers.
There is definitely pattern, that all of them are using just this 3 
hardcoded servers. Problem is that many customers are changing mac of 
router, so i cannot clearly

identify vendor by first mac nibbles.
He sent me 2 photos, one of them LB-Link (mac vendor lookup 20:f4:1b 
says Shenzhen Bilian electronic CO.,LTD), another is Tenda (c8:3a:35 is 
Tenda).

If it is necessary i can investigate further.

On 2016-12-19 20:33, Ca By wrote:
My WAG is that the one plus updated firmeware on that day and they 
baked in

the pool.

Complete WAG, but time and distributed sources including wireless 
networks

On Mon, Dec 19, 2016 at 10:30 AM Laurent Dumont 

wrote:

I also have a similar experience with an increased load.

I'm running a pretty basic Linode VPS and I had to fine tune a few

things in order to deal with the increased traffic. I can clearly see 
a

date around the 14-15 where my traffic increases to 3-4 times the 
usual

amounts.

I did a quick dump and in 60 seconds I was hit by slightly over 190K 
IPs

http://i.imgur.com/mygYINk.png

Weird stuff

Laurent

On 12/17/2016 10:25 PM, Gary E. Miller wrote:

> Yo All!

>

> On Sat, 17 Dec 2016 17:54:55 -0800

> "Gary E. Miller"  wrote:

>

>> # tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"

>>

>> And I do indeed get odd results.  Some on my local network...

> To follow up on my own post, so this can be promply laid to rest.

>

> After some discussion at NTPsec.  It seems that chronyd takes a lot

> of 'creative license' with RFC 5905 (NTPv4).  But it is not malicious,

> just 'odd', and not new.

>

> So, nothing see here, back to the hunt for the real cause of the new

> NTP traffic.

>

> RGDS

> GARY

>
---

> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703

>   g...@rellim.com  Tel:+1 541 382 8588

Re: Recent NTP pool traffic increase

2016-12-19 Thread Denys Fedoryshchenko

I noticed now many customers using tp-links reported issues with 
internet connection.
Analyzing internet traffic, i noticed that tp-link seems excessively 
requesting ntp from those ip addresses, and not trying others:


 > 192.5.41.40.123: NTPv3, Client, length 48
 > 192.5.41.41.123: NTPv3, Client, length 48
 > 133.100.9.2.123: NTPv3, Client, length 48

I'm asking customer to make photo of device, to retrieve model and 
revision, and checking other customers as well, if they are abusing same 
servers.


On 2016-12-19 20:33, Ca By wrote:
My WAG is that the one plus updated firmeware on that day and they 
baked in

the pool.

Complete WAG, but time and distributed sources including wireless 
networks



On Mon, Dec 19, 2016 at 10:30 AM Laurent Dumont 


wrote:


I also have a similar experience with an increased load.



I'm running a pretty basic Linode VPS and I had to fine tune a few

things in order to deal with the increased traffic. I can clearly see 
a


date around the 14-15 where my traffic increases to 3-4 times the 
usual


amounts.



I did a quick dump and in 60 seconds I was hit by slightly over 190K 
IPs




http://i.imgur.com/mygYINk.png



Weird stuff



Laurent





On 12/17/2016 10:25 PM, Gary E. Miller wrote:

> Yo All!

>

> On Sat, 17 Dec 2016 17:54:55 -0800

> "Gary E. Miller"  wrote:

>

>> # tcpdump -nvvi eth0 port 123 |grep "Originator - Transmit Timestamp:"

>>

>> And I do indeed get odd results.  Some on my local network...

> To follow up on my own post, so this can be promply laid to rest.

>

> After some discussion at NTPsec.  It seems that chronyd takes a lot

> of 'creative license' with RFC 5905 (NTPv4).  But it is not malicious,

> just 'odd', and not new.

>

> So, nothing see here, back to the hunt for the real cause of the new

> NTP traffic.

>

> RGDS

> GARY

>
---

> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703

>   g...@rellim.com  Tel:+1 541 382 8588

Re: Arista unqualified SFP

2016-08-18 Thread Denys Fedoryshchenko


No, this driver patch (or similar) wont work on new model.
But honestly, on my experience, X520 perform still better than 710 
series on 10G links.


https://sourceforge.net/p/e1000/mailman/message/34991760/
From: Wesley W. Terpstra <wesley@te...> - 2016-04-03 14:03:52
He did unlocked by modifying NVM in card(and i guess losing warranty, 
for sure)
Somehow it is even better, because X520 needed modification of driver, 
and that is not possible on "blackbox" software solutions using them.


On 2016-08-18 15:55, Mike Hammett wrote:

https://sourceforge.net/p/e1000/mailman/message/28698959/

That or similar doesn't work for that model?




-
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

- Original Message -----

From: "Denys Fedoryshchenko" <de...@visp.net.lb>
To: "Mike Hammett" <na...@ics-il.net>
Cc: "NANOG Mailing List" <nanog@nanog.org>
Sent: Thursday, August 18, 2016 7:51:13 AM
Subject: Re: Arista unqualified SFP

Not a case with Intel X*710 new chipset, check is in firmware.
Someone hacked it, but ...

On 2016-08-18 15:41, Mike Hammett wrote:

Intel does allow DAC of any vendor (assuming they properly identify as
DACs. You can also disable Intel's check in the Linux drivers.




-
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

- Original Message -

From: "Mikael Abrahamsson" <swm...@swm.pp.se>
To: "Mark Tinka" <mark.ti...@seacom.mu>
Cc: "nanog list" <nanog@nanog.org>
Sent: Thursday, August 18, 2016 7:32:55 AM
Subject: Re: Arista unqualified SFP

On Thu, 18 Aug 2016, Mark Tinka wrote:


All other vendors, explicitly or silently, adopt the same approach.


I've heard from people running Intel NICs and HP switches, that this
can't
be turned off there. You run into very interesting problems when 
you're

trying to use DAC cables between multi vendor.

Any pointers to how to turn this of on Intel NICs and HP switches?

Re: Arista unqualified SFP

2016-08-18 Thread Denys Fedoryshchenko


Not a case with Intel X*710 new chipset, check is in firmware.
Someone hacked it, but ...

On 2016-08-18 15:41, Mike Hammett wrote:

Intel does allow DAC of any vendor (assuming they properly identify as
DACs. You can also disable Intel's check in the Linux drivers.




-
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

- Original Message -

From: "Mikael Abrahamsson" 
To: "Mark Tinka" 
Cc: "nanog list" 
Sent: Thursday, August 18, 2016 7:32:55 AM
Subject: Re: Arista unqualified SFP

On Thu, 18 Aug 2016, Mark Tinka wrote:


All other vendors, explicitly or silently, adopt the same approach.


I've heard from people running Intel NICs and HP switches, that this 
can't

be turned off there. You run into very interesting problems when you're
trying to use DAC cables between multi vendor.

Any pointers to how to turn this of on Intel NICs and HP switches?

Re: Arista unqualified SFP

2016-08-18 Thread Denys Fedoryshchenko

Same here, i was considering Arista, because they are quite cost 
effective,feature rich, interesting hardware for developing some custom 
solutions. But no more, after reading about unreasonable vendor lock-in.
But such inflexibility are very bad sign, this "openness" looks like 
marketing only, under the hood it seems worse than other solutions on 
market. Also when support shows such inflexibility, it is very bad sign. 
And very sad.



On 2016-08-18 14:29, Dovid Bender wrote:

And I was about to jump on to the Arista train.

Regards,

Dovid

-Original Message-
From: Stanislaw 
Sender: "NANOG" Date: Thu, 18 Aug 2016 
13:24:05

To: nanog list
Subject: Re: Arista unqualified SFP

Hi all,
If somebody is following my epic adventure of getting uqualified SFP to
work on Aristas, here is the unhappy end of it.

I've written to Arista support and got the following dialogue:
Support guy:
Hi,
Thank you for contacting Arista Support. My name is  and I'll be
assisting you on this case.
Could you please provide the "show version" output from this switch?

Me:
Hi,
Here it is:


Support guy:
Hi,
Thank you for the information.
Unfortunately, we are unable to activate your 3rd party components. To
ensure ongoing quality, Arista devices are designed to support only
properly qualified transceivers.
Please let me know if you have any other questions.

Me:
I do not understand,
But there is a command which allows using non-Arista transceivers. Why
have you implemented it but don't provide an access key to your
customers when they ask for it?
If it is required to sign some papers which declare that I am aware of
all the risks and losing my warranty - I agree with that, lets do it.
Any way what are the conditions to receive that access key?

Support guy:
I'm afraid that there is nothing I'm able to do regarding this
situation. If you have any other questions regarding enabling 3rd party
options in Arista switches, I suggest to contact your local account 
team

(or sales) for further discussion on this matter.


Next, i've tried inserting various QSFP+ DAC cables I have - none of
them has been even detected on the switch, it was acting like nothing
has been inserted. I guess that even if I get the key, most of my
transceivers/DAC (which work like a champ in Juniper or Extreme
switches) cables wouldnt work.

I'm writing this post to make somebody who considers buying their
switches be aware of what they'd get. Just buy Juniper instead.


Stanislaw wrote at 2016-08-17 23:25:

Hi Tim,

Thanks for your expressive answer. Will try it :)

Tim Jackson писал 2016-08-17 22:57:


I'd suggest bitching and moaning at your account team & support until
they give you the key to unlock them..

--
Tim

On Wed, Aug 17, 2016 at 2:50 PM, Stanislaw  wrote:


Hi all,
Is there a way for unlocking off-brand transceivers usage on Arista
switches?

I've got an Arista 7050QX switch with 4.14 EOS version. Then it has
been found out that Arista switches seem to not have possibility to
unlock off-brand xcievers usage (by some service command or so).

I've patched /usr/lib/python2.7/site-packages/XcvrAgent.py, made the
checking function bypass the actual check and it helped: ports are
not in errdisable state anymore. But despite of xceivers are 
detected

correctly, links aren't coming up (they are in notconnect state).

If anyone possibly have does have a sacred knowledge of bringing
off-branded transceivers to life on Arista switches, your help'd be
very appreciated. Thanks.

Re: OT - Small DNS appliances for remote offices.

2015-02-19 Thread Denys Fedoryshchenko


On 2015-02-19 18:26, valdis.kletni...@vt.edu wrote:

On Thu, 19 Feb 2015 14:52:42 +, David Reader said:


I'm using several to connect sensors, actuators, and such to a private
network, which it's great for - but I'd think at least twice before 
deploying
one as a public-serving host in user-experience-critical role in a 
remote

location.


I have a Pi that's found a purpose in life as a remote smokeping sensor 
and

related network monitoring, a task it does quite nicely.

Note that they just released the Pi 2, which goes from the original 
single-core
ARM V6 to a quad-core ARM V7, and increases memory from 256M to1G. All 
at the
same price point.  That may change the calculus. I admit not having 
gotten one

in hand to play with yet.

Weird thing - it still has Ethernet over ugly USB 2.0
That kills any interest to run it for any serious networking 
applications.


---
Best regards,
Denys

Re: OT - Small DNS appliances for remote offices.

2015-02-19 Thread Denys Fedoryshchenko

Beaglebone has gigabit mac, but due some errata it is not used in 
gigabit mode, it is 100M (which is maybe enough for small office). But 
it is hardware mac.

Another hardware MAC on inexpensive board it is Odroid-C1.
But stability of all this boards in heavy networking use is under 
question, i didn't tested them yet intensively for same purpose.


On 2015-02-19 02:27, Geoff Mulligan wrote:

The BeagleBone Black uses flash memory to hold the system image which
allows it to boot quickly.  I'm running Ubuntu Trusty 14.04 and it
seems stable.

Geoff

*--
Presidential Innovation Fellow | The White House*

On 02/18/2015 05:20 PM, Bacon Zombie wrote:

You also have to watch out for issues with the Pi corrupting SD cards.
On 19 Feb 2015 01:04, Geoff Mulligan nano...@mulligan.org wrote:

I have used the BeagleBone to run a few simple servers.  I don't know 
if

the ethernet port on the Bone is on the USB bus. It is slightly more
expensive than a PI, but they have worked well for me.

 Geoff

On 02/18/2015 04:44 PM, Peter Loron wrote:

For any site where you would use a Pi as the DNS cache, it won't be 
an

issue. DNS isn't that heavy at those query rates.

Yeah, it would be awesome if they'd been able to get a SoC that 
included

ethernet.

-Pete

On 2015-02-18 15:08, Robert Webb wrote:

What I do not like about the Pi is the network port is on the USB 
bus

and thus limited to USB speeds.

div Original message /divdivFrom: Maxwell 
Cole

mcole.mailingli...@gmail.com /divdivDate:02/18/2015  4:30 PM
(GMT-05:00) /divdivTo: nanog@nanog.org  'NANOG list'
nanog@nanog.org /divdivSubject: Re: OT - Small DNS 
appliances

for remote offices. /divdiv
/div



---
Best regards,
Denys

Re: OT - Small DNS appliances for remote offices.

2015-02-19 Thread Denys Fedoryshchenko


On 2015-02-19 15:13, Rob Seastrom wrote:

Denys Fedoryshchenko de...@visp.net.lb writes:


Beaglebone has gigabit mac, but due some errata it is not used in
gigabit mode, it is 100M (which is maybe enough for small office). But
it is hardware mac.


The Beaglebone Black rev C BOM calls out the ethernet phy chip as
LAN8710A-EZC-TR which is 10/100 so there's your constraint.  The MAC
is built into the SoC and according to the datasheet the AM3358B is
10/100/1000.


Another hardware MAC on inexpensive board it is Odroid-C1.


Difficulty: hardware MAC tells you nothing about how it's connected,
either on the board or internally in the SoC.  Ethernet on Multibus
and Ethernet on PCIe (neither likely on an embedded ARM ;-) are both
hardware MAC yet the bus-constrained bandwidths will differ by
several orders of magnitude.

-r
Well, i guess for DNS it wont matter much(400Mbit or full capacity). But 
stability of driverand archievable pps rate on it,
due poor code - can be a question. And mostly this products are Network 
enabled, but networking are very lightly
used, not as it is used on appliances, 24/7 traffic, sometimes 
malicious.

About Beaglebone, probably reason is this errata:
While the AM335x GP EVM has a Gb Ethernet PHY, AR8031A, on the base 
board,
the PCB was designed to use internal clock delay mode of the RGMII 
interface and
the AM335x does not support the internal clock delay mode. Therefore, if 
operating
the Ethernet in Gb mode, there may be problems with the 
performance/function due
to this. The AR8031A PHY supports internal delay mode. This can be 
enabled by
software to guarantee Gb operation. However, this cannot be done to 
enable

internal delay mode for Ethernet booting of course. 
Or maybe they just put 100Mbit PHY to make BOM cost less.

As far as i know, Raspberry PI ethernet over USB might be fine for DNS 
too, but before it had issues with

large data transfers (ethernet driver hangs). No idea about now.


---
Best regards,
Denys

Re: DDOS, IDS, RTBH, and Rate limiting

2014-11-22 Thread Denys Fedoryshchenko


On 2014-11-22 18:00, freed...@freedman.net wrote:

 Cisco ASRs and MXs with inline jflow can do hundreds of K flows/second
 without affecting packet forwarding.

Yes, i agree,those are good for netflow, but when they already exist 
in

network.

Does it worth to buy ASR, if L3 switch already doing the job
(BGP/ACL/rate-limit/routing)?


Not suggesting that anyone should change out their gear though per my 
other
message, I've seen SPAN make things go wonky on almost every vendor 
that

ISPs use for switching.

Well, i always try to stay on safe side. Additionally, sure, i do
mirror for RX only, RX+TX often can exceed interface rate too :)




Well, if it is available, except hardware limitations, there is second
obstacle, software licensing cost. On latest JunOS, for example on 
EX2200,
you need to purchase license (EFL), and if am not wrong it is $3000 
for

48port units.

So if only sFlow feature is on stake, it worth to think, to purchase 
license,
or to purchase server. Prices for JFlow license on MX, just for 5/10G 
is way

above cost of very decent server.


I believe that smaller MXs can run it for free.  Larger providers we've
worked with often have magic cookies they can call in to get it 
enabled,
but I understand you're talking about the smaller-provider (or at least 
~

10gig per POP across multiple POPs) case.

We see a lot of Brocade for switching in hosting providers, which makes
sFlow easy, of course.
Oh, Brocade, recent experience with ServerIron taught me new lesson, 
that i can't
do bonding on ports as i want, it has limitations about even/odd port 
numbers and

etc.
Most amazing part i just forgot, that i have this ServerIron, and it is 
a place where
i run DDoS protection (but it works perfectly over tap way). Thanks 
for reminding

about this vendor :)




 And with the right setup you can run FastNetMon or other tools in
 addition to generating flow that can be of use for other purposes
 as well...

Technically there is ipt_NETFLOW, that can generate netflow on same 
box,
for statistical/telemetry purposes. But i am not sure it is possible 
to

run them together.


At frac 10gig you can just open pcap on a 10gig interface on a Linux
box getting a tap, of course.

What we did was use myricom cards and the myri_snf drivers and take 
from

the single-consumer ring buffers into large in-RAM ring buffers, and
make those ring buffers available via LD_PRELOAD or cli tools to allow
flow, snort, p0f, tcpdump, etc to all be run at the same time at 10gig.

The key for that is not going through the kernel IP stack, though.
Ntop's pf_ring, which is basically same idea, but can run on Intel 
cards.
Just maybe because never had myricom in hands, and it is difficult to 
obtain

them here.




 But taps can be difficult or at least time consuming for people to
 put in at scale.  Even, we've seen, for folks with 10G networks.
 Often because they can get 90% of what they need for 4 different
 business purposes from just flow :)

About scaling, i guess it depends on proper deployment strategy and
sysadmins/developers capabilities. For example to deploy new ruleset
for my pcap-based homemade analyser to 150 probes across the country 
-

is just one click.


Sounds cool.  You should write up that use case.  Hopefully you've 
secured

the metadata/command push channel well enough :)
For servers it is ssh with key authentication, and push system doesn't 
contain private key, it is forwarded
over ssh agent from developer pc. Sure, it is better also to sign by 
assymmetric crypto update also,

keep keys on smartcard, but in this case it is not necessary.


---
Best regards,
Denys

Re: DDOS, IDS, RTBH, and Rate limiting


On 2014-11-21 03:12, Roland Dobbins wrote:

On 21 Nov 2014, at 6:22, Denys Fedoryshchenko wrote:


Netflow is stateful stuff,


This is factually incorrect; NetFlow flows are unidirectional in
nature, and in any event have no effect on processing of data-plane
traffic.
Word stateful has nothing common with stateful firewall.Stateful 
protocol. a protocol which requires keeping of the internal state on 
the server is known as a stateful protocol. And sure 
unidirectional/bidirectional is totally unrelated.




and just to run it on wirespeed, on hardware, you need to utilise 
significant part of TCAM,


Again, this is factually incorrect.

http://en.wikipedia.org/wiki/NetFlow#NetFlow_support
Proof, that majority of solutions runs *flow not in software.

Cisco 65xx (yes, they are obsolete, but they run stuff wirespeed)
Aug 24 12:30:53: %EARL_NETFLOW-SP-4-TCAM_THRLD: Netflow TCAM threshold 
exceeded, TCAM Utilization [97%]
This is best example. Also on many Cisco's if you use UBRL, then you 
cannot use NetFlow, just because they use same part of TCAM resources. 
Others, for example Juniper, are using sampling (read - missing data), 
just to not overflow resources, and has various limitations, such as 
RE-DPC communication pps limit, licensing limit.
For example MS-DPC is pretty good one, few million flows in hardware, 
7-8Gbps of traffic, and... cost $12.


i am not talking that on some hardware it is just impossible to run 
it.


This is also factually incorrect.  Some platforms/linecards do not in
fact support NetFlow (or other varieties of flow telemetry) due to
hardware limitations.

But still they can run fine mirroring, and fastnetmon will do it's job.



And last thing, from one of public papers, netflow delaying factors:
1. Flow record expiration


This is tunable.
In certain limits. You can't set flow-active-timeout less than 60 
seconds in Junos 14 for example.
On some platforms even if you can, you just run in the limits of 
platforms again (forwarding - management communications).



• Typical delay: 15-60 sec.


This is an entirely subjective assessment, and does not reflect
operational realities.  These are typically *maximum values* - and
they are well within operationally-useful timeframes.  Also, the
effect of NetFlow cache size and resultant FIFOing of flow records is
not taken into account, nor is the effect on flow termination and
flow-record export of TCP FIN or RST flags denoting TCP traffic taken
into account.

So for a small hosting(up to 10G), i believe, FastNetMon is best 
solution.


This is a gross over-generalization unsupported by facts.  Many years
of operational experience with NetFlow and other forms of flow
telemetry by large numbers of network operators of all sizes and
varieties contract this over-generalization.

Fastnetmon and similar tools popularity says for itself.


It is generally unwise to make sweeping statements regarding
operational impact which are not borne out by significant operational
experience in production networks.
What can be asserted without evidence can be dismissed without 
evidence.





Faster, and no significant investments to equipment.


This statement indicates a lack of understanding of opex costs,
irrespective of capex costs.
Sweet marketing buzzwords, that is used together with some unclear 
calculations,
to sell suffering hosting providers various expensive tools, that is not 
necessary for them.
OPEX of fastnetmon is a small fee for qualified sysadmin, and often not 
required,

because already hosting operator should have him.



Bigger hosting providers might reuse their existing servers, segment 
the network, and implement inexpensive monitoring on aggregation 
switches without any additional cost again.


This statement indicates a lack of operational experience in networks
of even minimal scale.

Ah, and there is one more huge problem with netflow vs FastNetMon - 
netflow just by design cannot be adapted to run pattern matching, 
while it is trivial to patch FastNetMon for that, turning it to 
mini-IDS for free.


This statement betrays a lack of understanding of NetFlow-based (and
other flow telemetry-based) detection and classification, as well as
the undesirability and negative operational impact of stateful
IDS/'IPS' deployments in production networks.

You should also note that FastNetMon is far from unique; there are
multiple other open-source tools which provide the same type of
functionality, and none of them have replaced flow telemetry, either.
Thats a power of opensource. Since FastNetMon is not only tool, worth to 
mention others,
people here will benefit from using it, for free. And i'm sure, author 
of FastNetMon will

not feel offended at all.



Tools such as FastNetMon supplement flow telemetry, in situations in
which such tools can be deployed.  They do not begin to replace flow
telemetry, and they are not inherently superior to flow telemetry.

Again, I'm sure FastNetMon is a useful tool in many circumstances

Re: DDOS, IDS, RTBH, and Rate limiting


On 2014-11-21 06:45, freed...@freedman.net wrote:
Netflow is stateful stuff, and just to run it on wirespeed, on 
hardware,

you need to utilise significant part of TCAM,


Cisco ASRs and MXs with inline jflow can do hundreds of K flows/second
without affecting packet forwarding.
Yes, i agree,those are good for netflow, but when they already exist in 
network.
Does it worth to buy ASR, if L3 switch already doing the job 
(BGP/ACL/rate-limit/routing)?


i am not talking that on some hardware it is just impossible to run 
it.
So everything about netflow are built on assumption that hosting or 
ISP

can run it. And based on some observations, majority of small/middle
hosting providers are using minimal,just BGP capable L3 switch as 
core,
and cheapest but reliable L2/L3 on aggregation, and both are capable 
in

best case to run sampled sFlow.


Actually, sFlow from many vendors is pretty good (per your points about 
flow

burstiness and delays), and is good enough for dDoS detection.  Not for
security forensics, or billing at 99.99% accuracy, but good enough for
traffic visibility, peering analytics, and (d)DoS detection.
Well, if it is available, except hardware limitations, there is second 
obstacle,
software licensing cost. On latest JunOS, for example on EX2200, you 
need
to purchase license (EFL), and if am not wrong it is $3000 for 48port 
units.
So if only sFlow feature is on stake, it worth to think, to purchase 
license,
or to purchase server. Prices for JFlow license on MX, just for 5/10G is 
way above cost

of very decent server.



snip


So for a small hosting(up to 10G), i believe, FastNetMon is best
solution. Faster, and no significant investments to equipment. Bigger
hosting providers might reuse their existing servers, segment the
network, and implement inexpensive monitoring on aggregation switches
without any additional cost again.


It can be useful to have a 10G network monitoring box of course...

And with the right setup you can run FastNetMon or other tools in
addition to generating flow that can be of use for other purposes
as well...
Technically there is ipt_NETFLOW, that can generate netflow on same box, 
for
statistical/telemetry purposes. But i am not sure it is possible to run 
them

together.




Ah, and there is one more huge problem with netflow vs FastNetMon -
netflow just by design cannot be adapted to run pattern matching, 
while

it is trivial to patch FastNetMon for that, turning it to mini-IDS for
free.


It's true, having a network tap can be useful for doing PCAP-y stuff.

But taps can be difficult or at least time consuming for people to
put in at scale.  Even, we've seen, for folks with 10G networks.
Often because they can get 90% of what they need for 4 different
business purposes from just flow :)
About scaling, i guess it depends on proper deployment strategy and 
sysadmins/developers
capabilities. For example to deploy new ruleset for my pcap-based 
homemade

analyser to 150 probes across the country - is just one click.


---
Best regards,
Denys

Re: DDOS, IDS, RTBH, and Rate limiting


On 2014-11-21 14:50, Roland Dobbins wrote:

On 21 Nov 2014, at 15:17, Denys Fedoryshchenko wrote:

Word stateful has nothing common with stateful firewall.Stateful 
protocol. a protocol which requires keeping of the internal state on 
the server is known as a stateful protocol.


Correct - and NetFlow is not stateful, by this definition.

Not stateful, if you pick on server word.
To be able to make bytes/packets accounting for a flow, you need to keep 
this specific flow previous state. To be able to differentiate between 
flows with same src/dst ip+ports (if one is ended, next is started with 
same data) you need to track it's state, again. And just to keep track 
of _flows_ in packet switched network you need states. Surprising lack 
of knowledge.





And sure unidirectional/bidirectional is totally unrelated.


On the contrary, it is quite relevant.


Cisco 65xx (yes, they are obsolete, but they run stuff wirespeed)


They are not obsolete - they perform very well with Sup2T and
EARL8-based linecards.
Seems yes, i'm wrong on that point, i was not successful to run netflow 
reliable way , but it was before CSCul90377 and CSCui17732 fixed.





Others, for example Juniper, are using sampling (read - missing data),


The largest networks in the world use sampled NetFlow every hour of
every day for many purposes, including DDoS
detection/classification/traceback.  It works quite well for all those
purposes.
Use case of fastnetmon is not largest networks. Sampled netflow is 
useless for per-traffic billing purpose for example.




just to not overflow resources, and has various limitations, such as 
RE-DPC communication pps limit, licensing limit.
For example MS-DPC is pretty good one, few million flows in hardware, 
7-8Gbps of traffic, and... cost $12.


You get what you pay for.
While i can pay $1500 for a server, and get netflow and ~3second BGP 
blackholing with fastnetmon.




But still they can run fine mirroring, and fastnetmon will do it's 
job.


On the contrary - SPAN nee port mirroring cuts into the
frames-per-second budget of linecards, as the traffic is in essence
being duplicated.  It is not 'free', and it has a profound impact on
the the switch's data-plane traffic forwarding capacity.

Unlike NetFlow.
In hosting case mirroring usually done for uplink port, but i have to 
agree, it might be a problem.



Yes, it does - they are far less popular that NetFlow, because they do
not scale on networks of any size, nor do they provide traceback
(given your lack of comments on traceback elsewhere in this thread, it
appears that you aren't familiar with this concept).
You make my point very well, thank you.  There is overwhelming
evidence that NetFlow and similar forms of flow telemetry scale well
and provide real, measurable, actionable operational value on networks
of all types and sizes.  The reason for the popularity of flow
telemetry is that it is low-opex (no probes to deply); low-capex (no
probes to deploy); scales to tb/sec speeds; is practicable for large
networks (no probes to deploy); provides instantaneous traceback
(probes can't do this); and provides statistics on dropped traffic
(probes can't do this, either).
And again and again we are going to tb/s. I don't need TB/s, i dont need 
traceback,nor on relatively small ISP nor on VDS provider i dont need 
all that above. I just need inexpensive way to block attacked ip and/or 
announce it from different location within minimal timeframe, to 
minimize impact on other customers.
You might be highly professional with large scale operators, but small 
guys needs and capabilities are very different.
I had developed tool similar to fastnetmon for almost same purpose, 
detecting attacks and switching affected network by BGP to protected 
backbone. After calculating OPEX/CAPEX, capable server turned to be 
much cheaper alternative in short and long term than buying netflow 
capable hardware (and support for it) just for netflow purposes, and 
buying hardware for netflow collector.

Let's talk numbers.
My case is small hosting, 4G, C4948-10G, one 10G uplink, one 10G port is 
free. Switch is not capable to run sFlow or Netflow.
Decent server is available already, since it is hosting company, so the 
only expenses are 10G 82599 card, which is around $500. Even in case 
server is not available, based on data from fastnetmon author still 
total cost is within $1500. Deployment time - hours from installing 
hardware, without distrupting existing traffic.
Major expenses - tuning server according author recommendations, and 
writing shell script that will send to 4948 command to blackhope IP. For 
qualified sysadmin it is 2 hours of work, and $500 max as a labor 
cost. Thats it. What can be cheaper than $2000 in this case? I guess i 
wont get answer.



I'm uninterested in selling anyone anything.  What I'm interested in
doing is correcting the misinformation you are promulgating regarding
the utility of flow telemetry coupled with open-source flow

Re: DDOS, IDS, RTBH, and Rate limiting

On 2014-11-21 18:41, Peter Phaal wrote:
Actually, sFlow from many vendors is pretty good (per your points
about

flow
burstiness and delays), and is good enough for dDoS detection. Not
for
security forensics, or billing at 99.99% accuracy, but good enough
for

traffic visibility, peering analytics, and (d)DoS detection.

Well, if it is available, except hardware limitations, there is second
obstacle,
software licensing cost. On latest JunOS, for example on EX2200, you
need
to purchase license (EFL), and if am not wrong it is $3000 for 48port
units.

So if only sFlow feature is on stake, it worth to think, to purchase
license,
or to purchase server.

Juniper no longer charges for sFlow on the EX2200 (as of Junos 11.2):

http://www.juniper.net/techpubs/en_US/junos11.2/information-products/topic-collections/release-notes/11.2/junos-release-notes-11.2.pdf

I am not aware of any vendor requiring an additional license to enable
sFlow.

sFlow (packet sampling) works extremely well for the DDoS flood
detection / mitigation use case. The measurements are build into low
cost commodity switch hardware and can be enabled operationally
without adversely impacting switch performance. A flood attack
generates high packet rates and sampling a 10G port at 1-in-10,000
will reliably detect flood attacks within seconds.

For most use cases, it is much less expensive to use switches to
perform measurement than to attach taps / mirror port probes. If your
switches don't already support sFlow, you can buy a 10G capable white
box switch for a few thousand dollars that will let you monitor 1.2
Terabits/sec. If you go with an open platform such as Cumulus Linux,
you could even run your DDoS mitigation software on the switch and
dispense with the external server. Embedded instrumentation is simple
to deploy and reduces operational complexity and cost when compared to
add on probe solutions.

Peter Phaal
InMon Corp.
Wow, that's great news then, i'm using mostly Cisco gear now, but seems
will have to take a look to Juniper, thanks for information.
If it is free, then if EX2200 available, it is much easier to run sFlow
and write custom collector for it, than installing custom probe(in most
common cases).

---
Best regards,
Denys

Re: DDOS, IDS, RTBH, and Rate limiting