Re: GoDaddy : DDoS : : Contact

[Roland Dobbins]

On 3 Aug 2015, at 12:10, John Levine wrote:


Given how easy it still is to put a fake source address in an IP
packet, it seems optimistic to assume that just because the packets
all have the same return address, they're actually coming from the
same place.


Concur 100% - we see that from time to time, multiple sources spoofing 
the same source IP.


---
Roland Dobbins 

Re: GoDaddy : DDoS : : Contact

[John Levine]

>> DDoS = multiple IPs
>>
>> DoS = single IP
>
>It seems most people colloquially use DDoS for both, and reserve DoS for 
>magic-packet blocking exploits like the latest BIND CVE, FYI.

Given how easy it still is to put a fake source address in an IP
packet, it seems optimistic to assume that just because the packets
all have the same return address, they're actually coming from the
same place.

R's,
John

Re: GoDaddy : DDoS :: Contact

[tqr2813d376cjozqap1l]

3. Aug 2015 04:20 by valdis.kletni...@vt.edu:
On Mon, 03 Aug 2015 03:58:31 -, tqr2813d376cjozqa...@tutanota.com said:

>> > It seems most people colloquially use DDoS for both, and reserve DoS for
>> > magic-packet blocking exploits like the latest BIND CVE, FYI.
>> Then they are mistaken, unfortunately.
>
> Feel free to try to reclaim the old meaning of the word "hacker" while
> you're at it.  That ship sailed long ago, and so has the DoS/DDoS 
> distinction.




I suppose you're right. Let the 'wordification' of  DDoS continue.. it 
certainly isn't an acronym anymore.

Re: GoDaddy : DDoS :: Contact

[Valdis . Kletnieks]

On Mon, 03 Aug 2015 03:58:31 -, tqr2813d376cjozqa...@tutanota.com said:

> > It seems most people colloquially use DDoS for both, and reserve DoS for
> > magic-packet blocking exploits like the latest BIND CVE, FYI.

> Then they are mistaken, unfortunately.

Feel free to try to reclaim the old meaning of the word "hacker" while
you're at it.  That ship sailed long ago, and so has the DoS/DDoS distinction.


pgpsPRTuZq0tB.pgp
Description: PGP signature

Re: GoDaddy : DDoS :: Contact

[Roland Dobbins]

On 3 Aug 2015, at 10:58, tqr2813d376cjozqa...@tutanota.com wrote:


Then they are mistaken, unfortunately.


Bring pedantic for its own sake, when there's little possibility of 
confusion, isn't really constructive.  Everyone, including you, knew 
what he meant.


---
Roland Dobbins 

Re: GoDaddy : DDoS :: Contact

[tqr2813d376cjozqap1l]

3. Aug 2015 03:54 by rdobb...@arbor.net:


> On 3 Aug 2015, at 6:16, > tqr2813d376cjozqa...@tutanota.com>  wrote:
>
>> DDoS = multiple IPs
>>
>> DoS = single IP
>
> It seems most people colloquially use DDoS for both, and reserve DoS for 
> magic-packet blocking exploits like the latest BIND CVE, FYI.
>




Then they are mistaken, unfortunately.

Re: GoDaddy : DDoS :: Contact

[Roland Dobbins]

On 3 Aug 2015, at 6:16, tqr2813d376cjozqa...@tutanota.com wrote:


DDoS = multiple IPs

DoS = single IP


It seems most people colloquially use DDoS for both, and reserve DoS for 
magic-packet blocking exploits like the latest BIND CVE, FYI.


---
Roland Dobbins 

Re: Quakecon: Network Operations Center tour

[Roland Dobbins]

On 3 Aug 2015, at 8:47, Christopher Morrow wrote:


oh .. maybe they really are all gone :)


People still run things long after EoS, heh.

A 6500 *with a Sup2T* is OK at the edge, for now - it has decent ASICs 
which support critical edge features, unlike its predecessors.  Myself, 
I'd much rather use an ASR9K or CRS (I don't know much about Juniper 
routers) as an edge device.


---
Roland Dobbins 

Did *bufferbloat* cause the 2010 flashcrash?

[Jay Ashworth]

This guy seems to think so, and his arguments seem pretty convincing to me, but 
I don't understand the financial system as well as I might.

yarchive.net/blog/computers/flash_crash.html

Gettys is namechecked in the piece.

Cheers,
-- jra
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Re: Quakecon: Network Operations Center tour

[Christopher Morrow]

On Sun, Aug 2, 2015 at 9:46 PM, Christopher Morrow
 wrote:
> On Sun, Aug 2, 2015 at 6:57 PM, Nick Hilliard  wrote:
>> As anchors, I would be hard put to make a choice between a 6500 and a 7500,
>> which was a fine router in its day but alas only had a useful lifetime of a
>> small number of years.  Obsolescence happens.
>
> isn't some of L3's edge still 7500's? I think some of 703/702's edges
> are still 7500's even.

"Last Date of Support:
HW
The last date to receive service and support for the product. After
this date, all support services for the product are unavailable, and
the product becomes obsolete.
December 31, 2012"

oh .. maybe they really are all gone :)

Re: Quakecon: Network Operations Center tour

[Christopher Morrow]

On Sun, Aug 2, 2015 at 6:57 PM, Nick Hilliard  wrote:
> As anchors, I would be hard put to make a choice between a 6500 and a 7500,
> which was a fine router in its day but alas only had a useful lifetime of a
> small number of years.  Obsolescence happens.

isn't some of L3's edge still 7500's? I think some of 703/702's edges
are still 7500's even.

Re: GoDaddy : DoS :: Contact

[Mel Beckman]

Blackholing isn't what you want. That will still permit his source IP into your 
network, and only blackhole replies from your network, so the attack will still 
consume bandwidth. What you should request is a source IP ACL blocking that 
address at your upstream' border.

BGP is no help in these situations, unless you use a BGP-based DDoS protection 
service.

 -mel beckman

On Aug 2, 2015, at 5:17 PM, Jason LeBlanc 
mailto:jason.lebl...@infusionsoft.com>> wrote:

Thanks Mel.  You are not being difficult, I meant DoS.  The network I inherited 
doesn't have BGP yet so I have asked our upstream to blackhole it and I emailed 
abuse neither have happened yet.  I do block it but that's after it hits our 
side.

//Jason

From: Mel Beckman mailto:m...@beckman.org>>
Date: Sunday, August 2, 2015 at 4:20 PM
To: Jason LeBlanc 
mailto:jason.lebl...@infusionsoft.com>>
Cc: NANOG mailto:nanog@nanog.org>>
Subject: Re: GoDaddy : DDoS :: Contact

Not to be difficult, but how can it be a DDoS attack if it's coming from a 
single IP? Normally you would just block this IP at your borders or ask your 
upstreams to do so before it consumes your bandwidth. You still want to get 
GoDaddy to address the problem, of course, but you should do that via their 
ab...@godaddy.com contact, or their abuse page at 
https://supportcenter.godaddy.com/AbuseReport/Index (submit via the "malware" 
button).

 -mel

On Aug 2, 2015, at 12:59 PM, Jason LeBlanc 
mailto:jason.lebl...@infusionsoft.com>> wrote:

My company is being DDoS'd by a single IP from a GoDaddy customer.

I havent had success with the ab...@godaddy.com 
email.  Was hoping someone
that could help might be watching the list and could contact me off-list.


//Jason

Re: GoDaddy : DoS :: Contact

[Jason LeBlanc]

Thanks Mel.  You are not being difficult, I meant DoS.  The network I inherited 
doesn’t have BGP yet so I have asked our upstream to blackhole it and I emailed 
abuse neither have happened yet.  I do block it but that’s after it hits our 
side.

//Jason

From: Mel Beckman mailto:m...@beckman.org>>
Date: Sunday, August 2, 2015 at 4:20 PM
To: Jason LeBlanc 
mailto:jason.lebl...@infusionsoft.com>>
Cc: NANOG mailto:nanog@nanog.org>>
Subject: Re: GoDaddy : DDoS :: Contact

Not to be difficult, but how can it be a DDoS attack if it’s coming from a 
single IP? Normally you would just block this IP at your borders or ask your 
upstreams to do so before it consumes your bandwidth. You still want to get 
GoDaddy to address the problem, of course, but you should do that via their 
ab...@godaddy.com contact, or their abuse page at 
https://supportcenter.godaddy.com/AbuseReport/Index (submit via the “malware” 
button).

 -mel

On Aug 2, 2015, at 12:59 PM, Jason LeBlanc 
mailto:jason.lebl...@infusionsoft.com>> wrote:

My company is being DDoS'd by a single IP from a GoDaddy customer.

I havent had success with the ab...@godaddy.com 
email.  Was hoping someone
that could help might be watching the list and could contact me off-list.


//Jason

Re: GoDaddy : DDoS :: Contact

[Jason Hellenthal]

Just block it 

-- 
 Jason Hellenthal
 JJH48-ARIN

On Aug 2, 2015, at 14:59, Jason LeBlanc  wrote:

My company is being DDoS'd by a single IP from a GoDaddy customer.

I havent had success with the ab...@godaddy.com email.  Was hoping someone
that could help might be watching the list and could contact me off-list.


//Jason

Re: GoDaddy : DDoS :: Contact

[Mel Beckman]

Not to be difficult, but how can it be a DDoS attack if it’s coming from a 
single IP? Normally you would just block this IP at your borders or ask your 
upstreams to do so before it consumes your bandwidth. You still want to get 
GoDaddy to address the problem, of course, but you should do that via their 
ab...@godaddy.com contact, or their abuse page at 
https://supportcenter.godaddy.com/AbuseReport/Index (submit via the “malware” 
button).

 -mel

On Aug 2, 2015, at 12:59 PM, Jason LeBlanc 
mailto:jason.lebl...@infusionsoft.com>> wrote:

My company is being DDoS'd by a single IP from a GoDaddy customer.

I havent had success with the ab...@godaddy.com 
email.  Was hoping someone
that could help might be watching the list and could contact me off-list.


//Jason

Re: GoDaddy : DDoS :: Contact

[tqr2813d376cjozqap1l]

2. Aug 2015 19:59 by jason.lebl...@infusionsoft.com:


> My company is being DDoS'd by a single IP from a GoDaddy customer.
>




DDoS = multiple IPs

DoS = single IP

GoDaddy : DDoS :: Contact

[Jason LeBlanc]

My company is being DDoS'd by a single IP from a GoDaddy customer.

I havent had success with the ab...@godaddy.com email.  Was hoping someone
that could help might be watching the list and could contact me off-list.


//Jason

Re: Quakecon: Network Operations Center tour

[Nick Hilliard]

On 02/08/2015 23:30, Randy Bush wrote:
> otoh, i did not believe in the fad of using 65xxs at the bgp global
> edge.  while it was temporarily cheap, two years later not a lot of folk
> had that many boats which needed anchoring.

A juniper EX9200 is a switch and a cisco sup2t box is a router.  The vendor
said it so it must be true.

As anchors, I would be hard put to make a choice between a 6500 and a 7500,
which was a fine router in its day but alas only had a useful lifetime of a
small number of years.  Obsolescence happens.

The distinction between layer 2 and layer 3 capable kit is not that
important these days.  What's important is whether the device's packet or
frame forwarding capabilities are a good match for the expected workload
and that the total operating cost over the depreciation period works.

Nick

Re: Quakecon: Network Operations Center tour

[Randy Bush]

>> so it is heavily routed using L3 on the core 'switches'?  makes a lot
>> of sense.
> Lots of switches will happily forward layer 3 packets.

and a lot of so-called switches will happily *route* at L3, which is i
think the point.  in this case, heavily subnetting a LAN, it makes a lot
of sense.

otoh, i did not believe in the fad of using 65xxs at the bgp global
edge.  while it was temporarily cheap, two years later not a lot of folk
had that many boats which needed anchoring.

randy

Re: Quakecon: Network Operations Center tour

[Josh Hoppes]

On Sun, Aug 2, 2015 at 4:59 PM, Randy Bush  wrote:
> josh,
>
> thanks for the more technical scoop.  now i get it a bit better.
>
>> We also re-designed the LAN back in 2011 to break up the giant single
>> broadcast domain down to a subnet per table switch.
>
> so it is heavily routed using L3 on the core 'switches'?  makes a lot of
> sense.

Single core switch, the Cisco 6509 VE in the video, handles routing
between subnets. Table switches have an IP for management and
monitoring. We have some 3750Gs for additional routing in other parts
of the event.

Re: Quakecon: Network Operations Center tour

[Nick Hilliard]

On 02/08/2015 22:59, Randy Bush wrote:
> so it is heavily routed using L3 on the core 'switches'?  makes a lot of
> sense.

Lots of switches will happily forward layer 3 packets.

Nick

Re: Quakecon: Network Operations Center tour

[Randy Bush]

josh,

thanks for the more technical scoop.  now i get it a bit better.

> We also re-designed the LAN back in 2011 to break up the giant single
> broadcast domain down to a subnet per table switch.

so it is heavily routed using L3 on the core 'switches'?  makes a lot of
sense.

randy

Re: Quakecon: Network Operations Center tour

[Josh Hoppes]

Not that often you see a bunch of people talking about a video you're
in, especially so on NANOG. So here goes.

BYOC is around 2700 seats. Total attendance was around 11,000.

2Gbps has been saturated at some point every year we have had it.
Additional bandwidth is definitely a serious consideration going
forward. It is a lot better than the 45mbps or less we dealt with 2010
and prior, but better doesn't mean good enough. Many games these days
do depend upon online services, and forced us to look for options.
AT&T has been sponsoring since then and we do appreciate it.

We have had the potential for DDoS attacks on our minds. Our first
option in those cases is blackhole announcements to the carrier for
the targeted /32. AT&T did provide address space for us to use so the
BYOC was using public IPs, and hopefully the impact of blackholing a
single IP could be made minimal. Thankfully we have not yet been
targeted, and we can only keep hoping it stays that way.

We haven't tackled IPv6 yet since it adds complexity that our primary
focus doesn't significantly benefit from yet since most games just
don't support it. Our current table switches don't have an RA guard,
and will probably require replacement to get ones that are capable.

We also re-designed the LAN back in 2011 to break up the giant single
broadcast domain down to a subnet per table switch. This has
definitely gotten us some flack from the BYOC since it breaks their
LAN browsers, but we thought a stable network was more important with
how much games have become dependent on stable Internet connectivity.
Still trying to find a good way to provide a middle ground for
attendees on that one, but I'm sure everyone here would understand how
insane a single broadcast domain with 2000+ hosts that aren't under
your control is. We have tried to focus on latency on the LAN, however
when so many games are no longer LAN oriented Internet connectivity
became a dominant issue.

Some traffic is routed out a separate lower capacity connection to
keep saturation issues from impacting it during the event.

Squid and nginx do help with caching, and thankfully Steam migrated to
a http distribution method and allows for easy caching. Some other
services make it more difficult, but we try our best. Before Steam
changed to http distribution there were a few years they helped in
providing a local mirror but that seems to have been discontinued with
the migration to http. The cache pushed a little over 4Gbps of traffic
at peak at the event.

The core IT team which handles the network (L2 and above) is about 9
volunteers. The physical infrastructure is our IP & D team, which gets
a huge team of volunteers put together in order to get that 13 miles
of cable ready between Monday and Wednesday. The event is very
volunteer driven, like many LAN parties across the planet. We try to
reuse cable from year to year, including loading up the table runs
onto a pallet to be used in making new cables out of in future years.

I imagine I haven't answered everyone's questions, but hopefully that
fills in some of the blanks.

If this has anyone considering sponsorship interest in the event the
contact email is sponsors(at)quakecon.org. Information is also
available on the website http://www.quakecon.org/.

RE: Windows 10 Release

[Jay Ashworth]

An article in VARGuy said they'd booked 40 Tb/s of capacity from Akamai, 
Limelight, and four or five other CDNs that I did not recognize by name.

I presume each machine will have to contact at least one machine at 
microsoft.com to confirm signatures on downloaded packages, et alia.

- jra

On July 28, 2015 8:09:52 PM EDT, Erik Sundberg  wrote:
>Does anyone know if Microsoft will be hosting the downloads from there
>ASN 8075 or from an CDN Provider like Akamai?
>
>
>
>-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Curtis
>Maurand
>Sent: Tuesday, July 28, 2015 6:43 PM
>To: Niels Bakker ; nanog@nanog.org
>Subject: Re: Windows 10 Release
>
>Microsoft tells me 3.2 GB for win 10 pro 64 bit.
>
>On July 28, 2015 6:04:04 PM EDT, Niels Bakker 
>wrote:
>>* n...@flhsi.com (Nick Olsen) [Tue 28 Jul 2015, 22:46 CEST]:
>>>Being a 3-4GB download. Each device is moving more data than any
>Apple
>>
>>>update ever did.
>>
>>I'm not so sure of that.  The 10.9 install image clocked in at 4.9 GB,
>>and the Mac App Store for 10.10 Yosemite says "Size: 5.67 GB";
>>http://www.microsoft.com/en-us/windows/features says "3GB download
>>required" in the small print at the bottom.
>>
>>
>>   -- Niels.
>
>--
>Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
>
>
>CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
>files or previous e-mail messages attached to it may contain
>confidential information that is legally privileged. If you are not the
>intended recipient, or a person responsible for delivering it to the
>intended recipient, you are hereby notified that any disclosure,
>copying, distribution or use of any of the information contained in or
>attached to this transmission is STRICTLY PROHIBITED. If you have
>received this transmission in error please notify the sender
>immediately by replying to this e-mail. You must destroy the original
>transmission and its attachments without reading or saving in any
>manner. Thank you.

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Re: RE: Bright House IMAP highwater warning real?

[tqr2813d376cjozqap1l]

- Tell user that they're nearly out of storage. Specify how much they've used 
and how much they have total. Perhaps include a percentage
- Mention that they could delete email that isn't needed to recover space.
- (optional but nice) Show the subject and sender of the biggest
messages
- (optional but nice) Mention how big the trash folder is (with percentage) 
and tell them they could empty it
- Tell user to visit website or call if they have any questions or want to 
add more storage

None of this 'high water mark' crap

2. Aug 2015 19:44 by frnk...@iname.com:


> What do you think their message should say?  We struggled over this, too, 
> and settled on some soft language, included information on how to purchase 
> more storage, and also provided our email address and phone numbers.
>
> Frank
>
> -Original Message-
> From: NANOG [> mailto:nanog-boun...@nanog.org> ] On Behalf Of Jay Ashworth
> Sent: Sunday, August 02, 2015 1:55 PM
> To: > nanog@nanog.org
> Subject: Bright House IMAP highwater warning real?
>
> Any brighthouse email admins on the list? My sister got the following high 
> water warning message, with the included headers which, since they appear 
> to include no Received: headers, look like they actually came from 
> brighthouse's email cluster.
>
> If this is a real Bright House warning message, somebody should be flogged. 
> Teaching people which messages is to believe is hard enough...
>
> Cheers,
> -- jra
>
>
>  Original Message 
> Subject: Re: Fwd: ATTENTION: High Water Mark Notification, bytes in the 
> mailbox!
>
> I lied. The header to yours - which I finally found - is nice and long.
>  the header on this one is
>
> Return-Path: <>
> From: admin
> Subject: ATTENTION: High Water Mark Notification, bytes in the mailbox!
> Date: Sun, 2 Aug 2015 06:22:44 +
> Message-ID: e31468ce-38de-11e5-b0a6-17507733086b
>
> >>-Original Message-
> >>From: admin
> >>Sent: Sun, 02 Aug 2015 2:22 AM
> >>Subject: ATTENTION: High Water Mark Notification, bytes in the
> >mailbox!

> >>Your mailbox is over the high water mark.
> >>Please delete some messages from your mailbox.
> --
> Sent from my Android phone with K-9 Mail. Please excuse my brevity.

RE: Bright House IMAP highwater warning real?

[Jay Ashworth]

I think the body text of the message should identify it as coming from the 
Bright House email system? I think it should be written in standard USAdian 
English, which that is decidedly not.

Or perhaps the problem is that that subject line was supposed to be 
parameterized, and the number of bytes is missing for some reason. But in any 
event that is a common message to spoof, and the more bits of identity that are 
in it the harder it is to do so. That message format has almost zero bit of 
provider-identifiable data.

"""
Your Bright House Networks IMAP email storage for u...@domain.com is at 490MB, 
approaching your quota of 500MB.

IMAP email permits you to access all your mail folders by storing them on the 
mail server, but because of this, all mail in your folders contributes to your 
storage limit.

You can delete messages to reduce your storage, or move them to your PC. If you 
delete them, or have already deleted them, you usually must 'compact' each 
folder to reclaim the extra space.

Alternatively, you can contact Customer Care to see about having your quota 
increased.
"""

Cheers,
-- jra

On August 2, 2015 3:44:35 PM EDT, Frank Bulk  wrote:
>What do you think their message should say?  We struggled over this,
>too, and settled on some soft language, included information on how to
>purchase more storage, and also provided our email address and phone
>numbers.
>
>Frank
>
>-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jay Ashworth
>Sent: Sunday, August 02, 2015 1:55 PM
>To: nanog@nanog.org
>Subject: Bright House IMAP highwater warning real?
>
>Any brighthouse email admins on the list? My sister got the following
>high water warning message, with the included headers which, since they
>appear to include no Received: headers, look like they actually came
>from brighthouse's email cluster. 
>
>If this is a real Bright House warning message, somebody should be
>flogged. Teaching people which messages is to believe is hard enough...
>
>Cheers,
>-- jra
>
>
> Original Message 
>Subject: Re: Fwd: ATTENTION: High Water Mark Notification, bytes in the
>mailbox!
>
>I lied. The header to yours - which I finally found - is nice and long.
> the header on this one is
>
>Return-Path: <>
>From: admin
>Subject: ATTENTION: High Water Mark Notification, bytes in the mailbox!
>Date: Sun, 2 Aug 2015 06:22:44 +
>Message-ID: e31468ce-38de-11e5-b0a6-17507733086b
>
>>>-Original Message-
>>>From: admin
>>>Sent: Sun, 02 Aug 2015 2:22 AM
>>>Subject: ATTENTION: High Water Mark Notification, bytes in the
>>mailbox!
>>>
>>>Your mailbox is over the high water mark.
>>>Please delete some messages from your mailbox.
>-- 
>Sent from my Android phone with K-9 Mail. Please excuse my brevity.

-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

RE: Bright House IMAP highwater warning real?

[Frank Bulk]

What do you think their message should say?  We struggled over this, too, and 
settled on some soft language, included information on how to purchase more 
storage, and also provided our email address and phone numbers.

Frank

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jay Ashworth
Sent: Sunday, August 02, 2015 1:55 PM
To: nanog@nanog.org
Subject: Bright House IMAP highwater warning real?

Any brighthouse email admins on the list? My sister got the following high 
water warning message, with the included headers which, since they appear to 
include no Received: headers, look like they actually came from brighthouse's 
email cluster. 

If this is a real Bright House warning message, somebody should be flogged. 
Teaching people which messages is to believe is hard enough...

Cheers,
-- jra


 Original Message 
Subject: Re: Fwd: ATTENTION: High Water Mark Notification, bytes in the mailbox!

I lied. The header to yours - which I finally found - is nice and long.
 the header on this one is

Return-Path: <>
From: admin
Subject: ATTENTION: High Water Mark Notification, bytes in the mailbox!
Date: Sun, 2 Aug 2015 06:22:44 +
Message-ID: e31468ce-38de-11e5-b0a6-17507733086b

>>-Original Message-
>>From: admin
>>Sent: Sun, 02 Aug 2015 2:22 AM
>>Subject: ATTENTION: High Water Mark Notification, bytes in the
>mailbox!
>>
>>Your mailbox is over the high water mark.
>>Please delete some messages from your mailbox.
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Bright House IMAP highwater warning real?

[Jay Ashworth]

Any brighthouse email admins on the list? My sister got the following high 
water warning message, with the included headers which, since they appear to 
include no Received: headers, look like they actually came from brighthouse's 
email cluster. 

If this is a real Bright House warning message, somebody should be flogged. 
Teaching people which messages is to believe is hard enough...

Cheers,
-- jra


 Original Message 
Subject: Re: Fwd: ATTENTION: High Water Mark Notification, bytes in the mailbox!

I lied. The header to yours - which I finally found - is nice and long.
 the header on this one is

Return-Path: <>
From: admin
Subject: ATTENTION: High Water Mark Notification, bytes in the mailbox!
Date: Sun, 2 Aug 2015 06:22:44 +
Message-ID: e31468ce-38de-11e5-b0a6-17507733086b

>>-Original Message-
>>From: admin
>>Sent: Sun, 02 Aug 2015 2:22 AM
>>Subject: ATTENTION: High Water Mark Notification, bytes in the
>mailbox!
>>
>>Your mailbox is over the high water mark.
>>Please delete some messages from your mailbox.
-- 
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Re: Quakecon: Network Operations Center tour

[Roland Dobbins]

On 2 Aug 2015, at 23:49, Mike Hammett wrote:

If the core of the mission is local LAN play and your Internet 
connection fills up


You're assuming the DDoS attack originates from outside the local 
network(s).  I was curious as to whether they'd seen any *internal* DDoS 
attacks.


And again, external bandwidth doesn't matter for externally-sourced DDoS 
attacks.  If the attacker wishes to do so, he'll completely overwhelm 
your transit bandwidth.



 who gives a shit? The games play on.


No, they don't, if they require a connection across the Internet to game 
servers for matchmaking/auth purposes, etc.


---
Roland Dobbins 

Re: Quakecon: Network Operations Center tour

[Mike Hammett]

It most certainly does. If the core of the mission is local LAN play and your 
Internet connection fills up who gives a shit? The games play on. If your 
500 megabit corporate connection gets a 20 terabit DDoS, your RDP session to 
the finance department will continue to hum along just fine. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

- Original Message -

From: "Roland Dobbins"  
To: "nanog list"  
Sent: Sunday, August 2, 2015 11:23:18 AM 
Subject: Re: Quakecon: Network Operations Center tour 

On 2 Aug 2015, at 22:56, Mike Hammett wrote: 

> It's completely reasonable when the world at large is only secondary 
> to the local, on-net operations. 

It has nothing to do with DDoS. 

--- 
Roland Dobbins  

Re: Quakecon: Network Operations Center tour

[Mikael Abrahamsson]

On Sun, 2 Aug 2015, Dave Pooser wrote:

I wonder if that would be a reason for the relatively anemic 1Gb 
Internet pipe-- making sure that a DDoS couldn't push enough packets 
through to inconvenience the LAN party.


I was involved in delivering 1GigE to Dreamhack in 2001 which at the time 
(if I remember correctly), 4500 computers that participants brought with 
them.


Usually these events nowadays tend to use 5-20 gigabit/s for that amount 
of people, so 2x1GE is just not enough. Already in 2001 that GigE was 
fully loaded after 1-2 days.


--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: Quakecon: Network Operations Center tour

[Roland Dobbins]

On 2 Aug 2015, at 22:56, Alistair Mackenzie wrote:


I would assume this would a start to the problem if your attacks were
volumetric.


In a world of 430gb/sec reflection/amplification DDoS attacks, not 
really.


;>

Just increasing bandwidth has never been a viable DDoS defense tactic, 
due to the extreme asymmetry of resource ratios in favor of the 
attackers.


---
Roland Dobbins 

Re: Quakecon: Network Operations Center tour

[Roland Dobbins]

On 2 Aug 2015, at 22:56, Mike Hammett wrote:

It's completely reasonable when the world at large is only secondary 
to the local, on-net operations.


It has nothing to do with DDoS.

---
Roland Dobbins 

Re: Quakecon: Network Operations Center tour

[Laurent Dumont]

I recently wrapped up a 1300 players with gigabit connections where we 
had a single 5gig link. We never saturated the link and peaked at 
3.92Gbps for a new minutes. Bandwidth usage peaks on the first day and 
settles down after that (the event was during an entire weekend starting 
on friday). If I recall correctly, average was around 2Gpbs.


We did not have a steam/web cache and I expect it might reduce even more 
the actual load on actual BW usage.


On 8/2/2015 7:32 AM, Randy Bush wrote:

Also, 2 Gbps for 4,400 people?  Pretty lackluster compared to European
events.  30C3 had 100 Gbps to the conference building.  And no NAT:
every host got real IP addresses (IPv4 + IPv6).

ietf, >1k people, easily fits in 10g, but tries to have two for
redundancy.  also no nat, no firewall, and even ipv6.  but absorbing or
combatting scans and other attacks cause complexity one would prefer to
avoid.  in praha, there was even a tkip attack, or so it is  believed;
turned off tkip.

the quakecon net was explained very poorly.  what in particular provides
game-quality latency, or lack thereof?  with only 2g, i guess i can
understand the cache.  decent bandwidth would reduce complexity.  and
the network is flat?

randy

Re: Quakecon: Network Operations Center tour

[Mike Hammett]

It's completely reasonable when the world at large is only secondary to the 
local, on-net operations. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Roland Dobbins"  
To: "nanog list"  
Sent: Sunday, August 2, 2015 10:50:05 AM 
Subject: Re: Quakecon: Network Operations Center tour 

On 2 Aug 2015, at 22:44, Dave Pooser wrote: 

> I wonder if that would be a reason for the relatively anemic 1Gb 
> Internet 
> 
> pipe-- making sure that a DDoS couldn't push enough packets through to 
> inconvenience the LAN party. 

While increasing bandwidth is not a viable DDoS defense tactic, 
decreasing it isn't one, either. 

--- 
Roland Dobbins  

Re: Quakecon: Network Operations Center tour

[Alistair Mackenzie]

While increasing bandwidth to the endpoint isn't viable wouldn't increasing
the edge bandwidth out to the ISP be a start in the right direction?

I would assume this would a start to the problem if your attacks were
volumetric.

Once the bandwidth is there you can look at mitigation before it reaches
the endpoint, in this case the computers on the floor (assuming no NAT).
On 2 Aug 2015 16:51, "Roland Dobbins"  wrote:

> On 2 Aug 2015, at 22:44, Dave Pooser wrote:
>
> I wonder if that would be a reason for the relatively anemic 1Gb Internet
>>
>> pipe-- making sure that a DDoS couldn't push enough packets through to
>> inconvenience the LAN party.
>>
>
> While increasing bandwidth is not a viable DDoS defense tactic, decreasing
> it isn't one, either.
>
> ---
> Roland Dobbins 
>

Re: Quakecon: Network Operations Center tour

[Roland Dobbins]

On 2 Aug 2015, at 22:44, Dave Pooser wrote:

I wonder if that would be a reason for the relatively anemic 1Gb 
Internet


pipe-- making sure that a DDoS couldn't push enough packets through to
inconvenience the LAN party.


While increasing bandwidth is not a viable DDoS defense tactic, 
decreasing it isn't one, either.


---
Roland Dobbins 

Re: Quakecon: Network Operations Center tour

[Dave Pooser]

>>any security protections so competitors can't kill off their
>> competition?)
>
>It would be interesting to learn whether they saw any DDoS attacks or
>cheating attempts during competitive play, or even casual
>non-competitive play amongst attendees.

I wonder if that would be a reason for the relatively anemic 1Gb Internet
pipe-- making sure that a DDoS couldn't push enough packets through to
inconvenience the LAN party.

(Disclaimer: $DAYJOB did the audio/visual/lighting for QuakeCon but we had
nothing to do with the network and I was utterly uninvolved in any way,
so my speculation is based on no information obtained from outside my own
skull.)
-- 
Dave Pooser
Cat-Herder-in-Chief, Pooserville.com

Re: Quakecon: Network Operations Center tour

[Roland Dobbins]

On 2 Aug 2015, at 22:32, Christopher Morrow wrote:

any security protections so competitors can't kill off their 
competition?)


It would be interesting to learn whether they saw any DDoS attacks or 
cheating attempts during competitive play, or even casual 
non-competitive play amongst attendees.


---
Roland Dobbins 

Re: Quakecon: Network Operations Center tour

[Christopher Morrow]

On Sun, Aug 2, 2015 at 7:56 AM, Niels Bakker  wrote:
> I guess a tale of punching 300-odd patchpanels is not that captivating to
> everybody out there.

I find this hard to believe.

:)

I was hoping for more 'how the network is built' (flat? segmented? any
security protections so competitors can't kill off their competition?)
and ideally some discussion of why the decisions made a difference.
(what tradeoffs were made and why?)

Re: Quakecon: Network Operations Center tour

[Harald F. Karlsen]

On 01.08.2015 21:27, Sean Donelan wrote:

What Powers Quakecon | Network Operations Center Tour
https://www.youtube.com/watch?v=mOv62lBdlXU

Cool stuff!

For reference here are the blog for the tech-crew at the worlds second 
largest LAN-party, The Gathering:

http://technical.gathering.org/

A few highlights:
* Over 12,000 Gigabit ports, 500 * 10Gigabit ports, 50 * 40Gigabit ports 
(not all utilized of course).

* Gigabit to all participants.
* Dual-stack public IPv4 and IPv6 to all participants.
* 30Gbit internet connection (upgradeable if needed).
* Zero-touch provisioning of all edge switches.

Most of the NMS and provisioning systems are made in-house and are 
available on github (https://github.com/tech-server/) and all 
configuration files are released to the public after each event on 
ftp://ftp.gathering.org (seems to be down at the moment).


--
Harald

Re: Quakecon: Network Operations Center tour

[Sean Donelan]

On Sun, 2 Aug 2015, Niels Bakker wrote:
Also, 2 Gbps for 4,400 people?  Pretty lackluster compared to European 
events.  30C3 had 100 Gbps to the conference building.  And no NAT: every 
host got real IP addresses (IPv4 + IPv6).


Quakecon is essentially a giant LAN party.  Bring Your Own Computer 
(BYOC), and there are big gaming rigs at Quakecon, and compete on the LAN. 
There isn't that much Internet traffic.  There is only 100Mbps wired to

each gaming station.

I'm not a quake fanatic, I don't know what are the important network 
metrics for a good gaming experience.  But I assume the important metrics

are local, and they install a big central server complex in the center
of the room.  I'm assuming the critical lag is between the central
server and the competitors; not the Internet.   Otherwise they could
have all stayed home and played in their basements across the
Internet.  Latency is probably more important than bulk bandwidth.

Re: Quakecon: Network Operations Center tour

[Nikolay Shopik]

Steam moved to http streaming few years ago for exact that reason

> On 2 авг. 2015 г., at 4:51, Steven Miano  wrote:
> 
> historically steam/game downloads are not
> cahce'able

Re: Quakecon: Network Operations Center tour

[Niels Bakker]

* ra...@psg.com (Randy Bush) [Sun 02 Aug 2015, 13:37 CEST]:
ietf, >1k people, easily fits in 10g, but tries to have two for 
redundancy.  also no nat, no firewall, and even ipv6.  but absorbing 
or combatting scans and other attacks cause complexity one would 
prefer to avoid.  in praha, there was even a tkip attack, or so it 
is believed; turned off tkip.


Didn't the IETF already deprecate TKIP?


the quakecon net was explained very poorly.  what in particular 
provides game-quality latency, or lack thereof?  with only 2g, i 
guess i can understand the cache.  decent bandwidth would reduce 
complexity.  and the network is flat?


Cabling up 4,400 ports does take a lot of effort, though.

The QuakeCon video was typical for a server guy talking about network: 
with a focus on the network periphery, i.e. some servers supporting 
the network.  I guess a tale of punching 300-odd patchpanels is not 
that captivating to everybody out there.



-- Niels.

Re: Quakecon: Network Operations Center tour

[Randy Bush]

> Also, 2 Gbps for 4,400 people?  Pretty lackluster compared to European
> events.  30C3 had 100 Gbps to the conference building.  And no NAT:
> every host got real IP addresses (IPv4 + IPv6).

ietf, >1k people, easily fits in 10g, but tries to have two for
redundancy.  also no nat, no firewall, and even ipv6.  but absorbing or
combatting scans and other attacks cause complexity one would prefer to
avoid.  in praha, there was even a tkip attack, or so it is  believed;
turned off tkip.

the quakecon net was explained very poorly.  what in particular provides
game-quality latency, or lack thereof?  with only 2g, i guess i can
understand the cache.  decent bandwidth would reduce complexity.  and
the network is flat?

randy

Re: best practice for number of RR

[Mark Tinka]

On 1/Aug/15 18:34, marco da pieve wrote:
> Hi Shane,
> for the boxes that are currently installed in the network, this is not a
> valid option (politically/commercially speaking).

Well, Cisco, Juniper and ALU are shipping carrier-grade OS's that will
run on a server in a VM.

Brocade is also known to be doing good work there re: Vyatta, but I
don't know of anyone running that as an RR.

In 2015, I'd never spend money on a dedicated RR running on router
hardware. Server, VM, end of story.

Mark.

Re: best practice for number of RR

[Mark Tinka]

On 1/Aug/15 17:38, marco da pieve wrote:
> Hi all,
> this is my first time in asking for advices here and I hope not to bother
> you with this topic (if it has been already covered in the past, would you
> please please point me to that discussion?).
>
> Anyway, I need to decide whether to go for a BGP topology with a single
> cluster of 3 Route Reflectors (to overcome a dual point of failure issue)
> or maybe to two standalone clusters each with two RR (sacrificing half of
> the network in case two RR of the same cluster fail).
>
> To give you some input data:
>
> - 8000 actual VPNV4 prefixes
> - 180 BGP neighbors
>
> In case of the 3 RRs option, prefixes will become 24000 on the clients (24k
> received routes in total but 1/3 installed. No BGP multipath will be used).
> In this scenario considering network growth up to doubling the current
> number of VPNV4 prefixes, I would end up to have 16k actual vpnv4 prefixes
> and 48k vpnv4 prefixes received by the clients, which is almost the limit
> for the HW used.
>
> In the case of two standalone clusters each with two RRs, BGP neighborships
> will be halved among the two clusters and vpnv4 prefixes too. In case of
> network growth up to doubling the number of prefixes, the clients will
> receive up to 24k vpnv4 prefixes and this is still far below the HW limits.
> Of course this option will not prevent a dual failure in the single cluster
> and half of the network would end up in outage.
>
> My choice would be to go for the two clusters assuming that each RR has
> supervisor/controlling card protection capabilities.
>
> However I'd like to have a feedback on the pros and cons on the design
> itself if any. I know that design is planned on the resources available but
> just for discussing and abstracting from the HW, would there be any
> drawbacks in having an odd number of RR in the network? is one of the two
> option a no to go choice? what was your experience?

We deploy 2x RR's in each of our main PoP's.

All iBGP clients in that PoP speak to their local RR's.

The RR's all speak to one another in a full-mesh.

Each RR pair is its own cluster.

We run our RR's on Cisco's CSR1000v software, which is IOS XE in a VM
(VMware ESXi in our case). These are high-end servers, but we don't
worry too much about over-protecting one because there is a redundant
one in each cluster.

I once ran a network which ad 3x RR's per cluster. That is fine, but the
impact on the clients can become an issue over time.

Mark.

Re: Quakecon: Network Operations Center tour

[Niels Bakker]

* mian...@gmail.com (Steven Miano) [Sun 02 Aug 2015, 03:52 CEST]:

It would have been more interesting to see:

-- a network weather map
-- the ELK implementation
-- actual cache statistics (historically steam/game downloads are not
cahce'able)


Not quite true according to 
http://blog.multiplay.co.uk/2014/04/lancache-dynamically-caching-game-installs-at-lans-using-nginx/


Also, 2 Gbps for 4,400 people?  Pretty lackluster compared to European 
events.  30C3 had 100 Gbps to the conference building.  And no NAT: 
every host got real IP addresses (IPv4 + IPv6).



-- Niels.