Re: Cisco Tech needed in San Franscisco

[JoeSox]

Thanks everyone. I have some good leads. and FYI, I learned about this
NANOG page :)
http://nanog.cluepon.net/index.php/Hands
--
Thank You, Joe


On Tue, Aug 26, 2014 at 5:23 PM, JoeSox  wrote:

> Hi Everyone,
> Took over a new network and they have no onsite Cisco tech help for one of
> my remote locations.
> Can someone email me directly a company I can contract with to get help
> this Wed night in downtown San Francisco (we can push maintenance to Friday
> night as well  but shooting for tomorrow)?
>
> --
> Thank You, Joe
>

DC opinions?

[Jay Ashworth]

Anyone in Centurylink Ashburn or Seattle?  Do you have an opinion?

Customary protocols apply.

Cheers,
-- jra
-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: Best US Tunnelbroker for Youtube

[Mark Andrews]

In message 
, ITechGeek writes:
> Someone was telling me this weekend their entire network is native dual
> stack now.  I haven't had a chance to confirm it yet, but he said they are
> issuing /60's to residential users using DHCPv6.

The residential network in fully IPv6 capable.
The commercial network is still a work in progress.

This is based on the last notice I saw from Comcast.

Mark
 
> -
> --
> -ITG (ITechGeek)
> i...@itechgeek.com
> https://itg.nu/
> GPG Keys: https://itg.nu/contact/gpg-key
> Preferred GPG Key: Fingerprint: AB46B7E363DA7E04ABFA57852AA9910A DCB1191A
> Google Voice: +1-703-493-0128 / Twitter: ITechGeek / Facebook:
> http://fb.me/Jbwa.Net
> 
> 
> On Tue, Aug 26, 2014 at 8:42 PM, Owen DeLong  wrote:
> 
> > My understanding is that almost all of the Comcast network is now IPv6
> > capable.
> >
> > Owen
> >
> > On Aug 20, 2014, at 10:26 AM, Ryan Shea  wrote:
> >
> > > Not sure I've seen any evidence (or implied) that the tunnel was the
> > > problem. My issue as far as I know is at the application layer and other
> > > end-user experiences seemed a reasonable way to pick a direction. I will
> > > work with HE though and provide them some details.
> > >
> > > Agreed, from an end-user perspective it can be often be clear as mud
> > > whether I am using v6, or whether my Chromecast or Android device even
> > > implements happy eyeballs. The relatively new "experiencing problems?"
> > > butter bar that shows up beneath a video with notable buffering problems
> > > (even at low quality levels) sends the user through to details about the
> > > service provider, in this case HE. Over the past couple years YouTube has
> > > been my canary to know when I've received a new IP from Verizon and I
> > need
> > > to go fix my tunnel -- video loading takes frever on
> > > Android/Chromecast/GoogleTV (which hints that happy eyeballs, if it
> > exists
> > > for Android, isn't working so well for the YouTube app).
> > >
> > > I can't get native v6 at my home -- I'm probably not in a particularly
> > > unique situation. Not to rathole the dicsussion, but as far as I know
> > (save
> > > for some small DSL providers) unless I'm in a gFiber city or happen to be
> > > in the portion of the Comcast network that provides native v6 I'm out of
> > > luck. I don't plan on moving to solve this problem.
> > >
> > >
> > >
> > > On Wed, Aug 20, 2014 at 12:42 PM, Jeroen Massar 
> > wrote:
> > >
> > >> On 2014-08-20 18:21, Ryan Shea wrote:
> > >>> IRC is a good suggestion, thanks. They'll likely be helpful.
> > >>>
> > >>> I see no indication of any throttling from my ISP - I can blast data at
> > >>> full speed to my home from my server and work (with native v6
> > >>> connections).
> > >>
> > >> Does that path between your $home and $server go over the tunnel you
> > >> find so "slow"?
> > >>
> > >> If so, then you have just nicely excluded that the tunnel is NOT the
> > >> problem.
> > >>
> > >> Hence, why traceroutes would be so extremely useful.
> > >>
> > >>
> > >>> Contacting my ISP (Verizon FiOS) is virtually never a
> > >>> reasonable path to a solution.
> > >>
> > >> google(Verizon FiOS throttle) = 71.900 results. One would almost think
> > >> that there might sometimes be issues there.
> > >>
> > >> Also, do realize that the IPv6 path you are using goes over a shared
> > >> host (the Tunnel Broker PoP) that has IPv4 and IPv6 capacity that might
> > >> be shared in various points of the paths your packets cross.
> > >>
> > >> Did you test at the same time of your "blast data" that the IPv6 Youtube
> > >> was working fine?
> > >>
> > >> Another thing, as browsers now do "Happy Eyeballs" (which is really
> > >> horrible to diagnose issues with on OSX), did you check if everything is
> > >> really going over IPv6? (hence the tcpdump/wireshark).
> > >>
> > >> [..]
> > >>> To be clear, I was seeking opinions/experiences on a list that was
> > >>> likely to have a high occurrence of folk with v6 tunnels.
> > >>
> > >> Tunnels are for endusers who still are at ISPs who don't do IPv6
> > natively.
> > >>
> > >> NANOG has operators who have been running native IPv6 for over a decade.
> > >>
> > >> Hence, StackExchange might be useful for your purpose.
> > >>
> > >>> You have
> > >>> etiquette suggestions, but not YouTube over tunnelbroker suggestions. I
> > >>> apparently bring out your inner grump? Do you need a hug?
> > >>
> > >> My cat videos are streaming perfectly fine...
> > >>
> > >>> Burning Google engineering time would be a sub-optimal way to get HD
> > cat
> > >>> videos at home with the least time spent.
> > >>
> > >> Interesting, I was not aware they did not care about their eyeballs.
> > >>
> > >> Actually I am very confident lots of folks there would love to dig into
> > >> your issue to actually resolve it. As when it is hitting you, it might
> > >> hit other customers.
> > >>
> > >> Is that also not w

Re: Best US Tunnelbroker for Youtube

[ITechGeek]

On Tue, Aug 26, 2014 at 9:22 PM,  wrote:

> hope you
> get a level-1 guy who knows what IPv6 is
>

Is that possible?

---
-ITG (ITechGeek)
i...@itechgeek.com
https://itg.nu/
GPG Keys: https://itg.nu/contact/gpg-key
Preferred GPG Key: Fingerprint: AB46B7E363DA7E04ABFA57852AA9910A DCB1191A
Google Voice: +1-703-493-0128 / Twitter: ITechGeek / Facebook:
http://fb.me/Jbwa.Net

Re: Best US Tunnelbroker for Youtube

[Valdis . Kletnieks]

On Tue, 26 Aug 2014 21:17:14 -0400, ITechGeek said:
> Someone was telling me this weekend their entire network is native dual
> stack now.  I haven't had a chance to confirm it yet, but he said they are
> issuing /60's to residential users using DHCPv6.

I believe the status is "every residential customer should be able to get a /60
via DHCPv6-PD".  It's worked for me for a while, and somebody official from
Comcast posted a while ago that they'd finished rolling out to 100% of the
residential network. If it isn't working, it's time to complain (and hope you
get a level-1 guy who knows what IPv6 is. :)


pgpA0cly9t12J.pgp
Description: PGP signature

Re: Best US Tunnelbroker for Youtube

[ITechGeek]

Someone was telling me this weekend their entire network is native dual
stack now.  I haven't had a chance to confirm it yet, but he said they are
issuing /60's to residential users using DHCPv6.

---
-ITG (ITechGeek)
i...@itechgeek.com
https://itg.nu/
GPG Keys: https://itg.nu/contact/gpg-key
Preferred GPG Key: Fingerprint: AB46B7E363DA7E04ABFA57852AA9910A DCB1191A
Google Voice: +1-703-493-0128 / Twitter: ITechGeek / Facebook:
http://fb.me/Jbwa.Net


On Tue, Aug 26, 2014 at 8:42 PM, Owen DeLong  wrote:

> My understanding is that almost all of the Comcast network is now IPv6
> capable.
>
> Owen
>
> On Aug 20, 2014, at 10:26 AM, Ryan Shea  wrote:
>
> > Not sure I've seen any evidence (or implied) that the tunnel was the
> > problem. My issue as far as I know is at the application layer and other
> > end-user experiences seemed a reasonable way to pick a direction. I will
> > work with HE though and provide them some details.
> >
> > Agreed, from an end-user perspective it can be often be clear as mud
> > whether I am using v6, or whether my Chromecast or Android device even
> > implements happy eyeballs. The relatively new "experiencing problems?"
> > butter bar that shows up beneath a video with notable buffering problems
> > (even at low quality levels) sends the user through to details about the
> > service provider, in this case HE. Over the past couple years YouTube has
> > been my canary to know when I've received a new IP from Verizon and I
> need
> > to go fix my tunnel -- video loading takes frever on
> > Android/Chromecast/GoogleTV (which hints that happy eyeballs, if it
> exists
> > for Android, isn't working so well for the YouTube app).
> >
> > I can't get native v6 at my home -- I'm probably not in a particularly
> > unique situation. Not to rathole the dicsussion, but as far as I know
> (save
> > for some small DSL providers) unless I'm in a gFiber city or happen to be
> > in the portion of the Comcast network that provides native v6 I'm out of
> > luck. I don't plan on moving to solve this problem.
> >
> >
> >
> > On Wed, Aug 20, 2014 at 12:42 PM, Jeroen Massar 
> wrote:
> >
> >> On 2014-08-20 18:21, Ryan Shea wrote:
> >>> IRC is a good suggestion, thanks. They'll likely be helpful.
> >>>
> >>> I see no indication of any throttling from my ISP - I can blast data at
> >>> full speed to my home from my server and work (with native v6
> >>> connections).
> >>
> >> Does that path between your $home and $server go over the tunnel you
> >> find so "slow"?
> >>
> >> If so, then you have just nicely excluded that the tunnel is NOT the
> >> problem.
> >>
> >> Hence, why traceroutes would be so extremely useful.
> >>
> >>
> >>> Contacting my ISP (Verizon FiOS) is virtually never a
> >>> reasonable path to a solution.
> >>
> >> google(Verizon FiOS throttle) = 71.900 results. One would almost think
> >> that there might sometimes be issues there.
> >>
> >> Also, do realize that the IPv6 path you are using goes over a shared
> >> host (the Tunnel Broker PoP) that has IPv4 and IPv6 capacity that might
> >> be shared in various points of the paths your packets cross.
> >>
> >> Did you test at the same time of your "blast data" that the IPv6 Youtube
> >> was working fine?
> >>
> >> Another thing, as browsers now do "Happy Eyeballs" (which is really
> >> horrible to diagnose issues with on OSX), did you check if everything is
> >> really going over IPv6? (hence the tcpdump/wireshark).
> >>
> >> [..]
> >>> To be clear, I was seeking opinions/experiences on a list that was
> >>> likely to have a high occurrence of folk with v6 tunnels.
> >>
> >> Tunnels are for endusers who still are at ISPs who don't do IPv6
> natively.
> >>
> >> NANOG has operators who have been running native IPv6 for over a decade.
> >>
> >> Hence, StackExchange might be useful for your purpose.
> >>
> >>> You have
> >>> etiquette suggestions, but not YouTube over tunnelbroker suggestions. I
> >>> apparently bring out your inner grump? Do you need a hug?
> >>
> >> My cat videos are streaming perfectly fine...
> >>
> >>> Burning Google engineering time would be a sub-optimal way to get HD
> cat
> >>> videos at home with the least time spent.
> >>
> >> Interesting, I was not aware they did not care about their eyeballs.
> >>
> >> Actually I am very confident lots of folks there would love to dig into
> >> your issue to actually resolve it. As when it is hitting you, it might
> >> hit other customers.
> >>
> >> Is that also not why there is this huge SRE department with lots of IPv6
> >> knowledgeable folks?
> >>
> >> Greets,
> >> Jeroen
> >>
> >>
>
>

Re: Best US Tunnelbroker for Youtube

[Owen DeLong]

My understanding is that almost all of the Comcast network is now IPv6 capable.

Owen

On Aug 20, 2014, at 10:26 AM, Ryan Shea  wrote:

> Not sure I've seen any evidence (or implied) that the tunnel was the
> problem. My issue as far as I know is at the application layer and other
> end-user experiences seemed a reasonable way to pick a direction. I will
> work with HE though and provide them some details.
> 
> Agreed, from an end-user perspective it can be often be clear as mud
> whether I am using v6, or whether my Chromecast or Android device even
> implements happy eyeballs. The relatively new "experiencing problems?"
> butter bar that shows up beneath a video with notable buffering problems
> (even at low quality levels) sends the user through to details about the
> service provider, in this case HE. Over the past couple years YouTube has
> been my canary to know when I've received a new IP from Verizon and I need
> to go fix my tunnel -- video loading takes frever on
> Android/Chromecast/GoogleTV (which hints that happy eyeballs, if it exists
> for Android, isn't working so well for the YouTube app).
> 
> I can't get native v6 at my home -- I'm probably not in a particularly
> unique situation. Not to rathole the dicsussion, but as far as I know (save
> for some small DSL providers) unless I'm in a gFiber city or happen to be
> in the portion of the Comcast network that provides native v6 I'm out of
> luck. I don't plan on moving to solve this problem.
> 
> 
> 
> On Wed, Aug 20, 2014 at 12:42 PM, Jeroen Massar  wrote:
> 
>> On 2014-08-20 18:21, Ryan Shea wrote:
>>> IRC is a good suggestion, thanks. They'll likely be helpful.
>>> 
>>> I see no indication of any throttling from my ISP - I can blast data at
>>> full speed to my home from my server and work (with native v6
>>> connections).
>> 
>> Does that path between your $home and $server go over the tunnel you
>> find so "slow"?
>> 
>> If so, then you have just nicely excluded that the tunnel is NOT the
>> problem.
>> 
>> Hence, why traceroutes would be so extremely useful.
>> 
>> 
>>> Contacting my ISP (Verizon FiOS) is virtually never a
>>> reasonable path to a solution.
>> 
>> google(Verizon FiOS throttle) = 71.900 results. One would almost think
>> that there might sometimes be issues there.
>> 
>> Also, do realize that the IPv6 path you are using goes over a shared
>> host (the Tunnel Broker PoP) that has IPv4 and IPv6 capacity that might
>> be shared in various points of the paths your packets cross.
>> 
>> Did you test at the same time of your "blast data" that the IPv6 Youtube
>> was working fine?
>> 
>> Another thing, as browsers now do "Happy Eyeballs" (which is really
>> horrible to diagnose issues with on OSX), did you check if everything is
>> really going over IPv6? (hence the tcpdump/wireshark).
>> 
>> [..]
>>> To be clear, I was seeking opinions/experiences on a list that was
>>> likely to have a high occurrence of folk with v6 tunnels.
>> 
>> Tunnels are for endusers who still are at ISPs who don't do IPv6 natively.
>> 
>> NANOG has operators who have been running native IPv6 for over a decade.
>> 
>> Hence, StackExchange might be useful for your purpose.
>> 
>>> You have
>>> etiquette suggestions, but not YouTube over tunnelbroker suggestions. I
>>> apparently bring out your inner grump? Do you need a hug?
>> 
>> My cat videos are streaming perfectly fine...
>> 
>>> Burning Google engineering time would be a sub-optimal way to get HD cat
>>> videos at home with the least time spent.
>> 
>> Interesting, I was not aware they did not care about their eyeballs.
>> 
>> Actually I am very confident lots of folks there would love to dig into
>> your issue to actually resolve it. As when it is hitting you, it might
>> hit other customers.
>> 
>> Is that also not why there is this huge SRE department with lots of IPv6
>> knowledgeable folks?
>> 
>> Greets,
>> Jeroen
>> 
>> 

Re: where to go to understand DDoS attack vector

[Brian Rak]

On 8/26/2014 8:28 PM, Larry Sheldon wrote:

On 8/26/2014 08:31, Roland Dobbins wrote:


On Aug 26, 2014, at 8:26 PM, Stephen Satchell  wrote:


qotd17/udp  quote


No, that's the protocol number - 17 is UDP - not the port number.



Really?

http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

udp DID used to be protocol 17, but it is a fact that quotd runs on 
udp port 17.




Yes, he is correct.  This is not UDP port 17.

> 8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto 
UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1


Protocol: UDP (IP protocol 17)
Source Port: 2072
Dest Port: 27015

What protocol is UDP now, if it's not 17?

Re: where to go to understand DDoS attack vector

[Larry Sheldon]

On 8/26/2014 08:31, Roland Dobbins wrote:


On Aug 26, 2014, at 8:26 PM, Stephen Satchell  wrote:


qotd17/udp  quote


No, that's the protocol number - 17 is UDP - not the port number.



Really?

http://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers

udp DID used to be protocol 17, but it is a fact that quotd runs on udp 
port 17.



--
The unique Characteristics of System Administrators:

The fact that they are infallible; and,

The fact that they learn from their mistakes.

Cisco Tech needed in San Franscisco

[JoeSox]

Hi Everyone,
Took over a new network and they have no onsite Cisco tech help for one of
my remote locations.
Can someone email me directly a company I can contract with to get help
this Wed night in downtown San Francisco (we can push maintenance to Friday
night as well  but shooting for tomorrow)?

--
Thank You, Joe

Re: Urgent

[Owen DeLong]

No, the God’s no longer listen on that address… Try FF0E::6:6:6

Owen

On Aug 18, 2014, at 8:42 AM, Justin M. Streiner  wrote:

> On Mon, 18 Aug 2014, Daniel Roesen wrote:
> 
>> Just send your request to the all-gods well-known multicast group.
> 
> 224.6.6.6?
> 
> jms
> 
>> On Mon, Aug 18, 2014 at 05:20:48PM +, Kain, Rebecca (.) wrote:
>>> Which one?
>>> 
>>> -Original Message-
>>> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of ra...@psg.com
>>> Sent: Monday, August 18, 2014 1:00 PM
>>> To: nanog@nanog.org
>>> Subject: Urgent
>>> 
>>> Contact for God, please reach out to me offlist.
>>> 
>>> Regards,
>>> -AS666 NOC
>> 
>> -- 
>> CLUE-RIPE -- Jabber: d...@cluenet.de -- dr@IRCnet -- PGP: 0xA85C8AA0
>> 

Re: where to go to understand DDoS attack vector

[John]

On 8/26/2014 10:40 AM, Miles Fidelman wrote:
That's about as far as I've gotten.  What has me scratching my head is 
what is setting the source port.  This has all the earmarks of a 
reflection attack, except... I'm not running anything that presents as 
port 2072 (msync) - so either the attack is making very clever use of 
some other open server, or the board's BMC is infected by a bot. 
Unfortunately, with the port now blocked, and what was intermittent in 
any case - it's a little hard to monitor incoming traffic to see what 
might be trigger traffic.  Sigh...


From the traffic dump and description, this was highly likely to be a 
direct attack and not an amplification/reflection hit. I don't know of 
reflectors that run on port 2072; but, bots are routinely used to send 
UDP length 29 (payload length 1) packets.


Older Supermicro IPMI devices have multiple published exploits including 
the much-publicized port-49152 vulnerability that provides the admin 
password in the clear (described at 
http://blog.cari.net/carisirt-yet-another-bmc-vulnerability-and-some-added-extras/ 
and other places). Many device owners also never change the default u/p, 
in which case an exploit doesn't even need to be used. The attacker will 
typically use a tool to scan the IPv4 space for vulnerable hosts; the 
tool logs in and installs a bot that connects to a C&C server and is 
later used for attacks. The same procedure is followed with other 
easily-compromised devices, including Hikvision DVRs/NVRs and various 
routers including the Chinese Telecom F420. Resulting botnets can be 
tens or even hundreds of thousands of hosts in size.


IPMI devices have been used quite regularly for attacks for a couple of 
months now -- as soon as that vulnerability was made public, the 
toolmakers started using it. The best defense against current and 
yet-to-be-discovered IPMI vulnerabilities is to make sure that your IPMI 
devices are not open to the public internet, as Roland said.


-John

Re: network quality measurement probes+reporting

[Dan Snyder]

http://netbeez.net/




On Tue, Aug 26, 2014 at 1:13 PM, Saku Ytti  wrote:

> Anyone can recommend or even just name drop network quality measurement
> kit?
>
> I'm only familiar with IP SLA, RPM and Creanord (and various inhouse
> tools:)
>
> What I'd like to see
>   - 1us or better one-way jitter (no need for clock sync, just accurate
> clock)
>   - tens or 100us one-way latency (as good as cheaply can get sync, ntp is
> ok)
>   - 1us or better RTT
>   - any-to-any measurement, not just hub<->spoke (or sufficiently cheap
> hub)
>   - 100pps to 100 measurement points on 8 CoS (ish, less may be acceptable)
>   - randomized payload pattern + verification (to catch bit mangling)
>   - randomized sport/dport (to put traffic in each ECMP/LAG combo, long
> term)
>   - programmatic accesss and useful documents to measurement data (e.g.
> some OSS TSDB)
>   - high quality, fast, useful graphical reporting and alarming
>   - support for TWAMP and OWAMP responders
>   - indicative price <100kEUR CAPEX and <2000EUR YRC for 100 nodes solution
>
> --
>   ++ytti
>

Re: network quality measurement probes+reporting

[Tarko Tikan]

hey,


   - any-to-any measurement, not just hub<->spoke (or sufficiently cheap hub)


May I add:

- local data caching on probes (don't want to see holes in data if 
central collector/NMS is unavailable for short period)

- DHCPv4/6 support with possibility to do periodic release/renew

--
tarko

Re: network quality measurement probes+reporting

[Joe Loiacono]

Perhaps you've considered this already ... not sure if it gets 1us or not 
: bwctl?

http://software.internet2.edu/bwctl/

Joe



From:   Saku Ytti 
To: nanog@nanog.org
Date:   08/26/2014 01:15 PM
Subject:network quality measurement probes+reporting
Sent by:"NANOG" 



Anyone can recommend or even just name drop network quality measurement 
kit?

I'm only familiar with IP SLA, RPM and Creanord (and various inhouse 
tools:)

What I'd like to see
  - 1us or better one-way jitter (no need for clock sync, just accurate 
clock)
  - tens or 100us one-way latency (as good as cheaply can get sync, ntp is 
ok)
  - 1us or better RTT
  - any-to-any measurement, not just hub<->spoke (or sufficiently cheap 
hub)
  - 100pps to 100 measurement points on 8 CoS (ish, less may be 
acceptable)
  - randomized payload pattern + verification (to catch bit mangling)
  - randomized sport/dport (to put traffic in each ECMP/LAG combo, long 
term)
  - programmatic accesss and useful documents to measurement data (e.g. 
some OSS TSDB)
  - high quality, fast, useful graphical reporting and alarming
  - support for TWAMP and OWAMP responders
  - indicative price <100kEUR CAPEX and <2000EUR YRC for 100 nodes 
solution

-- 
  ++ytti

Re: where to go to understand DDoS attack vector

[Miles Fidelman]

me wrote:


On 08/26/2014 07:58 AM, Roland Dobbins wrote:
On Aug 26, 2014, at 8:37 PM, John York  
wrote:


In this case, 17 is both the protocol and port number. Confusing 
coincidence :)

Not in this output which the OP sent to the list:

8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], 
proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  
@^i.C...
0x0020:          
..


18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], 
proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  
@^i.C...
0x0020:          
..
18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], 
proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  
@^i.C...
0x0020:          
..
18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], 
proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  
@^i.C...
0x0020:          
..

Source port 2072, destination port 27015.


Been awhile since I got to dig into hex tcpdump but spent the time 
anyway. A UDP data segment that is 9 bytes long and only contains a 
"C" (0x43) ? And looks like to a Steam/Half-life (27015) gaming port. 
Not sure what the "C" is used for with those systems but guessing it's 
some sort of request?


That's about as far as I've gotten.  What has me scratching my head is 
what is setting the source port.  This has all the earmarks of a 
reflection attack, except... I'm not running anything that presents as 
port 2072 (msync) - so either the attack is making very clever use of 
some other open server, or the board's BMC is infected by a bot. 
Unfortunately, with the port now blocked, and what was intermittent in 
any case - it's a little hard to monitor incoming traffic to see what 
might be trigger traffic.  Sigh...


Thanks,

Miles

RE: network quality measurement probes+reporting

[James Sink]

The licenses can get pricey, but AppNeta is worth a look.

http://www.appneta.com/products/pathview/

-James

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Saku Ytti
Sent: Tuesday, August 26, 2014 10:13 AM
To: nanog@nanog.org
Subject: network quality measurement probes+reporting

Anyone can recommend or even just name drop network quality measurement kit?

I'm only familiar with IP SLA, RPM and Creanord (and various inhouse tools:)

What I'd like to see
  - 1us or better one-way jitter (no need for clock sync, just accurate clock)
  - tens or 100us one-way latency (as good as cheaply can get sync, ntp is ok)
  - 1us or better RTT
  - any-to-any measurement, not just hub<->spoke (or sufficiently cheap hub)
  - 100pps to 100 measurement points on 8 CoS (ish, less may be acceptable)
  - randomized payload pattern + verification (to catch bit mangling)
  - randomized sport/dport (to put traffic in each ECMP/LAG combo, long term)
  - programmatic accesss and useful documents to measurement data (e.g. some 
OSS TSDB)
  - high quality, fast, useful graphical reporting and alarming
  - support for TWAMP and OWAMP responders
  - indicative price <100kEUR CAPEX and <2000EUR YRC for 100 nodes solution

-- 
  ++ytti

Re: where to go to understand DDoS attack vector

[Brian Rak]

On 8/26/2014 12:52 PM, me wrote:


On 08/26/2014 07:58 AM, Roland Dobbins wrote:
On Aug 26, 2014, at 8:37 PM, John York  
wrote:


In this case, 17 is both the protocol and port number. Confusing 
coincidence :)

Not in this output which the OP sent to the list:

8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], 
proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  
@^i.C...
0x0020:          
..


18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], 
proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  
@^i.C...
0x0020:          
..
18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], 
proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  
@^i.C...
0x0020:          
..
18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], 
proto UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  
@^i.C...
0x0020:          
..

Source port 2072, destination port 27015.


Been awhile since I got to dig into hex tcpdump but spent the time 
anyway. A UDP data segment that is 9 bytes long and only contains a 
"C" (0x43) ? And looks like to a Steam/Half-life (27015) gaming port. 
Not sure what the "C" is used for with those systems but guessing it's 
some sort of request?


It's pretty tough to say without knowing exactly what game is running 
there.  While 27015 was originally used for Half Life, it's been used by 
a wide range of games at this point.  Pretty much all the Valve games 
use this port, as well as a number of third party games that are based 
on the Steamworks SDK.


Trying to figure out exactly what the game server thinks the packet is 
is not likely to help you figure out why it's being sent.  You should 
instead be figuring out why your IPMI controller is compromised.  It 
could also be reflection, 2072 is within the port range that is usually 
used for KVM or remote media by the IPMI controllers (though, they're 
usually TCP and not UDP).

network quality measurement probes+reporting

[Saku Ytti]

Anyone can recommend or even just name drop network quality measurement kit?

I'm only familiar with IP SLA, RPM and Creanord (and various inhouse tools:)

What I'd like to see
  - 1us or better one-way jitter (no need for clock sync, just accurate clock)
  - tens or 100us one-way latency (as good as cheaply can get sync, ntp is ok)
  - 1us or better RTT
  - any-to-any measurement, not just hub<->spoke (or sufficiently cheap hub)
  - 100pps to 100 measurement points on 8 CoS (ish, less may be acceptable)
  - randomized payload pattern + verification (to catch bit mangling)
  - randomized sport/dport (to put traffic in each ECMP/LAG combo, long term)
  - programmatic accesss and useful documents to measurement data (e.g. some 
OSS TSDB)
  - high quality, fast, useful graphical reporting and alarming
  - support for TWAMP and OWAMP responders
  - indicative price <100kEUR CAPEX and <2000EUR YRC for 100 nodes solution

-- 
  ++ytti

Re: Verizon (NY, LEC) issues around 1200 EDT?

[Christopher J. Pilkington]

Well, I need to wind my watch... we saw path AIS on our OCs out of that CO
at 12:23:39 EDT. Cleared ten seconds later. Voice PRIs took a bit longer to
recover.


On Tue, Aug 26, 2014 at 12:46 PM, Christopher J. Pilkington 
wrote:

> We saw multiple service outages from Verizon local in NYC, specifically
> our offices served out of Broad Street (NYCMNYBS). Lost multiple OC as well
> as local voice PRI.
>
> I haven't dug exact times out of the logs yet, but was around noon EDT.
>
> Anyone else see similar or know what went on?
>
> -cjp
>

Re: where to go to understand DDoS attack vector

[me]

On 08/26/2014 07:58 AM, Roland Dobbins wrote:

On Aug 26, 2014, at 8:37 PM, John York  wrote:


In this case, 17 is both the protocol and port number. Confusing coincidence :)

Not in this output which the OP sent to the list:


8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), 
length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c E.@.8 
.;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  @^i.C...
0x0020:          ..



18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), 
length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c E.@.8 
.;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  @^i.C...
0x0020:          ..
18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), 
length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c E.@.8 
.;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  @^i.C...
0x0020:          ..
18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP (17), 
length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c E.@.8 
.;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300  @^i.C...
0x0020:          ..

Source port 2072, destination port 27015.


Been awhile since I got to dig into hex tcpdump but spent the time 
anyway. A UDP data segment that is 9 bytes long and only contains a "C" 
(0x43) ? And looks like to a Steam/Half-life (27015) gaming port. Not 
sure what the "C" is used for with those systems but guessing it's some 
sort of request?


--john



--
Roland Dobbins  // 

Equo ne credite, Teucri.

  -- Laocoön

Verizon (NY, LEC) issues around 1200 EDT?

[Christopher J. Pilkington]

We saw multiple service outages from Verizon local in NYC, specifically our
offices served out of Broad Street (NYCMNYBS). Lost multiple OC as well as
local voice PRI.

I haven't dug exact times out of the logs yet, but was around noon EDT.

Anyone else see similar or know what went on?

-cjp

Re: where to go to understand DDoS attack vector

[Roland Dobbins]

On Aug 26, 2014, at 11:02 PM, valdis.kletni...@vt.edu wrote:

> Took me a few seconds to figure it out too, am a tad low on caffeine today. :)

doh, lack of proper sanitization/escaping strikes again!

--
Roland Dobbins  // 

   Equo ne credite, Teucri.

  -- Laocoön



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Optical Transport Platform

[Nick Colton]

We've been using Cyan for our transport/backbone for several years now and
have been very happy with them.  Of all the platforms we've used their
element management system has been the easiest to train.

Nick Colton
Director of Network Operations
ALLO Communications
610 Broadway - PO Box 1123
Imperial, NE 69033
308-633-7814

 






On Mon, Aug 25, 2014 at 1:59 PM, Kenneth McRae  wrote:

> Greetings Everyone..
>
> Here is one for the transport/backbone/data center guys.  Can you tell
> which manufactures you like for optical transport?  I am looking for a
> solution to provide 8 channels of DWDM using tuned optics in my network
> gear. DWDM transport will be used for multiplexing only (no amplification
> or wavelength generation).  So far, I have been looking at Fiberstore and
> PacketLight..
>
> Any suggestions?
>
> All feedback is greatly appreciated!
>
> Thanks
>
> Kenneth

Re: where to go to understand DDoS attack vector

[Valdis . Kletnieks]

On Tue, 26 Aug 2014 18:57:27 +0700, Roland Dobbins said:
>.  The 'mailto:' bit is interesting; it might work sort of like SNMP 
>reflection/amplificati

Nope.  It's a red herring, somebody's MUA trying to get *far* too clever with
the fact that there's a literal "@.8" in the ascii dump part of the packet.

Took me a few seconds to figure it out too, am a tad low on caffeine today. :)



pgp6An7RtuHdc.pgp
Description: PGP signature

Re: where to go to understand DDoS attack vector

[Roland Dobbins]

On Aug 26, 2014, at 8:37 PM, John York  wrote:

> In this case, 17 is both the protocol and port number. Confusing coincidence 
> :)

Not in this output which the OP sent to the list:

> 8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP 
> (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
>0x: 4500 001d  4000 3811 088c cf9a 3b8c E.@.8 
> .;.
>0x0010: 405e eebf 0818 6987 0009 10f8 4300  
> @^i.C...
>0x0020:          ..


> 18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP 
> (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
>0x: 4500 001d  4000 3811 088c cf9a 3b8c E.@.8 
> .;.
>0x0010: 405e eebf 0818 6987 0009 10f8 4300  
> @^i.C...
>0x0020:          ..
> 18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP 
> (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
>0x: 4500 001d  4000 3811 088c cf9a 3b8c E.@.8 
> .;.
>0x0010: 405e eebf 0818 6987 0009 10f8 4300  
> @^i.C...
>0x0020:          ..
> 18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto UDP 
> (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
>0x: 4500 001d  4000 3811 088c cf9a 3b8c E.@.8 
> .;.
>0x0010: 405e eebf 0818 6987 0009 10f8 4300  
> @^i.C...
>0x0020:          ..

Source port 2072, destination port 27015.

--
Roland Dobbins  // 

   Equo ne credite, Teucri.

  -- Laocoön

RE: where to go to understand DDoS attack vector

[John York]

In this case, 17 is both the protocol and port number. Confusing
coincidence :)

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Roland
> Dobbins
> Sent: Tuesday, August 26, 2014 8:32 AM
> To: nanog@nanog.org
> Subject: Re: where to go to understand DDoS attack vector
>
>
> On Aug 26, 2014, at 8:26 PM, Stephen Satchell  wrote:
>
> > qotd17/udp  quote
>
> No, that's the protocol number - 17 is UDP - not the port number.
>
> --
> Roland Dobbins  //
> 
>
>Equo ne credite, Teucri.
>
> -- Laocoön

Re: where to go to understand DDoS attack vector

[Roland Dobbins]

On Aug 26, 2014, at 8:26 PM, Stephen Satchell  wrote:

> qotd17/udp  quote

No, that's the protocol number - 17 is UDP - not the port number.

--
Roland Dobbins  // 

   Equo ne credite, Teucri.

  -- Laocoön

Re: where to go to understand DDoS attack vector

[Stephen Satchell]

qotd17/udp  quote

You're not blocking small services outbound at the edge?

On 08/26/2014 05:18 AM, Miles Fidelman wrote:
> Roland Dobbins wrote:
>> On Aug 26, 2014, at 6:48 PM, Miles Fidelman
>>  wrote:
>>
>>> Immediate issue is dealt with (at least for us, target seems to be
>>> off the air) - but want to understand this, report it, all of that.
>> IPMI boards are reported as being used in reflection/amplification
>> attacks of various kinds; the ntp one is straightforward, as you note.
>>
>> This may be some sort of chargen-like packet reflector that's either
>> built into the firmware, or that an attacker has managed to insert,
>> somehow.  The 'mailto:' bit is interesting; it might work sort of like
>> SNMP reflection/amplification attacks work, where the attacker is
>> using some sort of management functionality to walk the device config
>> or somesuch, packetize it, and blast it out as packet-padding.
> 
> Can you say a bit more about what I might look for in trying to track
> this down?
> 
>>
>> Does the target of the attack have flow telemetry records or complete
>> packets?  Because the one you posted looked incomplete (29 bytes?) . . .
>>
>>
> 
> Unfortunately, all I have is what they sent to our abuse address -
> understandably, they've been a bit busy and not as responsive to further
> inquiries as one might hope.
> 
> But, having said that, this looks like all they have.  They seem to be
> getting these from lots of different places around the net, they just
> sent a filtered excerpt - here's a larger sample:
> 
> 18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto
> UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
> 0x: 4500 001d  4000 3811 088c cf9a 3b8c
> E.@.8 .;.
> 0x0010: 405e eebf 0818 6987 0009 10f8 4300 
> @^i.C...
> 0x0020:         
> ..
> 18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto
> UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
> 0x: 4500 001d  4000 3811 088c cf9a 3b8c
> E.@.8 .;.
> 0x0010: 405e eebf 0818 6987 0009 10f8 4300 
> @^i.C...
> 0x0020:         
> ..
> 18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto
> UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
> 0x: 4500 001d  4000 3811 088c cf9a 3b8c
> E.@.8 .;.
> 0x0010: 405e eebf 0818 6987 0009 10f8 4300 
> @^i.C...
> 0x0020:         
> ..
> 
> On closer reading, what they captured does seem to be "proto UDP (17),
> length 29)" and "UDP, length 1"
> 
> Thanks!
> 
> Miles
> 

Re: where to go to understand DDoS attack vector

[Roland Dobbins]

On Aug 26, 2014, at 7:18 PM, Miles Fidelman  wrote:

> Can you say a bit more about what I might look for in trying to track this 
> down?

Fuzz your IPMI boards?

;>

My guess is that this is going to come to light sooner rather than later.  
We're getting reports of other DDoS attacks which seem to match this scenario 
involving IPMI boards.

There's no real reason to try and track it down, from an operational 
standpoint, is there>  Management-plane things like IMPI boards shouldn't be 
open to the general Internet; put them behind ACLs and use a VPN.  Problem 
solved.

--
Roland Dobbins  // 

   Equo ne credite, Teucri.

  -- Laocoön

Re: where to go to understand DDoS attack vector

[Miles Fidelman]

Roland Dobbins wrote:

On Aug 26, 2014, at 6:48 PM, Miles Fidelman  wrote:


Immediate issue is dealt with (at least for us, target seems to be off the air) 
- but want to understand this, report it, all of that.

IPMI boards are reported as being used in reflection/amplification attacks of 
various kinds; the ntp one is straightforward, as you note.

This may be some sort of chargen-like packet reflector that's either built into 
the firmware, or that an attacker has managed to insert, somehow.  The 
'mailto:' bit is interesting; it might work sort of like SNMP 
reflection/amplification attacks work, where the attacker is using some sort of 
management functionality to walk the device config or somesuch, packetize it, 
and blast it out as packet-padding.


Can you say a bit more about what I might look for in trying to track 
this down?




Does the target of the attack have flow telemetry records or complete packets?  
Because the one you posted looked incomplete (29 bytes?) . . .




Unfortunately, all I have is what they sent to our abuse address - 
understandably, they've been a bit busy and not as responsive to further 
inquiries as one might hope.


But, having said that, this looks like all they have.  They seem to be 
getting these from lots of different places around the net, they just 
sent a filtered excerpt - here's a larger sample:


18:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto 
UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300 
 @^i.C...
0x0020:          
..
18:33:58.484625 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto 
UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300 
 @^i.C...
0x0020:          
..
18:33:58.486137 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto 
UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300 
 @^i.C...
0x0020:          
..


On closer reading, what they captured does seem to be "proto UDP (17), 
length 29)" and "UDP, length 1"


Thanks!

Miles

--
In theory, there is no difference between theory and practice.
In practice, there is.    Yogi Berra

Re: where to go to understand DDoS attack vector

[Roland Dobbins]

On Aug 26, 2014, at 6:48 PM, Miles Fidelman  wrote:

> Immediate issue is dealt with (at least for us, target seems to be off the 
> air) - but want to understand this, report it, all of that.

IPMI boards are reported as being used in reflection/amplification attacks of 
various kinds; the ntp one is straightforward, as you note.

This may be some sort of chargen-like packet reflector that's either built into 
the firmware, or that an attacker has managed to insert, somehow.  The 
'mailto:' bit is interesting; it might work sort of like SNMP 
reflection/amplification attacks work, where the attacker is using some sort of 
management functionality to walk the device config or somesuch, packetize it, 
and blast it out as packet-padding.

Does the target of the attack have flow telemetry records or complete packets?  
Because the one you posted looked incomplete (29 bytes?) . . .

--
Roland Dobbins  // 

   Equo ne credite, Teucri.

  -- Laocoön

where to go to understand DDoS attack vector

[Miles Fidelman]

Hi Folks,

Possibly a little off-topic for nanog, but I couldn't think of anywhere 
else to ask this (suggestions please!):


We just discovered a vulnerability the hard way - someone used one of 
our IPMI boards as a vector for a DDoS attack (well, I guess the real 
hard way would be to have been on the receiving end, but...).


Anyway... aside from some obvious issues, I've been learning a lot about 
the vulnerabilities of Supermicro IPMI boards (and busily locking them 
down).  The one that's tricky, though, is that this was a 
reflection/amplification attack.


Conveniently, the attackee's data center operator managed to capture 
incoming packets with tcpdump, and they all looked like this:


8:33:58.482193 IP (tos 0x0, ttl 56, id 0, offset 0, flags [DF], proto 
UDP (17), length 29) x.x.x.x.2072 > x.x.x.x.27015: UDP, length 1
0x: 4500 001d  4000 3811 088c cf9a 3b8c 
E.@.8 .;.
0x0010: 405e eebf 0818 6987 0009 10f8 4300 
 @^i.C...
0x0020:          
..

(obviously, with the IP addresses removed).

It could be that someone planted a bot in the IPMI board (just starting 
to do some forensics - currently hampered by being on travel, and having 
blocked all the ports from the outside world - need to get to the 
datacenter and make some hardwired connections) - but it looks a lot 
more like a reflection/amplification attack - particularly since the 
target seems to have been a game host, and port 27015 is used by the 
game halflife.  But


Now I understand reflected DNS and NTP attacks - but the outbound port, 
2072 (registered for GlobeCom msync) is neither, nor is it anything that 
we're running - which kind of begs the question of how this might be 
working.  Any thoughts?  Any pointers? Any starting points?


Immediate issue is dealt with (at least for us, target seems to be off 
the air) - but want to understand this, report it, all of that.


Thanks very much,

Miles Fidelman

--
In theory, there is no difference between theory and practice.
In practice, there is.    Yogi Berra