Re: Consumer networking head scratcher

[Ryan Pugatch]

On Thu, Mar 2, 2017, at 12:24 AM, Roland Dobbins wrote:
> On 2 Mar 2017, at 9:55, Oliver O'Boyle wrote:
> 
> > Currently, I have 3 devices connected. :)
> 
> You could have one or more botted machines launching outbound DDoS 
> attacks, potentially filling up the NAT translation table and/or getting 
> squelched by your broadband access provider with layer-4 granularity.  
> And the boxes themselves could be churning away due to being compromised 
> (look at CPU and memory stats over time).  Aggressive horizontal 
> scanning is often a hallmark of botted machines, and it can interrupt 
> normal network access on the botted hosts themselves.
> 
> I don't actually think that's the case, given the symptomology you 
> report, but just wanted to put it out there for the list archive.
> 
> What about DNS issues?  Are you sure that you really have a networking 
> issue, or are you having intermittent DNS resolution problems caused by 
> flaky/overloaded/attacked recursivs, EDNS0 problems (i.e., filtering on 
> DNS responses > 512 bytes), or TCP/53 blockage?  Different host 
> OSes/browsers/apps exhibit differing re-query characteristics.  Are the 
> Windows boxes and the other boxes set to use the same recursors?  Can 
> you resolve DNS requests during the outages?
> 
> Are your boxes statically-addressed, or are they using DHCP?  
> Periodically-duplicate IPs can cause intermittent symptoms, too.  If 
> you're using the consumer router as a DHCP server, DHCP-lease nonsense 
> could be a contributing factor.
> 
> Are the Windows boxes running some common application/service which 
> updates and/or churns periodically?  Are they members of a Windows 
> workgroup?  All kinds of strange name-resolution stuff goes on with 
> Windows-specific networking.
> 
> Also, be sure to use -n with traceroute.  tcptraceroute is useful, too.  
> netstat -rn should work on Windows boxes, IIRC.
> 
> ---
> Roland Dobbins 

It isn't a DNS issue as trying to access resources via IP address
directly also have the issue.

What became clear to me last night is that this actually also impacts my
Mac, and that it has to do with traffic not properly making it back to
my machines.  When the issue occurs, my traffic makes it out to the
destination, the destination responds, but that packet never makes it to
my laptop, for example.  I tested by sending traffic to a server I
control and doing PCAPs on both ends.

Thanks,
Ryan

Re: Consumer networking head scratcher

[Ryan Pugatch]

On Thu, Mar 2, 2017, at 10:32 AM, Dann Schuler wrote:
> Just a quick sanity check here since I know we can occasionally overlook
> the simple things.  You have updated the firmware to the latest available
> version correct?  Have you checked for any odd services like QoS,
> parental controls or an IDS?  Have you tried wiping it to factory default
> and reconfiguring it?
> 
> What happens if you give the affected machine a new IP?  Could it be some
> service on the device affecting that specific IP?
> 

Yes, I've done all of these.  It was running the latest version of code
and I even tried rolling back.  Disabled SPI firewalls, ipv6, verified
QoS and parental controls are off, etc.

The issue impacts multiple device so doesn't appear specific to one IP.

Thanks

Re: Consumer networking head scratcher

[Ryan Pugatch]

On Wed, Mar 1, 2017, at 09:29 PM, Oliver O'Boyle wrote:

> Each device associated with the AP consumes memory. Small low-end
> routers don't typically come with much memory. If you've got a lot of
> devices associated with the AP you will run out of memory. I'm not
> sure how many devices you're connecting, though. Three will not cause
> this problem. 30 might.
> 

> O.

> 



Currently, I have 3 devices connected. :)

Re: Consumer networking head scratcher

[Ryan Pugatch]

On Wed, Mar 1, 2017, at 06:35 PM, Jean-Francois Mezei wrote:
> On 2017-03-01 11:28, Ryan Pugatch wrote:
> 
> > At random times, my Windows machines (Win 7 and Win 10, attached to the
> > network via WiFi, 5GHz) lose connectivity to the Internet. 
> 
> > For what it's worth, the router is a Linksys EA7300 that I just picked
> > up.
> 
> 
> Way back when, I have a netgear router. It ended having a limit on its
> NAT translation table, and when I had too many connections going at same
> time (or not yet timed out), I would lose connection. There was an
> unofficial patch to the firmware (litterally a patch in code that
> defined table size) to increase that table to 1000- as I recall.
> 
> Does the Linksys have a means to display the NAT translation table and
> see if maybe connections are lost when that table is full and lots of
> connections have not yet timed out ?
> 


It doesn't seem to provide visibility into the NAT tables.  However, I'm
starting to think you might be on to something.

The issue actually happened to my Mac tonight, and sure enough the
traceroute dies at the same time.  So, it isn't just the Windows
machines impacted.

I did a packet capture on my end, and on a server somewhere that I
control and sent pings from my laptop to the server.

The server received my ICMP packets and responded, but those responses
never made it back to my laptop.

Meanwhile, my Roku is actively streaming from the Internet, so it's not
like the Internet was down.

Re: Consumer networking head scratcher

[Ryan Pugatch]

On Wed, Mar 1, 2017, at 03:58 PM, iam...@gmail.com wrote:
> On many non-windows OS (Mac OSX, Linux, FreeBSD etc.) you can specify
> ICMP
> traceroute using -I:
> 
> traceroute -I google.com
> 
> I wonder if this would replicate your experience with Windows tracert


Definitely on my list to test.

Thanks.

Re: Consumer networking head scratcher

[Ryan Pugatch]

On Wed, Mar 1, 2017, at 02:57 PM, William Herrin wrote:
> On Wed, Mar 1, 2017 at 2:31 PM, Ryan Pugatch <r...@lp0.org> wrote:
> > So in that case, I would be back to my original issue where I stop being
> > able to pass traffic to the Internet, and when that happens my
> > traceroute always dies at the same hop.  After disconnecting and
> > reconnecting, the same traceroute will go all the way through.
> 
> Hi Ryan,
> 
> Next step: run Wireshark and see what you see during the traceroutes.
> Are they leaving with a reasonable TTL? Is it certain that nothing
> returns? Are the packets going to the ethernet MAC address you expect
> them to?
> 
> I had a fun problem once when I cloned some VMs but neglected to
> change the source MAC address. They all seemed to work under light
> load but get two downloading at once and suddenly they both
> experienced major packet loss.
> 
> Regards,
> Bill
> 

Definitely the direction I'm going.  Even aside from the traceroutes,
I'm going to capture some regular web traffic to see what is happening. 
Planning to send traffic to a machine I control to see if any packets
are actually making it through at all.

I'm not sure if this new Linksys router has any packet capture ability
that is exposed to the end user, but I'd also love be able to see what's
actually going through the router itself.

Thanks,
Ryan

Re: Consumer networking head scratcher

[Ryan Pugatch]

On Wed, Mar 1, 2017, at 02:04 PM, William Herrin wrote:
> > On Wed, Mar 1, 2017, at 01:23 PM, Aaron Gould wrote:
> >> That's strange... it's like the TTL on all Windows IP packets are
> >> decrementing more and more as time goes on causing you to get less and
> >> less hops into the internet
> 
> Hi Ryan,
> 
> Windows tracert uses ICMP echo-request packets to trace the path. It
> expects either an ICMP destination unreachable message or an ICMP echo
> response message to come back. The final hop in the trace will return
> an ICMP echo-response or an unreachable-prohibited. The ones prior to
> the final hop will return an unreachable-time-exceeded if they return
> anything at all.
> 
> If the destination does not respond to ping, if those pings are
> dropped, or if it responds with an unreachable that's dropped you will
> not receive a response and the tracert will not find its end. That's
> why you're seeing the "decrementing" behavior you describe.
> 
> I have no information about whether comcast blocks pings to its routers.
> 
> Regards,
> Bill Herrin
> 

I see what you're saying, and that could explain the decrementing
behavior I'm seeing which ultimately is not a real indicator of the
problem I am having.

So in that case, I would be back to my original issue where I stop being
able to pass traffic to the Internet, and when that happens my
traceroute always dies at the same hop.  After disconnecting and
reconnecting, the same traceroute will go all the way through.

Thanks for the thoughts.

Re: Consumer networking head scratcher

[Ryan Pugatch]

The issue doesn't happen with my previous router, and I've tested
multiple computers (one that isn't mine.)

It doesn't seem like it decrements over time.. it just dies sooner as I
trace further up the path.  I can consistently die at the 7th hop if I
try to go to Google, but if I trace to the 6th hop, it'll die at the 5th
hop!


On Wed, Mar 1, 2017, at 01:23 PM, Aaron Gould wrote:
> That's strange... it's like the TTL on all Windows IP packets are
> decrementing more and more as time goes on causing you to get less and
> less hops into the internet
> 
> I wonder if it's a bug/virus/malware affecting only your windows
> computers.
> 
> -Aaron
> 
> 

Consumer networking head scratcher

[Ryan Pugatch]

Hi everyone,

I've got a real head scratcher that I have come across after replacing
the router on my home network.

I thought I'd share because it is a fascinating issue to me.

At random times, my Windows machines (Win 7 and Win 10, attached to the
network via WiFi, 5GHz) lose connectivity to the Internet.  They can
continue to access internal resources, such as the router's admin
interface.  Other devices including Macs, iPhones, Android phones, and
Rokus never have this issue.

I realized that on the Windows machines, when the connection drops, if I
run a traceroute, it dies at a certain hop every time (out in Comcast's
network, who is my ISP) even though a Mac sitting right next to it is
able to go all the way through to the destination.

The even stranger thing I discovered last night is that if I trace to
the hop before the hop that it dies at, it then dies at the hop before
that (and as I trace to closer and closer hops, it dies the hop before
that!)

This is illustrated in the traces I've captured here:
http://pastebin.com/raw/R1UHLi0U

For what it's worth, the router is a Linksys EA7300 that I just picked
up.

I can't even imagine what would cause this issue at this point.  If
anyone has any thoughts, I'd love to hear them!

I'm going to start studying some packet captures to see if I can spot an
issue.

Best,
Ryan

Cisco WiFI expertise in NYC

[Ryan Pugatch]

Hi,

Looking for a company that has lots of experience with Cisco WiFi in
NYC.  Our office deployment is struggling due to all of the interfering
APs, so we're looking for some outside help to assist us in improving
our coverage.

If you have any recommendations, please contact me off list.

Thanks,

Ryan

Re: Level3 routing issues

[Ryan Pugatch]

We're actually seeing some problems coming from the Boston area, going
to Cisco WebEx.  We keep ending up going to Level 3 NY and dying,
intermittently.  Unfortunately, both of our peers, Level3 and XO, end up
going to the same place from Boston.  If I go to WebEx from a device in
Florida, we end up going via Level3 in Washington DC and are fine.

On Wed, Jul 29, 2015, at 07:52 AM, Robert Blayzor via NANOG wrote:
 On Jul 28, 2015, at 8:54 PM, Matt Hoppes mhop...@indigowireless.com
 wrote:
  
  Is anyone seeing packet loss or routing issues on the Level3 network on the 
  east coast right now?
  
  
 
 
 We’ve seen a slew of problems going west out of Level3 in NYC the last
 couple of nights. Last night was particularly bad to the point we had to
 shut our Level3 BGP sessions down to route around the issue. 
 
 --
 Robert
 inoc.net!rblayzor
 Jabber: rblayzor.AT.inoc.net
 PGP Key: 78BEDCE1 @ pgp.mit.edu
 
 
 
 

Re: DDOS Simulation

[Ryan Pugatch]

Hi Dovid,

I recommend checking out NimbusDDOS. http://www.nimbusddos.com/

I know that they have done exactly this for several notable customers,
and also provide insights into impacts (they don't just blindly run the
attacks for you, they provide intelligence behind what's happening to
help you make sense of what is going on.)

Contact me off list if you want me to set up an intro.

Ryan


On Mon, Jul 27, 2015, at 11:32 AM, Dovid Bender wrote:
 Hi All,
 
 We are looking into a few different DDOS solutions for a client. We need
 a
 LEGITIMATE company that can simulate some DDOS attacks (the generic +
 specific to the clients business). Anyone have any recommendations?
 
 Regards,
 
 Dovid

Re: A case against vendor-locking optical modules

[Ryan Pugatch]

On Mon, Nov 17, 2014, at 07:02 PM, Jérôme Nicolle wrote:

 
 It's probably fine in a pure DC environment with few locations and only
 one SFP+ type, but it's rapidly a total mess when you have to manage 40
 channels for 3 module types over dozens of locations AND the added
 manufacturer specific pain-in-the-ass.
 
 -- 
 Jérôme Nicolle
 +33 6 19 31 27 14

So my question is, to Patrick's point, if you factor in the costs of
managing this versus the costs of going with someone else who does not
lock you in, is it worth it?  Insert comment about TCO and ROI here,
etc.

-- 
Ryan Pugatch
r...@lp0.org
Boston, MA

on the web:
www.ryanp.com (homepage)
www.lp0.org (blog)

Re: Verizon / FiOS network

[Ryan Pugatch]

My original email wasn't too clear.  This host specifically does not allow
80, but does allow 443.  What I was trying to explain is that we are
seeing the issue occur on several hosts on both 80 and 443.

Sorry for the confusion.

Ryan


 HTTP doesnt appear to be open from any network I try Verizon or otherwise
 so
 I'm not sure its network related

 On Fri, Sep 23, 2011 at 9:35 AM, Ryan Rawdon r...@u13.net wrote:


 On Sep 22, 2011, at 9:32 PM, Ryan Pugatch wrote:

  On Thu, Sep 22, 2011 at 8:55 PM, Ryan Pugatch r...@linux.com wrote:
  Hi,
 
  Anyone noticing anything weird with the Verizon / FiOS network?
 
  Seems like many people on their network are having trouble getting
 to
 us
  (on Sidera / RCN) but not everyone.
 
 
  it's, obviously, simpler to help diagnose this when you provide some
  semblance of destination address, port, protocol...
 
  just sayin'!
 
  -chris
  (fios user who could help, if only there was enough info to go on)
 
 
 
  HTTP/HTTPS over 80, 443.  Sample IP: 146.115.38.21
 
 

 From FiOS and non-FiOS locations I get the same result:

 HTTP: timeout
 HTTPS: connects and loads (Zimbra webmail page)
 also can ping via ICMP just fine


 Traceroute from fios is via Level3 from the DC area to Boston where it
 is
 handed off to RCN and then 2 hops to the destination

Re: Verizon / FiOS network

[Ryan Pugatch]

 On Thu, Sep 22, 2011 at 8:55 PM, Ryan Pugatch r...@linux.com wrote:
 Hi,

 Anyone noticing anything weird with the Verizon / FiOS network?

 Seems like many people on their network are having trouble getting to us
 (on Sidera / RCN) but not everyone.


 it's, obviously, simpler to help diagnose this when you provide some
 semblance of destination address, port, protocol...

 just sayin'!

 -chris
 (fios user who could help, if only there was enough info to go on)



HTTP/HTTPS over 80, 443.  Sample IP: 146.115.38.21

Re: Hotmail?

[Ryan Pugatch]

 What about starting with Zimbra's Open Source edition, and building onto
 it?


Let me just step in here and say.. it's tough to build onto Zimbra.  At
work, we support ~1000 users on Zimbra (network edition), with hundreds of
thousands of messages flowing through daily, and it doesn't like you
tinkering with stuff under the hood.  Most of your customizations get
blown away when you upgrade.  That said, I know of some organizations who
customize it like crazy (I had heard that Lycos's free mail system is
Zimbra-based, and Yahoo as well).  Once you deviate, though, don't expect
to stick to Zimbra's releases.  It might be easier to just start fresh
with postfix, amavis, spamassassin, dovecot, etc.  We've also run into
some pain in scaling it out (they want you to use Red Hat Clustering, but
there's no great way to scale out the mail store regardless).

Ryan

Internap FCP

[Ryan Pugatch]

Hi,

We are currently looking into Internap FCP as we are in the process of 
redoing our network infrastructure and taking on managing BGP ourselves 
rather than relying on blended providers.  I am interested in hearing 
about experiences using it.  Is the reporting really that good?  Does it 
actually provide value?  Or, is there something out there that is better 
(or should we just continue to plan to manage it ourselves?)


Thanks in advance for the info.

Ryan