Re: Verizon FIOS troubleshooting
On 09/25/2012 7:11 pm, Bryan Seitz wrote: Recently began seeing things like this to the default GW from inside and outside the FIOS network. Called tech support but all they could do was put a ticket in for the NetEng team. http://pastie.org/4800421 http://www.bsd-unix.net/smokeping/smokeping.cgi?target=people.bryan I worked with Brian offline and can confirm there's definitely an issue, at least on his particular node/area (W.D.C.). Anyone from Verizon lurking? -- Randy M
Charter Blackholing AS29889
Hi guys (and sorry for the noise), It appears return traffic from Charter to our ASN is blackholed. According to all three of our upstreams they are delivering traffic but it's not coming back. Unfortunately I don't have a reverse traceroute (our emails to charter customers are bouncing) so I have no idea what transit path they are returning traffic on. I tried fiddling with our outbound paths to no avail. If someone on a Charter connection could shoot me a traceroute to 209.9.238.7 that would be great. Ultimately if someone from Charter is willing to help that would be awesome as well. Source IP: 209.9.238.7 (AS29889) Dest IP: 75.140.10.216 Via HE: [root@mon ~]# traceroute 75.140.10.216 traceroute to 75.140.10.216 (75.140.10.216), 30 hops max, 60 byte packets 1 209.9.238.1 (209.9.238.1) 0.551 ms 0.790 ms 0.512 ms 2 gige-g4-13.core1.ash1.he.net (216.66.0.225) 12.029 ms 12.094 ms 12.158 ms 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * Via Abovenet: [root@mon ~]# traceroute 75.140.10.216 traceroute to 75.140.10.216 (75.140.10.216), 30 hops max, 60 byte packets 1 209.9.238.1 (209.9.238.1) 0.544 ms 0.540 ms 0.573 ms 2 208.185.24.1 (208.185.24.1) 0.206 ms 0.218 ms 0.200 ms 3 xe-4-2-0.er1.iad10.us.above.net (64.125.29.198) 0.228 ms 0.232 ms 0.215 ms 4 above-telia.iad10.us.above.net (64.125.13.158) 117.943 ms 117.958 ms 117.763 ms 5 las-bb1-link.telia.net (80.91.246.71) 62.157 ms 62.162 ms 62.189 ms 6 cco-ic-151505-las-bb1.c.telia.net (213.248.79.102) 72.780 ms 70.183 ms 70.151 ms 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * -- Randy McAnally
Re: Charter Blackholing AS29889
On 09/25/2012 9:32 am, Randy McAnally wrote: Hi guys (and sorry for the noise), Thanks to all those who replied as well as Charter's help we defermined uRPF between Charter and some of their peers were filtering ICMP packets making traceroutes appear dead. Compounded by the fact our test server was blocking certain ICMP packets. The issue appears to have been a non issue from the beginning. Carry on folks :) -- Randy McAnally
Re: Verizon FIOS troubleshooting
On 09/25/2012 7:11 pm, Bryan Seitz wrote: All, Recently began seeing things like this to the default GW from inside and outside the FIOS network. Called tech support but all they could do was put a ticket in for the NetEng team. http://pastie.org/4800421 http://www.bsd-unix.net/smokeping/smokeping.cgi?target=people.bryan The pings jumping from an avg of 3ms to 80 is what gets me. Also my downloading / uploading on my segment doesn't seem to affect the latency jumps on the default GW either way (when testing from my COLO). Any thoughts or suggestions would be appreciated! Worry about a connected hosts, not the gateway router. If you see the same behavior between hosts then check your upstream/downstream rates since they will buffer your connection if you get close to the advertised rates, even for micro bursts. -- Randy M
Re: WW: Colo Vending Machine
Cage nuts. Sent from my IPhone (pardon the typo's) On Feb 17, 2012, at 1:35 PM, Jay Ashworth j...@baylink.com wrote: Please post your top 3 favorite components/parts you'd like to see in a vending machine at your colo; please be as specific as possible; don't let vendor specificity scare you off. Cheers, -- jra -- Jay R. Ashworth Baylink j...@baylink.com Designer The Things I Think RFC 2100 Ashworth Associates http://baylink.pitas.com 2000 Land Rover DII St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Re: Switch and router
On Tue, 7 Feb 2012 08:32:21 -0500, Ann Kwok wrote Hello Thank you for your help But we can't increase the pipe as we are using 10G switch. The congestion happens when the traffic is using 7G If you cannot increase bandwidth, then you must increase the TX queue (in QOS and/or port buffer). ~Randy
Re: IP KVM suggestions
+1 on lantronix. Also does serial console. Lots of settings. Beats the pants off other units in terms of flexibility and configuration options. Sent from my IPhone (pardon the typo's) On Jan 30, 2012, at 9:11 PM, Jeff Fisher na...@techmonkeys.org wrote: Lantronix Spider is a small, portable, affordable and web enabled IP KVM. Supports ISO mounting and has USB connections. http://www.lantronix.com/it-management/kvm-over-ip/securelinx-spider.html It is a single server unit. So if you want to connect many servers at the same time, it might not be the best option as the price quickly escalates. However, if you buy one and just move it from server to server (which is what I got from your email), then it is a pretty good fit. Java based web interface, not the greatest, but it works. I've got a few Lantronix Spiders and I love them; however, I would opt to get the external power adapter instead of just relying on the unit drawing power from the computer it's connected to. Also, there is a PS2 + USB model available that I'd recommend getting if you have any older gear which doesn't support USB keyboards while in the BIOS. I think they go for around $260 + another $20 or so for the external power adapter. Jeff
Re: F.ROOT-SERVERS.NET moved to Beijing?
On Sun, 2 Oct 2011 17:40:23 + (UTC), Janne Snabb wrote I happened to notice the following at three separate sites around the US and one site in Europe: Getting palo alto from east coast. 3 10gigabitethernet1-2.core1.atl1.he.net (2001:470:0:1b5::2) 8.166 ms 8.135 ms 8.103 ms 4 2001:470:0:ce::2 (2001:470:0:ce::2) 77.881 ms 77.866 ms 77.909 ms 5 iana.r1.atl1.isc.org (2001:500:61:6::1) 77.885 ms 77.924 ms 77.896 ms 6 int-0-5-0-1.r1.pao1.isc.org (2001:4f8:0:1::49:1) 76.846 ms 75.854 ms 75.819 ms 7 f.root-servers.net (2001:500:2f::f) 75.788 ms 75.756 ms 75.726 ms
Re: Verizon / FiOS network
Not able to connect to 146.115.38.21 via fios or verizon 3g so the problem doesn't seem to be fios specific. Sent from my IPhone (pardon the typo's) On Sep 22, 2011, at 9:32 PM, Ryan Pugatch r...@linux.com wrote: On Thu, Sep 22, 2011 at 8:55 PM, Ryan Pugatch r...@linux.com wrote: Hi, Anyone noticing anything weird with the Verizon / FiOS network? Seems like many people on their network are having trouble getting to us (on Sidera / RCN) but not everyone. it's, obviously, simpler to help diagnose this when you provide some semblance of destination address, port, protocol... just sayin'! -chris (fios user who could help, if only there was enough info to go on) HTTP/HTTPS over 80, 443. Sample IP: 146.115.38.21
Re: Verizon Issues? East Coast US
On Tue, 1 Mar 2011 11:47:39 -0500, Chris Tracy wrote In both cases, mtr shows ~50% loss beginning at google- gw.customer.alter.net (152.179.50.62), the first hop in AS15169. It's clear that I must be losing more ICMP than TCP packets given that google webpages come up fairly quickly, but youtube videos hang ever since this started. Anybody else seeing this? I've been seeing ~50% packet loss to google from FiOS (WDC area) for a while now. Youtube completely unusable during the day for the most part, but that has been going on for months to tell you the truth. ~Randy
Re: Howto for BGP black holing/null routing
On Tue, 22 Feb 2011 16:42:28 -0500, David Hubbard wrote I was wondering if anyone has a howto floating around on the step by step setup of having an internal bgp peer for sending quick updates to border routers to null route sources of undesirable traffic? I've seen it discussed on nanog from time to time, typically suggesting using Zebra, but could not search up a link on a step by step. Ultimately it depends on the transit provider. For example, some have you set up a separate BGP session with a black hole router. Any prefix sent will be blackholed network wide. Some, such as the case of Level3, they are looking for specific community tags on your primary BGP session. So in a nutshell...lets blackhole a host: ip route x.x.x.x 255.255.255.255 null0 tag 255 Then set up a static-to-bgp with route-map to add community strings (for example 3356: for level3) to your routes with tag 255. route-map STATIC-TO-BGP permit 10 match tag 255 set community 3356: set origin igp And in your BGP config: redistribute static route-map STATIC-TO-BGP Now, for the case of level3, you're already set (just be sure to apply send-community on the neighbor). Now for a provider having a unique blackhole BGP session, you want a special route-map to filter prefixes going out that session: ip community-list BLACKHOLE seq 10 permit 3356: route-map BLACKHOLE permit 10 match community BLACKHOLE Now for the blackhole session: neighbor blackhole_peer route-map out BLACKHOLE It can get more complicated than this (for example, you've got more than one EBGP router) but this is just a simple case. I hope it helps... ~Randy
Re: External sanity checks
On Thu, 3 Feb 2011 10:04:10 -0800 (PST), Philip Lavine wrote To all, Does any one know a Vendor (NOT Keynote) that can do sanity checks against your web/smtp/ftp farms with pings, traceroutes, latency checks as well as application checks (GET, POST, ESMTP, etc) I've had good results with hyperspin.com...never any false alarms for that matter. ~Randy
Re: Ipv6 for the content provider
On Mon, 31 Jan 2011 11:53:22 -0600, Blake Hudson wrote # ip6tables -A INPUT -m state --state ESTABLISHED -j ACCEPT I guess the next question is whether or not it actually works correctly You can open/shut ports but you can't do anything with connection state (RELATED, ESTABLISHED, ect). For example, you have to open all upper inbound ports manually if you want to complete outbound connections. The solution is to manually build your own kernel from a vanilla source, along with all the problems that entails. ~Randy
Re: DSL options in NYC for OOB access
On Sat, 29 Jan 2011 13:35:01 +, Andy Ashley wrote if you want the name). Also suggested to me was doing a swap with another provider in the facility but it seems as if cross connects may be prohibitively expensive between suites/floors there. Im going to wait for pricing on this and make a choice then. Have you looked into the cross connect cost for your DSL line? They typically aren't very cheap either. ~Randy
Re: Ipv6 for the content provider
On Wed, 26 Jan 2011 10:22:40 -0800, Charles N Wyble wrote For the most part, I'm a data center/application administrator/content provider kind of guy. As such, I want to provide all my web content over ipv6, and support ipv6 SMTP. What are folks doing in this regard? The only issue I've faced is RHEL/CentOS doesn't have stateful connection tracking for IPv6 - so ip6tables is practically worthless. ~Randy
Re: Ipv6 for the content provider
On Wed, 26 Jan 2011 13:56:05 -0800, Charles N Wyble wrote The only issue I've faced is RHEL/CentOS doesn't have stateful connection tracking for IPv6 - so ip6tables is practically worthless. H. Interesting. I wonder if this is specific to the RedHat kernel? I've worked around it by compiling custom (newer) Kernels on systems that need it. Apparently support was added some time around 2.6.20, but of course RHEL5 is still in the dark ages of 2.6.18. ~Randy
RE: Dual Homed BGP for failover
On Wed, 19 Jan 2011 10:23:47 -, Ahmed Yousuf wrote - Accept that we are never going to get an ideal distribution of traffic and continue monitoring and adjusting local pref/prepends etc. as and when we need to change the distribution of traffic. Hopefully we don't need to do this that often. ^ This. You're fighting a loosing battle with such slow links. Given the limited route capacity of your router you might as well set up statics aimed at each link and forget about BGP shaping. Just keep a floating default pointed at each peer. -Randy
RE: Dual Homed BGP for failover
On Wed, 19 Jan 2011 14:26:32 -, Ahmed Yousuf wrote We're doing BGP to announce our PI space and make sure that our PI space is reachable through both ISPs in case one link goes down. This is the primary need to do the BGP here. Unfortunately my boss has requested that we make use of the capacity of both links, rather than pref traffic out of the higher capacity link. Understood! you would _still_ take default BGP routes, I was implying more along the lines (in cisco speak): ! Tweak as necessary to get a good balance ip route 0.0.0.0 128.0.0.0 peer1 ip route 128.0.0.0 128.0.0.0 peer2 Set up SLA tracking on the peer IPs to retract the routes if either peer goes down. Either that or get more RAM on your router and go the BGP-only method. -Randy
Re: IPv6 - real vs theoretical problems
-- Original Message --- From: Jeff Wheeler j...@inconcepts.biz Sent: Thu, 6 Jan 2011 21:01:12 -0500 Are there any large transit networks doing /64 on point-to-point networks to BGP customers? Who are they? Add HE.net to the list. -Randy www.fastserv.com
Re: sudden low spam levels?
-- Original Message --- From: Ken Chase k...@sizone.org To: nanog@nanog.org Sent: Mon, 3 Jan 2011 13:04:55 -0500 Subject: sudden low spam levels? I have two independent mailservers, and two other customers that run their own servers, all largely unrelated infrastructures and target domains, suddenly experiencing low levels of spam. Total emails/day dropping from some 175,000-250,000ish to 50-75, 000ish (legit mail in the 2-5,000 per day, yes I have some high spam:legit customers...). 3 days in a row now at least, at quick glance. Did someone set up them the bomb? We filter spam for over 2000 domains and I don't see any noticeable drop in payload. I have noticed that over the past few months greylisting has become MUCH more effective than it used to be... looks like spam delivery is moving more from snowshoe infrastructure towards botnets. -- Randy M. www.FastServ.com
Re: The tale of a single MAC
-- Original Message --- From: Graham Wooden gra...@g-rock.net Hi there, I encountered an interesting issue today and I found it so bizarre so I thought I would share it. I brought online a spare server to help offload some of the recent VMs that I have been deploying. Around the same time this new machine (we¹ll call it Server-B) came online, another machine which has been online for about a year now stopped responding to our monitoring (and we¹ll name this Server-A). I logged into the switch and saw that the machine that stopped responding was in the same VLAN as this newly deployed, and then quickly noticed that Server- A¹s MAC address was now on Server-B¹s switch port. ³What the ...² was my initial response. Fresh OS install from scratch or did you load an image from an existing server? What make/model of on-board NICs? -- Randy M.
Re: Throttle traffic for a single local IP on a Linux router?
take a read on this link http://www.faqs.org/docs/Linux-HOWTO/Bandwidth-Limiting-HOWTO.html -beavis Another: http://djlab.com/2009/10/limiting-bandwidth-in-linux/ -- Randy
RE: Over a decade of DDOS--any progress yet?
Soon several providers will begin offering dedicated servers with a 10Gbps connection to a single machine. -Drew Several already do. -Randy
Re: wireless data caps [was: Level 3 Communications Issues Statement Concerning Comcast'sActions]
-- Original Message --- From: William Herrin b...@herrin.us Sent: Tue, 30 Nov 2010 13:17:45 -0500 I checked it out when I updated my credit card number online recently. The billing page has a place to describe a cap and overage charges. It's listed as unlimited. Not saying you're wrong. Just saying that the billing documentation disagrees. It's 'unlimited' up to 5Gb -- big lawyers make that work I guess. And yes I've also been grandfathered in from almost 8 years ago when I first got it -- for these types of accounts they shut you off instead of billing overusage. -Randy
RE: Outage between GBLX and HE?
We saw further evidence of this on paths traversing global crossing to a customer last night.I don't know about others but we are intending to make some efforts to move traffic other places, this type of repeated failure is just terrible, especially since they still continue to announce routes indicating reachability that does not exist. John @ AS11404 in Seattle. This has been going on for some months, moving from market to market. LAX, SEA, now ASH. -Randy
Verizon contact
Anyone with a Verizon network engineering contact on the list? There's a bad router/link in Reston, VA for the past 36 hours that we're having a real heck of a time trying to route around. Hoping we can get someone at Verizon to take a look at things. -- Randy
Re: FIOS Router
-- Original Message --- From: Brielle Bruns br...@2mbit.com See the response I just posted, but in all likely, he's being hampered by the fact the handoff from the ONT is 100BT ethernet and OpenRG (which bolts on top of a Linux OS and 'replaces' the functionality of iptables and such). I really meant a real Linux server (or desktop box loaded with CentOS, Deb, ect) with some basic IPtables rules and dual NIC. I never intended to use any kind of appliance or router device loaded with 'brand x' Linux. A 100bT hand-off should have NO issues reaching ~98Mbps without packet loss; just a little extra latency as you start filling buffers. Since the first day our FiOS was installed, we switched out the cruddy Dlink router (later swapped with Actiontec) with a Linux box running CentOS and a simple iptables script. I later added a Atheros-based wifi card with HostAP and madwifi to create an AP from the same box. Linux/Wifi is not for all of course, but the dual-nic and IPtables part pretty much anyone can do...you could just as easily hang a small wifi router off the box. -R
Re: FIOS Router
I've been using linux/iptables since day 1. 100Mbps is a walk in the park. -- Original Message --- From: Chris Burwell cburw...@gmail.com To: NANOG nanog@nanog.org Sent: Thu, 27 May 2010 10:21:01 -0400 Subject: FIOS Router I'm doing some research for a group that has a 100Mb FIOS Internet connection at their site. I was surprised to learn that Verizon supplied them with the same Actiontec router that they provided me with on my 10Mb connection at home. Needless to say the Actiontec router is not up to the task of moving all of that traffic (they are using about 80Mb now and sometimes max out their connection). Verizon has been good about replacing the router multiple time when they finally fail, however having to power-cycle the router multiple times per day is not acceptable. What I would like to do is set them up with a router/firewall that is capable of handling their current bandwidth needs as well as their anticipated future growth. My concern is terminating the FIOS connection from the ONT directly to something like a Cisco 3900 (Output from the ONT is CAT5 terminating to RJ-45). I have been searching around the Internet and found one discussion where someone claims to have been able to accomplish just this using a Cisco 871 router. Based on the loose discussions that I have read it seems that the FIOS connection configuration can vary from area to area. I am also aware that we can configure the Actiontec router as a bridge, but I would much rather remove it altogether particularly with the amount of traffic this group is moving. Has anyone been able to accomplish this or something similar with any hardware other then the router Verizon provides? Any insight on Verizon's official stance on this would be helpful. If there is someone from Verizon out there that can contact me about the technical aspects of doing this, that would be much appreciated as well. - Chris --- End of Original Message ---
