Thanks for all the information about this that everyone shared, I now know that I'm not crazy. Although it sounds like we may have this as more of a problem in the future....
Kurt Fankhauser Wavelinc Communications P.O. Box 126 Bucyrus, OH 44820 http://www.wavelinc.com tel. 419-562-6405 fax. 419-617-0110 On Mon, Apr 27, 2015 at 11:39 AM, Ken Hohhof <[email protected]> wrote: > I’ve spotted exactly as George describes, torch at our border router > sees way more traffic than I see at the SM or the customer’s router. > > I think one time I traced it to a Limelight Networks IP, but I didn’t > write it down so I’m relying on memory so I could well be wrong. > > > *From:* George Skorup (Cyber Broadcasting) <[email protected]> > *Sent:* Monday, April 27, 2015 10:29 AM > *To:* [email protected] > *Subject:* Re: [AFMUG] 450SM sustain bucket throttle not working.. > > Yeah, it's almost always streaming video. Except the one guy I found > running the "download manager" on his PC was doing USENET crap, probably > porn, movies, music, etc. > > It's fairly easy to see what's going on. For instance, a customer is on > 6x1 Canopy. There's 6Mbps RF downlink and 12Mbps Rx at AP's ethernet > interface. This is all fine and dandy with Canopy because the AP controls > the downlink traffic and doesn't put more on it than the QoS allows. Like > Kurt, the first time I saw this, I was running the MT torch tool and > thought the Canopy QoS was broken, too. > > Now, for UBNT or really anything else where the CPE does the limiting, the > AP is sending all that traffic to the CPE. So now you have to resort to > policing at an upstream router which I'd rather not do to keep the load off > of the little MT routers. Even still, after you do that, all that extra > traffic is still coming in on the backhaul, so all you've done is moved the > problem, which helps relieve the AP stress, but it still sucks to have to > take on double the traffic. You can move the limiting all the way to your > border(s), but that extra traffic is still taking up bandwidth somewhere. > > On 4/27/2015 9:48 AM, Chuck McCown wrote: > > I haven’t been following this thread closely, but has anyone identified > the traffic? > > *From:* Ken Hohhof <[email protected]> > *Sent:* Monday, April 27, 2015 8:46 AM > *To:* [email protected] > *Subject:* Re: [AFMUG] 450SM sustain bucket throttle not working.. > > If we could get Procera to create a signature for this type of > behavior, then we could create a rule to restrict this traffic to some > percentage less than 50% of the subscriber’s speed tier. This would > perhaps have several desirable results: > > - if the CDN algorithm aims to oversubscribe the customer’s pipe by 2X, > then making the pipe look 0.5X as small should counteract that > > - I would actually set it lower than 50%, the objective to have the > customer say “XYZ service sucks” and complain to them or stop using them, > rather than “my Internet sucks” > > - if we can pool our information about who is doing this (not just the CDN > but the content provider paying them), we could complain directly to them, > and if necessary take the position that blocking or throttling their > traffic would be reasonable network management and allowable under net > neutrality rules, since they are in effect attacking our network with a > flood of traffic beyond what the customer has subscribed to > > Another approach would be to rate limit this traffic with a queue that has > a very large buffer, introducing latency rather than packet loss, hoping > that their algorithm recognizes late ACKs as a sign of congestion and will > back off the sending rate. But when I asked Simon about buffer size in > Procera, my understanding was that it’s more of a traffic policing box than > a shaping box. > > > *From:* Wireless Admin <[email protected]> > *Sent:* Monday, April 27, 2015 9:08 AM > *To:* [email protected] > *Subject:* Re: [AFMUG] 450SM sustain bucket throttle not working.. > > > Ken, > > Your assessment of the problem is exactly correct. I was going to compare > it tor DoS as you did here. I don’t see an easy fix for this. > > > > Steve > > > ------------------------------ > > *From:* Af [mailto:[email protected] <[email protected]>] *On > Behalf Of *Ken Hohhof > *Sent:* Monday, April 27, 2015 10:00 AM > *To:* [email protected] > *Subject:* Re: [AFMUG] 450SM sustain bucket throttle not working.. > > > > I don’t think you’re understanding the situation that we are speculating > is happening. > > > > He is using Cambium QoS. However, he is seeing twice that amount of > traffic destined to the customer, the SM is throwing half of it away (as it > should), and as a result the customer’s service sucks. > > > > The problem is that the sender is not observing traditional congestion > control, it is not backing off the sending rate when it sees high packet > loss. > > > > That’s why I say it is similar to a DoS attack, someone sending far more > traffic than the subscriber can receive. > > > > > > *From:* David Milholen <[email protected]> > > *Sent:* Monday, April 27, 2015 7:05 AM > > *To:* [email protected] > > *Subject:* Re: [AFMUG] 450SM sustain bucket throttle not working.. > > > > This is why we like setting the QOS at the subscriber. If I remember the > cambium burst allocation ignores tcp and udp and work strictly on a token > bit system. > We do not receive these complaints. The only time I hear them is if we get > overloaded at the backhaul link. > > On 4/26/2015 8:58 PM, Ken Hohhof wrote: > > I think George forgot the sarcasm emoticon. > > > > Also note that the problem here is the edge provider is sending more than > the customer’s plan rate, ignoring TCP congestion control. Not only does > this consume Internet bandwidth over and above what the customer has > subscribed to, it makes anything else the customer is trying to do on the > Internet unusable because normal TCP is unusable with 50% packet loss. It > is not surprising the customer calls saying his Internet is slow. > > > > There’s a saying that comes to mind, involving a 5 pound bag. > > > > > > *From:* Faisal Imtiaz <[email protected]> > > *Sent:* Sunday, April 26, 2015 8:37 PM > > *To:* [email protected] > > *Subject:* Re: [AFMUG] 450SM sustain bucket throttle not working.. > > > > I see that the net neutrality is going to be the next boogieman under the > bed for WISP's from now on... > > > > Please, please, please, correct your understanding on Net-Neutrality... > > > > It allows for one to traffic shape any and all kinds of traffic, as long > as :- > > a) You declare your practice on your website. > > b) You DON"T DO IT specific to A SPECIFIC Network.. i.e. all VOIP, or > all Video, or ALL Streaming.. > > (applying a throttle on video to netflix while allowing Hulu would be > considered a violation, but applying throttle to all types of video content > is NOT !) > > > > > > :) > > > > > > Faisal Imtiaz > Snappy Internet & Telecom > 7266 SW 48 Street > Miami, FL 33155 > Tel: 305 663 5518 x 232 > > > > Help-desk: (305)663-5518 Option 2 or Email: [email protected] > > > ------------------------------ > > *From: *"George Skorup (Cyber Broadcasting)" mailto:[email protected] > <[email protected]> > *To: *[email protected] > *Sent: *Sunday, April 26, 2015 7:12:13 PM > *Subject: *Re: [AFMUG] 450SM sustain bucket throttle not working.. > > > > So you'd be purposely slowing down or blocking legitimate traffic from an > edge provider to the customer? Oh no, net neutrality violation! > > So when everyone starts with the 4k streaming and we're selling the > customer 20Mbps, then we have to take on 40Mbps because of this!? > > On 4/26/2015 5:58 PM, Ken Hohhof wrote: > > I could justify declaring such traffic an attack and blocking the > source as malicious. > > > > *From:* George Skorup (Cyber Broadcasting) <[email protected]> > > *Sent:* Sunday, April 26, 2015 4:30 PM > > *To:* [email protected] > > *Subject:* Re: [AFMUG] 450SM sustain bucket throttle not working.. > > > > Yep, I see this all the time and Ken is exactly right. The Canopy QoS > works exactly as designed, the AP is definitely not delivering more than > the sustained rate, but is instead discarding the extra 50%. I've tested > this situation thoroughly. Stick a MT simple queue in at the upstream > router and the 2X rate traffic stops hitting the AP's ethernet interface, > but it's still coming in at double the sustained rate farther upstream. > There's no way around it except throwing bandwidth at it. > > This is CDN traffic. And when the customer thinks they can install one of > those "internet download managers" to speed up their connection. The only > thing it does is screw with TCP acks or window sizes or something which > just puts more traffic on your transit just to be discarded at the > congestion point (SM, queue, Procera, whatever). Gotta love it. > > You'd think with 70% of the internets being streaming video they'd think > hmm.. maybe we can cut down on the peering congestion by NOT doing this > crap. But no. > > On 4/26/2015 11:01 AM, Ken Hohhof wrote: > > Sorry to answer a question with a question, but are you measuring at > the SM, or at some upstream router? > > > > The reason I ask, is I have seen some CDN traffic that does not seem to > follow traditional TCP congestion control. It will send at twice the rate > limit, causing 50% packet loss to its own traffic and everything else to > that same subscriber. Evidently some TCP geniuses have decided to use > latency rather than packet loss as the indicator of congestion, and that > the objective is goodput not throughput. Works for last mile technologies > like T1 and DSL with big buffers at the head end of the fixed speed serial > connection, not so good with the type of rate limit queues we tend to use > unless we can provision the queues with big buffers. > > > > Probably not your problem, but I thought I’d bring it up just in case. > > > > *From:* Kurt Fankhauser <[email protected]> > > *Sent:* Sunday, April 26, 2015 10:50 AM > > *To:* [email protected] > > *Subject:* [AFMUG] 450SM sustain bucket throttle not working.. > > > > I have a 450 SM that is rate limited in the SM to 1500kbps download on the > sustain side. I noticed last night that this customer was pulling a steady > almost 3mbps download for several hours on end. How is this possible? Is > there a problem with 13.2 firmware? Its a 3.65ghz SM. > > > > see attached. > > > > Kurt Fankhauser > > Wavelinc Communications > > P.O. Box 126 > > Bucyrus, OH 44820 > > http://www.wavelinc.com > > tel. 419-562-6405 > > fax. 419-617-0110 > > > > > > > > > > -- > > >
