Re: Speed Test Results
On Fri, Dec 23, 2011 at 10:13 AM, Livingood, Jason jason_living...@cable.comcast.com wrote: If you want to understand the issue in detail, check out the report from MIT this year, written by Steve Bauer and available at http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_Broadband_Speed_Measurem ents.pdf. They should have put a date on their paper, including when the measurements were done. It appears to me to have been done sometime after or around June of 2010.
RE: Speed Test Results
We host an Ookla Speedtest server onsite and find it a very reliable means to identify throughput issues. The source of any performance issues may or may not be ours, but if a customer says things are slow we can usually identify whether it's their PC or network (browsing is slow but speed test runs fine) or a local or regional network issue (speed test runs slow). If a customer gets less than 90% of the advertised throughput, we follow up on it. Frank -Original Message- From: Michael Holstein [mailto:michael.holst...@csuohio.edu] Sent: Friday, December 23, 2011 1:27 PM To: jacob miller Cc: nanog@nanog.org Subject: Re: Speed Test Results Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. They are excellent tools for generating user complaints. (just like the do traceroute and count the hops advice from gamer mags of old). (my $0.02) Michael Holstein Cleveland State University
RE: Speed Test Results
The MIT article is good read, thanks for sharing that. One thing to watch out for is if the last mile provider is the one hosting the speedtest site, that's another variable removed from the equation. In some cases that is a good thing, in others it's not, depending on what you are trying to measure. It's also theoretically possible (and in my opinion not only likely but probably fairly common) for some large residential ISP's to not rate-limit these on-net test sites (either by design or as a side result of at what point in the network they apply the rate limiting), thereby showing much higher results than the end user could ever possibly see in a real world scenario. Also, when using some of the popular public Ookla/speedtest.net sites, their FAQ clearly states that the tests are not suitable for certain connection types like high speed services and non-residential services in general. One good example is Speakeasy's site, which in my personal experience has been the one most commonly used by end users (especially those contacting us about speed problems): http://www.speakeasy.net/speedtest/issues.php Our speed test is tuned to measure residential broadband services up to 20 Mbps over HTTP. It takes a very customized installation to be able to accurately measure up to 100 Mbps over HTTP. -Scott -Original Message- From: Frank Bulk [mailto:frnk...@iname.com] Sent: Sunday, December 25, 2011 8:28 PM To: 'Michael Holstein'; jacob miller Cc: nanog@nanog.org Subject: RE: Speed Test Results We host an Ookla Speedtest server onsite and find it a very reliable means to identify throughput issues. The source of any performance issues may or may not be ours, but if a customer says things are slow we can usually identify whether it's their PC or network (browsing is slow but speed test runs fine) or a local or regional network issue (speed test runs slow). If a customer gets less than 90% of the advertised throughput, we follow up on it. Frank -Original Message- From: Michael Holstein [mailto:michael.holst...@csuohio.edu] Sent: Friday, December 23, 2011 1:27 PM To: jacob miller Cc: nanog@nanog.org Subject: Re: Speed Test Results Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. They are excellent tools for generating user complaints. (just like the do traceroute and count the hops advice from gamer mags of old). (my $0.02) Michael Holstein Cleveland State University
Re: Speed Test Results
Even though the faq's say they are only good for residential usage, i have had no problems with it at school. My college has 2x 100 Mb circuits from TW. When i run speed tests (I use speedtest.net) with the campus empty, i can get around 95Mb up. The bottleneck is the school's 100Mb switches. When the campus is filled (during the week) i can normally get close to 40 Mb down on a test. -Grant On Sun, Dec 25, 2011 at 8:10 PM, Scott Berkman sc...@sberkman.net wrote: The MIT article is good read, thanks for sharing that. One thing to watch out for is if the last mile provider is the one hosting the speedtest site, that's another variable removed from the equation. In some cases that is a good thing, in others it's not, depending on what you are trying to measure. It's also theoretically possible (and in my opinion not only likely but probably fairly common) for some large residential ISP's to not rate-limit these on-net test sites (either by design or as a side result of at what point in the network they apply the rate limiting), thereby showing much higher results than the end user could ever possibly see in a real world scenario. Also, when using some of the popular public Ookla/speedtest.net sites, their FAQ clearly states that the tests are not suitable for certain connection types like high speed services and non-residential services in general. One good example is Speakeasy's site, which in my personal experience has been the one most commonly used by end users (especially those contacting us about speed problems): http://www.speakeasy.net/speedtest/issues.php Our speed test is tuned to measure residential broadband services up to 20 Mbps over HTTP. It takes a very customized installation to be able to accurately measure up to 100 Mbps over HTTP. -Scott -Original Message- From: Frank Bulk [mailto:frnk...@iname.com] Sent: Sunday, December 25, 2011 8:28 PM To: 'Michael Holstein'; jacob miller Cc: nanog@nanog.org Subject: RE: Speed Test Results We host an Ookla Speedtest server onsite and find it a very reliable means to identify throughput issues. The source of any performance issues may or may not be ours, but if a customer says things are slow we can usually identify whether it's their PC or network (browsing is slow but speed test runs fine) or a local or regional network issue (speed test runs slow). If a customer gets less than 90% of the advertised throughput, we follow up on it. Frank -Original Message- From: Michael Holstein [mailto:michael.holst...@csuohio.edu] Sent: Friday, December 23, 2011 1:27 PM To: jacob miller Cc: nanog@nanog.org Subject: Re: Speed Test Results Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. They are excellent tools for generating user complaints. (just like the do traceroute and count the hops advice from gamer mags of old). (my $0.02) Michael Holstein Cleveland State University
Re: Speed Test Results
Basically it's a CYA statement on the part of Ookla/speedtest.net, since their test sites are of varying quality. The Radnor, OH test site sometimes can't even properly test a 10mbit SOHO broadband connection, where the Toledo site is consistently able to flood every available bit of capacity on my 50/5 home connection. It's just another tool that needs to be used intelligently. If I'm testing out a new ISP or a new speed level I've never had before, I wouldn't immediately complain if I didn't get the expected result on a public speed test site as it may be something outside of my ISP's control. On the other hand if things start dragging on my home connection or anywhere else that I know I can expect a certain result speedtest.net is usually my first stop. -- Sean Harlow s...@seanharlow.info On Dec 25, 2011, at 9:43 PM, Grant Ridder wrote: Even though the faq's say they are only good for residential usage, i have had no problems with it at school. My college has 2x 100 Mb circuits from TW. When i run speed tests (I use speedtest.net) with the campus empty, i can get around 95Mb up. The bottleneck is the school's 100Mb switches. When the campus is filled (during the week) i can normally get close to 40 Mb down on a test. -Grant
Speed Test Results
Hi, Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. Regards, Jacob
Re: Speed Test Results
They are completely unreliable and not to be trusted except for an occasional general indication of speed. -- Leigh Porter On 23 Dec 2011, at 09:20, jacob miller mmzi...@yahoo.com wrote: Hi, Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. Regards, Jacob __ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com __ __ This email has been scanned by the Symantec Email Security.cloud service. For more information please visit http://www.symanteccloud.com __
Re: Speed Test Results
Couldn't agree more, it's unfortunate that so many users take them as gospel! On 12/23/2011 04:23 AM, Leigh Porter wrote: They are completely unreliable and not to be trusted except for an occasional general indication of speed.
Re: Speed Test Results
They are very useful for like-for-like comparison, for an indication of where your minimum performance levels are probably at, for a quick check that things are working properly and as expected. To determine the exact max effective speed? To test qos policies? To determine whether you are meeting SLA's? Not so much. Joe Leigh Porter wrote: They are completely unreliable and not to be trusted except for an occasional general indication of speed.
RE: Speed Test Results
In my opinion they are only somewhat reliable if they are on your network or very close to your network -we operate one of the speedtest.net sites and for our own eyeball traffic find it to be a reasonable indicator of what kind of speeds the customer is getting. To put it a different way, if a customer is getting 20X1 Internet service and the speedtest shows 17 X 0.8 then case closed - if they are getting a speedtest result of 5 X 0.5 then our helpdesk will take a further look - this is really in rough terms... Paul -Original Message- From: jacob miller [mailto:mmzi...@yahoo.com] Sent: Friday, December 23, 2011 4:19 AM To: nanog@nanog.org Subject: Speed Test Results Hi, Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. Regards, Jacob
Re: Speed Test Results
Rolling your own with NDT (web100 tools), placing it on a common transit point within your network, and providing access to your customers will provide decent, consistent and pretty detailed results regarding TCP attributes. There are facilities within the software to perform packet captures for each test too. Works for our help desk to understand relative performance and respond accordingly. Trust nothing off your managed net. Tate On 12/23/2011 4:18 AM, jacob miller wrote: Hi, Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. Regards, Jacob -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: Speed Test Results
On Dec 23, 2011, at 8:07 AM, Paul Stewart wrote: In my opinion they are only somewhat reliable if they are on your network or very close to your network -we operate one of the speedtest.net sites and for our own eyeball traffic find it to be a reasonable indicator of what kind of speeds the customer is getting. To put it a different way, if a customer is getting 20X1 Internet service and the speedtest shows 17 X 0.8 then case closed - if they are getting a speedtest result of 5 X 0.5 then our helpdesk will take a further look - this is really in rough terms... Paul From the consumer viewpoint: No single data point should be extrapolated to infinity, but comparing problematic behavior with normal behavior is a standard process across all fields. Speed tests from several locations done regularly give a baseline for performance. Major departure from expected numbers from a set of speed test sites can be regarded as an indicator of local loop problems. Did you know that local loops suffer from backhoe fade? And, DSLAMS fail. In my home office, speed tests are just another useful diagnostic helping to locate problem areas - just like in Paul's example. DSLReports line monitoring service is a similarly useful tool. James R. Cutler james.cut...@consultant.com
RE: Speed Test Results
I love using speedtest. My FIOS at home is 25/25. And speedtest consistently hits that mark so I know FIOS is giving me what I paid for. When Verizon was having internet issues last week my numbers were bad. Like someone else said, I would not use it much more for quick gauge. To get more granular info you should be using other tools Subject: Re: Speed Test Results From: james.cut...@consultant.com Date: Fri, 23 Dec 2011 09:02:01 -0500 To: nanog@nanog.org On Dec 23, 2011, at 8:07 AM, Paul Stewart wrote: In my opinion they are only somewhat reliable if they are on your network or very close to your network -we operate one of the speedtest.net sites and for our own eyeball traffic find it to be a reasonable indicator of what kind of speeds the customer is getting. To put it a different way, if a customer is getting 20X1 Internet service and the speedtest shows 17 X 0.8 then case closed - if they are getting a speedtest result of 5 X 0.5 then our helpdesk will take a further look - this is really in rough terms... Paul From the consumer viewpoint: No single data point should be extrapolated to infinity, but comparing problematic behavior with normal behavior is a standard process across all fields. Speed tests from several locations done regularly give a baseline for performance. Major departure from expected numbers from a set of speed test sites can be regarded as an indicator of local loop problems. Did you know that local loops suffer from backhoe fade? And, DSLAMS fail. In my home office, speed tests are just another useful diagnostic helping to locate problem areas - just like in Paul's example. DSLReports line monitoring service is a similarly useful tool. James R. Cutler james.cut...@consultant.com
RE: Speed Test Results
Re: Speed Test Results
On Fri, 23 Dec 2011 01:18:40 -0800, jacob miller mmzi...@yahoo.com wrote: Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. They are just a measurement, which need to be correctly used and interpreted (that's the difficult part). Reading bad numbers is not necessarily an indication of a link problem. Reading good enough numbers is only meaningful for the duration of the test. To me, the big problem is that they don't state all the details of the tests (for example, how exactly to they do the transfer). Geographical location is good, but sometimes not enough. Do they use http, https, ftp or their own JS implementation of whatever weird protocol they though of? How do I know if I'm hitting my firewall, web cache or ALG? I only use them to get a generic overview of the link. -- Octavio. Twitter: @alvarezp2000 -- Identi.ca: @alvarezp
Re: Speed Test Results
Hello, On Fri, Dec 23, 2011 at 4:19 PM, Octavio Alvarez alvar...@alvarezp.ods.org wrote: On Fri, 23 Dec 2011 01:18:40 -0800, jacob miller mmzi...@yahoo.com wrote: Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. They are just a measurement, which need to be correctly used and interpreted (that's the difficult part). Reading bad numbers is not necessarily an indication of a link problem. Reading good enough numbers is only meaningful for the duration of the test. To me, the big problem is that they don't state all the details of the tests (for example, how exactly to they do the transfer). Geographical location is good, but sometimes not enough. Do they use http, https, ftp or their own JS implementation of whatever weird protocol they though of? How do I know if I'm hitting my firewall, web cache or ALG? I agree. But one that is fairly clear in what (and how) it tests (but to be fair isn't really a 'speed test') that I've come across is ICSI Netalyzr. It's pretty useful to give a first impression to a tech of what's going on with a link. Take a look at an example report (from a dodgy connection) I dug up: http://netalyzr.icsi.berkeley.edu/restore/id=43ca208a-28820-e88f1efc-a129-4c92-8968 More info and examples are at http://netalyzr.icsi.berkeley.edu/ I also think that sometimes having a 'speed test' or similar hosted on a network you are trying to connect to can be useful to find out if a link is congested, or other problems getting from you to that network. An example of this is The BBC's iPlayer diagnostic at http://www.bbc.co.uk/iplayer/diagnostics (think Hulu, but in the UK). It tests to all their CDNs (Akami, Limelight etc) using different streaming methods and gives the results. Only useful as an overview, but a decent first guide nevertheless . I only use them to get a generic overview of the link. Heck yes! Alex
Re: Speed Test Results
If you want to understand the issue in detail, check out the report from MIT this year, written by Steve Bauer and available at http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_Broadband_Speed_Measurem ents.pdf. - Jason On 12/23/11 4:18 AM, jacob miller mmzi...@yahoo.com wrote: Hi, Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. Regards, Jacob
Re: Speed Test Results
On Fri, Dec 23, 2011 at 2:18 AM, jacob miller mmzi...@yahoo.com wrote: Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. It's one data point of many. Depending on the speed test site, the protocols it uses, where the test is located, any local networking gear (I've seen transparent proxies get great speedtest ratings!), etc, they can be useful, particularly in verifying that a provider's off-net interconnects and partners are doing well. However, they are susceptible to things like wireless network issues, TCP limitations (one stream vs. many streams), and misconfiguration of devices at the customer location. And the speed test box isn't necessarily configured/speced correctly either. I second the thoughts on NDT and I like the ICSI Netalyzer. But I wouldn't necessarily put either tool in most end users' hands (I think they are too complex for most end users to interpret the results properly).
Re: Speed Test Results
Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. They are excellent tools for generating user complaints. (just like the do traceroute and count the hops advice from gamer mags of old). (my $0.02) Michael Holstein Cleveland State University
Re: Speed Test Results
On Fri, 23 Dec 2011 12:16:38 MST, Joel Maslak said: However, they are susceptible to things like wireless network issues, TCP limitations (one stream vs. many streams), and misconfiguration of devices at the customer location. And the speed test box isn't necessarily configured/speced correctly either. I have seen some surreal results reported by some of the speed test sites if you have a sufficiently fat pipe. Near as I could tell, every single hop was gigE or better all the way, the speedtest site then tried to apply a correction for the bottleneck it knew about on its local gigE nterface, and basically decided that the *rest* of the path must be near-infinite speed. ;) pgpog8XkLzv90.pgp Description: PGP signature
Re: Speed Test Results
On 12/23/11 11:16 , Joel Maslak wrote: On Fri, Dec 23, 2011 at 2:18 AM, jacob miller mmzi...@yahoo.com wrote: Am having a debate on the results of speed tests sites. Am interested in knowing the thoughts of different individuals in regards to this. It's one data point of many. Depending on the speed test site, the protocols it uses, where the test is located, any local networking gear (I've seen transparent proxies get great speedtest ratings!), etc, they can be useful, particularly in verifying that a provider's off-net interconnects and partners are doing well. However, they are susceptible to things like wireless network issues, TCP limitations (one stream vs. many streams), and misconfiguration of devices at the customer location. And the speed test box isn't necessarily configured/speced correctly either. I don't imagine it accounts for l3 emcp either... To be clear, what one is I assume generally looking for from a speed test is usable throughput from the vantage point of the end-user running it. I second the thoughts on NDT and I like the ICSI Netalyzer. But I wouldn't necessarily put either tool in most end users' hands (I think they are too complex for most end users to interpret the results properly).
Re: Speed Test Results
Just a note on this subject although not directly related to the original question - There some interesting tests available here: http://www.measurementlab.net/ -- Landon Stewart lstew...@superb.net Manager of Systems and Engineering Superb Internet Corp - 888-354-6128 x 4199 Web hosting and more Ahead of the Rest: http://www.superbhosting.net
Re: Speed Test Results
On 23/12/2011 21:26, Michael Holstein wrote: They are excellent tools for generating user complaints. I find that they are useful for filtering out some of the completely bogus complaints. We encourage customers to include some test results when they contact our NOC to avoid being ignored when they send an its slow complaint. That said - people get fixated on the numbers. 80% of the purchased speed on non-CIR services is cause for a complaint. Our biggest issue is people doing tests to destinations 300+ ms away that only last for a few seconds and then complaining about poor performance. As soon as you mention things like bandwidth delay product the eyes glaze over. Heavy use of lossy WISP access network providers doesn't help. -- Graham Beneke
Re: Speed Test Results
On Fri, Dec 23, 2011 at 9:35 PM, Graham Beneke gra...@apolix.co.za wrote: That said - people get fixated on the numbers. 80% of the purchased speed on non-CIR services is cause for a complaint. Our biggest issue is people doing tests to destinations 300+ ms away that only last for a few seconds and then complaining about poor performance. As soon as you mention things like bandwidth delay product the eyes glaze over. Heavy use of lossy WISP access network providers doesn't help. Or that most ADSL lines have about 20% ATM cell tax on them. I did get caught up on a speed test today. I was turning up a GBLX 100Mb circuit. I got the /30 and all the pings were good to the router. I then pinged some known hosts in the Westin (about a block away where GBLX's router was) and saw some not so nice ping times. I then ran a speedtest and only got about 2Mb/s. Come to find out that this was going to be an MPLS path to the company's California office. Since it hadn't been setup fully the router had found some path through it's management network to ping the world through the tester's DSL line on the other side. So, know the path you are testing. -- Joe Hamelin, W7COM, Tulalip, WA, 360-474-7474
