Re: Ready to get your federal computer license?
On 8/28/2009 6:11 PM, Peter Beckman wrote: On Fri, 28 Aug 2009, Hiers, David wrote: Governments already license stock brokers, pilots, commercial drivers, accountants, engineers, all sorts of people whose mistakes can be measured in the loss of hundreds of lives and millions of dollars. "'The power company allowed their network security to be comprimised by a single Windows computer connected to the Internet in the main control facility, so we unplugged the entire Internet to mitigate the attack,' said Senator Rockefeller, the author of the bill that enabled the President to take swift action after an unknown hacker used the Internet to break into Brominion Power's main control facility and turn off the power to the entire East Coast. 'It will remain unplugged and nobody in the US will be allowed to connect to the Internet until the power is back on and this hacker is brought to justice.' Authorities are having a difficult time locating the hacker due to the unavailability of the Internet and electricity, and cannot communicate with lawmakers via traditional means due to the outage. A formal request to turn the power and Internet back on was sent on a pony earlier this afternoon to lawmakers in DC." Can't wait. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ --- ROFL!
RE: Link capacity upgrade threshold
> > > If your 95th percentile utilization is at 80% capacity, > it's time to > > > start planning the upgrade. > > > > s/80/60/ > > > > the normal snmp and other averaging methods *really* miss > the bursts. > > s/60/40/ > What is this "upgrade" thing you all speak of? When your links become saturated, shouldn't you solve the problem by deploying DPI-based application-discriminatory throttling and start double-dipping your customers? After all, it's their fault for using up more bandwidth than your flawed business model told you they will use. (If you're not familiar with Bell Canada, it's OK if you don't get the joke).
Re: Ready to get your federal computer license?
On Sun, 30 Aug 2009 22:20:55 -0400 Eric Brunner-Williams wrote: > randy, > > moveon is a maine-based org. it is an effective, fund raising, > partisan organization. it is much more than a click-and-opine > vehicle, it puts hundreds of thousands of dollars into competitive > races, and has a competent political director. > > to create a "NagOn" we would have to hire or appoint a political > director, and a financial director, and charge each with framing the > issue, and executing a seven figure plan, and a communications > director, to put the message with the money in targeted media > markets, and finally, to show teeth, drop the margin of error, or on > the order of high five, low six figures, in targeted congressional > races, for challengers and incumbants. > > in about a year after starting down this path, the "Congressman, its > NagOn on line one" conversation would be slightly different from > today, and in several years time, more so. > "A journey of a thousand miles begins with a single step." I don't know that a NagOn is the best way or the only way to make progress. I do know that the most likely source of that kind of funding is (many of) our employers, who may not have technical excellence on the top of their lists. But I'm even more certain that if technical people never speak up, their message will never be heard, except perhaps by accident. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: Ready to get your federal computer license?
randy, moveon is a maine-based org. it is an effective, fund raising, partisan organization. it is much more than a click-and-opine vehicle, it puts hundreds of thousands of dollars into competitive races, and has a competent political director. to create a "NagOn" we would have to hire or appoint a political director, and a financial director, and charge each with framing the issue, and executing a seven figure plan, and a communications director, to put the message with the money in targeted media markets, and finally, to show teeth, drop the margin of error, or on the order of high five, low six figures, in targeted congressional races, for challengers and incumbants. in about a year after starting down this path, the "Congressman, its NagOn on line one" conversation would be slightly different from today, and in several years time, more so. eric Randy Bush wrote: I strongly second this. To quote a bumper sticker/slogan I've seen, "if you didn't vote, you shouldn't complain". Some prominent politicians have proposed something that we -- including me -- believe to be a bad idea, not just on ideological grounds but because we think that it won't accomplish its purported goals and may even be counterproductive. I don't see a lot of network operators in Congress -- if you know better, you really need to tell them. we need an easy way to click and opine, a la moveon.org, and other social and political orgs. maybe forwardon.org? randy
Re: Ready to get your federal computer license?
+1 I operate a Maine ISP/ASP, and Senator Snowe is my lobbying target. Steven M. Bellovin wrote: On Sun, 30 Aug 2009 19:46:19 -0400 (EDT) Sean Donelan wrote: On Sun, 30 Aug 2009, Jeff Young wrote: The more troubling parts of this bill had to do with the President, at his discretion, classifying parts of public networks as "critical infrastructure" and so on. Whatever your opinion, get involved. Let your representatives know about your better ideas. I strongly second this. To quote a bumper sticker/slogan I've seen, "if you didn't vote, you shouldn't complain". Some prominent politicians have proposed something that we -- including me -- believe to be a bad idea, not just on ideological grounds but because we think that it won't accomplish its purported goals and may even be counterproductive. I don't see a lot of network operators in Congress -- if you know better, you really need to tell them. Some folks on this list -- and I know there are a few, very specifically including myself -- spend more than a little bit of time not just worrying about public policy issues, but actually spending time and effort on the subject. (I'm in D.C. right now, largely because of a policy-related meeting on Tuesday.) I'll misuses a security slogan I've seen on mass transit facilities in the New York area: if you see something, say something. If no one tells Congress that this is a bad idea, how should they know? currently living overseas and finding all of this very amusing... If any other country has solved the problem of protecting Internet/data/cyber/critical/etc infrastructures and have some great ideas, it would be great to hear what those ideas are and how they did it. Indeed. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: Ready to get your federal computer license?
> I strongly second this. To quote a bumper sticker/slogan I've seen, > "if you didn't vote, you shouldn't complain". Some prominent > politicians have proposed something that we -- including me -- believe > to be a bad idea, not just on ideological grounds but because we think > that it won't accomplish its purported goals and may even be > counterproductive. I don't see a lot of network operators in Congress > -- if you know better, you really need to tell them. we need an easy way to click and opine, a la moveon.org, and other social and political orgs. maybe forwardon.org? randy
Re: Ready to get your federal computer license?
On Sun, 30 Aug 2009 19:46:19 -0400 (EDT) Sean Donelan wrote: > On Sun, 30 Aug 2009, Jeff Young wrote: > > The more troubling parts of this bill had to do with the President, > > at his discretion, classifying parts of public networks as "critical > > infrastructure" and so on. > > Whatever your opinion, get involved. Let your representatives know > about your better ideas. I strongly second this. To quote a bumper sticker/slogan I've seen, "if you didn't vote, you shouldn't complain". Some prominent politicians have proposed something that we -- including me -- believe to be a bad idea, not just on ideological grounds but because we think that it won't accomplish its purported goals and may even be counterproductive. I don't see a lot of network operators in Congress -- if you know better, you really need to tell them. Some folks on this list -- and I know there are a few, very specifically including myself -- spend more than a little bit of time not just worrying about public policy issues, but actually spending time and effort on the subject. (I'm in D.C. right now, largely because of a policy-related meeting on Tuesday.) I'll misuses a security slogan I've seen on mass transit facilities in the New York area: if you see something, say something. If no one tells Congress that this is a bad idea, how should they know? > > > currently living overseas and finding all of this very amusing... > > If any other country has solved the problem of protecting > Internet/data/cyber/critical/etc infrastructures and have some great > ideas, it would be great to hear what those ideas are and how they > did it. > Indeed. --Steve Bellovin, http://www.cs.columbia.edu/~smb
Re: Ready to get your federal computer license?
On Sun, 30 Aug 2009, Jeff Young wrote: The more troubling parts of this bill had to do with the President, at his discretion, classifying parts of public networks as "critical infrastructure" and so on. Whatever your opinion, get involved. Let your representatives know about your better ideas. currently living overseas and finding all of this very amusing... If any other country has solved the problem of protecting Internet/data/cyber/critical/etc infrastructures and have some great ideas, it would be great to hear what those ideas are and how they did it.
Re: Link capacity upgrade threshold
On Sun, Aug 30, 2009 at 01:03:35PM -0400, Patrick W. Gilmore wrote: > >Also, a gig link on a Cisco will do approx 93-94% of imix of a gig > >in the values presented via SNMP (around 930-940 megabit/s as seen > >in "show int") before it's full, because of IFG, ethernet header > >overhead etc. > > I've heard this said many times. I've also seen 'sho int' say > 950,000,000 bits/sec and not see packets get dropped. I was under the > impression "show int" showed -every- byte leaving the interface. I > could make an argument that IFG would not be included, but things like > ethernet headers better be. > > Does this change between IOS revisions, or hardware, or is it old > info, or ... what? Actually Cisco does count layer 2 header overhead in its snmp and show int results, it is Juniper who does not (for most platforms at any rate) due to their hw architecture. I did some tests regarding this a while back on j-nsp, you'll see different results for different platforms and depending on whether you're looking at the tx or rx. Also you'll see different results for vlan overhead and the like, which can further complicate things. That said, "show int" is an epic disaster for a significantly large percentage of the time. I've seen more bugs and false readings on that thing than I can possibly count, so you really shouldn't rely on it for rate readings. The problem is extra special bad on SVIs, where you might see a reading that is 20% high or low from reality at any given second, even on modern code. I'm not aware of any major issues detecting drops though, so you should at least be able to detect them when they happen (which isn't always at line rate). If you're on a 6500/7600 platform running anything SXF+ try "show platform hardware capacity interface" to look for interfaces with lots of drops globally. -- Richard A Steenbergenhttp://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
Re: Link capacity upgrade threshold
If your 95th percentile utilization is at 80% capacity... s/80/60/ s/60/40/ I would suggest that the reason each of you have a different number is because there's a different best number for each case. Looking for any single number to fit all cases, rather than understanding the underlying process, is unlikely to yield good results. First, different people have different requirements. Some people need lowest possible cost, some people need lowest cost per volume of bits delivered, some people need lowest cost per burst capacity, some need low latency, some need low jitter, some want good customer service, some want flexible payment terms, and undoubtedly there are a thousand other possible qualities. Second, this is a binary digital network. It's never 80% full, it's never 60% full, and it's never 40% full. It's always exactly 100% full or exactly 0% full. If SNMP tells you that you've moved 800 megabits in a second on a one-gigabit pipe, then, modulo any bad implementations of SNMP, your pipe was 100% full for eight-tenths of that second. SNMP does not "hide" anything. Applying any percentile function to your data, on the other hand, does hide data. Specifically, it discards all of your data except a single point, irreversibly. So if you want to know anything about your network, you won't be looking at percentiles. Having your circuit be 100% full is a good thing, presuming you're paying for it and the traffic has some value to you. Having it be 100% full as much of the time as possible is a good thing, because that gives you a high ratio of value to cost. Dropping packets, on the other hand, is likely to be a bad thing, both because each packet putatively had value, and because many dropped packets are likely to be resent, and a resent packet is one you've paid for twice, and that's precluded the sending of another new, paid-for packet in that timeframe. The cost of not dropping packets is not having buffers overflow, and the cost of not having buffers overflow is either having deep buffers, which means high latency, or having customers with a predictable flow of traffic. Which brings me to item three. In my experience, the single biggest contributor to buffer overflow is having in-feeding (or downstream customer) circuits which are of burst capacity too close to that of the out-feeding (or upstream transit) circuits. Let's say that your outbound circuit is a gigabit, you have two inbound circuits that are a gigabit and run at 100% utilization 10% of the time each, and you have a megabit of buffer memory allocated to the outbound circuit. 1% of the time, both of the inbound circuits will be at 100% utilization simultaneously. When that's happening, you'll have data flowing in at the rate of two gigabits per second, which will fill the buffer in one twentieth of a second, if it persists. And, just like Rosencrantz and Guildenstern flipping coins, such a run will inevitably persist longer than you'd desire, frequently enough. On the other hand, if you have twenty inbound circuits of 100 megabits each, which are transmitting at 100% of capacity 10% of the time each, you're looking at exactly the same amount of data, however it arrives _much more predictably_, since the 2-gigabit inflow would only occur 0.001% of the time, rather than 1% of the time. And it would also be proportionally unlikely to persist for the longer periods of time necessary to overflow the buffer. Thus Kevin's ESnet customers, who are much more likely to be 10gb or 40gb downstream circuits feeding into his 40gb upstream circuits, are much more likely to overflow buffers, than a consumer Internet provider who's feeding 1mb circuits into a gigabit circuit, even if the aggregation ratio of the latter is hundreds of times higher. So, in summary: Your dropped packet counters are the ones to be looking at as a measure of goodput, more than your utilization counters. And keep the size of your aggregation pipes as much bigger than the size of the pipes you aggregate into them as you can afford to. As always, my apologies to those of you for whom this is unnecessarily remedial, for using NANOG bandwidth and a portion of your Sunday morning. -Bill PGP.sig Description: This is a digitally signed message part
Re: Link capacity upgrade threshold
On Aug 30, 2009, at 1:23 AM, Mikael Abrahamsson wrote: On Sun, 30 Aug 2009, William Herrin wrote: If your 95th percentile utilization is at 80% capacity, it's time to start planning the upgrade. If your 95th percentile utilization is at 95% it's time to finish the upgrade. I now see why people at the IETF spoke in a way that "core network congestion" was something natural. If your MRTG graph is showing 95% load in 5 minute average, you're most likely congesting/buffering at some time during that 5 minute interval. If this is acceptable or not in your network (it's not in mine) that's up to you. Also, a gig link on a Cisco will do approx 93-94% of imix of a gig in the values presented via SNMP (around 930-940 megabit/s as seen in "show int") before it's full, because of IFG, ethernet header overhead etc. I've heard this said many times. I've also seen 'sho int' say 950,000,000 bits/sec and not see packets get dropped. I was under the impression "show int" showed -every- byte leaving the interface. I could make an argument that IFG would not be included, but things like ethernet headers better be. Does this change between IOS revisions, or hardware, or is it old info, or ... what? -- TTFN, patrick P.S. I agree that without perfect conditions (e.g. using an Ixia to test link speeds), you should upgrade WAY before 90-something percent. microbursts are real, and buffer space is small these days. I'm just asking what the counters -actually- show. So personally, I consider a gig link "in desperate need of upgrade" when it's showing around 850-880 megs of traffic in mrtg. -- Mikael Abrahamssonemail: swm...@swm.pp.se
Re: Link capacity upgrade threshold
On 30/08/2009 17:53, Shane Ronan wrote: What system were you using to monitor link usage? yrtg Nick
Re: Link capacity upgrade threshold
What system were you using to monitor link usage? Shane On Aug 30, 2009, at 8:26 AM, Nick Hilliard wrote: On 30/08/2009 13:04, Randy Bush wrote: the normal snmp and other averaging methods *really* miss the bursts. Definitely. For fun and giggles, I recently turned on 30 second polling on some kit and it turned up all sorts of interesting peculiarities that were completely blotted out in a 5 minute average. In order to get a really good idea of what's going on at a microburst level, you would need to poll as often as it takes to fill the buffer of the port in question. This is not feasible in the general case, which is why we resort to hacks like QoS to make sure that when there is congestion, it is handled semi-sensibly. There's a lot to the saying that QoS really means "Quantity of Service", because quality of service only ever becomes a problem if there is a shortfall in quantity. Nick
Re: Link capacity upgrade threshold
> Date: Sun, 30 Aug 2009 21:04:15 +0900 > From: Randy Bush > > > If your 95th percentile utilization is at 80% capacity, it's time to > > start planning the upgrade. > > s/80/60/ > > the normal snmp and other averaging methods *really* miss the bursts. s/60/40/ If you need to carry large TCP flows, say 2Gbps on a 10GE, dropping even a single packet due to congestion is unacceptable. Even with fast recovery, the average transmission rate will take a noticeable dip on every drop and even a drop rate under 1% will slow the flow dramatically. The point is, what is acceptable for one traffic profile may be unacceptable for another. Mail and web browsing are generally unaffected by light congestion. Other applications are not so forgiving. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: ober...@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751
Re: Link capacity upgrade threshold
If talking about just max capacity, I would agree with most of the statements of 80+% being in the right range, likely with a very fine line of when you actually start seeing a performance impact. Operationally, at least in our network, I'd never run anything at that level. Providers that are redundant for each other don't normally operate above 40-45%, in order to accommodate a failure. Other links that have a backup, but don't actively load share, normally run up to about 60-70% before being upgraded. By the time the upgrade is complete, it could be close to 80%. Tom Sands Rackspace Hosting William Herrin wrote: On Sat, Aug 29, 2009 at 11:50 PM, devang patel wrote: I just wanted to know what is Link capacity upgrade threshold in terms of % of link utilization? Just to get an idea... If your 95th percentile utilization is at 80% capacity, it's time to start planning the upgrade. If your 95th percentile utilization is at 95% it's time to finish the upgrade. If you average or median utilizations are at 80% capacity then as often as not it's time for your boss to fire you and replace you with someone who can do the job. Slight variations depending on the resource. Use absolute peak instead of 95th percentile for modem bank utilization -- under normal circumstances a modem bank should never ring busy. And a gig-e can run a little closer to the edge (percentage-wise) before folks notice slowness than a T1 can. Regards, Bill Herrin Confidentiality Notice: This e-mail message (including any attached or embedded documents) is intended for the exclusive and confidential use of the individual or entity to which this message is addressed, and unless otherwise expressly indicated, is confidential and privileged information of Rackspace. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at ab...@rackspace.com, and delete the original message. Your cooperation is appreciated.
Re: Link capacity upgrade threshold
Nick Hilliard wrote: Definitely. For fun and giggles, I recently turned on 30 second polling on some kit and it turned up all sorts of interesting peculiarities that were completely blotted out in a 5 minute average. Would RMON History and Alarms help? I've always considered rolling them out to some of my kit to catch microbursts. Poggs
Re: Link capacity upgrade threshold
On 30/08/2009 13:04, Randy Bush wrote: the normal snmp and other averaging methods *really* miss the bursts. Definitely. For fun and giggles, I recently turned on 30 second polling on some kit and it turned up all sorts of interesting peculiarities that were completely blotted out in a 5 minute average. In order to get a really good idea of what's going on at a microburst level, you would need to poll as often as it takes to fill the buffer of the port in question. This is not feasible in the general case, which is why we resort to hacks like QoS to make sure that when there is congestion, it is handled semi-sensibly. There's a lot to the saying that QoS really means "Quantity of Service", because quality of service only ever becomes a problem if there is a shortfall in quantity. Nick
Re: Link capacity upgrade threshold
> If your 95th percentile utilization is at 80% capacity, it's time to > start planning the upgrade. s/80/60/ the normal snmp and other averaging methods *really* miss the bursts. randy
