Re: 2.6.12 Performance problems
--- Ben Greear <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > > I didn't refuse. I just chose to take help > from > > Ben, because Ben took the time to reproduce > the > > problem and to provide useful settings that > made > > sense to me. There's nothing wrong with my > > machine. > > Well, I didn't see the slowdown on my system > when I tried 2.6 > v/s 2.4. You reported a 3x slowdown. In your > original mail, > you didn't mention trying to do compiles at the > same time > as your network test. I *did* see a pathetic > slowdown (from 250kpps > bridging to about 6kpps getting through the > bridge) when I coppied > a 512MB CDROM to the HD, but let's get the pure > network slowdown > on your system figured out before we start > looking at the impact > of disk IO. > > So, if you see a 3x slowdown on an unloaded > machine when going > from 2.4 to 2.6, then there is something unique > about your system > and we should figure out what it is. > > Also, my numbers (about 250kpps with moderate > amount of drops) > was a lot better than the 100kpps you reported > for 2.6. My > hardware is different (P-IV, HT, 3.0Ghz 1MB > Cache, 1GB RAM, > dual Intel pro/1000 NIC in PCI-X 66/133 slot), > but I would not > expect that to be 2x faster than your Opteron > (or whatever you had). > > You could mention what you are using to > generate traffic. (I was > using pktgen to generate a stream of 60 byte > pkts.) > > I think you should give us some better > information on exactly > what you are doing on 2.4 v/s 2.6. And for > heaven's sake, > don't hesitate to send kernel configs and > hardware listings > to folks that ask. Ruling out problems is as > useful as finding > problems in this sort of thing. Ok, good point, but I think the original issue has evolved into "why does linux drop packets even when its not close to capacity". Its not really a "slow down" now that we've been discussing it, its the point at which packet loss begins to occur. And since it appears that packet loss can occur at usage levels not even close to capacity, I can't really determine the capacity. Since CPU usage also doesn't seem to register correctly, I can't get a handle on the load for various activities. So it may not have anything to do with "slowness". I feel like I'm trying to weigh a big blob of Jell-o that keeps falling off the scale. The bottom line is that if I can't get it to stop dropping packets, the performance is moot, because its simply not suitable for my needs. DT Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- "Vladimir B. Savkin" <[EMAIL PROTECTED]> wrote: > On Wed, Aug 24, 2005 at 11:08:43PM -0700, > Danial Thom wrote: > > If your test is still set up, try compiling > > something large while doing the test. The > drops > > go through the roof in my tests. > > > Couldn't this happen because ksoftirqd by > default > has a nice value of 19? > > ~ > :wq > With > best regards, > > Vladimir Savkin. renicing ksoftirqd improves it a bit, but still tons of loss, even at only 120Kpps. Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Vladimir B. Savkin [EMAIL PROTECTED] wrote: On Wed, Aug 24, 2005 at 11:08:43PM -0700, Danial Thom wrote: If your test is still set up, try compiling something large while doing the test. The drops go through the roof in my tests. Couldn't this happen because ksoftirqd by default has a nice value of 19? ~ :wq With best regards, Vladimir Savkin. renicing ksoftirqd improves it a bit, but still tons of loss, even at only 120Kpps. Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine. Well, I didn't see the slowdown on my system when I tried 2.6 v/s 2.4. You reported a 3x slowdown. In your original mail, you didn't mention trying to do compiles at the same time as your network test. I *did* see a pathetic slowdown (from 250kpps bridging to about 6kpps getting through the bridge) when I coppied a 512MB CDROM to the HD, but let's get the pure network slowdown on your system figured out before we start looking at the impact of disk IO. So, if you see a 3x slowdown on an unloaded machine when going from 2.4 to 2.6, then there is something unique about your system and we should figure out what it is. Also, my numbers (about 250kpps with moderate amount of drops) was a lot better than the 100kpps you reported for 2.6. My hardware is different (P-IV, HT, 3.0Ghz 1MB Cache, 1GB RAM, dual Intel pro/1000 NIC in PCI-X 66/133 slot), but I would not expect that to be 2x faster than your Opteron (or whatever you had). You could mention what you are using to generate traffic. (I was using pktgen to generate a stream of 60 byte pkts.) I think you should give us some better information on exactly what you are doing on 2.4 v/s 2.6. And for heaven's sake, don't hesitate to send kernel configs and hardware listings to folks that ask. Ruling out problems is as useful as finding problems in this sort of thing. Ok, good point, but I think the original issue has evolved into why does linux drop packets even when its not close to capacity. Its not really a slow down now that we've been discussing it, its the point at which packet loss begins to occur. And since it appears that packet loss can occur at usage levels not even close to capacity, I can't really determine the capacity. Since CPU usage also doesn't seem to register correctly, I can't get a handle on the load for various activities. So it may not have anything to do with slowness. I feel like I'm trying to weigh a big blob of Jell-o that keeps falling off the scale. The bottom line is that if I can't get it to stop dropping packets, the performance is moot, because its simply not suitable for my needs. DT Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine. Well, I didn't see the slowdown on my system when I tried 2.6 v/s 2.4. You reported a 3x slowdown. In your original mail, you didn't mention trying to do compiles at the same time as your network test. I *did* see a pathetic slowdown (from 250kpps bridging to about 6kpps getting through the bridge) when I coppied a 512MB CDROM to the HD, but let's get the pure network slowdown on your system figured out before we start looking at the impact of disk IO. So, if you see a 3x slowdown on an unloaded machine when going from 2.4 to 2.6, then there is something unique about your system and we should figure out what it is. Also, my numbers (about 250kpps with moderate amount of drops) was a lot better than the 100kpps you reported for 2.6. My hardware is different (P-IV, HT, 3.0Ghz 1MB Cache, 1GB RAM, dual Intel pro/1000 NIC in PCI-X 66/133 slot), but I would not expect that to be 2x faster than your Opteron (or whatever you had). You could mention what you are using to generate traffic. (I was using pktgen to generate a stream of 60 byte pkts.) I think you should give us some better information on exactly what you are doing on 2.4 v/s 2.6. And for heaven's sake, don't hesitate to send kernel configs and hardware listings to folks that ask. Ruling out problems is as useful as finding problems in this sort of thing. Thanks, Ben -- Ben Greear <[EMAIL PROTECTED]> Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Danial Thom <[EMAIL PROTECTED]> wrote: > > > --- Ben Greear <[EMAIL PROTECTED]> wrote: > > > Danial Thom wrote: > > > > > > --- Ben Greear <[EMAIL PROTECTED]> > > wrote: > > > > > > > > >>Danial Thom wrote: > > >> > > >> > > >>>I think the concensus is that 2.6 has made > > >> > > >>trade > > >> > > >>>offs that lower raw throughput, which is > > what > > >> > > >>a > > >> > > >>>networking device needs. So as a router or > > >>>network appliance, 2.6 seems less > suitable. > > A > > >> > > >>raw > > >> > > >>>bridging test on a 2.0Ghz operton system: > > >>> > > >>>FreeBSD 4.9: Drops no packets at 900K pps > > >>>Linux 2.4.24: Starts dropping packets at > > 350K > > >> > > >>pps > > >> > > >>>Linux 2.6.12: Starts dropping packets at > > 100K > > >> > > >>pps > > >> > > >>I ran some quick tests using kernel 2.6.11, > > 1ms > > >>tick (HZ=1000), SMP kernel. > > >>Hardware is P-IV 3.0Ghz + HT on a new > > >>SuperMicro motherboard with 64/133Mhz > > >>PCI-X bus. NIC is dual Intel pro/1000. > > Kernel > > >>is close to stock 2.6.11. > > > > > What GigE adapters did you use? Clearly > every > > > driver is going to be different. My > > experience is > > > that a 3.4Ghz P4 is about the performance > of > > a > > > 2.0Ghz Opteron. I have to try your tuning > > script > > > tomorrow. > > > > Intel pro/1000, as I mentioned. I haven't > > tried any other > > NIC that comes close in performance to the > > e1000. > > > > > If your test is still set up, try compiling > > > something large while doing the test. The > > drops > > > go through the roof in my tests. > > > > Installing RH9 on the box now to try some > > tests... > > > > Disk access always robs networking, in my > > experience, so > > I am not supprised you see bad ntwk > performance > > while > > compiling. > > > > Ben > > It would be useful if there were some way to > find > out "what" is getting "robbed". If networking > has > priority, then what is keeping it from getting > back to processing the rx interrupts? > > Ah, the e1000 has built-in interrupt > moderation. > I can't get into my lab until tomorrow > afternoon, > but if you get a chance try setting ITR in > e1000_main.c to something larger, like 20K. and > see if it makes a difference. At 200K pps that > would cause an interrupt every 10 packets, > which > may allow the routine to grab back the cpu more > often. > > > Danial > Just FYI, setting interrupt moderation to 20,000 didn't make much difference. Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Adrian Bunk <[EMAIL PROTECTED]> wrote: > On Fri, Aug 26, 2005 at 10:06:51AM -0700, > Danial Thom wrote: > >... > > I don't think I'm obligated to answer every > > single person who pipes into a thread. People > who > > say "show me your config and dmesg" are not > > useful. Linux has long had a philisophical > > problem of dropping packets as a "performance > > feature", and we've already established I > think > > that you can't eliminate it altogether, if > you > > read the thread carefully. > >... > > You say you have observed for a long time a > problem. > > The only person participating in this thread > who is one of the > networking maintainers is Patrick McHardy. > > Did _he_ say this was a known and unfixable > problem? I don't know. He said that the network stack has absolute priority, so perhaps he can explain why compiling a kernel causes an exponential increase in packet loss? The FreeBSD "network maintainers" are largely clueless about just about every problem in 5.x, so I'm not sure the title qualifies someone as knowing all there is to know. I haven't heard him claim that he can do the things I'm doing without any receive errors or drops. I'd like to hear about what test he runs to test this sort of problem. Before you can solve a problem you have to have a clear understanding of what the problem is, and admit that you have a problem. > > He wanted to help you and you pissed him off > because you refuxed to give > him dmesg, .config and other information that > would have helped him to > debug your problem. If you don't feel obliged > to help the persons > responsible for the part of the kernel you have > a problem with to debug > your problem your whole initial mail was > pointless. I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine. I don't know who is who; I judge people based on the intelligence of their analysis. People responsible for the network stack may not be the right people, because this is more likely a scheduler issue. There's probably nothing wrong with the stack or the drivers; the machine just doesn't react fast enough to avert ring overruns. DT __ Yahoo! Mail Stay connected, organized, and protected. Take the tour: http://tour.mail.yahoo.com/mailtour.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Thu, Aug 25, 2005 at 09:55:27AM -0700, Ben Greear wrote: > Of course. Never found a motherboard yet with decent built-in > NICs. The built-ins on this board are tg3 and they must be on > a slow bus, because they cannot go faster than about 700Mbps > (using big pkts). There should be a number of decent boards out on the market these days. Try picking up one with a CSA gige adapter (a dedicated high bandwidth link to an e1000) or PCI Express (although that is harder to pick up on from the specifications of most motherboards). -ben - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Fri, Aug 26, 2005 at 10:06:51AM -0700, Danial Thom wrote: >... > I don't think I'm obligated to answer every > single person who pipes into a thread. People who > say "show me your config and dmesg" are not > useful. Linux has long had a philisophical > problem of dropping packets as a "performance > feature", and we've already established I think > that you can't eliminate it altogether, if you > read the thread carefully. >... You say you have observed for a long time a problem. The only person participating in this thread who is one of the networking maintainers is Patrick McHardy. Did _he_ say this was a known and unfixable problem? He wanted to help you and you pissed him off because you refuxed to give him dmesg, .config and other information that would have helped him to debug your problem. If you don't feel obliged to help the persons responsible for the part of the kernel you have a problem with to debug your problem your whole initial mail was pointless. > DT cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Adrian Bunk <[EMAIL PROTECTED]> wrote: > On Fri, Aug 26, 2005 at 08:34:14AM -0700, > Danial Thom wrote: > > > > --- Adrian Bunk <[EMAIL PROTECTED]> wrote: > > > > > > That's not always true. > > > > > > Imagine a slow computer with a GBit > ethernet > > > connection, where the user > > > is downloading files from a server that can > > > utilize the full > > > network connection while listening to music > > > from his local disk with > > > XMMS. > > > > > > In this case, the audio stream is not > depending > > > on the network > > > connection. And the user might prefer > dropped > > > packages over a stuttering > > > XMMS. > > > Audio connections are going to be > windowed/flowed > > in some way (thats how the internet works) so > >... > > I was talking about an audio stream coming from > a file on the > "local disk", IOW something like an mp3 file. > > But the most interesting thing about your email > is not what you were > answering to, but which part of my email you > silently omitted. Since you > are not answering questions that might help to > debug the problem you > claim to have, it seems your intention is not > getting a Linux problem > fixed... I don't think I'm obligated to answer every single person who pipes into a thread. People who say "show me your config and dmesg" are not useful. Linux has long had a philisophical problem of dropping packets as a "performance feature", and we've already established I think that you can't eliminate it altogether, if you read the thread carefully. Also, if you're on a slow machine you can't download faster than your machine can handle it, because the "server on a gig network" can't send faster then you tell it to, not matter how fast it is. You'll never overrun a machine with 1 or 2 or 5 connections. Having a clue how things work is important. DT Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Fri, Aug 26, 2005 at 08:34:14AM -0700, Danial Thom wrote: > > --- Adrian Bunk <[EMAIL PROTECTED]> wrote: > > > > That's not always true. > > > > Imagine a slow computer with a GBit ethernet > > connection, where the user > > is downloading files from a server that can > > utilize the full > > network connection while listening to music > > from his local disk with > > XMMS. > > > > In this case, the audio stream is not depending > > on the network > > connection. And the user might prefer dropped > > packages over a stuttering > > XMMS. > Audio connections are going to be windowed/flowed > in some way (thats how the internet works) so >... I was talking about an audio stream coming from a file on the "local disk", IOW something like an mp3 file. But the most interesting thing about your email is not what you were answering to, but which part of my email you silently omitted. Since you are not answering questions that might help to debug the problem you claim to have, it seems your intention is not getting a Linux problem fixed... > DT cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Adrian Bunk <[EMAIL PROTECTED]> wrote: > On Mon, Aug 22, 2005 at 08:41:11AM -0700, > Danial Thom wrote: > >... > > > > The issue I have with that logic is that you > seem > > to use "kernel" in a general sense without > regard > > to what its doing. Dropping packets is always > > detrimental to the user regardless of what > he's > > using the computer for. An audio stream that > > drops packets isn't going to be "smooth" at > the > > user level. > > > That's not always true. > > Imagine a slow computer with a GBit ethernet > connection, where the user > is downloading files from a server that can > utilize the full > network connection while listening to music > from his local disk with > XMMS. > > In this case, the audio stream is not depending > on the network > connection. And the user might prefer dropped > packages over a stuttering > XMMS. > Audio connections are going to be windowed/flowed in some way (thats how the internet works) so you're not dealing with real capacity issues in that example. You're not going to overrun your ethernet adapter with an application; the issue is running applications on servers that are also doing a lot of other things. You can't isolate a particular application at the device level, so you drop packets randomly; not just ones you don't care about. If your application can't run well on your machine without dropping real traffic, then you need a faster machine, not an OS that compensates in a negative way. DT __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Mon, Aug 22, 2005 at 08:41:11AM -0700, Danial Thom wrote: >... > > The issue I have with that logic is that you seem > to use "kernel" in a general sense without regard > to what its doing. Dropping packets is always > detrimental to the user regardless of what he's > using the computer for. An audio stream that > drops packets isn't going to be "smooth" at the > user level. That's not always true. Imagine a slow computer with a GBit ethernet connection, where the user is downloading files from a server that can utilize the full network connection while listening to music from his local disk with XMMS. In this case, the audio stream is not depending on the network connection. And the user might prefer dropped packages over a stuttering XMMS. > All of this aside, I need to measure the raw > capabilities of the kernel. With 'bsd OSes I can > tell what the breaking point is by driving the > machine to livelock. Linux seems to have a soft, > floating capacity in that it will drop packets > here and there for no isolatable reason. I'm > having difficulty making a case for its use in a > networking appliance, as dropped packets are not > acceptable. How do I tune the "its ok to drop > packets when x occurs" algorithm to be "its never > ok to drop packets unless x occurs" (such as a > queue depth)? Is it possible? What do you want to achieve with this thread? 1. Try to proof that Linux is inferior to *BSD? 2. Get help with your problem? If your goal is 1. feel free to do so, but please do so on more appropriate lists. If your goal is 2., you must help the people who are trying to help you with _your_ problem. E.g. Patrick McHardy asked you: <-- snip --> In that case please send more information, like both 2.4 and 2.6 configs, dmesg output from booting, lspci output, other hardware details, .. <-- snip --> Crystal balls are rare, and if your goal is really to get problem solved you should send the information other people require for debugging your problem. > Danial cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Mon, Aug 22, 2005 at 08:41:11AM -0700, Danial Thom wrote: ... The issue I have with that logic is that you seem to use kernel in a general sense without regard to what its doing. Dropping packets is always detrimental to the user regardless of what he's using the computer for. An audio stream that drops packets isn't going to be smooth at the user level. That's not always true. Imagine a slow computer with a GBit ethernet connection, where the user is downloading files from a server that can utilize the full network connection while listening to music from his local disk with XMMS. In this case, the audio stream is not depending on the network connection. And the user might prefer dropped packages over a stuttering XMMS. All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft, floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the its ok to drop packets when x occurs algorithm to be its never ok to drop packets unless x occurs (such as a queue depth)? Is it possible? What do you want to achieve with this thread? 1. Try to proof that Linux is inferior to *BSD? 2. Get help with your problem? If your goal is 1. feel free to do so, but please do so on more appropriate lists. If your goal is 2., you must help the people who are trying to help you with _your_ problem. E.g. Patrick McHardy asked you: -- snip -- In that case please send more information, like both 2.4 and 2.6 configs, dmesg output from booting, lspci output, other hardware details, .. -- snip -- Crystal balls are rare, and if your goal is really to get problem solved you should send the information other people require for debugging your problem. Danial cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Adrian Bunk [EMAIL PROTECTED] wrote: On Mon, Aug 22, 2005 at 08:41:11AM -0700, Danial Thom wrote: ... The issue I have with that logic is that you seem to use kernel in a general sense without regard to what its doing. Dropping packets is always detrimental to the user regardless of what he's using the computer for. An audio stream that drops packets isn't going to be smooth at the user level. That's not always true. Imagine a slow computer with a GBit ethernet connection, where the user is downloading files from a server that can utilize the full network connection while listening to music from his local disk with XMMS. In this case, the audio stream is not depending on the network connection. And the user might prefer dropped packages over a stuttering XMMS. Audio connections are going to be windowed/flowed in some way (thats how the internet works) so you're not dealing with real capacity issues in that example. You're not going to overrun your ethernet adapter with an application; the issue is running applications on servers that are also doing a lot of other things. You can't isolate a particular application at the device level, so you drop packets randomly; not just ones you don't care about. If your application can't run well on your machine without dropping real traffic, then you need a faster machine, not an OS that compensates in a negative way. DT __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Fri, Aug 26, 2005 at 08:34:14AM -0700, Danial Thom wrote: --- Adrian Bunk [EMAIL PROTECTED] wrote: That's not always true. Imagine a slow computer with a GBit ethernet connection, where the user is downloading files from a server that can utilize the full network connection while listening to music from his local disk with XMMS. In this case, the audio stream is not depending on the network connection. And the user might prefer dropped packages over a stuttering XMMS. Audio connections are going to be windowed/flowed in some way (thats how the internet works) so ... I was talking about an audio stream coming from a file on the local disk, IOW something like an mp3 file. But the most interesting thing about your email is not what you were answering to, but which part of my email you silently omitted. Since you are not answering questions that might help to debug the problem you claim to have, it seems your intention is not getting a Linux problem fixed... DT cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Adrian Bunk [EMAIL PROTECTED] wrote: On Fri, Aug 26, 2005 at 08:34:14AM -0700, Danial Thom wrote: --- Adrian Bunk [EMAIL PROTECTED] wrote: That's not always true. Imagine a slow computer with a GBit ethernet connection, where the user is downloading files from a server that can utilize the full network connection while listening to music from his local disk with XMMS. In this case, the audio stream is not depending on the network connection. And the user might prefer dropped packages over a stuttering XMMS. Audio connections are going to be windowed/flowed in some way (thats how the internet works) so ... I was talking about an audio stream coming from a file on the local disk, IOW something like an mp3 file. But the most interesting thing about your email is not what you were answering to, but which part of my email you silently omitted. Since you are not answering questions that might help to debug the problem you claim to have, it seems your intention is not getting a Linux problem fixed... I don't think I'm obligated to answer every single person who pipes into a thread. People who say show me your config and dmesg are not useful. Linux has long had a philisophical problem of dropping packets as a performance feature, and we've already established I think that you can't eliminate it altogether, if you read the thread carefully. Also, if you're on a slow machine you can't download faster than your machine can handle it, because the server on a gig network can't send faster then you tell it to, not matter how fast it is. You'll never overrun a machine with 1 or 2 or 5 connections. Having a clue how things work is important. DT Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Fri, Aug 26, 2005 at 10:06:51AM -0700, Danial Thom wrote: ... I don't think I'm obligated to answer every single person who pipes into a thread. People who say show me your config and dmesg are not useful. Linux has long had a philisophical problem of dropping packets as a performance feature, and we've already established I think that you can't eliminate it altogether, if you read the thread carefully. ... You say you have observed for a long time a problem. The only person participating in this thread who is one of the networking maintainers is Patrick McHardy. Did _he_ say this was a known and unfixable problem? He wanted to help you and you pissed him off because you refuxed to give him dmesg, .config and other information that would have helped him to debug your problem. If you don't feel obliged to help the persons responsible for the part of the kernel you have a problem with to debug your problem your whole initial mail was pointless. DT cu Adrian -- Is there not promise of rain? Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. Only a promise, Lao Er said. Pearl S. Buck - Dragon Seed - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Adrian Bunk [EMAIL PROTECTED] wrote: On Fri, Aug 26, 2005 at 10:06:51AM -0700, Danial Thom wrote: ... I don't think I'm obligated to answer every single person who pipes into a thread. People who say show me your config and dmesg are not useful. Linux has long had a philisophical problem of dropping packets as a performance feature, and we've already established I think that you can't eliminate it altogether, if you read the thread carefully. ... You say you have observed for a long time a problem. The only person participating in this thread who is one of the networking maintainers is Patrick McHardy. Did _he_ say this was a known and unfixable problem? I don't know. He said that the network stack has absolute priority, so perhaps he can explain why compiling a kernel causes an exponential increase in packet loss? The FreeBSD network maintainers are largely clueless about just about every problem in 5.x, so I'm not sure the title qualifies someone as knowing all there is to know. I haven't heard him claim that he can do the things I'm doing without any receive errors or drops. I'd like to hear about what test he runs to test this sort of problem. Before you can solve a problem you have to have a clear understanding of what the problem is, and admit that you have a problem. He wanted to help you and you pissed him off because you refuxed to give him dmesg, .config and other information that would have helped him to debug your problem. If you don't feel obliged to help the persons responsible for the part of the kernel you have a problem with to debug your problem your whole initial mail was pointless. I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine. I don't know who is who; I judge people based on the intelligence of their analysis. People responsible for the network stack may not be the right people, because this is more likely a scheduler issue. There's probably nothing wrong with the stack or the drivers; the machine just doesn't react fast enough to avert ring overruns. DT __ Yahoo! Mail Stay connected, organized, and protected. Take the tour: http://tour.mail.yahoo.com/mailtour.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine. Well, I didn't see the slowdown on my system when I tried 2.6 v/s 2.4. You reported a 3x slowdown. In your original mail, you didn't mention trying to do compiles at the same time as your network test. I *did* see a pathetic slowdown (from 250kpps bridging to about 6kpps getting through the bridge) when I coppied a 512MB CDROM to the HD, but let's get the pure network slowdown on your system figured out before we start looking at the impact of disk IO. So, if you see a 3x slowdown on an unloaded machine when going from 2.4 to 2.6, then there is something unique about your system and we should figure out what it is. Also, my numbers (about 250kpps with moderate amount of drops) was a lot better than the 100kpps you reported for 2.6. My hardware is different (P-IV, HT, 3.0Ghz 1MB Cache, 1GB RAM, dual Intel pro/1000 NIC in PCI-X 66/133 slot), but I would not expect that to be 2x faster than your Opteron (or whatever you had). You could mention what you are using to generate traffic. (I was using pktgen to generate a stream of 60 byte pkts.) I think you should give us some better information on exactly what you are doing on 2.4 v/s 2.6. And for heaven's sake, don't hesitate to send kernel configs and hardware listings to folks that ask. Ruling out problems is as useful as finding problems in this sort of thing. Thanks, Ben -- Ben Greear [EMAIL PROTECTED] Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Danial Thom [EMAIL PROTECTED] wrote: --- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: --- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. What GigE adapters did you use? Clearly every driver is going to be different. My experience is that a 3.4Ghz P4 is about the performance of a 2.0Ghz Opteron. I have to try your tuning script tomorrow. Intel pro/1000, as I mentioned. I haven't tried any other NIC that comes close in performance to the e1000. If your test is still set up, try compiling something large while doing the test. The drops go through the roof in my tests. Installing RH9 on the box now to try some tests... Disk access always robs networking, in my experience, so I am not supprised you see bad ntwk performance while compiling. Ben It would be useful if there were some way to find out what is getting robbed. If networking has priority, then what is keeping it from getting back to processing the rx interrupts? Ah, the e1000 has built-in interrupt moderation. I can't get into my lab until tomorrow afternoon, but if you get a chance try setting ITR in e1000_main.c to something larger, like 20K. and see if it makes a difference. At 200K pps that would cause an interrupt every 10 packets, which may allow the routine to grab back the cpu more often. Danial Just FYI, setting interrupt moderation to 20,000 didn't make much difference. Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Thu, Aug 25, 2005 at 09:55:27AM -0700, Ben Greear wrote: Of course. Never found a motherboard yet with decent built-in NICs. The built-ins on this board are tg3 and they must be on a slow bus, because they cannot go faster than about 700Mbps (using big pkts). There should be a number of decent boards out on the market these days. Try picking up one with a CSA gige adapter (a dedicated high bandwidth link to an e1000) or PCI Express (although that is harder to pick up on from the specifications of most motherboards). -ben - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > > > --- Ben Greear <[EMAIL PROTECTED]> > wrote: > > > > > >>Danial Thom wrote: > >> > >> > >>>I think the concensus is that 2.6 has made > >> > >>trade > >> > >>>offs that lower raw throughput, which is > what > >> > >>a > >> > >>>networking device needs. So as a router or > >>>network appliance, 2.6 seems less suitable. > A > >> > >>raw > >> > >>>bridging test on a 2.0Ghz operton system: > >>> > >>>FreeBSD 4.9: Drops no packets at 900K pps > >>>Linux 2.4.24: Starts dropping packets at > 350K > >> > >>pps > >> > >>>Linux 2.6.12: Starts dropping packets at > 100K > >> > >>pps > >> > >>I ran some quick tests using kernel 2.6.11, > 1ms > >>tick (HZ=1000), SMP kernel. > >>Hardware is P-IV 3.0Ghz + HT on a new > >>SuperMicro motherboard with 64/133Mhz > >>PCI-X bus. NIC is dual Intel pro/1000. > Kernel > >>is close to stock 2.6.11. > > > What GigE adapters did you use? Clearly every > > driver is going to be different. My > experience is > > that a 3.4Ghz P4 is about the performance of > a > > 2.0Ghz Opteron. I have to try your tuning > script > > tomorrow. > > Intel pro/1000, as I mentioned. I haven't > tried any other > NIC that comes close in performance to the > e1000. > > > If your test is still set up, try compiling > > something large while doing the test. The > drops > > go through the roof in my tests. > > Installing RH9 on the box now to try some > tests... > > Disk access always robs networking, in my > experience, so > I am not supprised you see bad ntwk performance > while > compiling. > > Ben It would be useful if there were some way to find out "what" is getting "robbed". If networking has priority, then what is keeping it from getting back to processing the rx interrupts? Ah, the e1000 has built-in interrupt moderation. I can't get into my lab until tomorrow afternoon, but if you get a chance try setting ITR in e1000_main.c to something larger, like 20K. and see if it makes a difference. At 200K pps that would cause an interrupt every 10 packets, which may allow the routine to grab back the cpu more often. Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > > The tests I reported where on UP systems. > Perhaps > > the default settings are better for this in > 2.4, > > since that is what I used, and you used your > > hacks for both. > > My modifications to the kernel are unlikely to > speed anything > up, and probably will slow things down ever so > slightly. > > I can try with a UP kernel, but my machine at > least has a single > processor. I'm using the SMP kernel to take > advantage of HT. > > > Are you getting drops or overruns (or both)? > I > > would assume drops is a decision to drop > rather > > than an overrun which is a ring overrun. > Overruns > > would imply more about performance than > tuning, > > I'd think. > > I was seeing lots of NIC errors...in fact, it > was showing a great many > more errors than packets sent to it, so I just > ignored them. > > I increased the TxDescriptors and RxDescriptors > and that helped a little. > > Increasing the transmit queue for the NIC to > 2000 also helped a little. > > > I wouldn't think that HT would be appropriate > for > > this sort of setup...? > > 2.6.11 seems to be faster when running SMP > kernel on this system. HT and SMP are not the same animal, are they? My understanding is that an HT aware scheduler is likely to make things worse most of the time, particularly for systems not running a lot of threads.. > > > > You're using a dual PCI-X NIC rather than the > > onboard ports? Supermicro runs their onboard > > Of course. Never found a motherboard yet with > decent built-in > NICs. The built-ins on this board are tg3 and > they must be on > a slow bus, because they cannot go faster than > about 700Mbps > (using big pkts). If its the P8SCI or the same design they are on a 1X PCIE thats shared with the PCI-X. Pretty hokey stuff. Its also a low-end controller amongst the broadcom parts. Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: The tests I reported where on UP systems. Perhaps the default settings are better for this in 2.4, since that is what I used, and you used your hacks for both. My modifications to the kernel are unlikely to speed anything up, and probably will slow things down ever so slightly. I can try with a UP kernel, but my machine at least has a single processor. I'm using the SMP kernel to take advantage of HT. Are you getting drops or overruns (or both)? I would assume drops is a decision to drop rather than an overrun which is a ring overrun. Overruns would imply more about performance than tuning, I'd think. I was seeing lots of NIC errors...in fact, it was showing a great many more errors than packets sent to it, so I just ignored them. I increased the TxDescriptors and RxDescriptors and that helped a little. Increasing the transmit queue for the NIC to 2000 also helped a little. I wouldn't think that HT would be appropriate for this sort of setup...? 2.6.11 seems to be faster when running SMP kernel on this system. You're using a dual PCI-X NIC rather than the onboard ports? Supermicro runs their onboard Of course. Never found a motherboard yet with decent built-in NICs. The built-ins on this board are tg3 and they must be on a slow bus, because they cannot go faster than about 700Mbps (using big pkts). I'll benchmark things again when 2.6.13 comes out and try to get some more detailed numbers... Thanks, Ben -- Ben Greear <[EMAIL PROTECTED]> Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > > > --- Ben Greear <[EMAIL PROTECTED]> > wrote: > > > > > >>Danial Thom wrote: > >> > >> > >>>I think the concensus is that 2.6 has made > >> > >>trade > >> > >>>offs that lower raw throughput, which is > what > >> > >>a > >> > >>>networking device needs. So as a router or > >>>network appliance, 2.6 seems less suitable. > A > >> > >>raw > >> > >>>bridging test on a 2.0Ghz operton system: > >>> > >>>FreeBSD 4.9: Drops no packets at 900K pps > >>>Linux 2.4.24: Starts dropping packets at > 350K > >> > >>pps > >> > >>>Linux 2.6.12: Starts dropping packets at > 100K > >> > >>pps > >> > >>I ran some quick tests using kernel 2.6.11, > 1ms > >>tick (HZ=1000), SMP kernel. > >>Hardware is P-IV 3.0Ghz + HT on a new > >>SuperMicro motherboard with 64/133Mhz > >>PCI-X bus. NIC is dual Intel pro/1000. > Kernel > >>is close to stock 2.6.11. > >> > >>I used brctl to create a bridge with the two > >>GigE adapters in it and > >>used pktgen to stream traffic through it > >>(250kpps in one direction, 1kpps in > >>the other.) > >> > >>I see a reasonable amount of drops at 250kpps > >>(60 byte packets): > >>about 60,000,000 packets received, 20,700 > >>dropped. > > I get slightly worse performance on this system > when running RH9 > with kernel 2.4.29 (my hacks, HZ=1000, SMP). > Tried increasing > e1000 descriptors to 2048 tx and rx, but that > didn't help, or at least > not much. > > Will try some other tunings, but I doubt it > will affect performance > enough to come close to the discrepency that > you show between 2.4 > and 2.6 kernels... > > I tried copying a 500MB CDROM to HD on my RH9 > system, and only 6kpps > of the 250kpps get through the bridge...btw. The tests I reported where on UP systems. Perhaps the default settings are better for this in 2.4, since that is what I used, and you used your hacks for both. Are you getting drops or overruns (or both)? I would assume drops is a decision to drop rather than an overrun which is a ring overrun. Overruns would imply more about performance than tuning, I'd think. I wouldn't think that HT would be appropriate for this sort of setup...? You're using a dual PCI-X NIC rather than the onboard ports? Supermicro runs their onboard controllers at 32bit/33mhz for some mindless reason. Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: --- Ben Greear <[EMAIL PROTECTED]> wrote: Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. I used brctl to create a bridge with the two GigE adapters in it and used pktgen to stream traffic through it (250kpps in one direction, 1kpps in the other.) I see a reasonable amount of drops at 250kpps (60 byte packets): about 60,000,000 packets received, 20,700 dropped. I get slightly worse performance on this system when running RH9 with kernel 2.4.29 (my hacks, HZ=1000, SMP). Tried increasing e1000 descriptors to 2048 tx and rx, but that didn't help, or at least not much. Will try some other tunings, but I doubt it will affect performance enough to come close to the discrepency that you show between 2.4 and 2.6 kernels... I tried copying a 500MB CDROM to HD on my RH9 system, and only 6kpps of the 250kpps get through the bridge...btw. Ben -- Ben Greear <[EMAIL PROTECTED]> Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: --- Ben Greear <[EMAIL PROTECTED]> wrote: Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. What GigE adapters did you use? Clearly every driver is going to be different. My experience is that a 3.4Ghz P4 is about the performance of a 2.0Ghz Opteron. I have to try your tuning script tomorrow. Intel pro/1000, as I mentioned. I haven't tried any other NIC that comes close in performance to the e1000. If your test is still set up, try compiling something large while doing the test. The drops go through the roof in my tests. Installing RH9 on the box now to try some tests... Disk access always robs networking, in my experience, so I am not supprised you see bad ntwk performance while compiling. Ben -- Ben Greear <[EMAIL PROTECTED]> Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > > I think the concensus is that 2.6 has made > trade > > offs that lower raw throughput, which is what > a > > networking device needs. So as a router or > > network appliance, 2.6 seems less suitable. A > raw > > bridging test on a 2.0Ghz operton system: > > > > FreeBSD 4.9: Drops no packets at 900K pps > > Linux 2.4.24: Starts dropping packets at 350K > pps > > Linux 2.6.12: Starts dropping packets at 100K > pps > > I ran some quick tests using kernel 2.6.11, 1ms > tick (HZ=1000), SMP kernel. > Hardware is P-IV 3.0Ghz + HT on a new > SuperMicro motherboard with 64/133Mhz > PCI-X bus. NIC is dual Intel pro/1000. Kernel > is close to stock 2.6.11. > > I used brctl to create a bridge with the two > GigE adapters in it and > used pktgen to stream traffic through it > (250kpps in one direction, 1kpps in > the other.) > > I see a reasonable amount of drops at 250kpps > (60 byte packets): > about 60,000,000 packets received, 20,700 > dropped. > > Interestingly, the system is about 60% idle > according to top, > and still dropping pkts, so it would seem that > the system could > be better utilized! > > Ben > What GigE adapters did you use? Clearly every driver is going to be different. My experience is that a 3.4Ghz P4 is about the performance of a 2.0Ghz Opteron. I have to try your tuning script tomorrow. If your test is still set up, try compiling something large while doing the test. The drops go through the roof in my tests. Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. I used brctl to create a bridge with the two GigE adapters in it and used pktgen to stream traffic through it (250kpps in one direction, 1kpps in the other.) I see a reasonable amount of drops at 250kpps (60 byte packets): about 60,000,000 packets received, 20,700 dropped. Interestingly, the system is about 60% idle according to top, and still dropping pkts, so it would seem that the system could be better utilized! Ben What GigE adapters did you use? Clearly every driver is going to be different. My experience is that a 3.4Ghz P4 is about the performance of a 2.0Ghz Opteron. I have to try your tuning script tomorrow. If your test is still set up, try compiling something large while doing the test. The drops go through the roof in my tests. Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: --- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. What GigE adapters did you use? Clearly every driver is going to be different. My experience is that a 3.4Ghz P4 is about the performance of a 2.0Ghz Opteron. I have to try your tuning script tomorrow. Intel pro/1000, as I mentioned. I haven't tried any other NIC that comes close in performance to the e1000. If your test is still set up, try compiling something large while doing the test. The drops go through the roof in my tests. Installing RH9 on the box now to try some tests... Disk access always robs networking, in my experience, so I am not supprised you see bad ntwk performance while compiling. Ben -- Ben Greear [EMAIL PROTECTED] Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: --- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. I used brctl to create a bridge with the two GigE adapters in it and used pktgen to stream traffic through it (250kpps in one direction, 1kpps in the other.) I see a reasonable amount of drops at 250kpps (60 byte packets): about 60,000,000 packets received, 20,700 dropped. I get slightly worse performance on this system when running RH9 with kernel 2.4.29 (my hacks, HZ=1000, SMP). Tried increasing e1000 descriptors to 2048 tx and rx, but that didn't help, or at least not much. Will try some other tunings, but I doubt it will affect performance enough to come close to the discrepency that you show between 2.4 and 2.6 kernels... I tried copying a 500MB CDROM to HD on my RH9 system, and only 6kpps of the 250kpps get through the bridge...btw. Ben -- Ben Greear [EMAIL PROTECTED] Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: --- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. I used brctl to create a bridge with the two GigE adapters in it and used pktgen to stream traffic through it (250kpps in one direction, 1kpps in the other.) I see a reasonable amount of drops at 250kpps (60 byte packets): about 60,000,000 packets received, 20,700 dropped. I get slightly worse performance on this system when running RH9 with kernel 2.4.29 (my hacks, HZ=1000, SMP). Tried increasing e1000 descriptors to 2048 tx and rx, but that didn't help, or at least not much. Will try some other tunings, but I doubt it will affect performance enough to come close to the discrepency that you show between 2.4 and 2.6 kernels... I tried copying a 500MB CDROM to HD on my RH9 system, and only 6kpps of the 250kpps get through the bridge...btw. The tests I reported where on UP systems. Perhaps the default settings are better for this in 2.4, since that is what I used, and you used your hacks for both. Are you getting drops or overruns (or both)? I would assume drops is a decision to drop rather than an overrun which is a ring overrun. Overruns would imply more about performance than tuning, I'd think. I wouldn't think that HT would be appropriate for this sort of setup...? You're using a dual PCI-X NIC rather than the onboard ports? Supermicro runs their onboard controllers at 32bit/33mhz for some mindless reason. Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: The tests I reported where on UP systems. Perhaps the default settings are better for this in 2.4, since that is what I used, and you used your hacks for both. My modifications to the kernel are unlikely to speed anything up, and probably will slow things down ever so slightly. I can try with a UP kernel, but my machine at least has a single processor. I'm using the SMP kernel to take advantage of HT. Are you getting drops or overruns (or both)? I would assume drops is a decision to drop rather than an overrun which is a ring overrun. Overruns would imply more about performance than tuning, I'd think. I was seeing lots of NIC errors...in fact, it was showing a great many more errors than packets sent to it, so I just ignored them. I increased the TxDescriptors and RxDescriptors and that helped a little. Increasing the transmit queue for the NIC to 2000 also helped a little. I wouldn't think that HT would be appropriate for this sort of setup...? 2.6.11 seems to be faster when running SMP kernel on this system. You're using a dual PCI-X NIC rather than the onboard ports? Supermicro runs their onboard Of course. Never found a motherboard yet with decent built-in NICs. The built-ins on this board are tg3 and they must be on a slow bus, because they cannot go faster than about 700Mbps (using big pkts). I'll benchmark things again when 2.6.13 comes out and try to get some more detailed numbers... Thanks, Ben -- Ben Greear [EMAIL PROTECTED] Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: The tests I reported where on UP systems. Perhaps the default settings are better for this in 2.4, since that is what I used, and you used your hacks for both. My modifications to the kernel are unlikely to speed anything up, and probably will slow things down ever so slightly. I can try with a UP kernel, but my machine at least has a single processor. I'm using the SMP kernel to take advantage of HT. Are you getting drops or overruns (or both)? I would assume drops is a decision to drop rather than an overrun which is a ring overrun. Overruns would imply more about performance than tuning, I'd think. I was seeing lots of NIC errors...in fact, it was showing a great many more errors than packets sent to it, so I just ignored them. I increased the TxDescriptors and RxDescriptors and that helped a little. Increasing the transmit queue for the NIC to 2000 also helped a little. I wouldn't think that HT would be appropriate for this sort of setup...? 2.6.11 seems to be faster when running SMP kernel on this system. HT and SMP are not the same animal, are they? My understanding is that an HT aware scheduler is likely to make things worse most of the time, particularly for systems not running a lot of threads.. You're using a dual PCI-X NIC rather than the onboard ports? Supermicro runs their onboard Of course. Never found a motherboard yet with decent built-in NICs. The built-ins on this board are tg3 and they must be on a slow bus, because they cannot go faster than about 700Mbps (using big pkts). If its the P8SCI or the same design they are on a 1X PCIE thats shared with the PCI-X. Pretty hokey stuff. Its also a low-end controller amongst the broadcom parts. Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: --- Ben Greear [EMAIL PROTECTED] wrote: Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. What GigE adapters did you use? Clearly every driver is going to be different. My experience is that a 3.4Ghz P4 is about the performance of a 2.0Ghz Opteron. I have to try your tuning script tomorrow. Intel pro/1000, as I mentioned. I haven't tried any other NIC that comes close in performance to the e1000. If your test is still set up, try compiling something large while doing the test. The drops go through the roof in my tests. Installing RH9 on the box now to try some tests... Disk access always robs networking, in my experience, so I am not supprised you see bad ntwk performance while compiling. Ben It would be useful if there were some way to find out what is getting robbed. If networking has priority, then what is keeping it from getting back to processing the rx interrupts? Ah, the e1000 has built-in interrupt moderation. I can't get into my lab until tomorrow afternoon, but if you get a chance try setting ITR in e1000_main.c to something larger, like 20K. and see if it makes a difference. At 200K pps that would cause an interrupt every 10 packets, which may allow the routine to grab back the cpu more often. Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. I used brctl to create a bridge with the two GigE adapters in it and used pktgen to stream traffic through it (250kpps in one direction, 1kpps in the other.) I see a reasonable amount of drops at 250kpps (60 byte packets): about 60,000,000 packets received, 20,700 dropped. Interestingly, the system is about 60% idle according to top, and still dropping pkts, so it would seem that the system could be better utilized! Ben -- Ben Greear <[EMAIL PROTECTED]> Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl <[EMAIL PROTECTED]> wrote: > On 8/24/05, Danial Thom <[EMAIL PROTECTED]> > wrote: > > --- Patrick McHardy <[EMAIL PROTECTED]> wrote: > > > > > Danial Thom wrote: > > > > I think part of the problem is the > continued > > > > misuse of the word "latency". Latency, in > > > > language terms, means "unexplained > delay". > > > Its > > > > wrong here because for one, its > explainable. > > > But > > > > it also depends on your perspective. The > > > > "latency" is increased for kernel tasks, > > > while it > > > > may be reduced for something that is > getting > > > the > > > > benefit of preempting the kernel. So you > > > really > > > > can't say "the price of reduced latency > is > > > lower > > > > throughput", because thats simply > backwards. > > > > You've increased the kernel tasks latency > by > > > > allowing it to be pre-empted. Reduced > latency > > > > implies higher efficiency. All you've > done > > > here > > > > is shift the latency from one task to > > > another, so > > > > there is no reduction overall, in fact > there > > > is > > > > probably a marginal increase due to the > > > overhead > > > > of pre-emption vs doing nothing. > > > > > > If instead of complaining you would provide > the > > > information > > > I've asked for two days ago someone might > > > actually be able > > > to help you. > > > > Because gaining an understanding of how the > > settings work is better than having 30 guys > > telling me to tune something that is only > going > > to make a marginal difference. I didn't ask > you > > to tell me what was wrong with my setup, only > > whether its expected that 2.6 would be less > > useful in a UP setup than 2.4, which I think > > you've answered. > > > > I hope you're implying that the answer is; no, > it's not expected that > 2.6 is less useful in a UP setup than 2.4 :-) I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps Now the 2.6.12 keyboard is always nice and snappy, but thats not what I need. I can't have a box drop traffic if some admin decides to recompile some application. Linux is fine on low-medium speed networks, but at a certain capacity, depending on the specs of the machine of course, linux drops packets. If I do a "make install" in BSD when on a busy network, it takes a long time, but it doesn't drop packets. Linux compiles a lot faster, but it drops buckets of packets. Its just not the priority thats needed for a networking device. Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl <[EMAIL PROTECTED]> wrote: > > > >>If you have preemtion enabled you could > > > disable > > > >>it. Low latency comes > > > >>at the cost of decreased throughput - > can't > > > >>have both. Also try using > > > >>a HZ of 100 if you are currently using > 1000, > > > >>that should also improve > > > >>throughput a little at the cost of > slightly > > > >>higher latencies. > > > >> > > > >>I doubt that it'll do any huge > difference, > > > but > > > >>if it does, then that > > > >>would probably be valuable info. > > > >> > > > >> > > > >> > > > >Ok, well you'll have to explain this one: > > > > > > > >"Low latency comes at the cost of > decreased > > > >throughput - can't have both" > > > > > > > > > > > Configuring "preempt" gives lower latency, > > > because then > > > almost anything can be interrupted > (preempted). > > > You can then > > > get very quick responses to some things, > i.e. > > > interrupts and such. > > > > I think part of the problem is the continued > > misuse of the word "latency". Latency, in > > language terms, means "unexplained delay". > Its > > wrong here because for one, its explainable. > But > > it also depends on your perspective. The > > "latency" is increased for kernel tasks, > while it > > may be reduced for something that is getting > the > > benefit of preempting the kernel. So you > really > > can't say "the price of reduced latency is > lower > > throughput", because thats simply backwards. > > You've increased the kernel tasks latency by > > allowing it to be pre-empted. Reduced latency > > implies higher efficiency. All you've done > here > > is shift the latency from one task to > another, so > > there is no reduction overall, in fact there > is > > probably a marginal increase due to the > overhead > > of pre-emption vs doing nothing. > > > > > No. That's simply not correct. > > 1. Preempt adds overhead in tracking when it's > safe to preempt and > when it is not and overhead for checking if > something needs to preempt > something else at various points. > This means more bookkeeping, which means more > work done by the kernel > and less work done on behalf of user processes > == lower overall > throughput / bandwith [1]. > > 2. Every time a process is preempted work has > to be done to switch to > the new process, caches will be flushed/cold, > stuff may have to paged > in/out, etc. > This means more copying of process related > data and more cache misses > == lower overall throughput. > > But, generally the overhead associated with > preemption is low, which > is also why I said in a previous email that on > many systems you won't > even notice it. > > > But, this whole thing has gotten sidetracked > into a discussion about; > is preempt good or bad, what does the word > "latency" means exactely > (and I don't agree with your definition) [2]. > > When I originally suggested to you to try a > non-preempt kernel (if > your current one is even using preempt, which > you haven't told us) I > was simply trying to gather a datapoint. > Since preemption is a major thing that has > changed from 2.4 to 2.6 and > since in some cases it *does* impact > performance I thought it would be > a valid thing to eliminate it from the list of > things that could be > causing your problem. I also believe I said > that I didn't think it > would make a big difference, but that it > /might/ and we might as well > see what difference it does make (if any). > > So, if instead of arguing the exact meaning of > a word, making > statements about you /assuming/ that HZ or > PREEMPT won't affect > things, you had instead just *tested* it then > we would have saved 2 > days of arguing and getting nowhere and could > instead have gotten that > little detail out of the way and then moved on > to the next thing to > check. > > When I made the suggestion I had hoped for a > reply along the lines of this : > > I just tried a HZ=1000 + PREEMPT vs a HZ=100 + > NO-PREEMPT kernel, and > the NO-PREEMPT kernel achieves x% higher > throughput. We seem to have a > problem with PREEMPT. > > or > > Sorry Jesper, you were wrong, booting a kernel > with HZ=100 and no > PREEMPT makes absolutely no difference to my > network throughput, so > the bug must lie elsewhere. > > > If you'd done that (and what would it have > taken? all of 30min perhaps > to build two kernels and test them), then > everyone would have had a > valid piece of information and could have gone > off looking for a > preempt related bug or put preempt out of their > minds and gone hunting > for the bug somewhere else. > That was my intention with the original > suggestion to test a no > preempt kernel, not to start arguing the > merrits of preemption in > general or the exact meaning of the word > "latency". I did do it, and there was no difference. It seemed that no one thought it was going to make a 3-fold difference, so it wasn't worthy of reporting. When someone suggests "try this, although I
Re: 2.6.12 Performance problems
--- Patrick McHardy <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > None of this is helpful, but since no one has > > been able to tell me how to tune it to > provide > > absolute priority to the network stack I'll > > assume it can't be done. > > The network stack already has priority over > user processes, > except when executed in process context, so > preemption has > no direct impact on briding or routing > performance. > > The reason why noone answered your question is > because you > don't ask but claim or assume. > No, its because guys like you snip out content when they reply, and/or only read the parts of messages that you want to read, so when other people enter a thread, they miss the questions that were asked long ago. Quoting my post on Aug 22: "All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft, floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the "its ok to drop packets when x occurs" algorithm to be "its never ok to drop packets unless x occurs" (such as a queue depth)? Is it possible?" Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On 8/24/05, Danial Thom <[EMAIL PROTECTED]> wrote: > --- Patrick McHardy <[EMAIL PROTECTED]> wrote: > > > Danial Thom wrote: > > > I think part of the problem is the continued > > > misuse of the word "latency". Latency, in > > > language terms, means "unexplained delay". > > Its > > > wrong here because for one, its explainable. > > But > > > it also depends on your perspective. The > > > "latency" is increased for kernel tasks, > > while it > > > may be reduced for something that is getting > > the > > > benefit of preempting the kernel. So you > > really > > > can't say "the price of reduced latency is > > lower > > > throughput", because thats simply backwards. > > > You've increased the kernel tasks latency by > > > allowing it to be pre-empted. Reduced latency > > > implies higher efficiency. All you've done > > here > > > is shift the latency from one task to > > another, so > > > there is no reduction overall, in fact there > > is > > > probably a marginal increase due to the > > overhead > > > of pre-emption vs doing nothing. > > > > If instead of complaining you would provide the > > information > > I've asked for two days ago someone might > > actually be able > > to help you. > > Because gaining an understanding of how the > settings work is better than having 30 guys > telling me to tune something that is only going > to make a marginal difference. I didn't ask you > to tell me what was wrong with my setup, only > whether its expected that 2.6 would be less > useful in a UP setup than 2.4, which I think > you've answered. > I hope you're implying that the answer is; no, it's not expected that 2.6 is less useful in a UP setup than 2.4 :-) -- Jesper Juhl <[EMAIL PROTECTED]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Sven-Thorsten Dietrich <[EMAIL PROTECTED]> wrote: > On Tue, 2005-08-23 at 13:10 -0700, Danial Thom > wrote: > > > > None of this is helpful, but since no one has > > been able to tell me how to tune it to > provide > > absolute priority to the network stack I'll > > assume it can't be done. > > History has proven that camp wrong almost 100% > of the time. > > You were told to turn off kernel preemption. > > A diligent comparison requires that, since 2.4 > does not support kernel > preemption, and a fair comparison requires > holding all other things > constant. > > In addition, there were several IP-level > features mentioned in emails, > that have been added to 2.6. > > You need to make sure those are all off by > default, to keep your > comparison relevant. > > All the answers are before you, review those > emails, turn all that stuff > off and retest. I had tried turning off pre-emption, with little difference. However linux had the same properties before there was such a setting (of liking to drop packets here and there for no apparent reason under heavy load), so I didn't expect it to make a huge difference. It seems typing on the keyboard has the same effect with or without pre-emption enabled. IP is not involved in this test, so no IP stack issues should be relevent. Danial __ Yahoo! Mail for Mobile Take Yahoo! Mail with you! Check email on your mobile phone. http://mobile.yahoo.com/learn/mail - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Patrick McHardy <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > I think part of the problem is the continued > > misuse of the word "latency". Latency, in > > language terms, means "unexplained delay". > Its > > wrong here because for one, its explainable. > But > > it also depends on your perspective. The > > "latency" is increased for kernel tasks, > while it > > may be reduced for something that is getting > the > > benefit of preempting the kernel. So you > really > > can't say "the price of reduced latency is > lower > > throughput", because thats simply backwards. > > You've increased the kernel tasks latency by > > allowing it to be pre-empted. Reduced latency > > implies higher efficiency. All you've done > here > > is shift the latency from one task to > another, so > > there is no reduction overall, in fact there > is > > probably a marginal increase due to the > overhead > > of pre-emption vs doing nothing. > > If instead of complaining you would provide the > information > I've asked for two days ago someone might > actually be able > to help you. Because gaining an understanding of how the settings work is better than having 30 guys telling me to tune something that is only going to make a marginal difference. I didn't ask you to tell me what was wrong with my setup, only whether its expected that 2.6 would be less useful in a UP setup than 2.4, which I think you've answered. D __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Patrick McHardy [EMAIL PROTECTED] wrote: Danial Thom wrote: I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. Its wrong here because for one, its explainable. But it also depends on your perspective. The latency is increased for kernel tasks, while it may be reduced for something that is getting the benefit of preempting the kernel. So you really can't say the price of reduced latency is lower throughput, because thats simply backwards. You've increased the kernel tasks latency by allowing it to be pre-empted. Reduced latency implies higher efficiency. All you've done here is shift the latency from one task to another, so there is no reduction overall, in fact there is probably a marginal increase due to the overhead of pre-emption vs doing nothing. If instead of complaining you would provide the information I've asked for two days ago someone might actually be able to help you. Because gaining an understanding of how the settings work is better than having 30 guys telling me to tune something that is only going to make a marginal difference. I didn't ask you to tell me what was wrong with my setup, only whether its expected that 2.6 would be less useful in a UP setup than 2.4, which I think you've answered. D __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Sven-Thorsten Dietrich [EMAIL PROTECTED] wrote: On Tue, 2005-08-23 at 13:10 -0700, Danial Thom wrote: None of this is helpful, but since no one has been able to tell me how to tune it to provide absolute priority to the network stack I'll assume it can't be done. History has proven that camp wrong almost 100% of the time. You were told to turn off kernel preemption. A diligent comparison requires that, since 2.4 does not support kernel preemption, and a fair comparison requires holding all other things constant. In addition, there were several IP-level features mentioned in emails, that have been added to 2.6. You need to make sure those are all off by default, to keep your comparison relevant. All the answers are before you, review those emails, turn all that stuff off and retest. I had tried turning off pre-emption, with little difference. However linux had the same properties before there was such a setting (of liking to drop packets here and there for no apparent reason under heavy load), so I didn't expect it to make a huge difference. It seems typing on the keyboard has the same effect with or without pre-emption enabled. IP is not involved in this test, so no IP stack issues should be relevent. Danial __ Yahoo! Mail for Mobile Take Yahoo! Mail with you! Check email on your mobile phone. http://mobile.yahoo.com/learn/mail - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On 8/24/05, Danial Thom [EMAIL PROTECTED] wrote: --- Patrick McHardy [EMAIL PROTECTED] wrote: Danial Thom wrote: I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. Its wrong here because for one, its explainable. But it also depends on your perspective. The latency is increased for kernel tasks, while it may be reduced for something that is getting the benefit of preempting the kernel. So you really can't say the price of reduced latency is lower throughput, because thats simply backwards. You've increased the kernel tasks latency by allowing it to be pre-empted. Reduced latency implies higher efficiency. All you've done here is shift the latency from one task to another, so there is no reduction overall, in fact there is probably a marginal increase due to the overhead of pre-emption vs doing nothing. If instead of complaining you would provide the information I've asked for two days ago someone might actually be able to help you. Because gaining an understanding of how the settings work is better than having 30 guys telling me to tune something that is only going to make a marginal difference. I didn't ask you to tell me what was wrong with my setup, only whether its expected that 2.6 would be less useful in a UP setup than 2.4, which I think you've answered. I hope you're implying that the answer is; no, it's not expected that 2.6 is less useful in a UP setup than 2.4 :-) -- Jesper Juhl [EMAIL PROTECTED] Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Patrick McHardy [EMAIL PROTECTED] wrote: Danial Thom wrote: None of this is helpful, but since no one has been able to tell me how to tune it to provide absolute priority to the network stack I'll assume it can't be done. The network stack already has priority over user processes, except when executed in process context, so preemption has no direct impact on briding or routing performance. The reason why noone answered your question is because you don't ask but claim or assume. No, its because guys like you snip out content when they reply, and/or only read the parts of messages that you want to read, so when other people enter a thread, they miss the questions that were asked long ago. Quoting my post on Aug 22: All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft, floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the its ok to drop packets when x occurs algorithm to be its never ok to drop packets unless x occurs (such as a queue depth)? Is it possible? Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl [EMAIL PROTECTED] wrote: If you have preemtion enabled you could disable it. Low latency comes at the cost of decreased throughput - can't have both. Also try using a HZ of 100 if you are currently using 1000, that should also improve throughput a little at the cost of slightly higher latencies. I doubt that it'll do any huge difference, but if it does, then that would probably be valuable info. Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Configuring preempt gives lower latency, because then almost anything can be interrupted (preempted). You can then get very quick responses to some things, i.e. interrupts and such. I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. Its wrong here because for one, its explainable. But it also depends on your perspective. The latency is increased for kernel tasks, while it may be reduced for something that is getting the benefit of preempting the kernel. So you really can't say the price of reduced latency is lower throughput, because thats simply backwards. You've increased the kernel tasks latency by allowing it to be pre-empted. Reduced latency implies higher efficiency. All you've done here is shift the latency from one task to another, so there is no reduction overall, in fact there is probably a marginal increase due to the overhead of pre-emption vs doing nothing. No. That's simply not correct. 1. Preempt adds overhead in tracking when it's safe to preempt and when it is not and overhead for checking if something needs to preempt something else at various points. This means more bookkeeping, which means more work done by the kernel and less work done on behalf of user processes == lower overall throughput / bandwith [1]. 2. Every time a process is preempted work has to be done to switch to the new process, caches will be flushed/cold, stuff may have to paged in/out, etc. This means more copying of process related data and more cache misses == lower overall throughput. But, generally the overhead associated with preemption is low, which is also why I said in a previous email that on many systems you won't even notice it. But, this whole thing has gotten sidetracked into a discussion about; is preempt good or bad, what does the word latency means exactely (and I don't agree with your definition) [2]. When I originally suggested to you to try a non-preempt kernel (if your current one is even using preempt, which you haven't told us) I was simply trying to gather a datapoint. Since preemption is a major thing that has changed from 2.4 to 2.6 and since in some cases it *does* impact performance I thought it would be a valid thing to eliminate it from the list of things that could be causing your problem. I also believe I said that I didn't think it would make a big difference, but that it /might/ and we might as well see what difference it does make (if any). So, if instead of arguing the exact meaning of a word, making statements about you /assuming/ that HZ or PREEMPT won't affect things, you had instead just *tested* it then we would have saved 2 days of arguing and getting nowhere and could instead have gotten that little detail out of the way and then moved on to the next thing to check. When I made the suggestion I had hoped for a reply along the lines of this : I just tried a HZ=1000 + PREEMPT vs a HZ=100 + NO-PREEMPT kernel, and the NO-PREEMPT kernel achieves x% higher throughput. We seem to have a problem with PREEMPT. or Sorry Jesper, you were wrong, booting a kernel with HZ=100 and no PREEMPT makes absolutely no difference to my network throughput, so the bug must lie elsewhere. If you'd done that (and what would it have taken? all of 30min perhaps to build two kernels and test them), then everyone would have had a valid piece of information and could have gone off looking for a preempt related bug or put preempt out of their minds and gone hunting for the bug somewhere else. That was my intention with the original suggestion to test a no preempt kernel, not to start arguing the merrits of preemption in general or the exact meaning of the word latency. I did do it, and there was no difference. It seemed that no one thought it was going to make a 3-fold difference, so it wasn't worthy of reporting. When someone suggests try this, although I don't think its going to make much of a difference, I usually don't get excited about spending time trying it out. But I did, and it made no difference. So I'm trying to get a handle on the philosophy of how things work, so I tune my tests without having to ask you, so I can get my boss off my
Re: 2.6.12 Performance problems
--- Jesper Juhl [EMAIL PROTECTED] wrote: On 8/24/05, Danial Thom [EMAIL PROTECTED] wrote: --- Patrick McHardy [EMAIL PROTECTED] wrote: Danial Thom wrote: I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. Its wrong here because for one, its explainable. But it also depends on your perspective. The latency is increased for kernel tasks, while it may be reduced for something that is getting the benefit of preempting the kernel. So you really can't say the price of reduced latency is lower throughput, because thats simply backwards. You've increased the kernel tasks latency by allowing it to be pre-empted. Reduced latency implies higher efficiency. All you've done here is shift the latency from one task to another, so there is no reduction overall, in fact there is probably a marginal increase due to the overhead of pre-emption vs doing nothing. If instead of complaining you would provide the information I've asked for two days ago someone might actually be able to help you. Because gaining an understanding of how the settings work is better than having 30 guys telling me to tune something that is only going to make a marginal difference. I didn't ask you to tell me what was wrong with my setup, only whether its expected that 2.6 would be less useful in a UP setup than 2.4, which I think you've answered. I hope you're implying that the answer is; no, it's not expected that 2.6 is less useful in a UP setup than 2.4 :-) I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps Now the 2.6.12 keyboard is always nice and snappy, but thats not what I need. I can't have a box drop traffic if some admin decides to recompile some application. Linux is fine on low-medium speed networks, but at a certain capacity, depending on the specs of the machine of course, linux drops packets. If I do a make install in BSD when on a busy network, it takes a long time, but it doesn't drop packets. Linux compiles a lot faster, but it drops buckets of packets. Its just not the priority thats needed for a networking device. Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: I think the concensus is that 2.6 has made trade offs that lower raw throughput, which is what a networking device needs. So as a router or network appliance, 2.6 seems less suitable. A raw bridging test on a 2.0Ghz operton system: FreeBSD 4.9: Drops no packets at 900K pps Linux 2.4.24: Starts dropping packets at 350K pps Linux 2.6.12: Starts dropping packets at 100K pps I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel. Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz PCI-X bus. NIC is dual Intel pro/1000. Kernel is close to stock 2.6.11. I used brctl to create a bridge with the two GigE adapters in it and used pktgen to stream traffic through it (250kpps in one direction, 1kpps in the other.) I see a reasonable amount of drops at 250kpps (60 byte packets): about 60,000,000 packets received, 20,700 dropped. Interestingly, the system is about 60% idle according to top, and still dropping pkts, so it would seem that the system could be better utilized! Ben -- Ben Greear [EMAIL PROTECTED] Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Patrick McHardy wrote: Danial Thom wrote: None of this is helpful, but since no one has been able to tell me how to tune it to provide absolute priority to the network stack I'll assume it can't be done. The network stack already has priority over user processes, except when executed in process context, so preemption has no direct impact on briding or routing performance. Well, NAPI puts a lot more of the packet processing in process context, including the code that would do routing I believe. To give networking more priority, you can re-nice the ksoftirqd process(es) to a high-priority level like -18 or so. You can also reserve more memory for the networking stack and increase the size of the permitted buffers. I am also interested to hear how your system responds to compiling w/out PREEMPT. Ben -- Ben Greear <[EMAIL PROTECTED]> Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
> > >>If you have preemtion enabled you could > > disable > > >>it. Low latency comes > > >>at the cost of decreased throughput - can't > > >>have both. Also try using > > >>a HZ of 100 if you are currently using 1000, > > >>that should also improve > > >>throughput a little at the cost of slightly > > >>higher latencies. > > >> > > >>I doubt that it'll do any huge difference, > > but > > >>if it does, then that > > >>would probably be valuable info. > > >> > > >> > > >> > > >Ok, well you'll have to explain this one: > > > > > >"Low latency comes at the cost of decreased > > >throughput - can't have both" > > > > > > > > Configuring "preempt" gives lower latency, > > because then > > almost anything can be interrupted (preempted). > > You can then > > get very quick responses to some things, i.e. > > interrupts and such. > > I think part of the problem is the continued > misuse of the word "latency". Latency, in > language terms, means "unexplained delay". Its > wrong here because for one, its explainable. But > it also depends on your perspective. The > "latency" is increased for kernel tasks, while it > may be reduced for something that is getting the > benefit of preempting the kernel. So you really > can't say "the price of reduced latency is lower > throughput", because thats simply backwards. > You've increased the kernel tasks latency by > allowing it to be pre-empted. Reduced latency > implies higher efficiency. All you've done here > is shift the latency from one task to another, so > there is no reduction overall, in fact there is > probably a marginal increase due to the overhead > of pre-emption vs doing nothing. > No. That's simply not correct. 1. Preempt adds overhead in tracking when it's safe to preempt and when it is not and overhead for checking if something needs to preempt something else at various points. This means more bookkeeping, which means more work done by the kernel and less work done on behalf of user processes == lower overall throughput / bandwith [1]. 2. Every time a process is preempted work has to be done to switch to the new process, caches will be flushed/cold, stuff may have to paged in/out, etc. This means more copying of process related data and more cache misses == lower overall throughput. But, generally the overhead associated with preemption is low, which is also why I said in a previous email that on many systems you won't even notice it. But, this whole thing has gotten sidetracked into a discussion about; is preempt good or bad, what does the word "latency" means exactely (and I don't agree with your definition) [2]. When I originally suggested to you to try a non-preempt kernel (if your current one is even using preempt, which you haven't told us) I was simply trying to gather a datapoint. Since preemption is a major thing that has changed from 2.4 to 2.6 and since in some cases it *does* impact performance I thought it would be a valid thing to eliminate it from the list of things that could be causing your problem. I also believe I said that I didn't think it would make a big difference, but that it /might/ and we might as well see what difference it does make (if any). So, if instead of arguing the exact meaning of a word, making statements about you /assuming/ that HZ or PREEMPT won't affect things, you had instead just *tested* it then we would have saved 2 days of arguing and getting nowhere and could instead have gotten that little detail out of the way and then moved on to the next thing to check. When I made the suggestion I had hoped for a reply along the lines of this : I just tried a HZ=1000 + PREEMPT vs a HZ=100 + NO-PREEMPT kernel, and the NO-PREEMPT kernel achieves x% higher throughput. We seem to have a problem with PREEMPT. or Sorry Jesper, you were wrong, booting a kernel with HZ=100 and no PREEMPT makes absolutely no difference to my network throughput, so the bug must lie elsewhere. If you'd done that (and what would it have taken? all of 30min perhaps to build two kernels and test them), then everyone would have had a valid piece of information and could have gone off looking for a preempt related bug or put preempt out of their minds and gone hunting for the bug somewhere else. That was my intention with the original suggestion to test a no preempt kernel, not to start arguing the merrits of preemption in general or the exact meaning of the word "latency". Finally, here's a little article you might like to read : http://www.linuxdevices.com/articles/AT5152980814.html [1] http://en.wikipedia.org/wiki/Comparison_of_latency_and_bandwidth [2] http://en.wikipedia.org/wiki/Latency_%28engineering%29 -- Jesper Juhl <[EMAIL PROTECTED]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at
Re: 2.6.12 Performance problems
Danial Thom wrote: > None of this is helpful, but since no one has > been able to tell me how to tune it to provide > absolute priority to the network stack I'll > assume it can't be done. The network stack already has priority over user processes, except when executed in process context, so preemption has no direct impact on briding or routing performance. The reason why noone answered your question is because you don't ask but claim or assume. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Tue, 2005-08-23 at 13:10 -0700, Danial Thom wrote: > > None of this is helpful, but since no one has > been able to tell me how to tune it to provide > absolute priority to the network stack I'll > assume it can't be done. History has proven that camp wrong almost 100% of the time. You were told to turn off kernel preemption. A diligent comparison requires that, since 2.4 does not support kernel preemption, and a fair comparison requires holding all other things constant. In addition, there were several IP-level features mentioned in emails, that have been added to 2.6. You need to make sure those are all off by default, to keep your comparison relevant. All the answers are before you, review those emails, turn all that stuff off and retest. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Sven-Thorsten Dietrich <[EMAIL PROTECTED]> wrote: > On Tue, 2005-08-23 at 10:10 -0700, Danial Thom > wrote: > > > > > > >Ok, well you'll have to explain this one: > > > > > > > >"Low latency comes at the cost of > decreased > > > >throughput - can't have both" > > > > > > > > > > > Configuring "preempt" gives lower latency, > > > because then > > > almost anything can be interrupted > (preempted). > > > You can then > > > get very quick responses to some things, > i.e. > > > interrupts and such. > > > > I think part of the problem is the continued > > misuse of the word "latency". Latency, in > > language terms, means "unexplained delay". > > latency > > n > 1: (computer science) the time it takes for a > specific block of data on > a data track to rotate around to the read/write > head [syn: rotational > latency] > 2: the time that elapses between a stimulus and > the response to it [syn: > reaction time, response time, latent period] > 3: the state of being not yet evident or active > > No apparent references to "unexplained" in > association with the word > latency. Teaching English is not my thing, but latent means "dormant", which means doing nothing. So the time it takes to perform a task is not latency. Its the time it really takes compared to the time it ought to take if there was no overhead. If you have a perfect implementation then you have no latency. Your definition implies that there is no way to have zero latency, which is just wrong. I've seen computer dictionaries that define latency as the time it takes to get a response. That would mean a network switch's latency is different with different sized packets, which is just plain stupid. None of this is helpful, but since no one has been able to tell me how to tune it to provide absolute priority to the network stack I'll assume it can't be done. DT __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Tue, 2005-08-23 at 10:10 -0700, Danial Thom wrote: > > > >Ok, well you'll have to explain this one: > > > > > >"Low latency comes at the cost of decreased > > >throughput - can't have both" > > > > > > > > Configuring "preempt" gives lower latency, > > because then > > almost anything can be interrupted (preempted). > > You can then > > get very quick responses to some things, i.e. > > interrupts and such. > > I think part of the problem is the continued > misuse of the word "latency". Latency, in > language terms, means "unexplained delay". latency n 1: (computer science) the time it takes for a specific block of data on a data track to rotate around to the read/write head [syn: rotational latency] 2: the time that elapses between a stimulus and the response to it [syn: reaction time, response time, latent period] 3: the state of being not yet evident or active No apparent references to "unexplained" in association with the word latency. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: > I think part of the problem is the continued > misuse of the word "latency". Latency, in > language terms, means "unexplained delay". Its > wrong here because for one, its explainable. But > it also depends on your perspective. The > "latency" is increased for kernel tasks, while it > may be reduced for something that is getting the > benefit of preempting the kernel. So you really > can't say "the price of reduced latency is lower > throughput", because thats simply backwards. > You've increased the kernel tasks latency by > allowing it to be pre-empted. Reduced latency > implies higher efficiency. All you've done here > is shift the latency from one task to another, so > there is no reduction overall, in fact there is > probably a marginal increase due to the overhead > of pre-emption vs doing nothing. If instead of complaining you would provide the information I've asked for two days ago someone might actually be able to help you. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Helge Hafting <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > >--- Jesper Juhl <[EMAIL PROTECTED]> wrote: > > > > > > > >>On 8/21/05, Danial Thom > <[EMAIL PROTECTED]> > >>wrote: > >> > >> > >>>I just started fiddling with 2.6.12, and > >>> > >>> > >>there > >> > >> > >>>seems to be a big drop-off in performance > >>> > >>> > >>from > >> > >> > >>>2.4.x in terms of networking on a > >>> > >>> > >>uniprocessor > >> > >> > >>>system. Just bridging packets through the > >>>machine, 2.6.12 starts dropping packets at > >>>~100Kpps, whereas 2.4.x doesn't start > >>> > >>> > >>dropping > >> > >> > >>>until over 350Kpps on the same hardware > >>> > >>> > >>(2.0Ghz > >> > >> > >>>Opteron with e1000 driver). This is pitiful > >>>prformance for this hardware. I've > >>>increased the rx ring in the e1000 driver to > >>> > >>> > >>512 > >> > >> > >>>with little change (interrupt moderation is > >>> > >>> > >>set > >> > >> > >>>to 8000 Ints/second). Has "tuning" for MP > >>>destroyed UP performance altogether, or is > >>> > >>> > >>there > >> > >> > >>>some tuning parameter that could make a > >>> > >>> > >>4-fold > >> > >> > >>>difference? All debugging is off and there > >>> > >>> > >>are > >> > >> > >>>no messages on the console or in the error > >>> > >>> > >>logs. > >> > >> > >>>The kernel is the standard kernel.org > dowload > >>>config with SMP turned off and the intel > >>> > >>> > >>ethernet > >> > >> > >>>card drivers as modules without any other > >>>changes, which is exactly the config for my > >>> > >>> > >>2.4 > >> > >> > >>>kernels. > >>> > >>> > >>> > >>If you have preemtion enabled you could > disable > >>it. Low latency comes > >>at the cost of decreased throughput - can't > >>have both. Also try using > >>a HZ of 100 if you are currently using 1000, > >>that should also improve > >>throughput a little at the cost of slightly > >>higher latencies. > >> > >>I doubt that it'll do any huge difference, > but > >>if it does, then that > >>would probably be valuable info. > >> > >> > >> > >Ok, well you'll have to explain this one: > > > >"Low latency comes at the cost of decreased > >throughput - can't have both" > > > > > Configuring "preempt" gives lower latency, > because then > almost anything can be interrupted (preempted). > You can then > get very quick responses to some things, i.e. > interrupts and such. I think part of the problem is the continued misuse of the word "latency". Latency, in language terms, means "unexplained delay". Its wrong here because for one, its explainable. But it also depends on your perspective. The "latency" is increased for kernel tasks, while it may be reduced for something that is getting the benefit of preempting the kernel. So you really can't say "the price of reduced latency is lower throughput", because thats simply backwards. You've increased the kernel tasks latency by allowing it to be pre-empted. Reduced latency implies higher efficiency. All you've done here is shift the latency from one task to another, so there is no reduction overall, in fact there is probably a marginal increase due to the overhead of pre-emption vs doing nothing. DT Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: --- Jesper Juhl <[EMAIL PROTECTED]> wrote: On 8/21/05, Danial Thom <[EMAIL PROTECTED]> wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has "tuning" for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. If you have preemtion enabled you could disable it. Low latency comes at the cost of decreased throughput - can't have both. Also try using a HZ of 100 if you are currently using 1000, that should also improve throughput a little at the cost of slightly higher latencies. I doubt that it'll do any huge difference, but if it does, then that would probably be valuable info. Ok, well you'll have to explain this one: "Low latency comes at the cost of decreased throughput - can't have both" Configuring "preempt" gives lower latency, because then almost anything can be interrupted (preempted). You can then get very quick responses to some things, i.e. interrupts and such. The cost comes, because _something_ was interrupted, something that instead would run to completion first in a kernel made without "preempt". So that other thing, whatever it is, got slower. And the problem is bigger than merely "things happens in a different order". Switching the cpu from one job to another have a big overhead. Particularly, the cpu caches have to be refilled more often, which takes time. Running one big job to completion fills the cache with that job's data _once_. If the job is preempted a couple of times you have to bring it into cache three times instead, and that will cost you, performance wise. This is not _necessarily_ your problem, but trying a 2.6 kernel without preempt and with hz=100 (both things configurable through normal kernel configuration) will clearly show if this is the problem in your case. If you're lucky, this is all you need to get your performance back. If not, then at least it is an important datapoint for those trying to figure it out. Nobody here want 2.6 to have 1/4 of the performance of 2.4! Helge Hafting - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: --- Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Danial Thom [EMAIL PROTECTED] wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has tuning for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. If you have preemtion enabled you could disable it. Low latency comes at the cost of decreased throughput - can't have both. Also try using a HZ of 100 if you are currently using 1000, that should also improve throughput a little at the cost of slightly higher latencies. I doubt that it'll do any huge difference, but if it does, then that would probably be valuable info. Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Configuring preempt gives lower latency, because then almost anything can be interrupted (preempted). You can then get very quick responses to some things, i.e. interrupts and such. The cost comes, because _something_ was interrupted, something that instead would run to completion first in a kernel made without preempt. So that other thing, whatever it is, got slower. And the problem is bigger than merely things happens in a different order. Switching the cpu from one job to another have a big overhead. Particularly, the cpu caches have to be refilled more often, which takes time. Running one big job to completion fills the cache with that job's data _once_. If the job is preempted a couple of times you have to bring it into cache three times instead, and that will cost you, performance wise. This is not _necessarily_ your problem, but trying a 2.6 kernel without preempt and with hz=100 (both things configurable through normal kernel configuration) will clearly show if this is the problem in your case. If you're lucky, this is all you need to get your performance back. If not, then at least it is an important datapoint for those trying to figure it out. Nobody here want 2.6 to have 1/4 of the performance of 2.4! Helge Hafting - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Helge Hafting [EMAIL PROTECTED] wrote: Danial Thom wrote: --- Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Danial Thom [EMAIL PROTECTED] wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has tuning for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. If you have preemtion enabled you could disable it. Low latency comes at the cost of decreased throughput - can't have both. Also try using a HZ of 100 if you are currently using 1000, that should also improve throughput a little at the cost of slightly higher latencies. I doubt that it'll do any huge difference, but if it does, then that would probably be valuable info. Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Configuring preempt gives lower latency, because then almost anything can be interrupted (preempted). You can then get very quick responses to some things, i.e. interrupts and such. I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. Its wrong here because for one, its explainable. But it also depends on your perspective. The latency is increased for kernel tasks, while it may be reduced for something that is getting the benefit of preempting the kernel. So you really can't say the price of reduced latency is lower throughput, because thats simply backwards. You've increased the kernel tasks latency by allowing it to be pre-empted. Reduced latency implies higher efficiency. All you've done here is shift the latency from one task to another, so there is no reduction overall, in fact there is probably a marginal increase due to the overhead of pre-emption vs doing nothing. DT Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. Its wrong here because for one, its explainable. But it also depends on your perspective. The latency is increased for kernel tasks, while it may be reduced for something that is getting the benefit of preempting the kernel. So you really can't say the price of reduced latency is lower throughput, because thats simply backwards. You've increased the kernel tasks latency by allowing it to be pre-empted. Reduced latency implies higher efficiency. All you've done here is shift the latency from one task to another, so there is no reduction overall, in fact there is probably a marginal increase due to the overhead of pre-emption vs doing nothing. If instead of complaining you would provide the information I've asked for two days ago someone might actually be able to help you. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Tue, 2005-08-23 at 10:10 -0700, Danial Thom wrote: Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Configuring preempt gives lower latency, because then almost anything can be interrupted (preempted). You can then get very quick responses to some things, i.e. interrupts and such. I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. latency n 1: (computer science) the time it takes for a specific block of data on a data track to rotate around to the read/write head [syn: rotational latency] 2: the time that elapses between a stimulus and the response to it [syn: reaction time, response time, latent period] 3: the state of being not yet evident or active No apparent references to unexplained in association with the word latency. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Sven-Thorsten Dietrich [EMAIL PROTECTED] wrote: On Tue, 2005-08-23 at 10:10 -0700, Danial Thom wrote: Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Configuring preempt gives lower latency, because then almost anything can be interrupted (preempted). You can then get very quick responses to some things, i.e. interrupts and such. I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. latency n 1: (computer science) the time it takes for a specific block of data on a data track to rotate around to the read/write head [syn: rotational latency] 2: the time that elapses between a stimulus and the response to it [syn: reaction time, response time, latent period] 3: the state of being not yet evident or active No apparent references to unexplained in association with the word latency. Teaching English is not my thing, but latent means dormant, which means doing nothing. So the time it takes to perform a task is not latency. Its the time it really takes compared to the time it ought to take if there was no overhead. If you have a perfect implementation then you have no latency. Your definition implies that there is no way to have zero latency, which is just wrong. I've seen computer dictionaries that define latency as the time it takes to get a response. That would mean a network switch's latency is different with different sized packets, which is just plain stupid. None of this is helpful, but since no one has been able to tell me how to tune it to provide absolute priority to the network stack I'll assume it can't be done. DT __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Tue, 2005-08-23 at 13:10 -0700, Danial Thom wrote: None of this is helpful, but since no one has been able to tell me how to tune it to provide absolute priority to the network stack I'll assume it can't be done. History has proven that camp wrong almost 100% of the time. You were told to turn off kernel preemption. A diligent comparison requires that, since 2.4 does not support kernel preemption, and a fair comparison requires holding all other things constant. In addition, there were several IP-level features mentioned in emails, that have been added to 2.6. You need to make sure those are all off by default, to keep your comparison relevant. All the answers are before you, review those emails, turn all that stuff off and retest. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: None of this is helpful, but since no one has been able to tell me how to tune it to provide absolute priority to the network stack I'll assume it can't be done. The network stack already has priority over user processes, except when executed in process context, so preemption has no direct impact on briding or routing performance. The reason why noone answered your question is because you don't ask but claim or assume. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
If you have preemtion enabled you could disable it. Low latency comes at the cost of decreased throughput - can't have both. Also try using a HZ of 100 if you are currently using 1000, that should also improve throughput a little at the cost of slightly higher latencies. I doubt that it'll do any huge difference, but if it does, then that would probably be valuable info. Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Configuring preempt gives lower latency, because then almost anything can be interrupted (preempted). You can then get very quick responses to some things, i.e. interrupts and such. I think part of the problem is the continued misuse of the word latency. Latency, in language terms, means unexplained delay. Its wrong here because for one, its explainable. But it also depends on your perspective. The latency is increased for kernel tasks, while it may be reduced for something that is getting the benefit of preempting the kernel. So you really can't say the price of reduced latency is lower throughput, because thats simply backwards. You've increased the kernel tasks latency by allowing it to be pre-empted. Reduced latency implies higher efficiency. All you've done here is shift the latency from one task to another, so there is no reduction overall, in fact there is probably a marginal increase due to the overhead of pre-emption vs doing nothing. No. That's simply not correct. 1. Preempt adds overhead in tracking when it's safe to preempt and when it is not and overhead for checking if something needs to preempt something else at various points. This means more bookkeeping, which means more work done by the kernel and less work done on behalf of user processes == lower overall throughput / bandwith [1]. 2. Every time a process is preempted work has to be done to switch to the new process, caches will be flushed/cold, stuff may have to paged in/out, etc. This means more copying of process related data and more cache misses == lower overall throughput. But, generally the overhead associated with preemption is low, which is also why I said in a previous email that on many systems you won't even notice it. But, this whole thing has gotten sidetracked into a discussion about; is preempt good or bad, what does the word latency means exactely (and I don't agree with your definition) [2]. When I originally suggested to you to try a non-preempt kernel (if your current one is even using preempt, which you haven't told us) I was simply trying to gather a datapoint. Since preemption is a major thing that has changed from 2.4 to 2.6 and since in some cases it *does* impact performance I thought it would be a valid thing to eliminate it from the list of things that could be causing your problem. I also believe I said that I didn't think it would make a big difference, but that it /might/ and we might as well see what difference it does make (if any). So, if instead of arguing the exact meaning of a word, making statements about you /assuming/ that HZ or PREEMPT won't affect things, you had instead just *tested* it then we would have saved 2 days of arguing and getting nowhere and could instead have gotten that little detail out of the way and then moved on to the next thing to check. When I made the suggestion I had hoped for a reply along the lines of this : I just tried a HZ=1000 + PREEMPT vs a HZ=100 + NO-PREEMPT kernel, and the NO-PREEMPT kernel achieves x% higher throughput. We seem to have a problem with PREEMPT. or Sorry Jesper, you were wrong, booting a kernel with HZ=100 and no PREEMPT makes absolutely no difference to my network throughput, so the bug must lie elsewhere. If you'd done that (and what would it have taken? all of 30min perhaps to build two kernels and test them), then everyone would have had a valid piece of information and could have gone off looking for a preempt related bug or put preempt out of their minds and gone hunting for the bug somewhere else. That was my intention with the original suggestion to test a no preempt kernel, not to start arguing the merrits of preemption in general or the exact meaning of the word latency. Finally, here's a little article you might like to read : http://www.linuxdevices.com/articles/AT5152980814.html [1] http://en.wikipedia.org/wiki/Comparison_of_latency_and_bandwidth [2] http://en.wikipedia.org/wiki/Latency_%28engineering%29 -- Jesper Juhl [EMAIL PROTECTED] Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Patrick McHardy wrote: Danial Thom wrote: None of this is helpful, but since no one has been able to tell me how to tune it to provide absolute priority to the network stack I'll assume it can't be done. The network stack already has priority over user processes, except when executed in process context, so preemption has no direct impact on briding or routing performance. Well, NAPI puts a lot more of the packet processing in process context, including the code that would do routing I believe. To give networking more priority, you can re-nice the ksoftirqd process(es) to a high-priority level like -18 or so. You can also reserve more memory for the networking stack and increase the size of the permitted buffers. I am also interested to hear how your system responds to compiling w/out PREEMPT. Ben -- Ben Greear [EMAIL PROTECTED] Candela Technologies Inc http://www.candelatech.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Sunday 21 August 2005 23:21, Danial Thom wrote: > > You problem could very well be something else > > entirely, but try a > > kernel build with PREEMPT_NONE and HZ=100 and > > see if it makes a big > > difference (or if that's your current config, > > then try the opposite, > > HZ=1000 and PREEMPT). If it does make a > > difference, then that's a > > valuable piece of information to report on the > > list. If it turns out > > it makes next to no difference at all, then > > that as well is relevant > > information as then people will know that HZ & > > preempt is not the > > cause and can focus on finding the problem > > elsewhere. > > Yes. Hz isn't going to make much difference on a > 2.0Ghz opteron, but I can see how premption can > cause packet loss. Shouldn't packet processing be > the highest priority process? It seems pointless > to "keep the audio buffers full" if you're > dropping packets as a result. > > Also some clown typing on the keyboard shouldn't > cause packet loss. Trading network integrity for > snappy responsiveness is a bad trade. You do not need to argue about usefulness of preempt (or lack thereof). You need to try non-PREEMPT kernel as suggested (if you really are interested in fixing performance degradation you observe, that is). http://www.catb.org/~esr/faqs/smart-questions.html -- vda - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
*confused by the top-posting..* --- Luigi Genoni <[EMAIL PROTECTED]> wrote: > maybe it is possible to be more clear. > > voluntary kernel preemption adds explicit > preemption points into the > kernel and full kernel preemption makes all > kernel code preemptible. This > way even when a process is executing some > syscall in kernel space, it can > be volontary or involontary preempted. > > For interactive users the systems seems to be > smarter, but when the system > is doing a lot of work in kernel space, then > you of course have to lose > something. > > Also just to check id it's the case or not to > preempt means you lose > something. This something is usually troughput. > In your case I would not > use a preemptible kernel. > > SOmething similar could be said about Timer > frequency, but here the lost > is connected to the number to interrupts you > have to manage. > > The point is that a desktop where the users > simple need a smooth sysstem > to be userd interactivelly, but not real CPU > power, and a server where you > need hourse power are different topics and need > different kernel > behaviour. > > > > On Sun, August 21, 2005 19:07, Danial Thom > wrote: > > > Ok, well you'll have to explain this one: > > > > > > "Low latency comes at the cost of decreased > > throughput - can't have both" > > > > Seems to be a bit backwards. Threading the > kernel > > adds latency, so its the additional latency > in the kernel that causes the > > drop in throughput. Do you mean that kernel > performance has been sacrificed > > in order to be able to service other threads > more quickly, even when there > > are no other threads to be serviced? > > > > Danial The issue I have with that logic is that you seem to use "kernel" in a general sense without regard to what its doing. Dropping packets is always detrimental to the user regardless of what he's using the computer for. An audio stream that drops packets isn't going to be "smooth" at the user level. All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft, floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the "its ok to drop packets when x occurs" algorithm to be "its never ok to drop packets unless x occurs" (such as a queue depth)? Is it possible? Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
*confused by the top-posting..* --- Luigi Genoni [EMAIL PROTECTED] wrote: maybe it is possible to be more clear. voluntary kernel preemption adds explicit preemption points into the kernel and full kernel preemption makes all kernel code preemptible. This way even when a process is executing some syscall in kernel space, it can be volontary or involontary preempted. For interactive users the systems seems to be smarter, but when the system is doing a lot of work in kernel space, then you of course have to lose something. Also just to check id it's the case or not to preempt means you lose something. This something is usually troughput. In your case I would not use a preemptible kernel. SOmething similar could be said about Timer frequency, but here the lost is connected to the number to interrupts you have to manage. The point is that a desktop where the users simple need a smooth sysstem to be userd interactivelly, but not real CPU power, and a server where you need hourse power are different topics and need different kernel behaviour. On Sun, August 21, 2005 19:07, Danial Thom wrote: Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Seems to be a bit backwards. Threading the kernel adds latency, so its the additional latency in the kernel that causes the drop in throughput. Do you mean that kernel performance has been sacrificed in order to be able to service other threads more quickly, even when there are no other threads to be serviced? Danial The issue I have with that logic is that you seem to use kernel in a general sense without regard to what its doing. Dropping packets is always detrimental to the user regardless of what he's using the computer for. An audio stream that drops packets isn't going to be smooth at the user level. All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft, floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the its ok to drop packets when x occurs algorithm to be its never ok to drop packets unless x occurs (such as a queue depth)? Is it possible? Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On Sunday 21 August 2005 23:21, Danial Thom wrote: You problem could very well be something else entirely, but try a kernel build with PREEMPT_NONE and HZ=100 and see if it makes a big difference (or if that's your current config, then try the opposite, HZ=1000 and PREEMPT). If it does make a difference, then that's a valuable piece of information to report on the list. If it turns out it makes next to no difference at all, then that as well is relevant information as then people will know that HZ preempt is not the cause and can focus on finding the problem elsewhere. Yes. Hz isn't going to make much difference on a 2.0Ghz opteron, but I can see how premption can cause packet loss. Shouldn't packet processing be the highest priority process? It seems pointless to keep the audio buffers full if you're dropping packets as a result. Also some clown typing on the keyboard shouldn't cause packet loss. Trading network integrity for snappy responsiveness is a bad trade. You do not need to argue about usefulness of preempt (or lack thereof). You need to try non-PREEMPT kernel as suggested (if you really are interested in fixing performance degradation you observe, that is). http://www.catb.org/~esr/faqs/smart-questions.html -- vda - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Patrick McHardy <[EMAIL PROTECTED]> wrote: > Danial Thom wrote: > > I just started fiddling with 2.6.12, and > there > > seems to be a big drop-off in performance > from > > 2.4.x in terms of networking on a > uniprocessor > > system. Just bridging packets through the > > machine, 2.6.12 starts dropping packets at > > ~100Kpps, whereas 2.4.x doesn't start > dropping > > until over 350Kpps on the same hardware > (2.0Ghz > > Opteron with e1000 driver). This is pitiful > > prformance for this hardware. I've > > increased the rx ring in the e1000 driver to > 512 > > with little change (interrupt moderation is > set > > to 8000 Ints/second). Has "tuning" for MP > > destroyed UP performance altogether, or is > there > > some tuning parameter that could make a > 4-fold > > difference? All debugging is off and there > are > > no messages on the console or in the error > logs. > > The kernel is the standard kernel.org dowload > > config with SMP turned off and the intel > ethernet > > card drivers as modules without any other > > changes, which is exactly the config for my > 2.4 > > kernels. > > Do you have netfilter enabled? Briging > netfilter was > added in 2.6, enabling it will influence > performance > negatively. Otherwise, is this performance drop > visible in other setups besides bridging as > well? > Yes, bridging is clean. I also routed with the same performance drop. Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl <[EMAIL PROTECTED]> wrote: > On 8/21/05, Danial Thom <[EMAIL PROTECTED]> > wrote: > > I just started fiddling with 2.6.12, and > there > > seems to be a big drop-off in performance > from > > 2.4.x in terms of networking on a > uniprocessor > > system. Just bridging packets through the > > machine, 2.6.12 starts dropping packets at > > ~100Kpps, whereas 2.4.x doesn't start > dropping > > until over 350Kpps on the same hardware > (2.0Ghz > > Opteron with e1000 driver). This is pitiful > > prformance for this hardware. I've > > increased the rx ring in the e1000 driver to > 512 > > with little change (interrupt moderation is > set > > to 8000 Ints/second). Has "tuning" for MP > > destroyed UP performance altogether, or is > there > > some tuning parameter that could make a > 4-fold > > difference? All debugging is off and there > are > > no messages on the console or in the error > logs. > > The kernel is the standard kernel.org dowload > > config with SMP turned off and the intel > ethernet > > card drivers as modules without any other > > changes, which is exactly the config for my > 2.4 > > kernels. > > > > If you have preemtion enabled you could disable > it. Low latency comes > at the cost of decreased throughput - can't > have both. Also try using > a HZ of 100 if you are currently using 1000, > that should also improve > throughput a little at the cost of slightly > higher latencies. > > I doubt that it'll do any huge difference, but > if it does, then that > would probably be valuable info. > Ok, well you'll have to explain this one: "Low latency comes at the cost of decreased throughput - can't have both" Seems to be a bit backwards. Threading the kernel adds latency, so its the additional latency in the kernel that causes the drop in throughput. Do you mean that kernel performance has been sacrificed in order to be able to service other threads more quickly, even when there are no other threads to be serviced? Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl <[EMAIL PROTECTED]> wrote: > On 8/21/05, Danial Thom <[EMAIL PROTECTED]> > wrote: > > I just started fiddling with 2.6.12, and > there > > seems to be a big drop-off in performance > from > > 2.4.x in terms of networking on a > uniprocessor > > system. Just bridging packets through the > > machine, 2.6.12 starts dropping packets at > > ~100Kpps, whereas 2.4.x doesn't start > dropping > > until over 350Kpps on the same hardware > (2.0Ghz > > Opteron with e1000 driver). This is pitiful > > prformance for this hardware. I've > > increased the rx ring in the e1000 driver to > 512 > > with little change (interrupt moderation is > set > > to 8000 Ints/second). Has "tuning" for MP > > destroyed UP performance altogether, or is > there > > some tuning parameter that could make a > 4-fold > > difference? All debugging is off and there > are > > no messages on the console or in the error > logs. > > The kernel is the standard kernel.org dowload > > config with SMP turned off and the intel > ethernet > > card drivers as modules without any other > > changes, which is exactly the config for my > 2.4 > > kernels. > > > > If you have preemtion enabled you could disable > it. Low latency comes > at the cost of decreased throughput - can't > have both. Also try using > a HZ of 100 if you are currently using 1000, > that should also improve > throughput a little at the cost of slightly > higher latencies. > > I doubt that it'll do any huge difference, but > if it does, then that > would probably be valuable info. > Ok, well you'll have to explain this one: "Low latency comes at the cost of decreased throughput - can't have both" Seems to be a bit backwards. Threading the kernel adds latency, so its the additional latency in the kernel that causes the drop in throughput. Do you mean that kernel performance has been sacrificed in order to be able to service other threads more quickly, even when there are no other threads to be serviced? Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: --- Patrick McHardy <[EMAIL PROTECTED]> wrote: Do you have netfilter enabled? Briging netfilter was added in 2.6, enabling it will influence performance negatively. Otherwise, is this performance drop visible in other setups besides bridging as well? Yes, bridging is clean. I also routed with the same performance drop. In that case please send more information, like both 2.4 and 2.6 configs, dmesg output from booting, lspci output, other hardware details, .. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has "tuning" for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. Do you have netfilter enabled? Briging netfilter was added in 2.6, enabling it will influence performance negatively. Otherwise, is this performance drop visible in other setups besides bridging as well? - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On 8/21/05, Danial Thom <[EMAIL PROTECTED]> wrote: > > > --- Jesper Juhl <[EMAIL PROTECTED]> wrote: > > > On 8/21/05, Jesper Juhl <[EMAIL PROTECTED]> > > wrote: > > > On 8/21/05, Danial Thom > > <[EMAIL PROTECTED]> wrote: > > > > > > > > > Ok, well you'll have to explain this one: > > > > > > > > "Low latency comes at the cost of decreased > > > > throughput - can't have both" > > > > > > > > Seems to be a bit backwards. Threading the > > kernel > > > > adds latency, so its the additional latency > > in > > > > the kernel that causes the drop in > > throughput. Do > > > > you mean that kernel performance has been > > > > sacrificed in order to be able to service > > other > > > > threads more quickly, even when there are > > no > > > > other threads to be serviced? > > > > > > > > > > Ok, let me start with the way HZ influences > > things. > > > > > [snip] > > > > A small followup. > > > > I'm not saying that the value of HZ or your > > preempt setting (whatever > > it may be) is definately the cause of your > > problem. All I'm saying is > > that it might be a contributing factor, so it's > > probably something > > that's worth a little bit of time testing. > > > > In many cases people running a server on > > resonably new hardware with > > HZ=1000 and full preempt won't even notice, but > > that's depending on > > the load on the server and what jobs it has. > > For some tasks it > > matters, for some the differences in > > performance is negligible. > > > > You problem could very well be something else > > entirely, but try a > > kernel build with PREEMPT_NONE and HZ=100 and > > see if it makes a big > > difference (or if that's your current config, > > then try the opposite, > > HZ=1000 and PREEMPT). If it does make a > > difference, then that's a > > valuable piece of information to report on the > > list. If it turns out > > it makes next to no difference at all, then > > that as well is relevant > > information as then people will know that HZ & > > preempt is not the > > cause and can focus on finding the problem > > elsewhere. > > > Yes. Hz isn't going to make much difference on a > 2.0Ghz opteron, but I can see how premption can > cause packet loss. Shouldn't packet processing be > the highest priority process? It seems pointless > to "keep the audio buffers full" if you're > dropping packets as a result. > > Also some clown typing on the keyboard shouldn't > cause packet loss. Trading network integrity for > snappy responsiveness is a bad trade. > Since you choose to reply to a public list on a private email I guess I should post the bit I'd [snip]'d above, so everyone else can get the whole picture : On 8/21/05, Jesper Juhl <[EMAIL PROTECTED]> wrote: > On 8/21/05, Danial Thom <[EMAIL PROTECTED]> wrote: > > > > > Ok, well you'll have to explain this one: > > > > "Low latency comes at the cost of decreased > > throughput - can't have both" > > > > Seems to be a bit backwards. Threading the kernel > > adds latency, so its the additional latency in > > the kernel that causes the drop in throughput. Do > > you mean that kernel performance has been > > sacrificed in order to be able to service other > > threads more quickly, even when there are no > > other threads to be serviced? > > > > Ok, let me start with the way HZ influences things. > > The value of HZ was increased from 100 to 1000 in 2.6.x (and later > made a config option so people can choose at compile time between 100, > 250 & 1000). > > This means that in 2.6 you have ten times as many timer interrupts > happening every second compared to 2.4. This is great for things that > need to be served on short notice, like audio buffers that need to be > filled *now* before the music skips and can't wait 10ms bacause the > user will notice. But, all those interrupts add overhead and thus the > total amount of work that can be done every second will be slightly > lower even though response times are good. For a desktop HZ=1000 is > usually a very nice thing, but for a server that doesn't need good > response times (low latency) HZ=100 or HZ=250 will provide better > overall throughput and is often preferable. > > It's more or less the same thing with kernel preemption. When the > kernel is fully preemtible it means that kernel functions can be > interrupted at (almost) any time if there happens to be a higher > priority process/thread that is runnable. When the kernel is not > preemtible, system calls etc always run to completion before being > interrupted. > > Preemption is great for interactive applications where you want user > interfaces to respond fast, want multimedia applications to be able to > refill buffers at regular intervals (with great precision on the > timing) etc. With preempt, such high applications will be able to > interrupt/preempt other lower priority processes (like a background > run of updatedb running from cron for example) even when those > processes are in the middle of a system call, whereas without >
Re: 2.6.12 Performance problems
Danial Thom <[EMAIL PROTECTED]> wrote: > > I just started fiddling with 2.6.12, and there > seems to be a big drop-off in performance from > 2.4.x in terms of networking on a uniprocessor > system. Just bridging packets through the > machine, 2.6.12 starts dropping packets at > ~100Kpps, whereas 2.4.x doesn't start dropping > until over 350Kpps on the same hardware (2.0Ghz > Opteron with e1000 driver). This is pitiful > prformance for this hardware. I've > increased the rx ring in the e1000 driver to 512 > with little change (interrupt moderation is set > to 8000 Ints/second). Has "tuning" for MP > destroyed UP performance altogether, or is there > some tuning parameter that could make a 4-fold > difference? All debugging is off and there are > no messages on the console or in the error logs. > The kernel is the standard kernel.org dowload > config with SMP turned off and the intel ethernet > card drivers as modules without any other > changes, which is exactly the config for my 2.4 > kernels. > (added netdev to cc) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl <[EMAIL PROTECTED]> wrote: > On 8/21/05, Jesper Juhl <[EMAIL PROTECTED]> > wrote: > > On 8/21/05, Danial Thom > <[EMAIL PROTECTED]> wrote: > > > > > > > Ok, well you'll have to explain this one: > > > > > > "Low latency comes at the cost of decreased > > > throughput - can't have both" > > > > > > Seems to be a bit backwards. Threading the > kernel > > > adds latency, so its the additional latency > in > > > the kernel that causes the drop in > throughput. Do > > > you mean that kernel performance has been > > > sacrificed in order to be able to service > other > > > threads more quickly, even when there are > no > > > other threads to be serviced? > > > > > > > Ok, let me start with the way HZ influences > things. > > > [snip] > > A small followup. > > I'm not saying that the value of HZ or your > preempt setting (whatever > it may be) is definately the cause of your > problem. All I'm saying is > that it might be a contributing factor, so it's > probably something > that's worth a little bit of time testing. > > In many cases people running a server on > resonably new hardware with > HZ=1000 and full preempt won't even notice, but > that's depending on > the load on the server and what jobs it has. > For some tasks it > matters, for some the differences in > performance is negligible. > > You problem could very well be something else > entirely, but try a > kernel build with PREEMPT_NONE and HZ=100 and > see if it makes a big > difference (or if that's your current config, > then try the opposite, > HZ=1000 and PREEMPT). If it does make a > difference, then that's a > valuable piece of information to report on the > list. If it turns out > it makes next to no difference at all, then > that as well is relevant > information as then people will know that HZ & > preempt is not the > cause and can focus on finding the problem > elsewhere. > Yes. Hz isn't going to make much difference on a 2.0Ghz opteron, but I can see how premption can cause packet loss. Shouldn't packet processing be the highest priority process? It seems pointless to "keep the audio buffers full" if you're dropping packets as a result. Also some clown typing on the keyboard shouldn't cause packet loss. Trading network integrity for snappy responsiveness is a bad trade. Danial __ Yahoo! Mail for Mobile Take Yahoo! Mail with you! Check email on your mobile phone. http://mobile.yahoo.com/learn/mail - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Danial Thom [EMAIL PROTECTED] wrote: Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Seems to be a bit backwards. Threading the kernel adds latency, so its the additional latency in the kernel that causes the drop in throughput. Do you mean that kernel performance has been sacrificed in order to be able to service other threads more quickly, even when there are no other threads to be serviced? Ok, let me start with the way HZ influences things. [snip] A small followup. I'm not saying that the value of HZ or your preempt setting (whatever it may be) is definately the cause of your problem. All I'm saying is that it might be a contributing factor, so it's probably something that's worth a little bit of time testing. In many cases people running a server on resonably new hardware with HZ=1000 and full preempt won't even notice, but that's depending on the load on the server and what jobs it has. For some tasks it matters, for some the differences in performance is negligible. You problem could very well be something else entirely, but try a kernel build with PREEMPT_NONE and HZ=100 and see if it makes a big difference (or if that's your current config, then try the opposite, HZ=1000 and PREEMPT). If it does make a difference, then that's a valuable piece of information to report on the list. If it turns out it makes next to no difference at all, then that as well is relevant information as then people will know that HZ preempt is not the cause and can focus on finding the problem elsewhere. Yes. Hz isn't going to make much difference on a 2.0Ghz opteron, but I can see how premption can cause packet loss. Shouldn't packet processing be the highest priority process? It seems pointless to keep the audio buffers full if you're dropping packets as a result. Also some clown typing on the keyboard shouldn't cause packet loss. Trading network integrity for snappy responsiveness is a bad trade. Danial __ Yahoo! Mail for Mobile Take Yahoo! Mail with you! Check email on your mobile phone. http://mobile.yahoo.com/learn/mail - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
On 8/21/05, Danial Thom [EMAIL PROTECTED] wrote: --- Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Danial Thom [EMAIL PROTECTED] wrote: Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Seems to be a bit backwards. Threading the kernel adds latency, so its the additional latency in the kernel that causes the drop in throughput. Do you mean that kernel performance has been sacrificed in order to be able to service other threads more quickly, even when there are no other threads to be serviced? Ok, let me start with the way HZ influences things. [snip] A small followup. I'm not saying that the value of HZ or your preempt setting (whatever it may be) is definately the cause of your problem. All I'm saying is that it might be a contributing factor, so it's probably something that's worth a little bit of time testing. In many cases people running a server on resonably new hardware with HZ=1000 and full preempt won't even notice, but that's depending on the load on the server and what jobs it has. For some tasks it matters, for some the differences in performance is negligible. You problem could very well be something else entirely, but try a kernel build with PREEMPT_NONE and HZ=100 and see if it makes a big difference (or if that's your current config, then try the opposite, HZ=1000 and PREEMPT). If it does make a difference, then that's a valuable piece of information to report on the list. If it turns out it makes next to no difference at all, then that as well is relevant information as then people will know that HZ preempt is not the cause and can focus on finding the problem elsewhere. Yes. Hz isn't going to make much difference on a 2.0Ghz opteron, but I can see how premption can cause packet loss. Shouldn't packet processing be the highest priority process? It seems pointless to keep the audio buffers full if you're dropping packets as a result. Also some clown typing on the keyboard shouldn't cause packet loss. Trading network integrity for snappy responsiveness is a bad trade. Since you choose to reply to a public list on a private email I guess I should post the bit I'd [snip]'d above, so everyone else can get the whole picture : On 8/21/05, Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Danial Thom [EMAIL PROTECTED] wrote: Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Seems to be a bit backwards. Threading the kernel adds latency, so its the additional latency in the kernel that causes the drop in throughput. Do you mean that kernel performance has been sacrificed in order to be able to service other threads more quickly, even when there are no other threads to be serviced? Ok, let me start with the way HZ influences things. The value of HZ was increased from 100 to 1000 in 2.6.x (and later made a config option so people can choose at compile time between 100, 250 1000). This means that in 2.6 you have ten times as many timer interrupts happening every second compared to 2.4. This is great for things that need to be served on short notice, like audio buffers that need to be filled *now* before the music skips and can't wait 10ms bacause the user will notice. But, all those interrupts add overhead and thus the total amount of work that can be done every second will be slightly lower even though response times are good. For a desktop HZ=1000 is usually a very nice thing, but for a server that doesn't need good response times (low latency) HZ=100 or HZ=250 will provide better overall throughput and is often preferable. It's more or less the same thing with kernel preemption. When the kernel is fully preemtible it means that kernel functions can be interrupted at (almost) any time if there happens to be a higher priority process/thread that is runnable. When the kernel is not preemtible, system calls etc always run to completion before being interrupted. Preemption is great for interactive applications where you want user interfaces to respond fast, want multimedia applications to be able to refill buffers at regular intervals (with great precision on the timing) etc. With preempt, such high applications will be able to interrupt/preempt other lower priority processes (like a background run of updatedb running from cron for example) even when those processes are in the middle of a system call, whereas without preemption they are forced to wait several milliseconds for the other apps syscall to complete before they can get a chance to run - which the user notices as sluggish gui response, music or video skips etc. There are a few prices you pay for preemption.
Re: 2.6.12 Performance problems
Danial Thom wrote: --- Patrick McHardy [EMAIL PROTECTED] wrote: Do you have netfilter enabled? Briging netfilter was added in 2.6, enabling it will influence performance negatively. Otherwise, is this performance drop visible in other setups besides bridging as well? Yes, bridging is clean. I also routed with the same performance drop. In that case please send more information, like both 2.4 and 2.6 configs, dmesg output from booting, lspci output, other hardware details, .. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has tuning for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. Do you have netfilter enabled? Briging netfilter was added in 2.6, enabling it will influence performance negatively. Otherwise, is this performance drop visible in other setups besides bridging as well? - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Danial Thom [EMAIL PROTECTED] wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has tuning for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. If you have preemtion enabled you could disable it. Low latency comes at the cost of decreased throughput - can't have both. Also try using a HZ of 100 if you are currently using 1000, that should also improve throughput a little at the cost of slightly higher latencies. I doubt that it'll do any huge difference, but if it does, then that would probably be valuable info. Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Seems to be a bit backwards. Threading the kernel adds latency, so its the additional latency in the kernel that causes the drop in throughput. Do you mean that kernel performance has been sacrificed in order to be able to service other threads more quickly, even when there are no other threads to be serviced? Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Jesper Juhl [EMAIL PROTECTED] wrote: On 8/21/05, Danial Thom [EMAIL PROTECTED] wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has tuning for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. If you have preemtion enabled you could disable it. Low latency comes at the cost of decreased throughput - can't have both. Also try using a HZ of 100 if you are currently using 1000, that should also improve throughput a little at the cost of slightly higher latencies. I doubt that it'll do any huge difference, but if it does, then that would probably be valuable info. Ok, well you'll have to explain this one: Low latency comes at the cost of decreased throughput - can't have both Seems to be a bit backwards. Threading the kernel adds latency, so its the additional latency in the kernel that causes the drop in throughput. Do you mean that kernel performance has been sacrificed in order to be able to service other threads more quickly, even when there are no other threads to be serviced? Danial Start your day with Yahoo! - make it your home page http://www.yahoo.com/r/hs - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
--- Patrick McHardy [EMAIL PROTECTED] wrote: Danial Thom wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has tuning for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. Do you have netfilter enabled? Briging netfilter was added in 2.6, enabling it will influence performance negatively. Otherwise, is this performance drop visible in other setups besides bridging as well? Yes, bridging is clean. I also routed with the same performance drop. Danial __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.6.12 Performance problems
Danial Thom [EMAIL PROTECTED] wrote: I just started fiddling with 2.6.12, and there seems to be a big drop-off in performance from 2.4.x in terms of networking on a uniprocessor system. Just bridging packets through the machine, 2.6.12 starts dropping packets at ~100Kpps, whereas 2.4.x doesn't start dropping until over 350Kpps on the same hardware (2.0Ghz Opteron with e1000 driver). This is pitiful prformance for this hardware. I've increased the rx ring in the e1000 driver to 512 with little change (interrupt moderation is set to 8000 Ints/second). Has tuning for MP destroyed UP performance altogether, or is there some tuning parameter that could make a 4-fold difference? All debugging is off and there are no messages on the console or in the error logs. The kernel is the standard kernel.org dowload config with SMP turned off and the intel ethernet card drivers as modules without any other changes, which is exactly the config for my 2.4 kernels. (added netdev to cc) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
