Re: Linux 2.4.5-ac2
yes I get a performance improvement of about 5% could you port your patches to the 2.4.5-ac4 kernel? I'd love to see if the ac improvements and yours add to each other. Thanks, - Fabio Jens Axboe wrote: > On Tue, May 29 2001, Fabio Riccardi wrote: > > "Leeuw van der, Tim" wrote: > > > > > But the claim was that 2.4.5-ac2 is faster than 2.4.5 plain, so which > > > changes are in 2.4.5-ac2 that would make it faster than 2.4.5 plain? Also, I > > > don't know if 2.4.5-ac1 is as fast as 2.4.5-ac2 for Fabio. If not, then it's > > > a change in the 2.4.5-ac2 changelog. If it is as fast, it is one of the > > > changes in the 2.4.5-ac1 changelog. > > > > 2.4.5-ac1 crashed on my machine, vanilla 2.4.5 worked but slower than 2.4.2 > > > > 2.4.5-ac2 is _a lot_ faster than all the 2.4.4 and of vanilla 2.4.5 > > > > please notice that I have a 4G machine, dual proc, and I run a very > > memory/IO/CPU intensive test, so your mileage may vary with different > > applications. > > Could you try the 4GB I/O patches and see if they boost performance of > such cases? > > *.kernel.org/pub/linux/kernel/people/axboe/patches/2.4.5/ > > -- > Jens Axboe > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.4.5-ac2
yes I get a performance improvement of about 5% could you port your patches to the 2.4.5-ac4 kernel? I'd love to see if the ac improvements and yours add to each other. Thanks, - Fabio Jens Axboe wrote: On Tue, May 29 2001, Fabio Riccardi wrote: Leeuw van der, Tim wrote: But the claim was that 2.4.5-ac2 is faster than 2.4.5 plain, so which changes are in 2.4.5-ac2 that would make it faster than 2.4.5 plain? Also, I don't know if 2.4.5-ac1 is as fast as 2.4.5-ac2 for Fabio. If not, then it's a change in the 2.4.5-ac2 changelog. If it is as fast, it is one of the changes in the 2.4.5-ac1 changelog. 2.4.5-ac1 crashed on my machine, vanilla 2.4.5 worked but slower than 2.4.2 2.4.5-ac2 is _a lot_ faster than all the 2.4.4 and of vanilla 2.4.5 please notice that I have a 4G machine, dual proc, and I run a very memory/IO/CPU intensive test, so your mileage may vary with different applications. Could you try the 4GB I/O patches and see if they boost performance of such cases? *.kernel.org/pub/linux/kernel/people/axboe/patches/2.4.5/ -- Jens Axboe - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.4.5-ac2
Ok, things are fast again now! :)) Performance is back to that of 2.4.2-ac26, and stability is a lot better. Under heavy FS pressure 2.4.5-ac2 is about 5-10% faster than vanilla 2.4.5, the aa1,2 kernels have the same performance of vanilla 2.4.5. Which one of your changes affected performance so much? BTW: the hangs that you are talking about in the 2.4.4 series, are they total freezes? I've sporadically observed in a few 2.4.4-acX kernels strange hung-ups where the system is still running, but very slowly, ps and other similar commands (top) hang and you cannot kill them anymore. I think that this is some sort of (memory?) resource deadlock. The only miraculous way to resurrect such a locked system is to start Mozilla... everything goes back to normal! For the curious, the latest Ximian Gnone has the terminal and the mozilla icons next to each other and I clicked on the wrong icon by mistake... - Fabio Alan Cox wrote: > ftp://ftp.kernel.org/pub/linux/kernel/people/alan/2.4/ > > Intermediate diffs are available from > http://www.bzimage.org > > In terms of going through the code audit almost all the sound drivers still > need fixing to lock against format changes during a read/write. Poll creating > and starting a buffer as write does and also mmap during write, write during > an mmap. > > 2.4.5-ac2 > o Restore lock_kernel on umount (Al Viro) > | Should cure Reiserfs crash in 2.4.5 > o Fix additional scsi_ioctl leak (John Martin) > o Clean up scsi_ioctl error handling (me) > o Configure.help typo fixes (Nerijus Baliunas) > o Fix hgafb problems with logos (Ferenc Bakonyi) > o Fix lock problems in the rio driver (Rasmus Andersen) > o Make new cmpci SMP safe (Carlos E Gorges) > o Fix missing restore flags in soundmodem (Rasmus Andersen) > o Set max sectors in ps2esdi (Paul Gortmaker) > o Fix interrupt restore problems in mixcom(Rasmus Andersen) > o Fix alpha compile on dp264/generic (Andrea Arcangeli) > o Fix irda irport locking restores(Rasmus Andersen) > o Fix failed kmalloc handling in hisax(Kai Germaschewski) > o Add missing memory barrier in qlogicisp (?) > o Fix missing restore_flags in eata_dma (Rasmus Andersen) > o Fix procfs locking in irttp (Rasmus Andersen) > o Winbond updates (Manfred Spraul) > o Stop network eating PF_MEMALLOC ram (Manfred Spraul) > o Drop fs/buffer.c low mem flush changes (me) > o Drop changes to mm/highmem.c(me) > | I don't think the Linus one is quite right but its easier > | for everyone to be working off one base > o Revert GFP_FAIL and some other alloc bits (me) > o Hopefully fix initrd problem(me) > o Fix kmalloc check in ide-tape (Rasmus Andersen) > o Fix irda irtty locking (Rasmus Andersen) > o Fix missing irq restore in qla1280 (Rasmus Andersen) > o Fix proc/pid/mem cross exec behaviour (Arjan van de Ven) > o Fix direct user space derefs in eicon (me) > | From Stanford checker > o Fix direct user space derefs in ipddp (me) > | From Stanford checker > o Fix direct user space derefs in ixj (me) > | From Stanford checker > o Fix direct user space derefs in decnet (me) > | From Stanford checker > > 2.4.5-ac1 > o Merge Linus 2.4.5 tree > > Summary of changes for Linux 2.4.5-ac versus Linus 2.4.5 > > o Fix memory leak in wanrouter > o Fix memory leak in wanmain > o Use non atomic memory for linearising NFS buffers as they are > done in task context > o Fix dereference of freed memory in NetROM drivers > o Fix writing to freed memory in ax25_ip > o Support debugging of slab pools > o NinjaSCSI pcmcia scsi driver > o Raw HID device for USB peripheral buttons/controllers > o Updated NTFS > o RAMfs with resource limits > o NMI watchdog available on uniprocessor x86 > o Update CMPCI drivers (not yet SMP safe) > o Configurable max_map_count > o Dynamic sysctl key registration > o SE401 USB camera driver > o Updated Zoran ZR3606x driver (replaces buz) > o w9966 parallel port camera driver (partially merged with Linus) > o Include headers in etags > o Don't delete empty directories on make distclean > o Fix halt/reboot handling on Alcor Alpha > o IDE driver support for Etrax E100 > o IDE infrastructure support
License Clarification [X15 beta 1 - source release]
Some people pointed out that I misused the term "Open Source" wrt our Source License. Our license doesn't meet the required criteria of the OSI since we restrict the usage of our sources for personal use only. As was already pointed out in my previous message people will have to buy a license for commercial exploitation of the code (i.e. running a commercial web site). Sorry for the inconvenience, - Fabio Fabio Riccardi wrote: > Dear all, > > I finally managed to package the X15 web accelerator for the first > source release. > > The current release includes a CGI module, an Apache configuration > module and several salability improvements. It is a beta 1, quite stable > but it may/will still contain a few bugs. The README is a bit outdated > and the code could use more comments... :) > > The code It is released under an Open Source license very much in the > spirit of Sun/Solaris, Apple/Darwin, etc., but less restrictive. > > Basically (for what I have been explained by our lawyers :) the license > says that: > > 1) you can peruse the code for your own use and/or research in any way > you like, > > 2) you can use X15 for your own non commercial use (i.e., you can have > the fastest family reunion pictures website of the planet), > > 3) if you want to use the software for any commercial exploitation you > have to buy a license from Chromium Communications, > > 4) if you make changes they belong to you, but if you send them back to > us than you agree to not claim anything for them. > > You can get the sources from: > http://www.chromium.com/cgi-bin/crosforum/YaBB.pl > > or (when the DNS gets propagated) from: http://source.chromium.com/ > > Our source site sports one of those nifty sourceforge-like interfaces > for discussions, bug reports and whatever, I'd prefer you to use that > instead of sending email directly to me or to this list. > > I'll build a proper "product" web page and add more documentation in the > next few days. > > - Fabio > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.4.5-ac1 won't boot with 4GB bigmem option
Same here, I have a dual 1GHz PIII with 4G, I don't get an oops but an infinite loop of: > mm: critical shortage of bounce buffers. Indeed this message has been pestering me in all the recent .4-acx kernels when the machine is under heavy FS pressure. In these kernels I observe a significative (5-10%) performance degradation as soon as the FS cache fills up all the available memory, at this moment "kswapd" starts to take lots of CPU time (10-20%) and I keep getting plenty of the above messages. I'm running SpecWeb with the X15 webserver, which uses sendfile to send its content, and a very large file set (8-9G, more than twice as much as the physical RAM). 2.4.2-acx and early 2.4.3-acx kernles were much better in this respect and a lot more stable. - Fabio Ben Twijnstra wrote: > Hi, > > I compiled and booted the 2.4.5-ac1 kernel with the CONFIG_HIGHMEM4G=y option > and got an oops in __alloc_pages() (called by alloc_bounce() called by > schedule()). Everything works fine if I turn the 4GB mode off. > > Machine is a Dell Precision with 2 Xeons and 2GB of RAM. > > 2.4.5 works fine with the 4GB. Any idea what changed between the two? > > Grtz, > > Ben > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: 2.4.5-ac1 won't boot with 4GB bigmem option
Same here, I have a dual 1GHz PIII with 4G, I don't get an oops but an infinite loop of: mm: critical shortage of bounce buffers. Indeed this message has been pestering me in all the recent .4-acx kernels when the machine is under heavy FS pressure. In these kernels I observe a significative (5-10%) performance degradation as soon as the FS cache fills up all the available memory, at this moment kswapd starts to take lots of CPU time (10-20%) and I keep getting plenty of the above messages. I'm running SpecWeb with the X15 webserver, which uses sendfile to send its content, and a very large file set (8-9G, more than twice as much as the physical RAM). 2.4.2-acx and early 2.4.3-acx kernles were much better in this respect and a lot more stable. - Fabio Ben Twijnstra wrote: Hi, I compiled and booted the 2.4.5-ac1 kernel with the CONFIG_HIGHMEM4G=y option and got an oops in __alloc_pages() (called by alloc_bounce() called by schedule()). Everything works fine if I turn the 4GB mode off. Machine is a Dell Precision with 2 Xeons and 2GB of RAM. 2.4.5 works fine with the 4GB. Any idea what changed between the two? Grtz, Ben - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
License Clarification [X15 beta 1 - source release]
Some people pointed out that I misused the term Open Source wrt our Source License. Our license doesn't meet the required criteria of the OSI since we restrict the usage of our sources for personal use only. As was already pointed out in my previous message people will have to buy a license for commercial exploitation of the code (i.e. running a commercial web site). Sorry for the inconvenience, - Fabio Fabio Riccardi wrote: Dear all, I finally managed to package the X15 web accelerator for the first source release. The current release includes a CGI module, an Apache configuration module and several salability improvements. It is a beta 1, quite stable but it may/will still contain a few bugs. The README is a bit outdated and the code could use more comments... :) The code It is released under an Open Source license very much in the spirit of Sun/Solaris, Apple/Darwin, etc., but less restrictive. Basically (for what I have been explained by our lawyers :) the license says that: 1) you can peruse the code for your own use and/or research in any way you like, 2) you can use X15 for your own non commercial use (i.e., you can have the fastest family reunion pictures website of the planet), 3) if you want to use the software for any commercial exploitation you have to buy a license from Chromium Communications, 4) if you make changes they belong to you, but if you send them back to us than you agree to not claim anything for them. You can get the sources from: http://www.chromium.com/cgi-bin/crosforum/YaBB.pl or (when the DNS gets propagated) from: http://source.chromium.com/ Our source site sports one of those nifty sourceforge-like interfaces for discussions, bug reports and whatever, I'd prefer you to use that instead of sending email directly to me or to this list. I'll build a proper product web page and add more documentation in the next few days. - Fabio - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: Linux 2.4.5-ac2
Ok, things are fast again now! :)) Performance is back to that of 2.4.2-ac26, and stability is a lot better. Under heavy FS pressure 2.4.5-ac2 is about 5-10% faster than vanilla 2.4.5, the aa1,2 kernels have the same performance of vanilla 2.4.5. Which one of your changes affected performance so much? BTW: the hangs that you are talking about in the 2.4.4 series, are they total freezes? I've sporadically observed in a few 2.4.4-acX kernels strange hung-ups where the system is still running, but very slowly, ps and other similar commands (top) hang and you cannot kill them anymore. I think that this is some sort of (memory?) resource deadlock. The only miraculous way to resurrect such a locked system is to start Mozilla... everything goes back to normal! For the curious, the latest Ximian Gnone has the terminal and the mozilla icons next to each other and I clicked on the wrong icon by mistake... - Fabio Alan Cox wrote: ftp://ftp.kernel.org/pub/linux/kernel/people/alan/2.4/ Intermediate diffs are available from http://www.bzimage.org In terms of going through the code audit almost all the sound drivers still need fixing to lock against format changes during a read/write. Poll creating and starting a buffer as write does and also mmap during write, write during an mmap. 2.4.5-ac2 o Restore lock_kernel on umount (Al Viro) | Should cure Reiserfs crash in 2.4.5 o Fix additional scsi_ioctl leak (John Martin) o Clean up scsi_ioctl error handling (me) o Configure.help typo fixes (Nerijus Baliunas) o Fix hgafb problems with logos (Ferenc Bakonyi) o Fix lock problems in the rio driver (Rasmus Andersen) o Make new cmpci SMP safe (Carlos E Gorges) o Fix missing restore flags in soundmodem (Rasmus Andersen) o Set max sectors in ps2esdi (Paul Gortmaker) o Fix interrupt restore problems in mixcom(Rasmus Andersen) o Fix alpha compile on dp264/generic (Andrea Arcangeli) o Fix irda irport locking restores(Rasmus Andersen) o Fix failed kmalloc handling in hisax(Kai Germaschewski) o Add missing memory barrier in qlogicisp (?) o Fix missing restore_flags in eata_dma (Rasmus Andersen) o Fix procfs locking in irttp (Rasmus Andersen) o Winbond updates (Manfred Spraul) o Stop network eating PF_MEMALLOC ram (Manfred Spraul) o Drop fs/buffer.c low mem flush changes (me) o Drop changes to mm/highmem.c(me) | I don't think the Linus one is quite right but its easier | for everyone to be working off one base o Revert GFP_FAIL and some other alloc bits (me) o Hopefully fix initrd problem(me) o Fix kmalloc check in ide-tape (Rasmus Andersen) o Fix irda irtty locking (Rasmus Andersen) o Fix missing irq restore in qla1280 (Rasmus Andersen) o Fix proc/pid/mem cross exec behaviour (Arjan van de Ven) o Fix direct user space derefs in eicon (me) | From Stanford checker o Fix direct user space derefs in ipddp (me) | From Stanford checker o Fix direct user space derefs in ixj (me) | From Stanford checker o Fix direct user space derefs in decnet (me) | From Stanford checker 2.4.5-ac1 o Merge Linus 2.4.5 tree Summary of changes for Linux 2.4.5-ac versus Linus 2.4.5 o Fix memory leak in wanrouter o Fix memory leak in wanmain o Use non atomic memory for linearising NFS buffers as they are done in task context o Fix dereference of freed memory in NetROM drivers o Fix writing to freed memory in ax25_ip o Support debugging of slab pools o NinjaSCSI pcmcia scsi driver o Raw HID device for USB peripheral buttons/controllers o Updated NTFS o RAMfs with resource limits o NMI watchdog available on uniprocessor x86 o Update CMPCI drivers (not yet SMP safe) o Configurable max_map_count o Dynamic sysctl key registration o SE401 USB camera driver o Updated Zoran ZR3606x driver (replaces buz) o w9966 parallel port camera driver (partially merged with Linus) o Include headers in etags o Don't delete empty directories on make distclean o Fix halt/reboot handling on Alcor Alpha o IDE driver support for Etrax E100 o IDE infrastructure support for IDE using non standard data transfers o Run ~/bin/installkernel if
X15 beta 1 - source release
Dear all, I finally managed to package the X15 web accelerator for the first source release. The current release includes a CGI module, an Apache configuration module and several salability improvements. It is a beta 1, quite stable but it may/will still contain a few bugs. The README is a bit outdated and the code could use more comments... :) The code It is released under an Open Source license very much in the spirit of Sun/Solaris, Apple/Darwin, etc., but less restrictive. Basically (for what I have been explained by our lawyers :) the license says that: 1) you can peruse the code for your own use and/or research in any way you like, 2) you can use X15 for your own non commercial use (i.e., you can have the fastest family reunion pictures website of the planet), 3) if you want to use the software for any commercial exploitation you have to buy a license from Chromium Communications, 4) if you make changes they belong to you, but if you send them back to us than you agree to not claim anything for them. You can get the sources from: http://www.chromium.com/cgi-bin/crosforum/YaBB.pl or (when the DNS gets propagated) from: http://source.chromium.com/ Our source site sports one of those nifty sourceforge-like interfaces for discussions, bug reports and whatever, I'd prefer you to use that instead of sending email directly to me or to this list. I'll build a proper "product" web page and add more documentation in the next few days. - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
X15 beta 1 - source release
Dear all, I finally managed to package the X15 web accelerator for the first source release. The current release includes a CGI module, an Apache configuration module and several salability improvements. It is a beta 1, quite stable but it may/will still contain a few bugs. The README is a bit outdated and the code could use more comments... :) The code It is released under an Open Source license very much in the spirit of Sun/Solaris, Apple/Darwin, etc., but less restrictive. Basically (for what I have been explained by our lawyers :) the license says that: 1) you can peruse the code for your own use and/or research in any way you like, 2) you can use X15 for your own non commercial use (i.e., you can have the fastest family reunion pictures website of the planet), 3) if you want to use the software for any commercial exploitation you have to buy a license from Chromium Communications, 4) if you make changes they belong to you, but if you send them back to us than you agree to not claim anything for them. You can get the sources from: http://www.chromium.com/cgi-bin/crosforum/YaBB.pl or (when the DNS gets propagated) from: http://source.chromium.com/ Our source site sports one of those nifty sourceforge-like interfaces for discussions, bug reports and whatever, I'd prefer you to use that instead of sending email directly to me or to this list. I'll build a proper product web page and add more documentation in the next few days. - Fabio - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
Hello, I have uploaded a new release of X15 that hopefully solves all the RFC bugs. I say hopefully because I haven't had the opportunity to fully test the request pipelining. Is there anything to automatize such tests? >From what I could measure X15 is still a good 5% faster than TUX. You can find the file at: http://www.chromium.com/X15-Alpha-4.tgz BTW: Next release (in a week or so) will be a beta and it will include source code! BTW2: I'll be away from my email in the next few days, so don't be amazed if I'll be kind of slow replying to messages... - Fabio Ingo Molnar wrote: > yet another anomaly i noticed. X15 does not appear to handle pipelined > HTTP/1.1 requests properly, it ignores the second request if two requests > arrive in the same packet. > > SPECweb99 does not send pipelined requests, but a number of RL web clients > do. (Mozilla, apt-get, etc.) > > Ingo > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
Hello, I have uploaded a new release of X15 that hopefully solves all the RFC bugs. I say hopefully because I haven't had the opportunity to fully test the request pipelining. Is there anything to automatize such tests? From what I could measure X15 is still a good 5% faster than TUX. You can find the file at: http://www.chromium.com/X15-Alpha-4.tgz BTW: Next release (in a week or so) will be a beta and it will include source code! BTW2: I'll be away from my email in the next few days, so don't be amazed if I'll be kind of slow replying to messages... - Fabio Ingo Molnar wrote: yet another anomaly i noticed. X15 does not appear to handle pipelined HTTP/1.1 requests properly, it ignores the second request if two requests arrive in the same packet. SPECweb99 does not send pipelined requests, but a number of RL web clients do. (Mozilla, apt-get, etc.) Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
ok, I'm totally ignorant here, what is a pipelined request? btw: please be kind with my mistakes, X15 _is_ alpha code anyway... :) - Fabio Ingo Molnar wrote: > yet another anomaly i noticed. X15 does not appear to handle pipelined > HTTP/1.1 requests properly, it ignores the second request if two requests > arrive in the same packet. > > SPECweb99 does not send pipelined requests, but a number of RL web clients > do. (Mozilla, apt-get, etc.) > > Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release
Ingo, I'm really impressed by your feedback! How do you manage to discover so many things? I fixed the bug, and checked that it hadn't affected my specweb results. Indeed specweb never issues closing 1.1 connections, it would use a 1.0 request with close in that case. Moreover even if a client says that it will close the connection and the server instead leaves it open, the client would just close the connection anyway, unless there is a (very contrived) bug in the client which would let itself be diverted from its original intention by an overly talkative server... X15 would be indeed negatively affected by these useless idle open connections cluttering the file descriptor table and consuming resources for nothing. I'll post the corrected version later on today. BTW: is there any _concise_ document specifying the HTTP protocol and its variants? - Fabio Ingo Molnar wrote: > Fabio, > > i noticed another RFC anomaly in X15. It ignores the "Connection: close" > request header passed by a HTTP/1.1 client. This behavior is against RFC > 2616, a server must not override the client's choice of non-persistent > connection. (there might be HTTP/1.1 clients that do not support > persistent connections and signal this via "Connection: close".) > > the rule is this: a request is either keepalive or non-keepalive. HTTP/1.0 > requests default to non-keepalive. HTTP/1.1 requests default to keepalive. > The default can be overriden via the "Connection: Keep-Alive" or > "Connection: close" header fields. > > if you fix this, does it impact SPECweb99 performance in any way? > > Ingo > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release
Ingo, I'm really impressed by your feedback! How do you manage to discover so many things? I fixed the bug, and checked that it hadn't affected my specweb results. Indeed specweb never issues closing 1.1 connections, it would use a 1.0 request with close in that case. Moreover even if a client says that it will close the connection and the server instead leaves it open, the client would just close the connection anyway, unless there is a (very contrived) bug in the client which would let itself be diverted from its original intention by an overly talkative server... X15 would be indeed negatively affected by these useless idle open connections cluttering the file descriptor table and consuming resources for nothing. I'll post the corrected version later on today. BTW: is there any _concise_ document specifying the HTTP protocol and its variants? - Fabio Ingo Molnar wrote: Fabio, i noticed another RFC anomaly in X15. It ignores the Connection: close request header passed by a HTTP/1.1 client. This behavior is against RFC 2616, a server must not override the client's choice of non-persistent connection. (there might be HTTP/1.1 clients that do not support persistent connections and signal this via Connection: close.) the rule is this: a request is either keepalive or non-keepalive. HTTP/1.0 requests default to non-keepalive. HTTP/1.1 requests default to keepalive. The default can be overriden via the Connection: Keep-Alive or Connection: close header fields. if you fix this, does it impact SPECweb99 performance in any way? Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
ok, I'm totally ignorant here, what is a pipelined request? btw: please be kind with my mistakes, X15 _is_ alpha code anyway... :) - Fabio Ingo Molnar wrote: yet another anomaly i noticed. X15 does not appear to handle pipelined HTTP/1.1 requests properly, it ignores the second request if two requests arrive in the same packet. SPECweb99 does not send pipelined requests, but a number of RL web clients do. (Mozilla, apt-get, etc.) Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
I have fixed the stale header cache problem. Files are statted on every request, no "practical" tricks. Performance doesn't seem to have suffered :) I also have added a cache garbage collector to close "old" file descriptors and remove even older header cache entries. This should make sure that you don't exceed your system resources. The definition of old and the sweep frequency are user configurable. You can download the new version from: http://www.chromium.com/X15-Alpha-3.tgz - Fabio Ingo Molnar wrote: > On Tue, 1 May 2001, Fabio Riccardi wrote: > > > This is actually a bug in the current X15, I know how to fix it (with > > negligible overhead) but I've been lazy :) > > yep, i think it's pretty straightforward: you have a cache of open file > descriptors (like Squid i think), and you can start a background 'cache > synchronization thread' that does a stat() on every open file's real VFS > path, every second or so. This should have small overhead (the number of > file descriptors cached should be limited anyway via some sort of LRU), > and guarantees 'practical coherency', without having the overhead of > immediate coherency. [total coherency is pointless anyway, not even the > kernel guarantees it to all parallel VFS users.] > > Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
I have fixed the stale header cache problem. Files are statted on every request, no practical tricks. Performance doesn't seem to have suffered :) I also have added a cache garbage collector to close old file descriptors and remove even older header cache entries. This should make sure that you don't exceed your system resources. The definition of old and the sweep frequency are user configurable. You can download the new version from: http://www.chromium.com/X15-Alpha-3.tgz - Fabio Ingo Molnar wrote: On Tue, 1 May 2001, Fabio Riccardi wrote: This is actually a bug in the current X15, I know how to fix it (with negligible overhead) but I've been lazy :) yep, i think it's pretty straightforward: you have a cache of open file descriptors (like Squid i think), and you can start a background 'cache synchronization thread' that does a stat() on every open file's real VFS path, every second or so. This should have small overhead (the number of file descriptors cached should be limited anyway via some sort of LRU), and guarantees 'practical coherency', without having the overhead of immediate coherency. [total coherency is pointless anyway, not even the kernel guarantees it to all parallel VFS users.] Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
Our intention is to release X15 with an open source license. This will happen as soon as the codebase stabilizes a bit, that is when we go beta (in two - three weeks). At the moment we just don't have the time... The reason why I released the alpha binary version is that several people would not believe that a user-space server with this level of performance would be possible at all and several statements that I made on this list were challenged. Besides I really appreciate the feedback that I received so far from Ingo and others, and I'd be very curious to know if anybody did any performance evaluation at all. - Fabio Zach Brown wrote: > I've always been tempted to go back and take a real swing at a > nice content server, but there's only so many hours in the day, and > apache+thttpd+tux complete the problem space. If X15 isn't released > with an open license, I may be tempted yet again :) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space (fwd)
>From my experience system calls are not an issue. What costs a lot is moving data around, since modern CPUs spend most of their time in memory/bus wait cycles... - Fabio Linus Torvalds wrote: > >I think that applies to all really high-performance servers. > > Note that it is definitely not always true. > > Linux system calls are reasonably light-weight. And sometimes trying to > avoid them ends up beaing _more_ work - because you might have to worry > about synchronization and cache coherency in user mode. > > So the rule should be "avoid _unnecessary_ system calls". > > Linus > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space (fwd)
From my experience system calls are not an issue. What costs a lot is moving data around, since modern CPUs spend most of their time in memory/bus wait cycles... - Fabio Linus Torvalds wrote: I think that applies to all really high-performance servers. Note that it is definitely not always true. Linux system calls are reasonably light-weight. And sometimes trying to avoid them ends up beaing _more_ work - because you might have to worry about synchronization and cache coherency in user mode. So the rule should be avoid _unnecessary_ system calls. Linus - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
Our intention is to release X15 with an open source license. This will happen as soon as the codebase stabilizes a bit, that is when we go beta (in two - three weeks). At the moment we just don't have the time... The reason why I released the alpha binary version is that several people would not believe that a user-space server with this level of performance would be possible at all and several statements that I made on this list were challenged. Besides I really appreciate the feedback that I received so far from Ingo and others, and I'd be very curious to know if anybody did any performance evaluation at all. - Fabio Zach Brown wrote: I've always been tempted to go back and take a real swing at a nice content server, but there's only so many hours in the day, and apache+thttpd+tux complete the problem space. If X15 isn't released with an open license, I may be tempted yet again :) - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
This is actually a bug in the current X15, I know how to fix it (with negligible overhead) but I've been lazy :) give me a few days... - Fabio Ingo Molnar wrote: > hm, another X15 caching artifact i noticed: i've changed the index.html > file, and while X15 was still serving the old copy, TUX served the new > file immediately. > > its cache is apparently not coherent with actual VFS contents. (ie. it's a > read-once cache apparently?) TUX has some (occasionally significant) > overhead due to non-cached VFS-lookups. > > Ingo > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
Ok I fixed it, the header date timestamp is updated with every request. Performance doesn't seem to have suffered significantly (less than 1%). You can find the new release at: http://www.chromium.com/X15-Alpha-2.tgz BTW: Don't call me slime, I wasn't trying to cheat, I just didn't know that the date stamp was required to be really up-to-date. - Fabio dean gaudet wrote: > On Sun, 29 Apr 2001, Fabio Riccardi wrote: > > > I can disable header caching and see what happens, I'll add an option > > for this in the next X15 release. > > heh, well to be honest, i'd put the (permanent) caching of the Date header > into the very slimy, benchmark-only trick category. (right up there > alongside running the HTTP and TCP stacks inside the NIC interrupt > handler, which folks have done to get even better scores.) > > > Nevertheless I don't know how much this is interesting in real life, > > since on the internet most static pages are cached on proxies. I agree > > that the RFC asks for a date for the original response, but once the > > response is cached what does this date mean? > > the Date is always the time the response was generated on the origin > server. (there are exceptions for clockless servers, but such servers > have other limitations you wouldn't want -- notably they MUST NOT generate > Last-Modified.) > > one example oddity you might experience with a non-increasing Date > surrounds Last-Modified and Date, see section 13.3.3. note that the rfc > indicates that if Last-Modified is less than 60 seconds earlier than Date > then Last-Modified is only a weak validator rather than a strong > validator. this would complicate range requests -- because weak > validators can't be used with range requests. if your server never > updates the Date after the first time it serves an object then you'd > potentially never get out of this 60 second window. > > (strong validators are guaranteed to change whenever the object changes... > and Last-Modified isn't strong until some time has passed -- consider NFS > mounted docroots, clock skew in the origin network, multiple file updates > within a second, etc.) > > there are a bunch of other things that Date is used for, all of them are > related to caching heuristics and rules. > > in theory you could claim that you're implementing a cache server rather > than an origin server... i dunno what the SPEC committee will think when > you try to submit results though :) > > so way back when sendfile() was being created i brought up the Date issue > and pointed out that we needed more than a single "void *, size_t" to take > care of headers. eventually this discussion lead creation of TCP_CORK -- > so that a http server could use writev() to send a two element iov for the > headers -- one element with everything that doesn't need to change, the > next element with the Date. > > i also kind of expected the high performance servers to rebuild a Date: > header every second for all of its threads to use. (of course it's not > that simple, you really want to keep a circular list of N dates... and > just assume that after N seconds no thread could still be accessing an old > Date.) > > is this too slow for some reason? (does it play well with zero-copy?) > > -dean > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
Ok I fixed it, the header date timestamp is updated with every request. Performance doesn't seem to have suffered significantly (less than 1%). You can find the new release at: http://www.chromium.com/X15-Alpha-2.tgz BTW: Don't call me slime, I wasn't trying to cheat, I just didn't know that the date stamp was required to be really up-to-date. - Fabio dean gaudet wrote: On Sun, 29 Apr 2001, Fabio Riccardi wrote: I can disable header caching and see what happens, I'll add an option for this in the next X15 release. heh, well to be honest, i'd put the (permanent) caching of the Date header into the very slimy, benchmark-only trick category. (right up there alongside running the HTTP and TCP stacks inside the NIC interrupt handler, which folks have done to get even better scores.) Nevertheless I don't know how much this is interesting in real life, since on the internet most static pages are cached on proxies. I agree that the RFC asks for a date for the original response, but once the response is cached what does this date mean? the Date is always the time the response was generated on the origin server. (there are exceptions for clockless servers, but such servers have other limitations you wouldn't want -- notably they MUST NOT generate Last-Modified.) one example oddity you might experience with a non-increasing Date surrounds Last-Modified and Date, see section 13.3.3. note that the rfc indicates that if Last-Modified is less than 60 seconds earlier than Date then Last-Modified is only a weak validator rather than a strong validator. this would complicate range requests -- because weak validators can't be used with range requests. if your server never updates the Date after the first time it serves an object then you'd potentially never get out of this 60 second window. (strong validators are guaranteed to change whenever the object changes... and Last-Modified isn't strong until some time has passed -- consider NFS mounted docroots, clock skew in the origin network, multiple file updates within a second, etc.) there are a bunch of other things that Date is used for, all of them are related to caching heuristics and rules. in theory you could claim that you're implementing a cache server rather than an origin server... i dunno what the SPEC committee will think when you try to submit results though :) so way back when sendfile() was being created i brought up the Date issue and pointed out that we needed more than a single void *, size_t to take care of headers. eventually this discussion lead creation of TCP_CORK -- so that a http server could use writev() to send a two element iov for the headers -- one element with everything that doesn't need to change, the next element with the Date. i also kind of expected the high performance servers to rebuild a Date: header every second for all of its threads to use. (of course it's not that simple, you really want to keep a circular list of N dates... and just assume that after N seconds no thread could still be accessing an old Date.) is this too slow for some reason? (does it play well with zero-copy?) -dean - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
I can disable header caching and see what happens, I'll add an option for this in the next X15 release. Nevertheless I don't know how much this is interesting in real life, since on the internet most static pages are cached on proxies. I agree that the RFC asks for a date for the original response, but once the response is cached what does this date mean? - Fabio Ingo Molnar wrote: > Fabio, > > i noticed one weirdness in the Date-field handling of X15. X15 appears to > cache the Date field too, which is contrary to RFCs: > > earth2:~> wget -s http://localhost/index.html -O - 2>/dev/null | grep Date > Date: Sat Apr 28 10:15:14 2001 > earth2:~> date > Sat Apr 28 10:32:40 CEST 2001 > > ie. there is already a 15 minutes difference between the 'origin date of > the reply' and the actual date of the reply. (i started X15 up 15 minutes > ago.) > > per RFC 2616: > . > The Date general-header field represents the date and time at which the > message was originated, [...] > > Origin servers MUST include a Date header field in all responses, [...] > . > > i considered the caching of the Date field for TUX too, and avoided it > exactly due to this issue, to not violate this 'MUST' item in the RFC. It > can be reasonably expected from a web server to have a 1-second accurate > Date: field. > > the header-caching in X15 gives it an edge against TUX, obviously, but IMO > it's a questionable practice. > > if caching of headers was be allowed then we could the obvious trick of > sendfile()ing complete web replies (first header, then body). > > Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
Linux 2.4 is surely one of the most advanced OSs ever happened, especially from the optimization point of view and for the admirable economy of concepts on which it lies. I definitively hope that X15 helps reinforcing the success to this amazing system. TUX has definitively been my performance yardstick for the development of X15, but I had many sources of inspiration for the X15 architecture. Maybe the most relevant are the Flash Web Server (Pai, Druschel, Zwaenepoel), several Linus observations on this list about (web) server architecture and kernnel services, and the reading of the Hennessy & Patterson architecture books. Last but not least, aside from some heated discussions, research in microkernel architecture has taught us many lessons on how to achieve an efficient model of interaction across separate addressing spaces. If i have to make some sort of educated guess and point at where the current bottleneck lies for web server performance, I would say that it is somewhere between the memory subsystem and the PCI bus. With zero-copy sendfile data movement is not an issue anymore, asynchronous network IO allows for really inexpensive thread scheduling, and system call invocation adds a very negligible overhead in Linux. What we are left with now is purely wait cycles, the CPUs and the NICs are contending for memory and bus bandwidth. It would be really interesting to see where the network shifts now that faster machines are becoming available. On my whish list for future kernel developments I would definitively put disk asynchronous IO and a more decent file descriptor passing implementation. I'll detail this in subsequent messages. I'll surely check out the impact of Ingo's patches on TUX performance sometime this week. I'd also like to reiterate my request for help for testing X15 on higher end server architectures. X15 is still very young alpha code and I can surely improve its performance in many ways. - Fabio Ingo Molnar wrote: > On Fri, 27 Apr 2001, Fabio Riccardi wrote: > > > I'd like to announce the first release of X15 Alpha 1, a _user space_ > > web server that is as fast as TUX. > > great, the first TUX clone! ;-) > > This should put the accusations to rest that Linux got the outstandingly > high SPECweb99 scores only because the webserver was in kernel-space. It's > the 2.4 kernel's high performance that enabled those results, having the > web-server in kernel-space didnt have much effect. TUX was and remains a > testbed to test high-performance webserving (and FTP serving), without the > API-exporting overhead of userspace. > > [i suspect the small performance advantage of X15 is due to subtle > differences in the SPECweb99 user-space module: eg. while the TUX code was > written, tested and ready to use mmap()-enabled > TUXAPI_alloc_read_objectbuf(), it wasnt enabled actually. I sent Fabio a > mail how to enable it, perhaps he can do some tests to confirm this > suspicion?] > > doing a TUX 2.0 SPECweb99 benchmark on the latest -ac kernels, 86% of time > is spent in generic parts of the kernel, 12% of time is spent in the > user-space SPECweb99 module, and only 2% of time is spent in TUX-specific > kernel code. > > doing the same test with the original TUX 1.0 code shows that more than > 50% of CPU time was spent in TUX-specific code. > > what does this mean? In the roughly 6 months since TUX 1.0 was released, > we moved much of the TUX 1.0 -only improvements into the generic kernel > (most of which was made available to user-space as well), and TUX itself > became smaller and smaller (and used more and more generic parts of the > kernel). So in effect X15 is executing 50% TUX code :-) > > (there are still a number of performance improvement patches pending that > are not integrated yet: the pagecache extreme-scalability patch and the > smptimers patch. These patches speed both X15 and TUX up.) > > (there is one thing though that can never be 'exported to user-space': to > isolate possibly untrusted binary application code from the server itself, > without performance degradation. So we always have to be mentally open to > the validity of kernel-space services.) > > Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
Linux 2.4 is surely one of the most advanced OSs ever happened, especially from the optimization point of view and for the admirable economy of concepts on which it lies. I definitively hope that X15 helps reinforcing the success to this amazing system. TUX has definitively been my performance yardstick for the development of X15, but I had many sources of inspiration for the X15 architecture. Maybe the most relevant are the Flash Web Server (Pai, Druschel, Zwaenepoel), several Linus observations on this list about (web) server architecture and kernnel services, and the reading of the Hennessy Patterson architecture books. Last but not least, aside from some heated discussions, research in microkernel architecture has taught us many lessons on how to achieve an efficient model of interaction across separate addressing spaces. If i have to make some sort of educated guess and point at where the current bottleneck lies for web server performance, I would say that it is somewhere between the memory subsystem and the PCI bus. With zero-copy sendfile data movement is not an issue anymore, asynchronous network IO allows for really inexpensive thread scheduling, and system call invocation adds a very negligible overhead in Linux. What we are left with now is purely wait cycles, the CPUs and the NICs are contending for memory and bus bandwidth. It would be really interesting to see where the network shifts now that faster machines are becoming available. On my whish list for future kernel developments I would definitively put disk asynchronous IO and a more decent file descriptor passing implementation. I'll detail this in subsequent messages. I'll surely check out the impact of Ingo's patches on TUX performance sometime this week. I'd also like to reiterate my request for help for testing X15 on higher end server architectures. X15 is still very young alpha code and I can surely improve its performance in many ways. - Fabio Ingo Molnar wrote: On Fri, 27 Apr 2001, Fabio Riccardi wrote: I'd like to announce the first release of X15 Alpha 1, a _user space_ web server that is as fast as TUX. great, the first TUX clone! ;-) This should put the accusations to rest that Linux got the outstandingly high SPECweb99 scores only because the webserver was in kernel-space. It's the 2.4 kernel's high performance that enabled those results, having the web-server in kernel-space didnt have much effect. TUX was and remains a testbed to test high-performance webserving (and FTP serving), without the API-exporting overhead of userspace. [i suspect the small performance advantage of X15 is due to subtle differences in the SPECweb99 user-space module: eg. while the TUX code was written, tested and ready to use mmap()-enabled TUXAPI_alloc_read_objectbuf(), it wasnt enabled actually. I sent Fabio a mail how to enable it, perhaps he can do some tests to confirm this suspicion?] doing a TUX 2.0 SPECweb99 benchmark on the latest -ac kernels, 86% of time is spent in generic parts of the kernel, 12% of time is spent in the user-space SPECweb99 module, and only 2% of time is spent in TUX-specific kernel code. doing the same test with the original TUX 1.0 code shows that more than 50% of CPU time was spent in TUX-specific code. what does this mean? In the roughly 6 months since TUX 1.0 was released, we moved much of the TUX 1.0 -only improvements into the generic kernel (most of which was made available to user-space as well), and TUX itself became smaller and smaller (and used more and more generic parts of the kernel). So in effect X15 is executing 50% TUX code :-) (there are still a number of performance improvement patches pending that are not integrated yet: the pagecache extreme-scalability patch and the smptimers patch. These patches speed both X15 and TUX up.) (there is one thing though that can never be 'exported to user-space': to isolate possibly untrusted binary application code from the server itself, without performance degradation. So we always have to be mentally open to the validity of kernel-space services.) Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
I can disable header caching and see what happens, I'll add an option for this in the next X15 release. Nevertheless I don't know how much this is interesting in real life, since on the internet most static pages are cached on proxies. I agree that the RFC asks for a date for the original response, but once the response is cached what does this date mean? - Fabio Ingo Molnar wrote: Fabio, i noticed one weirdness in the Date-field handling of X15. X15 appears to cache the Date field too, which is contrary to RFCs: earth2:~ wget -s http://localhost/index.html -O - 2/dev/null | grep Date Date: Sat Apr 28 10:15:14 2001 earth2:~ date Sat Apr 28 10:32:40 CEST 2001 ie. there is already a 15 minutes difference between the 'origin date of the reply' and the actual date of the reply. (i started X15 up 15 minutes ago.) per RFC 2616: . The Date general-header field represents the date and time at which the message was originated, [...] Origin servers MUST include a Date header field in all responses, [...] . i considered the caching of the Date field for TUX too, and avoided it exactly due to this issue, to not violate this 'MUST' item in the RFC. It can be reasonably expected from a web server to have a 1-second accurate Date: field. the header-caching in X15 gives it an edge against TUX, obviously, but IMO it's a questionable practice. if caching of headers was be allowed then we could the obvious trick of sendfile()ing complete web replies (first header, then body). Ingo - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
In both cases (X15 and TUX) the CPU utilization is 100% There are no IO bottlenecks on disk or on the net side. I think that the major bottleneck is the speed of RAM and the PCI bus, wait cycles. We are basically going at the speed of the hardware. - Fabio "David S. Miller" wrote: > Fabio Riccardi writes: > > On my Dell 4400 with 2G of RAM and 2 933MHz PIII and NetGear 2Gbit NICs > > I achieve about 2500 SpecWeb99 connections, with both X15 and > > TUX (actually X15 is sligtly faster, some 20 connections more... ;) > > What is the CPU utilization like in X15 vs. TUX during > these runs? > > Later, > David S. Miller > [EMAIL PROTECTED] > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
X15 alpha release: as fast as TUX but in user space
Dear All, I'd like to announce the first release of X15 Alpha 1, a _user space_ web server that is as fast as TUX. On my Dell 4400 with 2G of RAM and 2 933MHz PIII and NetGear 2Gbit NICs I achieve about 2500 SpecWeb99 connections, with both X15 and TUX (actually X15 is sligtly faster, some 20 connections more... ;) Given the limitations of my experimental setup I'd like to ask if some of you could help me testing my software on some higher end machines. I'm interested to see what happens on 4-8 processors in terms of scalability etc. You can download X15 Alpha 1 from here: http://www.chromium.com/X15-Alpha-1.tgz The the README file in the tarball should contain sufficient information to run the thing, I also included a support module for running the SpecWeb benchmark. TIA, ciao, - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: X15 alpha release: as fast as TUX but in user space
In both cases (X15 and TUX) the CPU utilization is 100% There are no IO bottlenecks on disk or on the net side. I think that the major bottleneck is the speed of RAM and the PCI bus, wait cycles. We are basically going at the speed of the hardware. - Fabio David S. Miller wrote: Fabio Riccardi writes: On my Dell 4400 with 2G of RAM and 2 933MHz PIII and NetGear 2Gbit NICs I achieve about 2500 SpecWeb99 connections, with both X15 and TUX (actually X15 is sligtly faster, some 20 connections more... ;) What is the CPU utilization like in X15 vs. TUX during these runs? Later, David S. Miller [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: numbers?
Ingo Molnar wrote: > > On a Dell PowerEdge 1550/1000 the published TUX 2 result is 2765. > > > > If you take into account the fact that the 1550 has a faster processor > > (1GHz) and a more modern bus architecture (Serverworks HE with memory > > interleaving and a triple PCI bus), the performance is roughly the > > same. > > the system was IO-limited (given that a ~9 GB fileset was running on a 2 > GB RAM system), so CPU speed has not a big impact. I'd say it makes no > sense to compare different systems. >From what I've seen the major impact comes from the disk IO bandwidth to memory size ratio and from the PCI bus to memory bandwidth. I agree that comparing different hardware architectures is a tricky business, but you asked me to comment on some of the comparisons that you made... > > The static pages work fine, the dynamic module gets executed, but for > > some reason it fails to open the postlog file and to spawn the spec > > utility tasks at reset time. > > the newest TUX code chroots into docroot, so you should either use "/" as > the docroot, or put /lib libraries into your docroot. Oh, the docs don't mention anything of that... I'll try to set the docroot as you say > > I'll make an alpha release of X15 available for download by the end of > > next week, so people will be able to test it independently. > > (will source code be available so we can see whether it's an apples to > apples thing?) I'll release the source for the SPEC dynamic code dll, which indeed is just a straight porting of the TUX dynamic code from the SPEC site. - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: numbers?
Alan, SPEC connections are cumulative of static (70%) and dynamic (30%) pages, with the dynamic using quite a bit of CPU (25%-30%) and the static pages dataset of several (6-8) gigabytes. The chromium server is actually much faster than thttpd and it is a complete web server. - Fabio Alan Cox wrote: > > Incidentally the same server running on a kernel with a multiqueue scheduler > > achieves 1600 connections per second on the same machine, that was the original > > reason for my message for a better scheduler. > > I get 2000 connections a second with a single threaded server called thttpd > on my setup. Thats out of the box on 2.4.2ac with zero copy/sendfile. > > I've never had occasion to frob with tux or specweb - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: numbers?
The current chromium server is based on Apache 1.3, and it inherits its threading limitations. Incidentally the same server running on a kernel with a multiqueue scheduler achieves 1600 connections per second on the same machine, that was the original reason for my message for a better scheduler. In any case the chromium server is a substantially faster apache (more than a factor of two on spec, possibly much more in real life), and given that Apache is the most widespread server on the planet, having something that makes it faster is quite handy. You don't need any further training for users, all standard modules work, we even fixed the performance problems that afflicted Tomcat Jakarta. A convenient solution. Our forthcoming server (dubbed X15) is a completely new thing, but it still sits in user space. X15 is the server I was referring to and as far as I can measure I get very much the same performance as TUX. On a Dell 4400 (933 MHz PIII, 2G of RAM, 5 9G disks) I get 2450 connections/second. On a Dell PowerEdge 1550/1000 the published TUX 2 result is 2765. If you take into account the fact that the 1550 has a faster processor (1GHz) and a more modern bus architecture (Serverworks HE with memory interleaving and a triple PCI bus), the performance is roughly the same. I'd love to try TUX and X15 head to head on the same hardware, indeed I've spent the last two days trying to get TUX to run on my Dell 4400, but I wasn't very lucky. The static pages work fine, the dynamic module gets executed, but for some reason it fails to open the postlog file and to spawn the spec utility tasks at reset time. I tried the latest TUX based on 2.4.2-ac26 from your home site at RedHat, and I've used the latest TUX dynamic code that is published on the SPEC site (Compaq-20010122-DL320-API.tar.gz). Are they compatible with each other? I'll try again today with the TUX that comes with the new RH 7.1 The PowerEdge 2500 for which the TUX result is 3225 (I think that this is the result you were quoting) is a much modern machine than the 4400, with a much higher memory bandwidth, that could explain the performance difference (four NICs sucking data at the same time require quite a bit of bandwidth). I'll make an alpha release of X15 available for download by the end of next week, so people will be able to test it independently. - Fabio Ingo Molnar wrote: > On Fri, 23 Mar 2001, Fabio Riccardi wrote: > > > I'm building an alternative web server that is entirely in _user > > space_ and that achieves the same level of performance as TUX. > > Presently I can match TUX performance within 10-20%, and I still have > > quite a few improvements in my pocket. > > very interesting statement, which appears to be contradicted by numbers on > your website. Your website says you get a 1375 SPECweb99 connections > result on a dual 1 GHz, 4 GB, PIII system: > > http://www.chromium.com/cr_hp.html > > the best TUX 2.0 result published so far, on a very similar system (same > CPU speed, same amount of RAM, same number and type of network cards) is > 3222 connections: > > http://www.spec.org/osg/web99/results/res2001q2/web99-20010319-00100.html > > the difference between 1375 and 3222 is quite substantial, TUX is 134% > faster (2.3 times the performance of your server). I'm sure a userspace > webserver can get quite close to TUX in simple static benchmarks (in fact > phttpd should be very close), but SPECweb99 is far from simple. When > saying you are 10-20% close to TUX, did you refer to SPECweb99 results? > > Ingo > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to [EMAIL PROTECTED] > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: numbers?
Ingo Molnar wrote: On a Dell PowerEdge 1550/1000 the published TUX 2 result is 2765. If you take into account the fact that the 1550 has a faster processor (1GHz) and a more modern bus architecture (Serverworks HE with memory interleaving and a triple PCI bus), the performance is roughly the same. the system was IO-limited (given that a ~9 GB fileset was running on a 2 GB RAM system), so CPU speed has not a big impact. I'd say it makes no sense to compare different systems. From what I've seen the major impact comes from the disk IO bandwidth to memory size ratio and from the PCI bus to memory bandwidth. I agree that comparing different hardware architectures is a tricky business, but you asked me to comment on some of the comparisons that you made... The static pages work fine, the dynamic module gets executed, but for some reason it fails to open the postlog file and to spawn the spec utility tasks at reset time. the newest TUX code chroots into docroot, so you should either use "/" as the docroot, or put /lib libraries into your docroot. Oh, the docs don't mention anything of that... I'll try to set the docroot as you say I'll make an alpha release of X15 available for download by the end of next week, so people will be able to test it independently. (will source code be available so we can see whether it's an apples to apples thing?) I'll release the source for the SPEC dynamic code dll, which indeed is just a straight porting of the TUX dynamic code from the SPEC site. - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: numbers?
Alan, SPEC connections are cumulative of static (70%) and dynamic (30%) pages, with the dynamic using quite a bit of CPU (25%-30%) and the static pages dataset of several (6-8) gigabytes. The chromium server is actually much faster than thttpd and it is a complete web server. - Fabio Alan Cox wrote: Incidentally the same server running on a kernel with a multiqueue scheduler achieves 1600 connections per second on the same machine, that was the original reason for my message for a better scheduler. I get 2000 connections a second with a single threaded server called thttpd on my setup. Thats out of the box on 2.4.2ac with zero copy/sendfile. I've never had occasion to frob with tux or specweb - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: numbers?
The current chromium server is based on Apache 1.3, and it inherits its threading limitations. Incidentally the same server running on a kernel with a multiqueue scheduler achieves 1600 connections per second on the same machine, that was the original reason for my message for a better scheduler. In any case the chromium server is a substantially faster apache (more than a factor of two on spec, possibly much more in real life), and given that Apache is the most widespread server on the planet, having something that makes it faster is quite handy. You don't need any further training for users, all standard modules work, we even fixed the performance problems that afflicted Tomcat Jakarta. A convenient solution. Our forthcoming server (dubbed X15) is a completely new thing, but it still sits in user space. X15 is the server I was referring to and as far as I can measure I get very much the same performance as TUX. On a Dell 4400 (933 MHz PIII, 2G of RAM, 5 9G disks) I get 2450 connections/second. On a Dell PowerEdge 1550/1000 the published TUX 2 result is 2765. If you take into account the fact that the 1550 has a faster processor (1GHz) and a more modern bus architecture (Serverworks HE with memory interleaving and a triple PCI bus), the performance is roughly the same. I'd love to try TUX and X15 head to head on the same hardware, indeed I've spent the last two days trying to get TUX to run on my Dell 4400, but I wasn't very lucky. The static pages work fine, the dynamic module gets executed, but for some reason it fails to open the postlog file and to spawn the spec utility tasks at reset time. I tried the latest TUX based on 2.4.2-ac26 from your home site at RedHat, and I've used the latest TUX dynamic code that is published on the SPEC site (Compaq-20010122-DL320-API.tar.gz). Are they compatible with each other? I'll try again today with the TUX that comes with the new RH 7.1 The PowerEdge 2500 for which the TUX result is 3225 (I think that this is the result you were quoting) is a much modern machine than the 4400, with a much higher memory bandwidth, that could explain the performance difference (four NICs sucking data at the same time require quite a bit of bandwidth). I'll make an alpha release of X15 available for download by the end of next week, so people will be able to test it independently. - Fabio Ingo Molnar wrote: On Fri, 23 Mar 2001, Fabio Riccardi wrote: I'm building an alternative web server that is entirely in _user space_ and that achieves the same level of performance as TUX. Presently I can match TUX performance within 10-20%, and I still have quite a few improvements in my pocket. very interesting statement, which appears to be contradicted by numbers on your website. Your website says you get a 1375 SPECweb99 connections result on a dual 1 GHz, 4 GB, PIII system: http://www.chromium.com/cr_hp.html the best TUX 2.0 result published so far, on a very similar system (same CPU speed, same amount of RAM, same number and type of network cards) is 3222 connections: http://www.spec.org/osg/web99/results/res2001q2/web99-20010319-00100.html the difference between 1375 and 3222 is quite substantial, TUX is 134% faster (2.3 times the performance of your server). I'm sure a userspace webserver can get quite close to TUX in simple static benchmarks (in fact phttpd should be very close), but SPECweb99 is far from simple. When saying you are 10-20% close to TUX, did you refer to SPECweb99 results? Ingo - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: a quest for a better scheduler
I was actually suspecting that the extra lines in your patch were there for a reason :) A few questions: What is the real impact of a (slight) change in scheduling semantics? Under which situation one should notice a difference? As you state in your papers the global decision comes with a cost, is it worth it? Could you make a port of your thing on recent kernels? I tried and I failed and I don't have enough time to figure out why, that should be trivial for you though. TIA, ciao, - Fabio Mike Kravetz wrote: > On Tue, Apr 03, 2001 at 05:18:03PM -0700, Fabio Riccardi wrote: > > > > I have measured the HP and not the "scalability" patch because the two do more > > or less the same thing and give me the same performance advantages, but the > > former is a lot simpler and I could port it with no effort on any recent > > kernel. > > Actually, there is a significant difference between the HP patch and > the one I developed. In the HP patch, if there is a schedulable task > on the 'local' (current CPU) runqueue it will ignore runnable tasks on > other (remote) runqueues. In the multi-queue patch I developed, the > scheduler always attempts to make the same global scheduling decisions > as the current scheduler. > > -- > Mike Kravetz [EMAIL PROTECTED] > IBM Linux Technology Center - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: a quest for a better scheduler
Alan Cox wrote: > > for the "normal case" performance see my other message. > > I did - and with a lot of interest thanks! :) > > I agree that a better threading model would surely help in a web server, but to > > me this is not an excuse to live up with a broken scheduler. > > The problem has always been - alternative scheduler, crappier performance for > 2 tasks running (which is most boxes). If your numbers are right then the > HP patch is working as well for 1 or 2 tasks too Please verify them if you have a couple of spare hours. BTW: I measured similar results for the "scalability" patches on a 2.4.1 kernel, it would be worth the effort to seriously compare them from an architectural point of view, but I don't have the time right now... > > Unless we want to maintain the position tha the only way to achieve good > > performance is to embed server applications in the kernel, some minimal help > > should be provided to goodwilling user applications :) > > Indeed. I'd love to see you beat tux entirely in userspace. It proves the > rest of the API for the kernel is right Indeed, I'm using RT sigio/sigwait event scheduling, bare clone threads and zero-copy io. If only I had a really asynchronous sendfile, or a smarter madvise that wouldn't require to map files :) My server cannot execute dynamic stuff yet, it relies on Apache for that. Running X15 and TUX in the same conditions (i.e. dynamic code in Apache) I get exactly the same score in both cases. I'm adding a TUX-like dynamic interface, I hope to get it to work by next week, then I'll make a real confrontation. Regards, ciao, - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: a quest for a better scheduler
Alan, for the "normal case" performance see my other message. I agree that a better threading model would surely help in a web server, but to me this is not an excuse to live up with a broken scheduler. The X15 server I'm working on now is a sort of user-space TUX, it uses only 8 threads per CPU and it achieves the same level of performance of the kernel space TUX. Even in this case the performance advantage of the multiqueue scheduler is remarkable, especially on a multi-CPU (> 2) architecture. To achieve some decent disk/CPU/network overlapping with the current linux blocking disk IO limitations there is no way to avoid a "bunch of server threads". I've (experimentally) figured out that 8-16 threads per CPU can assure some reasonable overlapping, depending on the memory size and disk subsystem speed. On a 8-way machine this means 64-128 active tasks, a total disaster with the current scheduler. Unless we want to maintain the position tha the only way to achieve good performance is to embed server applications in the kernel, some minimal help should be provided to goodwilling user applications :) TIA, ciao, - Fabio Alan Cox wrote: > > Is there any special reason why any of those patches didn't make it to > > the mainstream kernel code? > > All of them are worse for the normal case. Also 1500 running apache's isnt > a remotely useful situation, you are thrashing the cache even if you are now > not thrashing the scheduler. Use an httpd designed for that situation. Then > you can also downgrade to a cheap pentium class box for the task ;) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: a quest for a better scheduler
Dear all,
I've spent my afternoon running some benchmarks to see if MQ patches would
degrade performance in the "normal case".
To measure performance I've used the latest lmbench and I have mesured the
kernel compile times on a dual pentium III box runing at 1GHz with an 133MHz
bus.
Results (attached) show that there is no measurable difference in performace
between a vanilla scheduler and a multiqueue scheduler when running only few
processes (the compilation benchmark runs essentially two processes, one per
CPU).
I have measured the HP and not the "scalability" patch because the two do more
or less the same thing and give me the same performance advantages, but the
former is a lot simpler and I could port it with no effort on any recent
kernel. It is indeed interesting to see that this patch was originally designed
for a 2.4.0-test10 kernel, and still works fine on the latest kernels, only a
minor change (one line) was required. A version of the patch for the 2.4.2-ac26
kernel is attached to this message.
Given the zero impact on "normal case" performance and the huge positive impact
(> 20%) in the heavy load case (70-80 runnable processess on a load of about
1400 total) I don't see why such a thing shouldn't be accepted in the
mainstream scheduler.
- Fabio
--- sched.c.origTue Mar 27 17:30:58 2001
+++ sched.c Tue Apr 3 16:45:21 2001
@@ -34,6 +34,7 @@
extern void timer_bh(void);
extern void tqueue_bh(void);
extern void immediate_bh(void);
+static inline void hop_queues(struct task_struct *, int);
/*
* scheduler variables
@@ -90,7 +91,8 @@
spinlock_t runqueue_lock __cacheline_aligned = SPIN_LOCK_UNLOCKED; /* inner */
rwlock_t tasklist_lock __cacheline_aligned = RW_LOCK_UNLOCKED; /* outer */
-static LIST_HEAD(runqueue_head);
+static struct list_head runqueue_head[NR_CPUS] = {
+LIST_HEAD_INIT((runqueue_head[0]))};
+static LIST_HEAD(rt_queue_head);
/*
* We align per-CPU scheduling data on cacheline boundaries,
@@ -100,12 +102,15 @@
struct schedule_data {
struct task_struct * curr;
cycles_t last_schedule;
+ struct list_head runqueue_head;
} schedule_data;
char __pad [SMP_CACHE_BYTES];
-} aligned_data [NR_CPUS] __cacheline_aligned = { {{_task,0}}};
+} aligned_data [NR_CPUS] __cacheline_aligned = { {{_task,0,
+ LIST_HEAD_INIT((aligned_data[0].schedule_data.runqueue_head))}}};
#define cpu_curr(cpu) aligned_data[(cpu)].schedule_data.curr
#define last_schedule(cpu) aligned_data[(cpu)].schedule_data.last_schedule
+#define cpu_rq(cpu) (aligned_data[(cpu)].schedule_data.runqueue_head)
struct kernel_stat kstat;
@@ -199,6 +204,33 @@
return goodness(p, cpu, prev->active_mm) - goodness(prev, cpu,
prev->active_mm);
}
+
+static inline int other_goodness(struct task_struct * p, int this_cpu, struct
+mm_struct *this_mm)
+{
+ int weight;
+
+ /*
+* select the current process after every other
+* runnable process, but before the idle thread.
+* Also, dont trigger a counter recalculation.
+*
+* Give the process a first-approximation goodness value
+* according to the number of clock-ticks it has left.
+*
+* Don't do any other calculations if the time slice is
+* over..
+*/
+ weight = p->counter;
+ if (!weight)
+ goto out2;
+
+ /* .. and a slight advantage to the current MM */
+ if (p->mm == this_mm || !p->mm)
+ weight += 1;
+ weight += 20 - p->nice;
+out2:
+ return weight;
+}
/*
* This is ugly, but reschedule_idle() is very timing-critical.
* We are called with the runqueue spinlock held and we must
@@ -266,6 +298,10 @@
} else {
if (oldest_idle == -1ULL) {
int prio = preemption_goodness(tsk, p, cpu);
+ /*
+* this will never be true for < 400 HZ non
+* realtime. optimize this? SAR
+*/
if (prio > max_prio) {
max_prio = prio;
@@ -277,6 +313,10 @@
tsk = target_tsk;
if (tsk) {
if (oldest_idle != -1ULL) {
+ /* push onto best queue */
+ if (p->policy == SCHED_OTHER){
+ hop_queues(p, tsk->processor);
+ }
best_cpu = tsk->processor;
goto send_now_idle;
}
@@ -306,20 +346,28 @@
*/
static inline void add_to_runqueue(struct task_struct * p)
{
- list_add(>run_list, _head);
+ if (p->policy == SCHED_OTHER){
+ list_add(>run_list, _rq(p->processor));
+ } else list_add(>run_list, _queue_head);
nr_running++;
}
-static inline
Re: a quest for a better scheduler
Dear all,
I've spent my afternoon running some benchmarks to see if MQ patches would
degrade performance in the "normal case".
To measure performance I've used the latest lmbench and I have mesured the
kernel compile times on a dual pentium III box runing at 1GHz with an 133MHz
bus.
Results (attached) show that there is no measurable difference in performace
between a vanilla scheduler and a multiqueue scheduler when running only few
processes (the compilation benchmark runs essentially two processes, one per
CPU).
I have measured the HP and not the "scalability" patch because the two do more
or less the same thing and give me the same performance advantages, but the
former is a lot simpler and I could port it with no effort on any recent
kernel. It is indeed interesting to see that this patch was originally designed
for a 2.4.0-test10 kernel, and still works fine on the latest kernels, only a
minor change (one line) was required. A version of the patch for the 2.4.2-ac26
kernel is attached to this message.
Given the zero impact on "normal case" performance and the huge positive impact
( 20%) in the heavy load case (70-80 runnable processess on a load of about
1400 total) I don't see why such a thing shouldn't be accepted in the
mainstream scheduler.
- Fabio
--- sched.c.origTue Mar 27 17:30:58 2001
+++ sched.c Tue Apr 3 16:45:21 2001
@@ -34,6 +34,7 @@
extern void timer_bh(void);
extern void tqueue_bh(void);
extern void immediate_bh(void);
+static inline void hop_queues(struct task_struct *, int);
/*
* scheduler variables
@@ -90,7 +91,8 @@
spinlock_t runqueue_lock __cacheline_aligned = SPIN_LOCK_UNLOCKED; /* inner */
rwlock_t tasklist_lock __cacheline_aligned = RW_LOCK_UNLOCKED; /* outer */
-static LIST_HEAD(runqueue_head);
+static struct list_head runqueue_head[NR_CPUS] = {
+LIST_HEAD_INIT((runqueue_head[0]))};
+static LIST_HEAD(rt_queue_head);
/*
* We align per-CPU scheduling data on cacheline boundaries,
@@ -100,12 +102,15 @@
struct schedule_data {
struct task_struct * curr;
cycles_t last_schedule;
+ struct list_head runqueue_head;
} schedule_data;
char __pad [SMP_CACHE_BYTES];
-} aligned_data [NR_CPUS] __cacheline_aligned = { {{init_task,0}}};
+} aligned_data [NR_CPUS] __cacheline_aligned = { {{init_task,0,
+ LIST_HEAD_INIT((aligned_data[0].schedule_data.runqueue_head))}}};
#define cpu_curr(cpu) aligned_data[(cpu)].schedule_data.curr
#define last_schedule(cpu) aligned_data[(cpu)].schedule_data.last_schedule
+#define cpu_rq(cpu) (aligned_data[(cpu)].schedule_data.runqueue_head)
struct kernel_stat kstat;
@@ -199,6 +204,33 @@
return goodness(p, cpu, prev-active_mm) - goodness(prev, cpu,
prev-active_mm);
}
+
+static inline int other_goodness(struct task_struct * p, int this_cpu, struct
+mm_struct *this_mm)
+{
+ int weight;
+
+ /*
+* select the current process after every other
+* runnable process, but before the idle thread.
+* Also, dont trigger a counter recalculation.
+*
+* Give the process a first-approximation goodness value
+* according to the number of clock-ticks it has left.
+*
+* Don't do any other calculations if the time slice is
+* over..
+*/
+ weight = p-counter;
+ if (!weight)
+ goto out2;
+
+ /* .. and a slight advantage to the current MM */
+ if (p-mm == this_mm || !p-mm)
+ weight += 1;
+ weight += 20 - p-nice;
+out2:
+ return weight;
+}
/*
* This is ugly, but reschedule_idle() is very timing-critical.
* We are called with the runqueue spinlock held and we must
@@ -266,6 +298,10 @@
} else {
if (oldest_idle == -1ULL) {
int prio = preemption_goodness(tsk, p, cpu);
+ /*
+* this will never be true for 400 HZ non
+* realtime. optimize this? SAR
+*/
if (prio max_prio) {
max_prio = prio;
@@ -277,6 +313,10 @@
tsk = target_tsk;
if (tsk) {
if (oldest_idle != -1ULL) {
+ /* push onto best queue */
+ if (p-policy == SCHED_OTHER){
+ hop_queues(p, tsk-processor);
+ }
best_cpu = tsk-processor;
goto send_now_idle;
}
@@ -306,20 +346,28 @@
*/
static inline void add_to_runqueue(struct task_struct * p)
{
- list_add(p-run_list, runqueue_head);
+ if (p-policy == SCHED_OTHER){
+ list_add(p-run_list, cpu_rq(p-processor));
+ } else list_add(p-run_list, rt_queue_head);
nr_running++;
}
Re: a quest for a better scheduler
Alan, for the "normal case" performance see my other message. I agree that a better threading model would surely help in a web server, but to me this is not an excuse to live up with a broken scheduler. The X15 server I'm working on now is a sort of user-space TUX, it uses only 8 threads per CPU and it achieves the same level of performance of the kernel space TUX. Even in this case the performance advantage of the multiqueue scheduler is remarkable, especially on a multi-CPU ( 2) architecture. To achieve some decent disk/CPU/network overlapping with the current linux blocking disk IO limitations there is no way to avoid a "bunch of server threads". I've (experimentally) figured out that 8-16 threads per CPU can assure some reasonable overlapping, depending on the memory size and disk subsystem speed. On a 8-way machine this means 64-128 active tasks, a total disaster with the current scheduler. Unless we want to maintain the position tha the only way to achieve good performance is to embed server applications in the kernel, some minimal help should be provided to goodwilling user applications :) TIA, ciao, - Fabio Alan Cox wrote: Is there any special reason why any of those patches didn't make it to the mainstream kernel code? All of them are worse for the normal case. Also 1500 running apache's isnt a remotely useful situation, you are thrashing the cache even if you are now not thrashing the scheduler. Use an httpd designed for that situation. Then you can also downgrade to a cheap pentium class box for the task ;) - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: a quest for a better scheduler
Alan Cox wrote: for the "normal case" performance see my other message. I did - and with a lot of interest thanks! :) I agree that a better threading model would surely help in a web server, but to me this is not an excuse to live up with a broken scheduler. The problem has always been - alternative scheduler, crappier performance for 2 tasks running (which is most boxes). If your numbers are right then the HP patch is working as well for 1 or 2 tasks too Please verify them if you have a couple of spare hours. BTW: I measured similar results for the "scalability" patches on a 2.4.1 kernel, it would be worth the effort to seriously compare them from an architectural point of view, but I don't have the time right now... Unless we want to maintain the position tha the only way to achieve good performance is to embed server applications in the kernel, some minimal help should be provided to goodwilling user applications :) Indeed. I'd love to see you beat tux entirely in userspace. It proves the rest of the API for the kernel is right Indeed, I'm using RT sigio/sigwait event scheduling, bare clone threads and zero-copy io. If only I had a really asynchronous sendfile, or a smarter madvise that wouldn't require to map files :) My server cannot execute dynamic stuff yet, it relies on Apache for that. Running X15 and TUX in the same conditions (i.e. dynamic code in Apache) I get exactly the same score in both cases. I'm adding a TUX-like dynamic interface, I hope to get it to work by next week, then I'll make a real confrontation. Regards, ciao, - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: a quest for a better scheduler
I was actually suspecting that the extra lines in your patch were there for a reason :) A few questions: What is the real impact of a (slight) change in scheduling semantics? Under which situation one should notice a difference? As you state in your papers the global decision comes with a cost, is it worth it? Could you make a port of your thing on recent kernels? I tried and I failed and I don't have enough time to figure out why, that should be trivial for you though. TIA, ciao, - Fabio Mike Kravetz wrote: On Tue, Apr 03, 2001 at 05:18:03PM -0700, Fabio Riccardi wrote: I have measured the HP and not the "scalability" patch because the two do more or less the same thing and give me the same performance advantages, but the former is a lot simpler and I could port it with no effort on any recent kernel. Actually, there is a significant difference between the HP patch and the one I developed. In the HP patch, if there is a schedulable task on the 'local' (current CPU) runqueue it will ignore runnable tasks on other (remote) runqueues. In the multi-queue patch I developed, the scheduler always attempts to make the same global scheduling decisions as the current scheduler. -- Mike Kravetz [EMAIL PROTECTED] IBM Linux Technology Center - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
a quest for a better scheduler
Hello, I sent a message a few days ago about some limitations I found in the linux scheduler. In servers like Apache where a large (> 1000) number of processes can be running at the same time and where many of them are runnable at the same time, the default Linux scheduler just starts trashing and the machine becomes very rapidly unusable. Performance degradations are quite noticeable on a two-way SMP machine (20-30% of the CPU gets lost) and are even more glaring on a multi-cpu machine. As an example, an 8-way Compaq Proliant just crawls with linux. >From the feedback I received I realized that there are at least two possible solutions to the problem: http://lse.sourceforge.net/scheduling/ http://resourcemanagement.unixsolutions.hp.com/WaRM/schedpolicy.html Indeed I've tried the patches available on the sites for the multi-queue scheduler and I was amazed by the performance improvement that I got. Both patches allow me to get to a 100% real CPU utilization on a two way machine running ~1500 processes. What those patches do is quite simple, instead of having the single global process queue present in the normal Linux scheduler, they add multiple queues (one per CPU). In this way the scheduling decision can be greatly simplified and almost made local to each CPU. No hotspots, no global locks (well, almost). Although some scalability problems are still there (there still is a global decision to make), the performance improvement obtained and the simplicity of the solution are remarkable. The HP patch is probably the most interesting, since it consists of really a few lines of code and it gets (for what I could measure) the same kind of performance improvement of the more elaborate (but still quite simple) sourceforge patch. Is there any special reason why any of those patches didn't make it to the mainstream kernel code? TIA, ciao, - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
a quest for a better scheduler
Hello, I sent a message a few days ago about some limitations I found in the linux scheduler. In servers like Apache where a large ( 1000) number of processes can be running at the same time and where many of them are runnable at the same time, the default Linux scheduler just starts trashing and the machine becomes very rapidly unusable. Performance degradations are quite noticeable on a two-way SMP machine (20-30% of the CPU gets lost) and are even more glaring on a multi-cpu machine. As an example, an 8-way Compaq Proliant just crawls with linux. From the feedback I received I realized that there are at least two possible solutions to the problem: http://lse.sourceforge.net/scheduling/ http://resourcemanagement.unixsolutions.hp.com/WaRM/schedpolicy.html Indeed I've tried the patches available on the sites for the multi-queue scheduler and I was amazed by the performance improvement that I got. Both patches allow me to get to a 100% real CPU utilization on a two way machine running ~1500 processes. What those patches do is quite simple, instead of having the single global process queue present in the normal Linux scheduler, they add multiple queues (one per CPU). In this way the scheduling decision can be greatly simplified and almost made local to each CPU. No hotspots, no global locks (well, almost). Although some scalability problems are still there (there still is a global decision to make), the performance improvement obtained and the simplicity of the solution are remarkable. The HP patch is probably the most interesting, since it consists of really a few lines of code and it gets (for what I could measure) the same kind of performance improvement of the more elaborate (but still quite simple) sourceforge patch. Is there any special reason why any of those patches didn't make it to the mainstream kernel code? TIA, ciao, - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: linux scheduler limitations?
Hi Mike, somebody else on the list already pointed me at your stuff and I quickly downloaded your multiqueue patch for 2.4.1 to try it out. It works great! I finally manage to have 100% CPU utilization and keep the machine decently responsive. On a two 1GHz pentium box i went from 1300 specweb to 1600. That's pretty amazing. There is a bit more overhead though, I'd say arount 5%, when the CPU is not fully loaded. What is the status of your code? Is it going to end-up in the mainstream kernel? Do you have a port to the 2.4.2x kernels? In my enthousiasm I tried to port the patch to 2.4.2-ac26 but I broke something and it didn't work anymore... :) I havent't tried the pooling patch yet, it didn't seem to make much sense on a 2-way box. I have an 8-way on which I'm planning to bench my web server enhancements, I'll try the pooling stuff on it. BTW: interested in the fastest linux web server? BTW2: what about the HP scheduler patches? Thanks, ciao, - Fabio Mike Kravetz wrote: > On Thu, Mar 29, 2001 at 01:55:11PM -0800, Fabio Riccardi wrote: > > I'm using 2.4.2-ac26, but I've noticed the same behavior with all the 2.4 > > kernels I've seen so far. > > > > I haven't even tried on 2.2 > > > > - Fabio > > Fabio, > > Just for fun, you might want to try out some of our scheduler patches > located at: > > http://lse.sourceforge.net/scheduling/ > > I would be interested in your observations. > > -- > Mike Kravetz [EMAIL PROTECTED] > IBM Linux Technology Center - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: linux scheduler limitations?
"J . A . Magallon" wrote: > It all depends on your app, as every parallel algorithm. In a web-ftp-whatever > server, you do not need any synchro. You can start threads in free run and > let them die alone. even if you don't need synchronization you pay for it anyway, since you will have to use the pthread version of libc that is reentrant. Moreover many calls (i.e. accept) are "scheduling points" for pthreads, whenever you call them the runtime will perform quite a bit of bookeeping. it is instructive to use a profiler on your application and see what happens when you use pthreads... > You said, 'mapped'. > AFAIK, that is the advantage, you can avoid the VM switch by sharing memory. If your application uses lots of memory than I agree, Apache only uses a tiny amount of RAM per instance though, so I don't think that that is my case. - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: linux scheduler limitations?
Apache uses a pre-fork "threading" mechanism, it spawns (fork()s) new instances of itself whenever it finds out that the number of idle "threads" is below a certain (configurable) threshold. Despite of all apparences this method performs beautifully on Linux, pthreads are actually slower in many cases, since you will incur some additional overhead due to thread synchronization and scheduling. The problem is that beyond a certain number of processes the scheduler just goes bananas, or so it seems to me. Since Linux threads are mapped on processes, I don't think that (p)threads woud help in any way, unless it is the VM context switch overhead that is playing a role here, which I wouldn't think is the case. - Fabio "J . A . Magallon" wrote: > On 03.29 Fabio Riccardi wrote: > > > > I've found a (to me) unexplicable system behaviour when the number of > > Apache forked instances goes somewhere beyond 1050, the machine > > suddently slows down almost top a halt and becomes totally unresponsive, > > until I stop the test (SpecWeb). > > > > Have you though about pthreads (when you talk about fork, I suppose you > say literally 'fork()') ? > > I give a course on Parallel Programming at the university and the practical > work was done with POSIX threads. One of my students caught the idea and > used it to modify his assignment from one other matter on Networks, and > changed the traditional 'fork()' in a simple ftp server he had to implement > by 'pthread_create' and got a 10-30 speedup (conns per second). > > And you will get rid of some process-per-user limit. But you will fall into > an threads-per-user limit, if there is any. > > And you cal also control its scheduling, to make each thread fight against > the whole system or only its siblings. > > -- > J.A. Magallon # Let the source > mailto:[EMAIL PROTECTED] # be with you, Luke... > > Linux werewolf 2.4.2-ac28 #1 SMP Thu Mar 29 16:41:17 CEST 2001 i686 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: linux scheduler limitations?
I'm using 2.4.2-ac26, but I've noticed the same behavior with all the 2.4 kernels I've seen so far. I haven't even tried on 2.2 - Fabio David Lang wrote: > 2.2 or 2.4 kernel? > > the 2.4 does a MUCH better job of dealing with large numbers of processes. > > David Lang > > On Thu, 29 Mar 2001, Fabio Riccardi wrote: > > > Date: Thu, 29 Mar 2001 13:19:05 -0800 > > From: Fabio Riccardi <[EMAIL PROTECTED]> > > To: [EMAIL PROTECTED] > > Subject: linux scheduler limitations? > > > > Hello, > > > > I'm working on an enhanced version of Apache and I'm hitting my head > > against something I don't understand. > > > > I've found a (to me) unexplicable system behaviour when the number of > > Apache forked instances goes somewhere beyond 1050, the machine > > suddently slows down almost top a halt and becomes totally unresponsive, > > until I stop the test (SpecWeb). > > > > Profiling the kernel shows that the scheduler and the interrupt handler > > are taking most of the CPU time. > > > > I understand that there must be a limit to the number of processes that > > the scheduler can efficiently handle, but I would expect some sort of > > gradual performance degradation when increasing the number of tasks, > > instead I observe that by increasing Apache's MaxClient linit by as > > little as 10 can cause a sudden transition between smooth working with > > lots (30-40%) of CPU idle to a total lock-up. > > > > Moreover the max number of processes is not even constant. If I increase > > the server load gradually then I manage to have 1500 processes running > > with no problem, but if the transition is sharp (the SpecWeb case) than > > I end-up having a lock up. > > > > Anybody seen this before? Any clues? > > > > - Fabio > > > > > > - > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to [EMAIL PROTECTED] > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > > - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
linux scheduler limitations?
Hello, I'm working on an enhanced version of Apache and I'm hitting my head against something I don't understand. I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb). Profiling the kernel shows that the scheduler and the interrupt handler are taking most of the CPU time. I understand that there must be a limit to the number of processes that the scheduler can efficiently handle, but I would expect some sort of gradual performance degradation when increasing the number of tasks, instead I observe that by increasing Apache's MaxClient linit by as little as 10 can cause a sudden transition between smooth working with lots (30-40%) of CPU idle to a total lock-up. Moreover the max number of processes is not even constant. If I increase the server load gradually then I manage to have 1500 processes running with no problem, but if the transition is sharp (the SpecWeb case) than I end-up having a lock up. Anybody seen this before? Any clues? - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
linux scheduler limitations?
Hello, I'm working on an enhanced version of Apache and I'm hitting my head against something I don't understand. I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb). Profiling the kernel shows that the scheduler and the interrupt handler are taking most of the CPU time. I understand that there must be a limit to the number of processes that the scheduler can efficiently handle, but I would expect some sort of gradual performance degradation when increasing the number of tasks, instead I observe that by increasing Apache's MaxClient linit by as little as 10 can cause a sudden transition between smooth working with lots (30-40%) of CPU idle to a total lock-up. Moreover the max number of processes is not even constant. If I increase the server load gradually then I manage to have 1500 processes running with no problem, but if the transition is sharp (the SpecWeb case) than I end-up having a lock up. Anybody seen this before? Any clues? - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: linux scheduler limitations?
I'm using 2.4.2-ac26, but I've noticed the same behavior with all the 2.4 kernels I've seen so far. I haven't even tried on 2.2 - Fabio David Lang wrote: 2.2 or 2.4 kernel? the 2.4 does a MUCH better job of dealing with large numbers of processes. David Lang On Thu, 29 Mar 2001, Fabio Riccardi wrote: Date: Thu, 29 Mar 2001 13:19:05 -0800 From: Fabio Riccardi [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: linux scheduler limitations? Hello, I'm working on an enhanced version of Apache and I'm hitting my head against something I don't understand. I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb). Profiling the kernel shows that the scheduler and the interrupt handler are taking most of the CPU time. I understand that there must be a limit to the number of processes that the scheduler can efficiently handle, but I would expect some sort of gradual performance degradation when increasing the number of tasks, instead I observe that by increasing Apache's MaxClient linit by as little as 10 can cause a sudden transition between smooth working with lots (30-40%) of CPU idle to a total lock-up. Moreover the max number of processes is not even constant. If I increase the server load gradually then I manage to have 1500 processes running with no problem, but if the transition is sharp (the SpecWeb case) than I end-up having a lock up. Anybody seen this before? Any clues? - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: linux scheduler limitations?
Apache uses a pre-fork "threading" mechanism, it spawns (fork()s) new instances of itself whenever it finds out that the number of idle "threads" is below a certain (configurable) threshold. Despite of all apparences this method performs beautifully on Linux, pthreads are actually slower in many cases, since you will incur some additional overhead due to thread synchronization and scheduling. The problem is that beyond a certain number of processes the scheduler just goes bananas, or so it seems to me. Since Linux threads are mapped on processes, I don't think that (p)threads woud help in any way, unless it is the VM context switch overhead that is playing a role here, which I wouldn't think is the case. - Fabio "J . A . Magallon" wrote: On 03.29 Fabio Riccardi wrote: I've found a (to me) unexplicable system behaviour when the number of Apache forked instances goes somewhere beyond 1050, the machine suddently slows down almost top a halt and becomes totally unresponsive, until I stop the test (SpecWeb). Have you though about pthreads (when you talk about fork, I suppose you say literally 'fork()') ? I give a course on Parallel Programming at the university and the practical work was done with POSIX threads. One of my students caught the idea and used it to modify his assignment from one other matter on Networks, and changed the traditional 'fork()' in a simple ftp server he had to implement by 'pthread_create' and got a 10-30 speedup (conns per second). And you will get rid of some process-per-user limit. But you will fall into an threads-per-user limit, if there is any. And you cal also control its scheduling, to make each thread fight against the whole system or only its siblings. -- J.A. Magallon # Let the source mailto:[EMAIL PROTECTED] # be with you, Luke... Linux werewolf 2.4.2-ac28 #1 SMP Thu Mar 29 16:41:17 CEST 2001 i686 - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: linux scheduler limitations?
"J . A . Magallon" wrote: It all depends on your app, as every parallel algorithm. In a web-ftp-whatever server, you do not need any synchro. You can start threads in free run and let them die alone. even if you don't need synchronization you pay for it anyway, since you will have to use the pthread version of libc that is reentrant. Moreover many calls (i.e. accept) are "scheduling points" for pthreads, whenever you call them the runtime will perform quite a bit of bookeeping. it is instructive to use a profiler on your application and see what happens when you use pthreads... You said, 'mapped'. AFAIK, that is the advantage, you can avoid the VM switch by sharing memory. If your application uses lots of memory than I agree, Apache only uses a tiny amount of RAM per instance though, so I don't think that that is my case. - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: linux scheduler limitations?
Hi Mike, somebody else on the list already pointed me at your stuff and I quickly downloaded your multiqueue patch for 2.4.1 to try it out. It works great! I finally manage to have 100% CPU utilization and keep the machine decently responsive. On a two 1GHz pentium box i went from 1300 specweb to 1600. That's pretty amazing. There is a bit more overhead though, I'd say arount 5%, when the CPU is not fully loaded. What is the status of your code? Is it going to end-up in the mainstream kernel? Do you have a port to the 2.4.2x kernels? In my enthousiasm I tried to port the patch to 2.4.2-ac26 but I broke something and it didn't work anymore... :) I havent't tried the pooling patch yet, it didn't seem to make much sense on a 2-way box. I have an 8-way on which I'm planning to bench my web server enhancements, I'll try the pooling stuff on it. BTW: interested in the fastest linux web server? BTW2: what about the HP scheduler patches? Thanks, ciao, - Fabio Mike Kravetz wrote: On Thu, Mar 29, 2001 at 01:55:11PM -0800, Fabio Riccardi wrote: I'm using 2.4.2-ac26, but I've noticed the same behavior with all the 2.4 kernels I've seen so far. I haven't even tried on 2.2 - Fabio Fabio, Just for fun, you might want to try out some of our scheduler patches located at: http://lse.sourceforge.net/scheduling/ I would be interested in your observations. -- Mike Kravetz [EMAIL PROTECTED] IBM Linux Technology Center - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: user space web server accelerator support
Dave, Zach, thanks for your help, I've implemented a file descriptor passing mechanism very similar to that of Zach's and it worked. The problem now is performance, fd passing is utterly slow! On my system (a 1GHz Pentium III + 2G RAM) I can do 1300 SpecWeb99 with a khttp-like socket passing mechanism, while I only get something like 500 using file descriptor passing. Indeed with fd passing I decrease Apache's performance instead of increasing it! I've checked my code several times and I don't believe that I have introduced any specific bottleneck of my own (the code actually is quite trivial). I've profiled the kernel and some interesting differences show: With direct socket passing, 1300 SpecWeb load: 9759 total 0.0071 902 handle_IRQ_event 7.5167 256 skb_clone 0.6957 256 do_tcp_sendpages 0.0954 239 tcp_v4_rcv 0.1572 238 schedule 0.1766 226 __kfree_skb0.9741 207 skb_release_data 1.7845 204 tcp_transmit_skb 0.1541 199 d_lookup 0.6910 190 path_walk 0.0973 181 ip_output 0.6754 168 fget 2.2105 165 do_softirq 1.1786 158 do_generic_file_read 0.1287 With file descriptor passing, 500 SpecWeb load: 8621 total 0.0063 7037 schedule 5.2203 462 handle_IRQ_event 3.8500 188 __wake_up 0.9216 114 unix_stream_data_wait 0.4191 81 __switch_to0.3750 58 schedule_timeout 0.3718 25 d_lookup 0.0868 20 skb_clone 0.0543 19 path_walk 0.0097 17 tcp_transmit_skb 0.0128 17 do_tcp_sendpages 0.0063 17 do_softirq 0.1214 15 system_call0.2679 15 sys_rt_sigtimedwait0.0207 Zach, have you ever noticed such a performance bottleneck in your phhttpd? SpecWeb has about 30% of its load as dynamic requests, so the amount of forwarding is definitively significative in my case. Sime time ago I measured khttp's impact in socket passing and I found that it was negligible (forwarding everything to Apache instead of having it directly listening on the socket had an impact of a few percent). My impression from a first look to the profiling data is that the kernel is doing a very poor job of scheduling and is ping-ponging between processes... like it is not doing any buffering whatsoever and it is doing a contect switch for every passed file descriptor. Any thoughts? - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: user space web server accelerator support
Dave, Zach, thanks for your help, I've implemented a file descriptor passing mechanism very similar to that of Zach's and it worked. The problem now is performance, fd passing is utterly slow! On my system (a 1GHz Pentium III + 2G RAM) I can do 1300 SpecWeb99 with a khttp-like socket passing mechanism, while I only get something like 500 using file descriptor passing. Indeed with fd passing I decrease Apache's performance instead of increasing it! I've checked my code several times and I don't believe that I have introduced any specific bottleneck of my own (the code actually is quite trivial). I've profiled the kernel and some interesting differences show: With direct socket passing, 1300 SpecWeb load: 9759 total 0.0071 902 handle_IRQ_event 7.5167 256 skb_clone 0.6957 256 do_tcp_sendpages 0.0954 239 tcp_v4_rcv 0.1572 238 schedule 0.1766 226 __kfree_skb0.9741 207 skb_release_data 1.7845 204 tcp_transmit_skb 0.1541 199 d_lookup 0.6910 190 path_walk 0.0973 181 ip_output 0.6754 168 fget 2.2105 165 do_softirq 1.1786 158 do_generic_file_read 0.1287 With file descriptor passing, 500 SpecWeb load: 8621 total 0.0063 7037 schedule 5.2203 462 handle_IRQ_event 3.8500 188 __wake_up 0.9216 114 unix_stream_data_wait 0.4191 81 __switch_to0.3750 58 schedule_timeout 0.3718 25 d_lookup 0.0868 20 skb_clone 0.0543 19 path_walk 0.0097 17 tcp_transmit_skb 0.0128 17 do_tcp_sendpages 0.0063 17 do_softirq 0.1214 15 system_call0.2679 15 sys_rt_sigtimedwait0.0207 Zach, have you ever noticed such a performance bottleneck in your phhttpd? SpecWeb has about 30% of its load as dynamic requests, so the amount of forwarding is definitively significative in my case. Sime time ago I measured khttp's impact in socket passing and I found that it was negligible (forwarding everything to Apache instead of having it directly listening on the socket had an impact of a few percent). My impression from a first look to the profiling data is that the kernel is doing a very poor job of scheduling and is ping-ponging between processes... like it is not doing any buffering whatsoever and it is doing a contect switch for every passed file descriptor. Any thoughts? - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: user space web server accelerator support
Fantastic! I was not aware of it, sorry... where can I find some doc? - Fabio "David S. Miller" wrote: > Fabio Riccardi writes: > > How can Apache "grab" the file descriptor? > > > > My understanding is that file descriptors are data structures private to > > a process... > > > > Am I missing something? > > Unix sockets allow one processes to "give" a file descriptor to > another process via a facility called "file descriptor passing". > > Later, > David S. Miller > [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: user space web server accelerator support
How can Apache "grab" the file descriptor? My understanding is that file descriptors are data structures private to a process... Am I missing something? - Fabio "David S. Miller" wrote: > Fabio Riccardi writes: > > How could this be fixed? > > Why not pass the filedescriptors to apache over a UNIX domain > socket? I see no need for a new facility. > > Later, > David S. Miller > [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
user space web server accelerator support
Hi, I've been working for a while on a user-space web server accelerator (as opposed to a kernel space accelerator, like TUX). So far I've had very promising results and I can achieve performance (spec) figures comparable to those of TUX. Although my implementation is entirely sitting in user space, I need some cooperation form the kernel for efficiently forwarding network connections from the accelerator to the full-fledged Apache server. I've made a little kernel hack (mostly lifted out of the TUX and khttpd code) to forward a live socket connection from an application to another. I'd like to clean this up such that my users don't have to mock with their kernel to get my accelerator to work. Would it be a major heresy to ask for a new system call? If so I could still hide my stuff in a kernel module and snatch an unused kernel call for my private use (such as the one allotted for tux). The problem with this is that the kernel only exposes the "right" symbols to the modules if either khttp or ipv6 are compiled as modules. How could this be fixed? TIA, ciao, - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
user space web server accelerator support
Hi, I've been working for a while on a user-space web server accelerator (as opposed to a kernel space accelerator, like TUX). So far I've had very promising results and I can achieve performance (spec) figures comparable to those of TUX. Although my implementation is entirely sitting in user space, I need some cooperation form the kernel for efficiently forwarding network connections from the accelerator to the full-fledged Apache server. I've made a little kernel hack (mostly lifted out of the TUX and khttpd code) to forward a live socket connection from an application to another. I'd like to clean this up such that my users don't have to mock with their kernel to get my accelerator to work. Would it be a major heresy to ask for a new system call? If so I could still hide my stuff in a kernel module and snatch an unused kernel call for my private use (such as the one allotted for tux). The problem with this is that the kernel only exposes the "right" symbols to the modules if either khttp or ipv6 are compiled as modules. How could this be fixed? TIA, ciao, - Fabio - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
