Re: NET.ISR and CPU utilization performance w/ HP DL 585 using FreeBSD 7.1 Beta2
> - Original Message > From: Ivan Voras <[EMAIL PROTECTED]> > To: freebsd-hackers@freebsd.org > Sent: Tuesday, November 18, 2008 5:21:04 AM > Subject: Re: NET.ISR and CPU utilization performance w/ HP DL 585 using > FreeBSD 7.1 Beta2 > > Won De Erick wrote: > > > I compiled the following em driver for Intel NIC Pro (82571) w/ FreeBSD 7.1 > > Beta 2 on HPDL 585 machine having 16CPUs. > > > > http://people.yandex-team.ru/~wawa/ > > > > With net.isr.direct=1, I made some changes on kthreads(default=2) for em0 > > and em1's rx. > > > > dev.em.0.rx_kthreads: 6 > > > > dev.em.1.rx_kthreads: 6 > > > > With these settings, the result is: > > > > CPU: 0.0% user, 0.0% nice, 57.2% system, 3.6% interrupt, 39.2% idle > > Mem: 17M Active, 7228K Inact, 156M Wired, 76K Cache, 21M Buf, 31G Free > > Swap: 4096M Total, 4096M Free > > > > PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND > >52 root1 43- 0K16K CPU12 c 41:38 100.00% > > em0_rx_kthread_1 > >51 root1 43- 0K16K CPU3 3 41:38 100.00% > > em0_rx_kthread_0 > >54 root1 -68- 0K16K CPU2 2 39:39 100.00% > > em1_txcleaner > > 1283 root1 43- 0K16K CPU1 1 38:55 100.00% > > em0_rx_kthread_3 > > 1282 root1 43- 0K16K CPU10 a 38:55 100.00% > > em0_rx_kthread_2 > > 1344 root1 43- 0K16K CPU9 9 25:51 100.00% > > em0_rx_kthread_5 > > 1343 root1 43- 0K16K CPU4 4 25:51 100.00% > > em0_rx_kthread_4 > >12 root1 171 ki31 0K16K CPU14 e 44:28 91.70% idle: > > cpu14 > > This is very interesting. Do you see real performance (network > throughput) increase? > There's a little improvement on the throughput, but packet errors occurred on both interfaces. # netstat -I em1 -w 1 -d input (em1) output packets errs bytespackets errs bytes colls drops 32494 483 23083087 15681 0 23719154 082 30547 330 23104447 16062 0 23077442 044 # netstat -I em0 -w 1 -d input (em0) output packets errs bytespackets errs bytes colls drops 19889 640 24144754 21307 08719922 0 0 18071 2436 25966238 21088 08766995 0 0 Is there any other thing that I can tweak to solve the problem? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: How can I add new binaries to the mfsroot image?
>I'll have to check this out. I'm not getting anywhere with trying to >customize mfsroot with my current approach... The goal we are trying to achieve btw is to make gmirror available during an install so that the file systems are mirrored right from the get-go, so that we can avoid having to go through the process of converting a system as a post operation. The standard slicing/partition commands of sysinstall do *not* support the creation of a mirrored file system though, so our idea was to run a script via install.cfg to take care of fdisk/bsdlabel/gmirror phase, and then install the packages in the normal fashion via subsequent steps in install.cfg. Is this something that can be done via sysinstall? If not, what's the best alternative? This whole process is targeted to be on a PXE boot server so we can configure our systems in a completely automated hands-off manner. We have 200+ FreeBSD systems and we definitely need an automated process. We already have it working fine, but without mirroring. We can upgrade doezens of systems at a time simply by making them boot from our PXE server. We now need to tweak this process so that we can establish the mirrored file systems as part of the automated install. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: How can I add new binaries to the mfsroot image?
>I'll have to check this out. I'm not getting anywhere with trying to >customize mfsroot with my current approach... The goal we are trying to achieve btw is to make gmirror available during an install so that the file systems are mirrored right from the get-go, so that we can avoid having to go through the process of converting a system as a post operation. The standard slicing/partition commands of sysinstall do support the creation of a mirrored file system though, so our idea was to run a script via install.cfg to take care of fdisk/bsdlabel/gmirror phase, and then install the packages in the normal fashion via subsequent steps in install.cfg. Is this something that can be done via sysinstall? If not, what's the best alternative? This whole process is targeted to be on a PXE boot server so we can configure our systems in a completely automated hands-off manner. We have 200+ FreeBSD systems and we definitely need an automated process. We already have it working fine, but without mirroring. We can upgrade doezens of systems at a time simply by making them boot from our PXE server. We now need to tweak this process so that we can establish the mirrored file systems as part of the automated install. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: NET.ISR and CPU utilization performance w/ HP DL 585 using FreeBSD 7.1 Beta2
Won De Erick wrote: > I compiled the following em driver for Intel NIC Pro (82571) w/ FreeBSD 7.1 > Beta 2 on HPDL 585 machine having 16CPUs. > > http://people.yandex-team.ru/~wawa/ > > With net.isr.direct=1, I made some changes on kthreads(default=2) for em0 and > em1's rx. > > dev.em.0.rx_kthreads: 6 > > dev.em.1.rx_kthreads: 6 > > With these settings, the result is: > > CPU: 0.0% user, 0.0% nice, 57.2% system, 3.6% interrupt, 39.2% idle > Mem: 17M Active, 7228K Inact, 156M Wired, 76K Cache, 21M Buf, 31G Free > Swap: 4096M Total, 4096M Free > > PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND >52 root1 43- 0K16K CPU12 c 41:38 100.00% > em0_rx_kthread_1 >51 root1 43- 0K16K CPU3 3 41:38 100.00% > em0_rx_kthread_0 >54 root1 -68- 0K16K CPU2 2 39:39 100.00% > em1_txcleaner > 1283 root1 43- 0K16K CPU1 1 38:55 100.00% > em0_rx_kthread_3 > 1282 root1 43- 0K16K CPU10 a 38:55 100.00% > em0_rx_kthread_2 > 1344 root1 43- 0K16K CPU9 9 25:51 100.00% > em0_rx_kthread_5 > 1343 root1 43- 0K16K CPU4 4 25:51 100.00% > em0_rx_kthread_4 >12 root1 171 ki31 0K16K CPU14 e 44:28 91.70% idle: cpu14 This is very interesting. Do you see real performance (network throughput) increase? signature.asc Description: OpenPGP digital signature
Re: Giant lock, bce and uhc using the same irq
On Monday 17 November 2008, Ed Schouten wrote: > * Kris Kennaway <[EMAIL PROTECTED]> wrote: > > P.S. In 8.0 there is a new USB stack that is Giant-free. > > I'm not sure this is completely true. Maybe HPS could explain it in more > detail, but Giant still seems to be used pretty often. All the interrupt handlers of the Host- and Device-controller drivers are free from locking Giant when executing. --HPS ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Giant lock, bce and uhc using the same irq
* Kris Kennaway <[EMAIL PROTECTED]> wrote: > P.S. In 8.0 there is a new USB stack that is Giant-free. I'm not sure this is completely true. Maybe HPS could explain it in more detail, but Giant still seems to be used pretty often. -- Ed Schouten <[EMAIL PROTECTED]> WWW: http://80386.nl/ pgpCr3BoUF1P0.pgp Description: PGP signature
Re: Ответ: KLD loading, liking
* Aleksandr Litvinov <[EMAIL PROTECTED]> wrote: > Hello, > You can receive a little information about KLD from the book > "designing BSD rootkits". I don't own this book myself, but a colleague at Snow B.V. once showed it to me. I only looked through it a couple of minutes, but it seemed like a book nice to have. It also shows some techniques on how to hide KLD's. -- Ed Schouten <[EMAIL PROTECTED]> WWW: http://80386.nl/ pgpWDDjidvjLc.pgp Description: PGP signature
Re: KLD loading, liking
On Mon, Nov 17, 2008 at 02:34:28PM +0100, Dag-Erling Smørgrav wrote: > Alexej Sokolov <[EMAIL PROTECTED]> writes: > > Not only that. The use of the MODULE_DEPEND macro allows one module to > > access the variables of modules on which it depends. > > No. Any module X can access any public variable or function in the > kernel or in any other module Y, but loading X will fail if Y is not > already loaded. The only effect of MODULE_DEPEND is to tell the loader > that Y must be loaded before X. No, Example: two modules kld.c and kld1.1: kld:http://pastebin.com/m67799565Makefile: http://pastebin.com/m5418e5a7 kld1: http://pastebin.com/d154e8474Makefile: http://pastebin.com/m79723138 In kld is public int var_from_kld declared. To access this variable from kld1 you have to uncomment macro MODULE_DEPEND My system: % uname -v FreeBSD 7.0-RELEASE-p5 #0: Tue Oct 7 19:05:20 CEST 2008 Tell me please if I do something wrong! Thanx > > DES > -- > Dag-Erling Smørgrav - [EMAIL PROTECTED] > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "[EMAIL PROTECTED]" -- Alexandre Fiveg <[EMAIL PROTECTED]> Key fingerprint = 0B23 EB52 3944 E440 CFF3 C1F1 7D05 8D00 34F7 A6BD ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
RE: How can I add new binaries to the mfsroot image?
>I'm not sure, but probably the installation CD doesn't carry shared libraries at all? All binaries in /stand are >static-linked ones. Yeah, that is absolutely the problem--no shared libraries are available when sysinstall is running. >You could also try scripts from mfsbsd project: >http://people.freebsd.org/~mm/mfsbsd/ > >These works for me fine for building custom installation CDs. I'll have to check this out. I'm not getting anywhere with trying to customize mfsroot with my current approach... ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Giant lock, bce and uhc using the same irq
On Mon, Nov 17, 2008 at 05:14:21PM +0200, Murat Balaban wrote: > Hello hackers, > > In one of my production servers (64-bit Intel Xeon machine) running > > 6.3-RELEASE-p4 (amd64) FreeBSD 6.3-RELEASE-p4 #0: Fri Sep 12 17:07:19 > EEST 2008 > > I see this "top -S" output excerpt: > > 32 root 1 -68 -187 0K16K *Giant 0 48.3H 3.08% irq17: > bce1 uhci1 > > What I get from above is that bce and uhc drivers both use irq17. I > assume > bce is SMPng'ed, and uhc is not. > > Does uhc being Giant locked, affect bce performance, because they use > the same > interrupt handler thread? bce will not need to acquire Giant at all so they will not fight for the lock. However both of them will have to wake up to check each interrupt so there is some time spent there. Also if you have some other workload on the system that is still heavily dependent onGiant (e.g. MSDOSFS, etc) then that will interfere with uhci while interrupts are coming in. Kris P.S. In 8.0 there is a new USB stack that is Giant-free. -- In God we Trust -- all others must submit an X.509 certificate. -- Charles Forsythe <[EMAIL PROTECTED]> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Ответ: KLD loading, liking
2008/11/17, Alexej Sokolov <[EMAIL PROTECTED]>: > On Sun, Nov 16, 2008 at 11:09:00AM +0100, Ed Schouten wrote: >> * Alexej Sokolov <[EMAIL PROTECTED]> wrote: > >> > What exact does the macro MODULE_DEPEND ? The man page is to short, and >> > I >> > guess it tell no all things that the macro does. >> >> MODULE_DEPEND is used to say: this kernel module also depends on another >> module (i.e. the USB printer module depends on the USB code). Tools like >> kldload can then automatically load the missing modules. > Not only that. The use of the MODULE_DEPEND macro allows one module to > access > the variables of modules on which it depends. But man page of > MODULE_DEPEND doesn't tell anything about this functionality. Hence I > am looking for any good documentation of KLD loader. But I didn't find > anything. May be looking in the source code is the best solution. > >> >> -- >> Ed Schouten <[EMAIL PROTECTED]> >> WWW: http://80386.nl/ > > > > -- > Alexej Sokolov <[EMAIL PROTECTED]> > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "[EMAIL PROTECTED]" > Hello, You can receive a little information about KLD from the book "designing BSD rootkits". -- -- Good Luck. -- Litvinov Aleksandr. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Giant lock, bce and uhc using the same irq
Hello hackers, In one of my production servers (64-bit Intel Xeon machine) running 6.3-RELEASE-p4 (amd64) FreeBSD 6.3-RELEASE-p4 #0: Fri Sep 12 17:07:19 EEST 2008 I see this "top -S" output excerpt: 32 root 1 -68 -187 0K16K *Giant 0 48.3H 3.08% irq17: bce1 uhci1 What I get from above is that bce and uhc drivers both use irq17. I assume bce is SMPng'ed, and uhc is not. Does uhc being Giant locked, affect bce performance, because they use the same interrupt handler thread? -- Murat http://www.enderunix.org/murat/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: KLD loading, liking
Alexej Sokolov <[EMAIL PROTECTED]> writes: > Not only that. The use of the MODULE_DEPEND macro allows one module to > access the variables of modules on which it depends. No. Any module X can access any public variable or function in the kernel or in any other module Y, but loading X will fail if Y is not already loaded. The only effect of MODULE_DEPEND is to tell the loader that Y must be loaded before X. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: NET.ISR and CPU utilization performance w/ HP DL 585 using FreeBSD 7.1 Beta2
> - Original Message > From: Won De Erick <[EMAIL PROTECTED]> > To: Jeremy Chadwick <[EMAIL PROTECTED]> > Cc: [EMAIL PROTECTED]; freebsd-hackers@freebsd.org > Sent: Sunday, November 16, 2008 7:18:46 PM > Subject: Re: NET.ISR and CPU utilization performance w/ HP DL 585 using > FreeBSD 7.1 Beta2 > > > - Original Message > > > From: Jeremy Chadwick <[EMAIL PROTECTED]> > > To: Won De Erick <[EMAIL PROTECTED]> > > Cc: [EMAIL PROTECTED]; freebsd-hackers@freebsd.org > > Sent: Saturday, November 15, 2008 10:16:31 PM > > Subject: Re: NET.ISR and CPU utilization performance w/ HP DL 585 using > > FreeBSD 7.1 Beta2 > > > > On Sat, Nov 15, 2008 at 04:59:16AM -0800, Won De Erick wrote: > > > Hello, > > > > > > I tested HP DL 585 (16 CPUs, w/ built-in Broadcom NICs) running FreeBSD > > > 7.1 Beta2 under heavy network traffic (TCP). > > > > > > SCENARIO A : Bombarded w/ TCP traffic: > > > > > > When net.isr.direct=1, > > > > > > PID USERNAME THR PRI NICE SIZERES STATE C TIME WCPU COMMAND > > >52 root1 -68- 0K16K CPU11 b 38:43 95.36% irq32: > > > bce1 > > >51 root1 -68- 0K16K CPU10 a 25:50 85.16% irq31: > > > bce0 > > >16 root1 171 ki31 0K16K RUNa 65:39 15.97% idle: > > > cpu10 > > >28 root1 -32- 0K16K WAIT 8 12:28 5.18% swi4: > > > clock sio > > >15 root1 171 ki31 0K16K RUNb 52:46 3.76% idle: > > > cpu11 > > >45 root1 -64- 0K16K WAIT 7 7:29 1.17% irq17: > > > uhci0 > > >47 root1 -64- 0K16K WAIT 6 1:11 0.10% irq16: > > > ciss0 > > >27 root1 -44- 0K16K WAIT 0 28:52 0.00% swi1: > > > net > > > > > > When net.isr.direct=0, > > > > > >16 root1 171 ki31 0K16K CPU10 a 106:46 92.58% idle: > > > cpu10 > > >19 root1 171 ki31 0K16K CPU7 7 133:37 89.16% idle: > > > cpu7 > > >27 root1 -44- 0K16K WAIT 0 52:20 76.37% swi1: > > > net > > >25 root1 171 ki31 0K16K RUN1 132:30 70.26% idle: > > > cpu1 > > >26 root1 171 ki31 0K16K CPU0 0 111:58 64.36% idle: > > > cpu0 > > >15 root1 171 ki31 0K16K CPU11 b 81:09 57.76% idle: > > > cpu11 > > >52 root1 -68- 0K16K WAIT b 64:00 42.97% irq32: > > > bce1 > >51 root1 -68- 0K16K WAIT a 38:22 12.26% irq31: > > bce0 > > >45 root1 -64- 0K16K WAIT 7 11:31 12.06% irq17: > > > uhci0 > > >47 root1 -64- 0K16K WAIT 6 1:54 3.66% irq16: > > > ciss0 > > >28 root1 -32- 0K16K WAIT 8 16:01 0.00% swi4: > > > clock sio > > > > > > Overall CPU utilization has significantly dropped, but I noticed that > > > swi1 has taken CPU0 with high utilization when the net.isr.direct=0. > > > What does this mean? > > > > > > SCENARIO B : Bombarded w/ more TCP traffic: > > > > > > Worst thing, the box has become unresponsive (can't be PINGed, > > > inaccessible through SSH) after more traffic was added retaining > > > net.isr.direct=0. > > > This is due maybe to the 100% utilization on CPU0 for sw1:net (see below > > > result, first line). bce's and swi's seem to race each other based on the > > > > result when net.isr.direct=1, swi1 . > > > The rest of the CPUs are sitting pretty (100% Idle). Can you shed some > > > lights on this? > > > > > > When net.isr.direct=0: > > >27 root1 -44- 0K16K CPU0 0 5:45 100.00% swi1: > > > net > > >11 root1 171 ki31 0K16K CPU15 0 0:00 100.00% idle: > > > cpu15 > > >13 root1 171 ki31 0K16K CPU13 0 0:00 100.00% idle: > > > cpu13 > > >17 root1 171 ki31 0K16K CPU9 0 0:00 100.00% idle: > > > cpu9 > > >18 root1 171 ki31 0K16K CPU8 0 0:00 100.00% idle: > > > cpu8 > > >21 root1 171 ki31 0K16K CPU5 5 146:17 99.17% idle: > > > cpu5 > > >22 root1 171 ki31 0K16K CPU4 4 146:17 99.07% idle: > > > cpu4 > > >14 root1 171 ki31 0K16K CPU12 0 0:00 99.07% idle: > > > cpu12 > > >16 root1 171 ki31 0K16K CPU10 a 109:33 98.88% idle: > > > cpu10 > > >15 root1 171 ki31 0K16K CPU11 b 86:36 93.55% idle: > > > cpu11 > > >52 root1 -68- 0K16K WAIT b 59:42 13.87% irq32: > > > bce1 > > > > > > When net.isr.direct=1, > > >52 root1 -68- 0K16K CPU11 b 55:04 97.66% irq32: > > > bce1 > > >51 root1 -68- 0K16K CPU10 a 33:52 73.88% irq31: > > > bce0 > > >16 root1 171 ki31 0K16K RUNa 102:42 26.86% idle: > > > cpu10 > > >15 root1 171 ki31 0K16K RUNb 81:20 3.17% idle: > > > cpu11 > > >28 root1 -32-
Re: KLD loading, liking
On Sun, Nov 16, 2008 at 11:09:00AM +0100, Ed Schouten wrote: > * Alexej Sokolov <[EMAIL PROTECTED]> wrote: > > What exact does the macro MODULE_DEPEND ? The man page is to short, and I > > guess it tell no all things that the macro does. > > MODULE_DEPEND is used to say: this kernel module also depends on another > module (i.e. the USB printer module depends on the USB code). Tools like > kldload can then automatically load the missing modules. Not only that. The use of the MODULE_DEPEND macro allows one module to access the variables of modules on which it depends. But man page of MODULE_DEPEND doesn't tell anything about this functionality. Hence I am looking for any good documentation of KLD loader. But I didn't find anything. May be looking in the source code is the best solution. > > -- > Ed Schouten <[EMAIL PROTECTED]> > WWW: http://80386.nl/ -- Alexej Sokolov <[EMAIL PROTECTED]> ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: How can I add new binaries to the mfsroot image?
On Sun, 16 Nov 2008 14:59:29 -0800 "Peter Steele" <[EMAIL PROTECTED]> mentioned: > > What's weird is that I can open a fixit shell after the install.cfg > script fails and then run the same commands interactively and they work > fine. Why would work these commands work in an interactive fixit shell > but not during the automated sysinstall session? > I'm not sure, but probably the installation CD doesn't carry shared libraries at all? All binaries in /stand are static-linked ones. You could also try scripts from mfsbsd project: http://people.freebsd.org/~mm/mfsbsd/ These works for me fine for building custom installation CDs. -- Stanislav Sedov ST4096-RIPE pgpua59y8DHtG.pgp Description: PGP signature
