Re: [call for helpers!] Tuning for the Beaver Challenge
Beaver Challenge 2004 is coming!. Details in http://osuosl.org/benchmarks/bc/ We are preparing the tuning guide. Definitely we need suggestions and comments. does the sysctl 'machdep.cpu_idle_hlt' still have any effect? see: http://www.FreeBSD.org/cgi/getmsg.cgi?fetch=36729+40701+/usr/local/www/db/text/2003/freebsd-hackers/20031012.freebsd-hackers Timestamp: 0x40288509 [SorAlx] http://cydem.org.ua/ ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Re: [call for helpers!] Tuning for the Beaver Challenge
Thanks for comment.
- don't use apm or acpi on 4.x
There will be no benefit in terms of performance using acpi on 4.x.
And apm is just unnecessary.
Patrick
-Original Message-
From: [EMAIL PROTECTED] (Dag-Erling Smrgrav)
To: Dung Patrick [EMAIL PROTECTED]
Date: Mon, 09 Feb 2004 19:27:08 +0100
Subject: Re: [call for helpers!] Tuning for the Beaver Challenge
Dung Patrick [EMAIL PROTECTED] writes:
For those optionts CPU_FASTER_5X86_FPU through CPU_UPGRADE_HW_CACHE,
it is suggested by an user in the forum. I would like to see the
comments from the mailing list. If those options are dangerous, then
don't use them.
CPU_FASTER_5X86_FPU is not likely to have any positive impact on
performance, and fairly likely to render the system unbootable.
CPU_UPGRADE_HW_CACHE has absolutely no effect on non-PC98 systems.
further comments -
- symlink /usr/{src,obj,ports} to /slow/{src,obj,ports} where /slow
is a filesystem placed on the outside edge of the disk (to free up
space near the spindle for the filesystems actually used by the
benchmark)
- find out what file system the Linux people are using. if they are
using ext2fs or ext3fs, mount your filesystems async.
- most of what you put in sysctl.conf is completely irrelevant. the
rest needs to be tuned according to the actual needs of the
benchmark.
- you should not tune kern.maxfiles etc. unless the benchmark
actually hits those limits. increasing these numbers reduces the
amount of kernel memory available for other purposes.
btw, maxfiles and maxfilesperproc are tunable at run time.
- most of what you put in make.conf is bogus. just use
CPUTYPE ?= pentiumpro
CFLAGS = -O -pipe
COPTFLAGS = -O -pipe
- NOPROFILE has absolutely no impact on performance (except that it
shortens 'make world' a little)
- you *must* use -CURRENT and not 5.2 as the latter has issues with
the aac driver.
- don't use apm or acpi on 4.x.
- regarding jdk 1.4.2, just use the linux version (and make sure to
mount linprocfs). I very much doubt you'll notice a difference in
performance.
- mysql buffer and cache sizes etc. should imho be the same on all
test systems.
- some of the papers you reference ([3] and [4]) contain more
incorrect and dangerous information than useful advice.
DES
--
Dag-Erling Smrgrav - [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Re: [call for helpers!] Tuning for the Beaver Challenge
Is vm.max_proc_mmap autotune by maxusers? If maxusers=0, it determine proc_mmap from RAM size? Could you please explain the use of net.inet.tcp.tcbhashsize? Patrick -Original Message- From: Yaoping Ruan [EMAIL PROTECTED] To: Dung Patrick [EMAIL PROTECTED] Date: Mon, 09 Feb 2004 12:45:46 -0500 Subject: Re: [call for helpers!] Tuning for the Beaver Challenge In section 2.2.3, /etc/sysctl.conf: To our experience, it also helps to tune inode cache behavior by setting: vfs.vmiodirenable=0; (maybe) vfs.nameileafonly=-1 On a server with large memory, if apache 1.x is tested, maybe it is also necessary to increase: vm.max_proc_mmap In section 2.2.4, /boot/loader.conf: Do you need to increase net.inet.tcp.tcbhashsize ? - Yaoping Dung Patrick wrote: Hi Beaver Challenge 2004 is coming!. Details in http://osuosl.org/benchmarks/bc/ We are preparing the tuning guide. Definitely we need suggestions and comments. Please see this forum to view the latest tuning guide: http://osuosl.org/forums/viewforum.php?f=8 Attached is a ver0.4 of the tuning guide. Regards Patrick Name: bc-fbsd-tune guide.txt bc-fbsd-tune guide.txtType: Plain Text (text/plain) Encoding: BASE64 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
[call for helpers!] Tuning for the Beaver Challenge
Hi Beaver Challenge 2004 is coming!. Details in http://osuosl.org/benchmarks/bc/ We are preparing the tuning guide. Definitely we need suggestions and comments. Please see this forum to view the latest tuning guide: http://osuosl.org/forums/viewforum.php?f=8 Attached is a ver0.4 of the tuning guide. Regards Patrick == FreeBSD tuning guide for the beaver challenge 2004 For stock class Draft ver 0.4 == = Changelog = 0.1 Initial release 0.2 Add some notes about the packages for the benchmarking 0.3 sysctl.conf, loader.conf, more tuning on Apache and other minor updates 0.4 Partition order, 2.2.1(cron), 2.2.3 (sysctl), 2.2.5(flags), 2.2.6, 2.2.7 = 1. Background information = The machine: Dell PowerEdge 2650 2 - 2.8Ghz Intel Xeon processors 2GB of RAM 5 - 36GB U320 disks @ 10,000 RPM configured as RAID0 stripe Adaptec AAC raid controller It's full spec can be downloaded in here: http://www.dell.com/downloads/global/products/pedge/en/2650_specs.pdf From http://osuosl.org/benchmarks/bc/methodology/ .They will run these benchmarks: Bonnie++ - file system performance. lmbench - bandwidth and latency performance. VolanoMark - a Java benchmark. mySQL server throughput Apache server throughput Their testing script is also avaliable on their web page. For Bonnie, there testing parameters is this: my $file_size = 4g; my $file_num= 128; my $small_min_size = 16; my $small_max_size = 2; my $large_min_size = 2; my $large_max_size = 10; my $num_dir = 512; my $test_runs = 1; my $symlink_runs = 1; my $hardlink_runs = 1; = 2. Tuning section = In this section, we are going to tune the system to provide the best benchmark. 2.1 Installation Partitioning There will be about 180GB harddisk space and 2GB RAM. How are we going to partition the FreeBSD system? Suggested layout be like this (in order) /home 4GB (their bonnie++ script benchmark on this partition, if we do not create /home, then /home will be a symobilc link to /usr/home) / xxGB swap4GB (It have 2GB RAM) /usrxxGB (need some space to hold the ports, and possibly to build the jdk) The /home is the first because we want to use the outside part of the hard disk. (So /home will be a separate partition instead of a slide?) Use UFS1 for FBSD4 (no choice), and UFS2 for FBSD5. Turn on soft updates for all partitions. Some thought: 1) /home can be reformatted (to use different block/fragment size). By looking at their bonnie script, it seems increasing the blocksize (default is 16KB) would benefit. But I do not have actual testing or supporting materials. Unless we have any good reasons, we would stick to the default. 2) In the long run, will those /usr/src, /usr/ports, /usr/obj make the /usr partition becomes more and more fragments? Should we make separate partitions for them? -- 2.2 Tweaking (First stage) -- After the installation, we are going to tune in different areas. 2.2.1 Turn off all unneeded services Remember for stock class: The tweaks must reflect a system that could be placed into production. Turn off run these services: inetd portmap sendmail sshd (we use serial console access?) cron usbd? (I doubt it has any USB devices in the BC) Question: Turn these off for the sake of the bechmarking? (which should not be done in a production server) syslog 2.2.2 /etc/fstab By default, the file system will be mounted with atime. Let's change it. For more information, refer to man 8 mount. To do it without reboot # mount -o update,noatime / Then, add 'noatime' to the appropriate place in /etc/fstab. So that it will be done automatically in next reboot. 2.2.3 /etc/sysctl.conf -- This is a very critical file that many tweaking will be done in here. To be completed, refer to Tuning man page TCP man page More parameters to tune in http://sinuspl.net/txt/ (look for sysctl.txt) kern.ipc.somaxconn=8192 kern.ipc.maxsockbuf=2097152 net.inet.ip.portrange.last=3 net.inet.tcp.rfc1323=1 net.inet.tcp.delayed_ack=0 net.inet.tcp.sendspace=32768 net.inet.tcp.recvspace=32768 net.inet.udp.recvspace=32768 net.inet.icmp.icmplim=0 # We will stick to the defaults unless we have any good reasons #net.inet.udp.maxdgram= #net.local.stream.sendspace= #net.local.stream.recvspace= #net.local.dgram.maxdgram= #net.local.dgram.recvspace= # Disable features that we are not going to use in the BC net.inet.icmp.drop_redirect=1 net.inet.icmp.log_redirect=0 net.inet.ip.redirect=0 net.inet6.ip6.redirect=0 net.inet.ip.sourceroute=0 net.inet.ip.accept_sourceroute=0 # Turn off ARP wrong interface log messages net.link.ether.inet.log_arp_wrong_iface=0 Question: 1) Would it be
Re: [call for helpers!] Tuning for the Beaver Challenge
Bascially everything in section 2.2.3 (/etc/sysctl.conf) is wrong; some values may need tweaking, but changing them blindly will not be helpful. The one setting that I do suggest you keep is: kern.ipc.somaxconn=512 (128 may be too low for http testing) The settings from section 2.2.4 will _probably_ be ok, but you'll have to run netstat -m after a benchmarking run to really know. One setting which you'll need to change for the apache2 run is kern.ipc.nsfbufs. Unfortunately, -stable doesn't have any way to tell if you're running out, so you'll have to just guess there. Under -current, netstat -m will tell you what you need to know. Also, you'll probably want to increase KVA_PAGES from 256 to 512 so that the kernel does not run out of memory. Under section 2.2.7, take out the section talking about CPU_FASTER_5X86_FPU through CPU_UPGRADE_HW_CACHE - none of these options is going to help, and it will just increase the likelyhood that something goes wrong. In general, it appears that you can run most of the benchmarks locally, so I suggest that you do that when trying to decide how to tweak settings. For the threads-related benchmarks (volcanomark and apache2 with worker) start asking for help on the -threads mailing list. This is the one thing that is likely to benefit from tweaking. Mike Silby Silbersack On Tue, 10 Feb 2004, Dung Patrick wrote: Hi Beaver Challenge 2004 is coming!. Details in http://osuosl.org/benchmarks/bc/ We are preparing the tuning guide. Definitely we need suggestions and comments. Please see this forum to view the latest tuning guide: http://osuosl.org/forums/viewforum.php?f=8 Attached is a ver0.4 of the tuning guide. Regards Patrick ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
Hello, The one setting that I do suggest you keep is: kern.ipc.somaxconn=512 (128 may be too low for http testing) In our experience with Apache and clients that do not use Keep Alive (are short lived), 512 is also very low. It causes listen queue overflows and leads to a very low throughput. Another place that we had to look at was the IP sw interrupt queue length. # IP sw interrupt queue len, default 50; # check queue drops stats in net.inet.ip.intr_queue_drops net.inet.ip.intr_queue_maxlen=1000 Hope this helps Aniruddha ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Re: [call for helpers!] Tuning for the Beaver Challenge
So you suggested to use the default values for section 2.2.3. I think increasing maxsockbuf, and portrange would be helpful. Without testing, it is hard to choose a value for kern.ipc.nsfbufs. For those optionts CPU_FASTER_5X86_FPU through CPU_UPGRADE_HW_CACHE, it is suggested by an user in the forum. I would like to see the comments from the mailing list. If those options are dangerous, then don't use them. Yes, I can test things locally. But I don't have too much time... Patrick -Original Message- From: Mike Silbersack [EMAIL PROTECTED] To: Dung Patrick [EMAIL PROTECTED] Date: Mon, 9 Feb 2004 11:12:38 -0600 (CST) Subject: Re: [call for helpers!] Tuning for the Beaver Challenge Bascially everything in section 2.2.3 (/etc/sysctl.conf) is wrong; some values may need tweaking, but changing them blindly will not be helpful. The one setting that I do suggest you keep is: kern.ipc.somaxconn=512 (128 may be too low for http testing) The settings from section 2.2.4 will _probably_ be ok, but you'll have to run netstat -m after a benchmarking run to really know. One setting which you'll need to change for the apache2 run is kern.ipc.nsfbufs. Unfortunately, -stable doesn't have any way to tell if you're running out, so you'll have to just guess there. Under -current, netstat -m will tell you what you need to know. Also, you'll probably want to increase KVA_PAGES from 256 to 512 so that the kernel does not run out of memory. Under section 2.2.7, take out the section talking about CPU_FASTER_5X86_FPU through CPU_UPGRADE_HW_CACHE - none of these options is going to help, and it will just increase the likelyhood that something goes wrong. In general, it appears that you can run most of the benchmarks locally, sodsuggest that you do that when trying to decide how to tweak settings. For the threads-related benchmarks (volcanomark and apache2 with worker) start asking for help on the -threads mailing list. This is the one thing that is likely to benefit from tweaking. Mike Silby Silbersack On Tue, 10 Feb 2004, Dung Patrick wrote: Hi Beaver Challenge 2004 is coming!. Details in http://osuosl.org/benchmarks/bc/ We are preparing the tuning guide. Definitely we need suggestions and comments. Please see this forum to view the latest tuning guide: http://osuosl.org/forums/viewforum.php?f=8 Attached is a ver0.4 of the tuning guide. Regards Patrick ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
In section 2.2.3, /etc/sysctl.conf: To our experience, it also helps to tune inode cache behavior by setting: vfs.vmiodirenable=0; (maybe) vfs.nameileafonly=-1 On a server with large memory, if apache 1.x is tested, maybe it is also necessary to increase: vm.max_proc_mmap In section 2.2.4, /boot/loader.conf: Do you need to increase net.inet.tcp.tcbhashsize ? - Yaoping Dung Patrick wrote: Hi Beaver Challenge 2004 is coming!. Details in http://osuosl.org/benchmarks/bc/ We are preparing the tuning guide. Definitely we need suggestions and comments. Please see this forum to view the latest tuning guide: http://osuosl.org/forums/viewforum.php?f=8 Attached is a ver0.4 of the tuning guide. Regards Patrick Name: bc-fbsd-tune guide.txt bc-fbsd-tune guide.txtType: Plain Text (text/plain) Encoding: BASE64 ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
Dung Patrick [EMAIL PROTECTED] writes:
For those optionts CPU_FASTER_5X86_FPU through CPU_UPGRADE_HW_CACHE,
it is suggested by an user in the forum. I would like to see the
comments from the mailing list. If those options are dangerous, then
don't use them.
CPU_FASTER_5X86_FPU is not likely to have any positive impact on
performance, and fairly likely to render the system unbootable.
CPU_UPGRADE_HW_CACHE has absolutely no effect on non-PC98 systems.
further comments -
- symlink /usr/{src,obj,ports} to /slow/{src,obj,ports} where /slow
is a filesystem placed on the outside edge of the disk (to free up
space near the spindle for the filesystems actually used by the
benchmark)
- find out what file system the Linux people are using. if they are
using ext2fs or ext3fs, mount your filesystems async.
- most of what you put in sysctl.conf is completely irrelevant. the
rest needs to be tuned according to the actual needs of the
benchmark.
- you should not tune kern.maxfiles etc. unless the benchmark
actually hits those limits. increasing these numbers reduces the
amount of kernel memory available for other purposes.
btw, maxfiles and maxfilesperproc are tunable at run time.
- most of what you put in make.conf is bogus. just use
CPUTYPE ?= pentiumpro
CFLAGS = -O -pipe
COPTFLAGS = -O -pipe
- NOPROFILE has absolutely no impact on performance (except that it
shortens 'make world' a little)
- you *must* use -CURRENT and not 5.2 as the latter has issues with
the aac driver.
- don't use apm or acpi on 4.x.
- regarding jdk 1.4.2, just use the linux version (and make sure to
mount linprocfs). I very much doubt you'll notice a difference in
performance.
- mysql buffer and cache sizes etc. should imho be the same on all
test systems.
- some of the papers you reference ([3] and [4]) contain more
incorrect and dangerous information than useful advice.
DES
--
Dag-Erling Smørgrav - [EMAIL PROTECTED]
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
For your consideration, bearing in mind I haven't read all details of the challenge or the systems and therefore may suggest inappropriate, wrong or dangerous things: Custom kernel enabling only the hardware needed, all compiled in (no modules), but be careful with fancy looking options in LINT/NOTES. It may be prudent to hard code interrupts, flags and other parameters to avoid potential suboptimal automatic allocations (eg. all PCI devs on irq 9). Mount all filesystems with async, noatime and *enable softupdates*. Use carefully sized (ie. just big enough) mfs filesystems for any area with dynamically generated data such as temporary files, server scoreboards, etc. /tmp and /var/run, at least. If there's a foo_enable=NO you put in rc.conf, then do so. Disable as much logging as possible at system and application level, at source or config or by logging to /dev/null. If you have to log, trim I/O to the minimum, maximise buffering and minimise overhead (eg. -n for syslog). If any DNS activity is needed, a local cache (eg. dnscache) can help, but so can a nicely populated /etc/hosts. Depending on the tests, it may be a good idea to have the hostname (fully and unqualified) associated with the loopback rather than the (primary) network interface. For the Java benchmarks, try all the implementations which work, including the emulated ones. If the storage is going to be striped over all five disks in hardware by the controller, then notion of putting data on the outside of the disk doesn't apply. In fact in this configuration it _may_ make sense to use one big filesystem and leave it to the OS to optimise the filesystem I/O. -Andrew- -- ___ | -Andrew J. Caines- Unix Systems Engineer [EMAIL PROTECTED] | | They that can give up essential liberty to obtain a little temporary | | safety deserve neither liberty nor safety - Benjamin Franklin, 1759 | ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
Andrew J Caines [EMAIL PROTECTED] writes: Mount all filesystems with async, noatime and *enable softupdates*. async and softupdates are mutually exclusive. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
Andrew J Caines [EMAIL PROTECTED] writes: If any DNS activity is needed, a local cache (eg. dnscache) can help, but so can a nicely populated /etc/hosts. Depending on the tests, it may be a good idea to have the hostname (fully and unqualified) associated with the loopback rather than the (primary) network interface. that won't make any difference, the host's own IP addresses are automatically routed through lo0. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
On Mon, Feb 09, 2004 at 07:27:08PM +0100 I heard the voice of Dag-Erling Sm?rgrav, and lo! it spake thus: CPU_FASTER_5X86_FPU is not likely to have any positive impact on performance, and fairly likely to render the system unbootable. I would guess just from the name that this (and some similarly named options) apply only to Cyrix 5x86 processors. Somehow, I don't think you'll run into too many of them in benchmarks these days. Just a hunch. CPUTYPE ?= pentiumpro I recall a thread somewhere recently about pentiumpro being decidedly suboptimal for some new CPUs. Although, on 4.x with the older version of gcc, it may not matter. -- Matthew Fuller (MF4839) | [EMAIL PROTECTED] Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ The only reason I'm burning my candle at both ends, is because I haven't figured out how to light the middle yet ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
This might also be an opportune time to test the benefits of compilers other than gcc, such as Intel's reputedly super-optimised-vs-gcc C/C++ compiler (lang/icc). I've not tried icc or any of the other non-gcc compilers and I'm not suggesting it will help, or even what one might try to compile, but maybe some others can offer their experiences? -Andrew- -- ___ | -Andrew J. Caines- Unix Systems Engineer [EMAIL PROTECTED] | | They that can give up essential liberty to obtain a little temporary | | safety deserve neither liberty nor safety - Benjamin Franklin, 1759 | ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: [call for helpers!] Tuning for the Beaver Challenge
Matthew D. Fuller [EMAIL PROTECTED] writes: CPUTYPE ?= pentiumpro I recall a thread somewhere recently about pentiumpro being decidedly suboptimal for some new CPUs. Although, on 4.x with the older version of gcc, it may not matter. I believe the reverse is true - pentiumpro proved to be superior to p3 and p4 even on p3s and p4s. DES -- Dag-Erling Smørgrav - [EMAIL PROTECTED] ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to [EMAIL PROTECTED]
