Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server
Hmm... No sure what happened there again. What I sent (pulled from my Sent folder... === Thanks for the comment Arnaud. For comparative benchmarking on Phoronix.com http://Phoronix.com, Michael invariable leaves it in the default configuration 'in the way the developers or vendor wanted it for production'. This is by rule. However, invariable the community or vendor for platforms that post poor scores on benchmark cry foul about using the default config. 'it should be tuned, no-one deploys an untuned system' or 'the system is configured for a different workload'. The response from us to this comes in two forms. 1) If it is the wrong workload for the platform, do a public post explaining and analysing the results. Highlighting the rationale for the concious reduction in performance (ie: journaling filesystems with barriers suffer in some write benchmarks for the sake of filesystem integrity. 2) If tuning can have a material impact on the results, post a tuning guide with step by step and rationale. Ie: educate the community and users. Michael and I have had many discussions with vendors and communities on this. In almost all cases, the vendor has either changed the default configuration or accepted the results as valid. As a service to the community or vendor that publishes the tuning guide, Michael is more than willing to redo a tuned vs untuned comparison. To date, the communities have never taken us up on that offer. In part, this affects Phoronix.com http://Phoronix.com's perception in the public, but that is more of a result of a one sided discussion by a party external to a particular community (with a healthy touch of journalisticly pumped compare contrast). For the FreeBSD community, who else outside of the FreeBSD community actually runs public comparisons of FreeBSD against anything? Matthew === On 01/04/2012 03:49 PM, Alexander Kabaev wrote: On Wed, 04 Jan 2012 14:31:55 -0800 matt...@phoronix.com wrote: Thanks for the comment Arnaud. For comparative benchmarking on[1]Phoronix.com, Michael inva configuration 'in the way the developers or production'. This is by rule. However, i poor scores on be 'it should be tuned, is configured for a diffe The response from us to this comes in two forms.nb 1) If it is the wrong workload for the platform, do a public pos explaining and analysing the results. Highlighting the rationale fo r the concious reduction in performance (ie: journaling filesystems with ba filesystem integrity 2) If tuning can have a material impact on the results, post a t uning guide with step by step and rationale. Ie: educate the communit Michael and I have had many discussions with vendors an on this. In almost all cases, the vendor has either cha default configuration or accepted the results as valid. Asguide, Micha comparison. To dat offer. In part, thi public, but that is more of a result of a one sided d party external to a particular community (with a healthy tou journalisticly pumped compare contrast). For the FreeBSD community, who else outside of the FreeBSD community actually runs public c Matthew Not really related to the discussion on hand, but the above about the most unreadable email I am yet to read on the public mailing list. ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server
Bottom post this time to follow Oliver :). On 12/20/2011 02:54 PM, O. Hartmann wrote: On 12/20/11 22:45, Samuel J. Greear wrote: http://www.osnews.com/story/25334/DragonFly_BSD_MP_Performance_Significantly_Improved PostgreSQL tests, see the linked PDF for #'s on FreeBSD, DragonFly, Linux and Solaris. Steps to reproduce these benchmarks provided. Sam There are still possible issues with those benchmarks. The Xeon has known problems scaling from 6 to 12 cores (well enabling the hyperthreading), so you may find that some platforms are penalized in performance if HT is turned on. See the scaling that Phoronix has done in http://openbenchmarking.org/result/1112166-AR-1112153AR03 Most systems are good with scaling on real cores, the hyperthreading (and for that matter the Bulldozer thread affinity) can really break performance. Different platforms have different behaviours. Benchmarking is a mucky business.. Note that the benchmarks with Phoronix test suite are repeatable, once installed, you can just run ./phoronix-test-suite benchmark 1112113-AR-ORACLELIN37 to repeat (as close as the system allows) the benchmarks that started this thread. Is the postgresql benchmark the only way to benchmark? pgbench is already included in the Phoronix Test Suite (at least 9.0.1 TPC-B benchmark. Well, this inspires me to gather together all the benchmarks someone could find. There were lots of compalins about FreeBSD's poor performance with BIND - once a domain of FreeBSD. Network performance seems also to be an issue if it comes to scalability. It would be nice to see what portion of the raw CPU/GPU power the OS (FreeBSD, Linux ...) delivers to scientific applications. I only know some kind of benchmarks, BYTE UNIX benchmark, LINPACK test ... Does someone know a site to look for a couple of benchmarks to test a) memory system b) scalability (apart from pgbench) c) network performance/throughput/network scalability d) portion of CPU performance the system delivers for numerical applications to the user apart from the system's own consumption e) disk I/O performance and scalability The majority of these benchmarks are already in Phoronix Test Suite. There is monitoring capability (temp, load, CPU states, etc). The question is the mapping from system attribute to benchmark, as well as determine what the ambigious terms mean (scaling can mean on increasing workloads, as memory is increased, as cpus are increased). it would also be nice to discuss some nice settings and performance tunings for FreeBSD for several scenarios. I guess, starting developing benchmarking test scenarios for several purposes would lead faster to real numbers and non polemic than weird discussions ... This is what Michael and I are wanting to see. Adrian Chadd has offerered to help facilitate within the FreeBSD community. As mentioned before, what I'd like to see is 1) Recommendations for more rounded benchmarks from the FreeBSD perspective 2) Tuning guide documented somewhere within the community 3) Comparative results based on the communities testing. All concrete, and all achievable. Regards, Matthew ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server
For such a system, the greatest immediate value would be to attempt to reproduce the benchmarks in question. Install PTS from www.phoronix-test-suite.com or freshports.org. Run the benchmark against those used in the article phoronix-test-suite benchmark 1112113-AR-ORACLELIN37 You will be asked to push the comparison up to openbenchmarking at the end. Matthew On 12/20/2011 01:39 PM, O. Hartmann wrote: On 12/20/11 21:20, Igor Mozolevsky wrote: Interestingly, while people seem to be (arguably rightly) focused on criticising Phoronix's benchmarking, nobody has offered an alternative benchmark; and while (again, arguably rightly) it is important to benchmark real world performance, equally, nobody has offered any numbers in relation to, for example, HTTP or SMTP, or any other real world-application torture tests done on the aforementioned two platforms... IMO, this just goes to show that doing is hard and criticising is much easier (yes, I am aware of the irony involved in making this statement, but someone has to!) Cheers, Igor M :-) Unfortunately, M. Larabel is the only one who's performing benchmarks on FreeBSD, comparing its performance to the Linux-opponents. Adn indeed, there is a lot of criticism, but no alternative. I said unfortunately - not offensive - since Larabel and Phoronix are sadly the only ones who do actually such bechmarking. It would be much more nicer and kind to support those people. Well, in January/February we get new hardware. One box is supposed to do number crunching via 12 cores and a TESLA GPU. My colleague is developing a high parallelized peice of software for satellite data transformation. The software package is CPU bound, partially GPU, but massively memory hungry (96 to 128 GB RAM is needed). What I can offer is, since I will also work on that machine and I've free hand to administer, in the spare time of doing my PhD, installing FreeBSD 9.0/10.0 besides SuSe Linux and looking forward having one ZFS data storage drive for homes, so both systems can perform on a most recent ZFS. I'm new to Linux, not a BSD guru, nor I'm a professional programmer/developer. My skills are sufficient for the daily scientific work. So, without pressure, I'm willing to perform some HPC benchmarks under advice if the day comes and those interested in bare numbers of FreeBSD vs. Linux performance with a real-world-scientific application. I would appreciate to see some of the developers and/or FreeBSD hackers to help Phoronix setting up a proper testenvironment instead of bashing M. Larabel and his fellows. Regards, Oliver ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
