On 2009-04-03 18:06, Thomas Wouters wrote: > On Fri, Apr 3, 2009 at 11:27, Antoine Pitrou <solip...@pitrou.net> wrote: > >> Thomas Wouters <thomas <at> python.org> writes: >>> >>> Pystone is pretty much a useless benchmark. If it measures anything, it's >> the >> speed of the bytecode dispatcher (and it doesn't measure it particularly >> well.) >> PyBench isn't any better, in my experience. >> >> I don't think pybench is useless. It gives a lot of performance data about >> crucial internal operations of the interpreter. It is of course very little >> real-world, but conversely makes you know immediately where a performance >> regression has happened. (by contrast, if you witness a regression in a >> high-level benchmark, you still have a lot of investigation to do to find >> out >> where exactly something bad happened) > > > Really? Have you tried it? I get at least 5% noise between runs without any > changes. I have gotten results that include *negative* run times.
On which platform ? pybench 2.0 works reasonably well on Linux and Windows, but of course can't do better than the timers available for those platforms. If you have e.g. NTP running and it uses wall clock timers, it is possible that you get negative round times. If you don't and still get negative round times, you have to change the test parameters (see below). > And yes, I > tried all the different settings for calibration runs and timing mechanisms. > The tests in PyBench are not micro-benchmarks (they do way too much for > that), they don't try to minimize overhead or noise, That is not true. They were written as micro-benchmarks and adjusted to have a high signal-noise ratio. For some operations this isn't easy to do, but I certainly tried hard to get the overhead low (note that the overhead is listed in the output). That said, please keep in mind that the settings in pybench were last adjusted some years ago to have the tests all run in more or less the same wall clock time. CPUs have evolved a lot since then and this shows. > but they are also not > representative of real-world code. True and they never were meant for that, since I was frustrated by other benchmarks at the time and the whole approach in general. Each of the tests checks one specific aspect of Python. If your application happens to use a lot of dictionary operations, you'll be mostly interested in those. If you do a lot of simple arithmetic, there's another test for that. On top of that the application is written to be easily extensible, so it's easy to add new tests specific to whatever application space you're after. > That doesn't just mean "you can't infer > the affected operation from the test name", but "you can't infer anything." > You can just be looking at differently borrowed runtime. I have in the past > written patches to Python that improved *every* micro-benchmark and *every* > real-world measurement I made, except PyBench. Trying to pinpoint the > slowdown invariably lead to tests that did too much in the measurement loop, > introduced too much noise in the "calibration" run or just spent their time > *in the measurement loop* on doing setup and teardown of the test. pybench calibrates itself to remove that kind of noise from the output. Each test has a .calibrate() method which does all the setup and tear down minus the actual benchmark operations. If you get wrong numbers, try adjusting the parameters and add more "packets" of operations. Don't forget to adjust the version number to not compare apples and orange, though. Perhaps it's time to readjust the pybench parameters to todays CPUs. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 03 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2009-03-19: Released mxODBC.Connect 1.0.1 http://python.egenix.com/ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
