Re: [Speed] New benchmark suite for Python
On Sun, Aug 21, 2016 at 7:38 AM, Nick Coghlan wrote: > On 20 August 2016 at 02:50, Maciej Fijalkowski wrote: >> Very likely just pyc import time > > As one of the import system maintainers, that's a number I consider > quite interesting and worth benchmarking :) > > It's also one of the key numbers for Linux distro Python usage, since > it impacts how responsive the system shell feels to developers and > administrators - an end user can't readily tell the difference between > "this shell is slow" and "this particular command I am running is > using a language interpreter with a long startup time", but an > interpreter benchmark suite can. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia Fair point, let's have such a benchmark. Let's not have it called "bzr" though because it gives the wrong impression. The same way unladen swallow added a benchmark and called it "django" while not representing django very well. That said, likely not very many people use bzr, but still, would be good if it's called bzr-pyc or simpler - have a benchmark that imports a whole bunch of pyc from a big project (e.g. pypy :-) Cheers, fijal ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] New benchmark suite for Python
Very likely just pyc import time On Fri, Aug 19, 2016 at 6:48 PM, Alex Gaynor wrote: > It's probably an ok benchmark of warmup. > > Alex > > On Fri, Aug 19, 2016 at 12:47 PM, Maciej Fijalkowski > wrote: >> >> On Fri, Aug 19, 2016 at 1:20 PM, Nick Coghlan wrote: >> > On 19 August 2016 at 01:55, Victor Stinner >> > wrote: >> >> 2016-08-18 8:48 GMT+02:00 Armin Rigo : >> >>> Indeed, bzr cannot be installed on PyPy because it uses Cython in a >> >>> strange way: it declares and directly pokes inside PyListObjects from >> >>> a .pyx file. But note that bzr (seems to) have systematically a pure >> >>> Python version of all its .pyx files. (...) >> >> >> >> bazar is only used for a "startup" benchmark. I don't think that such >> >> benchmark is very interesting... I would prefer to see a benchmark on >> >> a less dummy operation on the repository than displaying the help... >> > >> > Simple commands like displaying help messages are where interpreter >> > startup time dominates the end user experience for applications >> > written in Python, though. For example, improvements to import system >> > performance tend to mostly show up there - for longer running >> > benchmarks, changes in startup time tend to get swamped by the actual >> > runtime speed, while the baseline "python -c 'pass'" mainly varies >> > based on how many modules we're implicitly importing at startup rather >> > than how well the import system is performing . >> > >> > Cheers, >> > Nick. >> >> I would still argue that displaying help is not a very good benchmark :-) >> ___ >> Speed mailing list >> Speed@python.org >> https://mail.python.org/mailman/listinfo/speed > > > > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: D1B3 ADC0 E023 8CA6 > ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] New benchmark suite for Python
On Fri, Aug 19, 2016 at 1:20 PM, Nick Coghlan wrote: > On 19 August 2016 at 01:55, Victor Stinner wrote: >> 2016-08-18 8:48 GMT+02:00 Armin Rigo : >>> Indeed, bzr cannot be installed on PyPy because it uses Cython in a >>> strange way: it declares and directly pokes inside PyListObjects from >>> a .pyx file. But note that bzr (seems to) have systematically a pure >>> Python version of all its .pyx files. (...) >> >> bazar is only used for a "startup" benchmark. I don't think that such >> benchmark is very interesting... I would prefer to see a benchmark on >> a less dummy operation on the repository than displaying the help... > > Simple commands like displaying help messages are where interpreter > startup time dominates the end user experience for applications > written in Python, though. For example, improvements to import system > performance tend to mostly show up there - for longer running > benchmarks, changes in startup time tend to get swamped by the actual > runtime speed, while the baseline "python -c 'pass'" mainly varies > based on how many modules we're implicitly importing at startup rather > than how well the import system is performing . > > Cheers, > Nick. I would still argue that displaying help is not a very good benchmark :-) ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] New benchmark suite for Python
Hey Victor Did you look into integrating the pypy benchmarks that we added over the years? On Thu, Aug 18, 2016 at 9:37 AM, Victor Stinner wrote: > Hi, > > After a few months of work, I created a "new" benchmark suite for Python: > >https://github.com/python/benchmarks > > It's based on: >https://hg.python.org/sandbox/benchmarks_perf > which is my fork the CPython benchmark suite: >https://hg.python.org/benchmarks > which is based on Unladen Swallow's benchmark suite (if I understood > correctly). > > Major differences: > > * Use the perf module to run benchmarks in multiple processes and > store results as JSON > * Create virtual environments using requirements.txt to download > dependencies from PyPI (rather than using old copies of libraries) > * Many libraries have been upgraded: see requirements.txt > > The project works on Python 2 and Python 3 (I tested 2.7 and 3.6). > > Known regressions: > > * Memory tracking is broken > * run_compare command is currently broken: use run (store result into > a file) + compare manually > * Some benchmarks have been removed: rietveld, spitfire (not on PyPI), > pystone, gcbench, tuple_gc_hell > * I only tested Linux, I expect issues on Windows. (I didn't try my > perf module on Windows yet.) > > I already allowed all Python core developers to push to the GitHub > project. We can create a new "benchmarks" (or "Performance" maybe?) > team if we want to allow more contributors who are not core > developers. > > PyPy, Pyston, Pyjion, Numba, etc. : Hey! it's now time to start to > take a look at my project and test it ;-) Tell me what is broken, what > is missing, and I will try to help you to move your project to this > new benchmark suite! > > As requested (suggested?) by Brett Canon, the Git repository has no > history, it only contains 1 commit! I'm really sorry of loosing all > the history and all authors, but it allows to start with a much > smaller repository: around 2 MB. The current benchmark repository is > more around 200 MB! > > TODO: > > * continue to upgrade libraries in requirements.txt. I failed to > upgrade Django to 1.10, it complains about a missing template engine > config setting. > * convert more code to the perf module, like "startup" tests > * run benchmarks and analyze results ;-) > * write more documentation explaining how to run reliable benchmarks > * ... > > Victor > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] External sources of noise changing call_simple "performance"
On Wed, May 18, 2016 at 1:16 PM, Victor Stinner wrote: > 2016-05-18 8:55 GMT+02:00 Maciej Fijalkowski : >> I think you misunderstand how caches work. The way caches work depends >> on the addresses of memory (their value) which even with ASLR disabled >> can differ between runs. Then you either do or don't have cache >> collisions. > > Ok. I'm not sure yet that it's feasible to get exactly the same memory > addresses for "hot" objects allocated by Python between two versions > of the code (especially when testing a small patch). Not only the > addresses look to depend on external parameters, but the patch can > also adds or avoids some memory allocations. > > The concrete problem is that the benchmark depends on such low-level > CPU feature and the perf.py doesn't ignore minor delta in performance, > no? > > Victor Well the answer is to do more statistics really in my opinion. That is, perf should report average over multiple runs in multiple processes. I started a branch for pypy benchmarks for that, but never finished it actually. ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] External sources of noise changing call_simple "performance"
> => The performance of the benchmark depends on the usage of low-level > memory caches (L1, L2, L3). > > I understand that in some cases, more memory fits into the fatest > caches, and so the benchmark is faster. But sometimes, all memory > doesn't fit, and so the benchmark is slower. > > Maybe the problem is that memory is close to memory pages boundaries, > or doesn't fit into L1 cache lines, or something like that. > > Victor I think you misunderstand how caches work. The way caches work depends on the addresses of memory (their value) which even with ASLR disabled can differ between runs. Then you either do or don't have cache collisions. How about you just accept the fact that there is a statistical distribution of the results on not the concrete "right" result? I tried to explain to you before that even if you get the "right" result, it'll still be at best just one sample of the statistics. ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Adapting pypy benchmark suite
Hi
I'm willing to put some work after I'm back from holiday (mid-May)
On Tue, May 3, 2016 at 12:24 AM, Brett Cannon wrote:
>
>
> On Mon, 2 May 2016 at 15:18 Kevin Modzelewski wrote:
>>
>> I'm definitely interested and willing to clean up + contribute our
>> benchmarks.
>>
>> On a side note, I'm a bit skeptical that there can be a single benchmark
>> suite that satisfies everyone. I would imagine that there will still be
>> projects with specific use-cases they prioritize (such as Pyston with
>> webserver workloads), or that have some idea that their users will be
>> "non-representative" in some way. One example of that is the emphasis on
>> warmup vs steady-state performance, which can be reflected in different
>> measurement methodologies -- I don't think there's a single right answer to
>> the question "how much does warmup matter".
>
>
> Totally agree. I think the general thinking is to at have a central
> repository and a flexible enough benchmark runner that people can benchmark
> whatever they find important to them. That way if e.g. Pyston adds nice web
> server benchmarks other implementations can use them or users can decide
> that's a workload they care about and make an informed decision of what
> Python implementations may work for them (before testing their own workload
> :).
>
> -Brett
>
>>
>>
>> But anyway, I'm still definitely +1 on the idea of merging all the
>> benchmarks together, and I think that that will be better than the current
>> situation. I'm imagining that we can at least have a common language for
>> discussing these things ("Pyston prefers to use the flags `--webserver
>> --include-warmup`"). I also see quite a few blog posts / academic papers on
>> Python performance that seem to get led astray by the confusing benchmark
>> situation, and I think having a blessed set of benchmarks (even if different
>> people use them in different ways) would still be a huge step forward.
>>
>> kmod
>>
>> On Mon, May 2, 2016 at 9:25 AM, Brett Cannon wrote:
>>>
>>>
>>>
>>> On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski wrote:
>>>>
>>>> On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou
>>>> wrote:
>>>> > On Wed, 13 Apr 2016 20:57:35 +0200
>>>> > Maciej Fijalkowski
>>>> > wrote:
>>>> >> Hi
>>>> >>
>>>> >> I have a radical idea: to take a pypy benchmark suite, update the
>>>> >> libraries to newer ones and replace python benchmarks with that. The
>>>> >> main reason being that pypy has a much better coverage of things that
>>>> >> are not microbenchmarks, the list (in json):
>>>> >
>>>> > So why not consolidate all benchmarks together, instead of throwing
>>>> > away work already done?
>>>> >
>>>> > Regards
>>>> >
>>>> > Antoine.
>>>>
>>>> Yeah, you can call it that too.
>>>
>>>
>>> I also reached out to Pyston at https://gitter.im/dropbox/pyston over the
>>> weekend to see if they would want to participate as well.
>>>
>>> So are we actually going to try and make this happen? I guess we should
>>> get people to vote on whether they like the idea enough before we hash out
>>> how we want to structure the new repository and benchmark suite.
>>>
>>> I'm +1 on the idea, but I currently don't have the time to help beyond
>>> helping drive the email conversation.
>>>
>>> ___
>>> Speed mailing list
>>> Speed@python.org
>>> https://mail.python.org/mailman/listinfo/speed
>>>
>
___
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Disable hash randomization to get reliable benchmarks
On Tue, Apr 26, 2016 at 6:36 PM, Antoine Pitrou wrote: > On Tue, 26 Apr 2016 18:28:32 +0200 > Maciej Fijalkowski > wrote: >> >> taking the minimum is a terrible idea anyway, none of the statistical >> discussion makes sense if you do that > > The minimum is a reasonable metric for quick throwaway benchmarks as > timeit is designed for, as it has a better hope of alleviating the > impact of system load (as such throwaway benchmarks are often run on > the developer's workstation). > > For a persistent benchmarks suite, where we can afford longer > benchmark runtimes and are able to keep system noise to a minimum, we > might prefer another metric. > > Regards > > Antoine. No, it's not Antoine. Minimum is not better than one random measurment. We had this discussion before, but you guys are happily dismissing all the papers written on the subject. It *does* get rid of random system stuff, but it *also* does get rid of all the effects related to gc/malloc/caches and infinite details that are not working in the same predictable fashion. ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Disable hash randomization to get reliable benchmarks
On Tue, Apr 26, 2016 at 11:46 AM, Victor Stinner wrote: > Hi, > > 2016-04-26 10:56 GMT+02:00 Armin Rigo : >> Hi, >> >> On 25 April 2016 at 08:25, Maciej Fijalkowski wrote: >>> The problem with disabled ASLR is that you change the measurment from >>> a statistical distribution, to one draw from a statistical >>> distribution repeatedly. There is no going around doing multiple runs >>> and doing an average on that. >> >> You should mention that it is usually enough to do the following: >> instead of running once with PYTHONHASHSEED=0, run five or ten times >> with PYTHONHASHSEED in range(5 or 10). In this way, you get all >> benefits: not-too-long benchmarking, no randomness, but still some >> statistically relevant sampling. > > I guess that the number of required runs to get a nice distribution > depends on the size of the largest dictionary in the benchmark. I > mean, the dictionaries that matter in performance. > > The best would be to handle this transparently in perf.py. Either > disable all source of randomness, or run mutliple processes to have an > uniform distribution, rather than on only having one sample for one > specific config. Maybe it could be an option: by default, run multiple > processes, but have an option to only run one process using > PYTHONHASHSEED=0. > > By the way, timeit has a very similar issue. I'm quite sure that most > Python developers run "python -m timeit ..." at least 3 times and take > the minimum. "python -m timeit" could maybe be modified to also spawn > child processes to get a better distribution, and maybe also modified > to display the minimum, the average and the standard deviation? (not > only the minimum) taking the minimum is a terrible idea anyway, none of the statistical discussion makes sense if you do that > > Well, the question is also if it's a good thing to have such really > tiny microbenchmark like bm_call_simple in the Python benchmark suite. > I spend 2 or 3 days to analyze CPython running bm_call_simple with > Linux perf tool, callgrind and cachegrind. I'm still unable to > understand the link between my changes on the C code and the result. > IMHO this specific benchmark depends on very low-level things like the > CPU L1 cache. Maybe bm_call_simple helps in some very specific use > cases, like trying to make Python function calls faster. But in other > cases, it can be a source of noise, confusion and frustration... > > Victor maybe it's just a terrible benchmark (it surely is for pypy for example) ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Disable hash randomization to get reliable benchmarks
Hi Victor The problem with disabled ASLR is that you change the measurment from a statistical distribution, to one draw from a statistical distribution repeatedly. There is no going around doing multiple runs and doing an average on that. Essentially for the same reason why using min is much worse than using average, with ASLR say you get: 2.0+-0.3 which we run 5 times and 1.8, 1.9, 2.2, 2.1, 2.1 now if you disable ASLR, you get one draw repeated 5 times, which might be 2.0, but also might be 1.8, 5 times. That just hides the problem, but does not actually fix it (because if you touch something, stuff might be allocated in a different order and then you get a different draw) On Mon, Apr 25, 2016 at 12:49 AM, Victor Stinner wrote: > Hi, > > Last months, I spent a lot of time on microbenchmarks. Probably too > much time :-) I found a great Linux config to get a much more stable > system to get reliable microbenchmarks: > https://haypo-notes.readthedocs.org/microbenchmark.html > > * isolate some CPU cores > * force CPU to performance > * disable ASLR > * block IRQ on isolated CPU cores > > With such Linux config, the system load doesn't impact benchmark results at > all. > > Last days, I almost lost my mind trying to figure out why a very tiny > change in C code makes a difference up to 8% slower. > > My main issue was to get reliable benchmark since running the same > microbenchmark using perf.py gave me "random" results. > > I finished to run directly the underlying script bm_call_simple.py: > > taskset -c 7 ./python ../benchmarks/performance/bm_call_simple.py -n 5 > --timer perf_counter > > In a single run, timings of each loop iteration is very stable. Example: > > 0.22682707803323865 > 0.22741253697313368 > 0.227521265973337 > 0.22750743699725717 > 0.22752994997426867 > 0.22753606992773712 > 0.22742654103785753 > 0.22750875598285347 > 0.22752253606449813 > 0.22718404198531061 > > Problem: each new run gives a different result. Example: > > * run 1: 0.226... > * run 2: 0.255... > * run 3: 0.248... > * run 4: 0.258... > * etc. > > I saw 3 groups of values: ~0.226, ~0.248, ~0.255. > > I didn't understand how running the same program can give so different > result. The reply is the randomization of the Python hash function. > Aaah! The last source of entropy in my microbenchmark! > > The performance difference can be seen by forcing a specific hash function: > > PYTHONHASHSEED=2 => 0.254... > PYTHONHASHSEED=1 => 0.246... > PYTHONHASHSEED=5 => 0.228... > > Sadly, perf.py and timeit don't disable hash randomization for me. I > hacked perf.py to set PYTHONHASHSEED=0 and magically the result became > super stable! > > Multiple runs of the command: > > $ taskset_isolated.py python3 perf.py ../default/python-ref > ../default/python -b call_simple --fast > > Outputs: > > ### call_simple ### > Min: 0.232621 -> 0.247904: 1.07x slower > Avg: 0.232628 -> 0.247941: 1.07x slower > Significant (t=-591.78) > Stddev: 0.1 -> 0.00010: 13.7450x larger > > ### call_simple ### > Min: 0.232619 -> 0.247904: 1.07x slower > Avg: 0.232703 -> 0.247955: 1.07x slower > Significant (t=-190.58) > Stddev: 0.00029 -> 0.00011: 2.6336x smaller > > ### call_simple ### > Min: 0.232621 -> 0.247903: 1.07x slower > Avg: 0.232629 -> 0.247918: 1.07x slower > Significant (t=-5896.14) > Stddev: 0.1 -> 0.1: 1.3350x larger > > Even with --fast, the result is *very* stable. See the very good > standard deviation. In 3 runs, I got exactly the same "1.07x". Average > timings are the same +/-1 up to 4 digits! > > No need to use the ultra slow --rigourous option. This option is > probably designed to hide the noise of a very unstable system. But > using my Linux config, it doesn't seem to be needed anymore, at least > on this very specific microbenchmark. > > Ok, now I can investigate why my change on the C code introduced a > performance regression :-D > > Victor > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Adapting pypy benchmark suite
On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou wrote: > On Wed, 13 Apr 2016 20:57:35 +0200 > Maciej Fijalkowski > wrote: >> Hi >> >> I have a radical idea: to take a pypy benchmark suite, update the >> libraries to newer ones and replace python benchmarks with that. The >> main reason being that pypy has a much better coverage of things that >> are not microbenchmarks, the list (in json): > > So why not consolidate all benchmarks together, instead of throwing > away work already done? > > Regards > > Antoine. Yeah, you can call it that too. ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Adapting pypy benchmark suite
On Thu, Apr 14, 2016 at 6:51 PM, Brett Cannon wrote: > > > On Wed, 13 Apr 2016 at 13:33 Zachary Ware > wrote: >> >> On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski >> wrote: >> > Hi >> > >> > I have a radical idea: to take a pypy benchmark suite, update the >> > libraries to newer ones and replace python benchmarks with that. The >> > main reason being that pypy has a much better coverage of things that >> > are not microbenchmarks, the list (in json): >> > >> > http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/ >> > >> > Which is much more extensive than this: >> > >> > https://hg.python.org/benchmarks/file/tip/performance >> > >> > I'm willing to put *some* effort, what do people think? >> >> I'm in favor. My support has two conditions, though: >> >> 1) at least a majority of the benchmarks must be Python3 compatible. >> Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are >> 2 compatible anyway. > > > Agreed (although I don't care about the 2/3 compatibility, just the 3 compat > ;) . > >> >> >> 2) and there should be an easy way to run the benchmarks against >> exactly 1 interpreter (for use with speed.python.org). I initially >> tried to set up speed.python.org using the PyPy benchmarks, but >> quickly ran into issues with trying to use 'nullpython.py' as the >> baseline Python. When I switched to using h.p.o/benchmarks, I added >> the '--raw' flag to perf.py which allows the benchmarks to be run on >> one interpreter instead of two. It was just a quick hack, though; I >> have no problems with that feature completely changing (even invoking >> it a different way is ok), so long as it exists. >> >> This project could probably start its life as >> github.com/python/benchmarks and save us from having to migrate >> h.p.o/benchmarks to GitHub. > > > Yep, I'm willing to postpone moving the benchmarks repo from hg.python.org > if works starts on this idea and then not move the old repo at all if this > succeeds. We could then make the people who care about the benchmarks the > maintainers of the new repository and contribute to it directly (which means > interested people from CPython, PyPy, Pyston, IronPython, and Jython). That > way we all get exposure to everyone's benchmarks and there's no more > benchmark fragmentation because some people disagree with another's approach > (i.e. no more benchmark silos among the Python implementations). > > And just because we're talking a new repo, would it be worth considering > relying on pip to grab libraries instead of embedding them in the > repository, hence shrinking the overall size of the repo? > Both make sense - to benchmark against the latest lib X and to benchmark against a pinned lib X ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Adapting pypy benchmark suite
On Wed, Apr 13, 2016 at 10:33 PM, Zachary Ware wrote: > On Wed, Apr 13, 2016 at 1:57 PM, Maciej Fijalkowski wrote: >> Hi >> >> I have a radical idea: to take a pypy benchmark suite, update the >> libraries to newer ones and replace python benchmarks with that. The >> main reason being that pypy has a much better coverage of things that >> are not microbenchmarks, the list (in json): >> >> http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/ >> >> Which is much more extensive than this: >> >> https://hg.python.org/benchmarks/file/tip/performance >> >> I'm willing to put *some* effort, what do people think? > > I'm in favor. My support has two conditions, though: > > 1) at least a majority of the benchmarks must be Python3 compatible. > Preferably 2/3 compatible, but I assume all of the PyPy benchmarks are > 2 compatible anyway. The 3-compatible is likely about updating the libs > > 2) and there should be an easy way to run the benchmarks against > exactly 1 interpreter (for use with speed.python.org). I initially > tried to set up speed.python.org using the PyPy benchmarks, but > quickly ran into issues with trying to use 'nullpython.py' as the > baseline Python. When I switched to using h.p.o/benchmarks, I added > the '--raw' flag to perf.py which allows the benchmarks to be run on > one interpreter instead of two. It was just a quick hack, though; I > have no problems with that feature completely changing (even invoking > it a different way is ok), so long as it exists. That is something that we're tackling on "single-run" branch, check it out, I will finish it, maybe that's a good reason to finish it > > This project could probably start its life as > github.com/python/benchmarks and save us from having to migrate > h.p.o/benchmarks to GitHub. > > -- > Zach > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
[Speed] Adapting pypy benchmark suite
Hi I have a radical idea: to take a pypy benchmark suite, update the libraries to newer ones and replace python benchmarks with that. The main reason being that pypy has a much better coverage of things that are not microbenchmarks, the list (in json): http://paste.pound-python.org/show/4YVq0fv6pv8rVOSmCTag/ Which is much more extensive than this: https://hg.python.org/benchmarks/file/tip/performance I'm willing to put *some* effort, what do people think? Cheers, fijal ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Performance comparison of regular expression engines
this is really difficult to read, can you tell me which column am I looking at?
On Sun, Mar 6, 2016 at 11:21 AM, Serhiy Storchaka wrote:
> On 06.03.16 09:14, Maciej Fijalkowski wrote:
>> Any chance you can rerun this on pypy?
>
> Results on PyPy 2.2.1 (I'm not sure I could build the last PyPy on my
> computer):
>
>re str.find
>
> Twain 5 5.469 3.852
> (?i)Twain 10 8.646
> [a-z]shing165 17.24
> Huck[a-zA-Z]+|Saw[a-zA-Z]+ 52 7.763
> \b\w+nn\b 32 101
> [a-q][^u-z]{13}x 445 167.6
> Tom|Sawyer|Huckleberry|Finn 314 8.583
> (?i)Tom|Sawyer|Huckleberry|Finn 47716.3
> .{0,2}(Tom|Sawyer|Huckleberry|Finn) 314 270.9
> .{2,4}(Tom|Sawyer|Huckleberry|Finn) 237 262
> Tom.{10,25}river|river.{10,25}Tom 1 8.461
> [a-zA-Z]+ing10079 348
> \s[a-zA-Z]{0,12}ing\s7160 115.8
> ([A-Za-z]awyer|[A-Za-z]inn)\s 50 16.62
> ["'][^"']{0,30}[?!\.]["']1618 14.45
>
> Alternative regular expression engines need extension modules and don't work
> on PyPy for me.
>
> For comparison results on CPython 2.7.11+:
>
>re regexre2 pcre
> str.find
>
> Twain 5 4.423 2.699 8.045 93.4 4.181
> (?i)Twain 10 50.07 3.563 20.35 185.6
> [a-z]shing165 98.68 6.365 23.71 2886
> Huck[a-zA-Z]+|Saw[a-zA-Z]+ 52 58.97 50.26 19.52 1016
> \b\w+nn\b 32 130.1 416.5 18.38 740.7
> [a-q][^u-z]{13}x 445 406.6 7.935 5886 7137
> Tom|Sawyer|Huckleberry|Finn 314 53.09 59.1 20.33 5377
> (?i)Tom|Sawyer|Huckleberry|Finn 477 281.2 338.5 23.77 7895
> .{0,2}(Tom|Sawyer|Huckleberry|Finn) 314 419.5 1142 20.69 6423
> .{2,4}(Tom|Sawyer|Huckleberry|Finn) 237 410.9 1013 18.99 5224
> Tom.{10,25}river|river.{10,25}Tom 1 63.17 58.31 18.94 260.2
> [a-zA-Z]+ing10079 203.8 363.8 43.78 1.583e+05
> \s[a-zA-Z]{0,12}ing\s7160 127.1 26.65 34.23 1.114e+05
> ([A-Za-z]awyer|[A-Za-z]inn)\s 50 147.6 412.4 21.57 1172
> ["'][^"']{0,30}[?!\.]["']1618 85.88 86.55 22.22 2.576e+04
>
> And on Jython 2.5.3 with JRE 7:
>
>re str.find
>
> Twain 5 34 3
> (?i)Twain 10 251
> [a-z]shing165 564
> Huck[a-zA-Z]+|Saw[a-zA-Z]+ 52 281
> \b\w+nn\b 32 510
> [a-q][^u-z]{13}x 4451786
> Tom|Sawyer|Huckleberry|Finn 314 102
> (?i)Tom|Sawyer|Huckleberry|Finn 4771232
> .{0,2}(Tom|Sawyer|Huckleberry|Finn) 3141345
> .{2,4}(Tom|Sawyer|Huckleberry|Finn) 2371353
> Tom.{10,25}river|river.{10,25}Tom 1 305
> [a-zA-Z]+ing100791211
> \s[a-zA-Z]{0,12}ing\s7160 571
> ([A-Za-z]awyer|[A-Za-z]inn)\s 50 676
> ["'][^"']{0,30}[?!\.]["']1618 431
>
>
> ___
> Speed mailing list
> Speed@python.org
> https://mail.python.org/mailman/listinfo/speed
___
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Performance comparison of regular expression engines
Hi serhiy
Any chance you can rerun this on pypy?
On Sat, Mar 5, 2016 at 8:35 PM, Serhiy Storchaka wrote:
> I have wrote a benchmark for comparing different regular expression engines
> available in Python. It uses tests and data from [1], that were itself
> inspired by Boost's benchmark [2].
>
> Tested engines are:
>
> * re, standard regular expression module
> * regex, alternative regular expression module [3]
> * re2, Python wrapper for Google's RE2 [4]
> * pcre, Python PCRE bindings [5]
>
> Running tests for all 20MB text file takes too long time, here are results
> (time in millisecons) for 2MB chunk (600:800):
>
>re regexre2 pcre
> str.find
>
> Twain 5 2.866 2.118 12.47 3.911 2.72
> (?i)Twain 10 84.42 4.366 24.76 17.12
> [a-z]shing165 125 5.466 27.78 180.6
> Huck[a-zA-Z]+|Saw[a-zA-Z]+ 52 57.11 72.16 23.87234
> \b\w+nn\b 32 239.5 427.6 23.18 251.9
> [a-q][^u-z]{13}x 445 381.8 5.537 5843 224.9
> Tom|Sawyer|Huckleberry|Finn 314 52.73 58.45 24.39 422.5
> (?i)Tom|Sawyer|Huckleberry|Finn 477 445.6 522.1 27.73 415.4
> .{0,2}(Tom|Sawyer|Huckleberry|Finn) 314 451.2 1113 24.38 1497
> .{2,4}(Tom|Sawyer|Huckleberry|Finn) 237 450.1 1000 24.3 1549
> Tom.{10,25}river|river.{10,25}Tom 1 61.55 58.11 24.97 233.8
> [a-zA-Z]+ing10079 189.4 350.3 47.41 357.6
> \s[a-zA-Z]{0,12}ing\s7160 115.7 23.65 37.74 237.6
> ([A-Za-z]awyer|[A-Za-z]inn)\s 50 153.7 430.4 27.86 425.3
> ["'][^"']{0,30}[?!\.]["']1618 83.12 77.39 26.96 157.6
>
> There is no absolute leader. All engines have its weak spots. For re these
> are case-insensitive search and search a pattern that starts with a set.
>
> pcre is very data-sensitive. For other 2Mb chunk (800:1000) results
> are 1-2 orders slower:
>
>re regexre2 pcre
> str.find
>
> Twain 33 2.671 2.209 16.6 413.6 2.75
> (?i)Twain 35 90.21 4.36 27.65 459.4
> [a-z]shing120 112.7 2.667 30.94 1895
> Huck[a-zA-Z]+|Saw[a-zA-Z]+ 61 57.12 49.9 26.76 1152
> \b\w+nn\b 33 238 401.4 26.93 763.7
> [a-q][^u-z]{13}x 481 387.7 5.694 5915 6979
> Tom|Sawyer|Huckleberry|Finn 845 52.89 59.61 28.42 1.228e+04
> (?i)Tom|Sawyer|Huckleberry|Finn 988 452.3 523.4 32.15 1.426e+04
> .{0,2}(Tom|Sawyer|Huckleberry|Finn) 845 421.1 1105 29.01 1.343e+04
> .{2,4}(Tom|Sawyer|Huckleberry|Finn) 625 398.6 985.6 29.19 9878
> Tom.{10,25}river|river.{10,25}Tom 161.6 58.33 26.59 254.1
> [a-zA-Z]+ing10109 194.5 349.7 50.85 1.445e+05
> \s[a-zA-Z]{0,12}ing\s7286 120.1 23.73 42.04 1.051e+05
> ([A-Za-z]awyer|[A-Za-z]inn)\s 43 170.6 402.9 30.84 1119
> ["'][^"']{0,30}[?!\.]["']168686.5 110.2 30.62 2.369e+04
>
> [1] http://sljit.sourceforge.net/regex_perf.html
> [2] http://www.boost.org/doc/libs/1_36_0/libs/regex/doc/vc71-performance.html
> [3] https://pypi.python.org/pypi/regex/2016.03.02
> [4] https://pypi.python.org/pypi/re2/0.2.22
> [5] https://pypi.python.org/pypi/python-pcre/0.7
>
> ___
> Speed mailing list
> Speed@python.org
> https://mail.python.org/mailman/listinfo/speed
>
___
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark
Hi. Disabling ASLR means you get more repeatable benchmarks, of course, but also means that on another identical machine (or a bit different circumstances), you can get different results, hence you moved statistical error to a more systematic one. I don't think that's a win On Fri, Feb 12, 2016 at 8:42 AM, Patrascu, Alecsandru wrote: > Hi, > > Some of the things we do here at Intel, in our Languages Performance Lab > [1,2], is to disable ASLR as you get more reliable results. This can be > achieved on Linux by running echo 0 > /proc/sys/kernel/randomize_va_space. > Also, setting the CPU frequency at a fixed frequency, disabling Turbo Boost > and Hyper Threading, also helps for benchmark stability. > > From my experience, the isolcpus feature is useful when you have a lot of > cores on your machine because the kernel will have other cores on which it > can schedule its work; furthermore, it is a best effort situation and it is > not an absolute guarantee that the kernel will not use the cores specified if > you have a lot of processes running (for example, if you benchmark on a > machine with 2 physical cores and you isolate one of the cores, there is a > big chance that the kernel will schedule processes on this core also, even it > is for a small amount of time). Nevertheless, for machines with more physical > cores, it can be good to have dedicated core(s) on which we do benchmarking. > > [1] http://languagesperformance.intel.com/ > [2] https://lists.01.org/pipermail/langperf/ > > Thank you, > Alecsandru > >> -Original Message- >> From: Speed [mailto:speed- >> bounces+alecsandru.patrascu=intel@python.org] On Behalf Of Victor >> Stinner >> Sent: Friday, February 12, 2016 12:54 AM >> To: speed@python.org >> Subject: [Speed] Linux tip: use isolcpus to have (more) reliable benchmark >> >> Hi, >> >> I'm sharing with you my notes (tricks) to get more reliable benchmarks on >> Linux if your CPU have multiple cores: >> >> https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micro- >> benchmarks >> >> FYI perf.py recently got a new --affinity= optional parameter. I plan to >> send a patch to automatically use /sys/devices/system/cpu/isolated if it's >> not empty. >> >> What are your "tricks" to get reliable benchmarks? >> >> Victor >> ___ >> Speed mailing list >> Speed@python.org >> https://mail.python.org/mailman/listinfo/speed > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Experiences with Microbenchmarking
On Fri, Feb 12, 2016 at 7:00 PM, Armin Rigo wrote: > Hi Paul, > > On Fri, Feb 12, 2016 at 5:00 PM, Paul wrote: >>> PyPy typically needs more than 2000 iterations to be warmed up. >> >> Same goes for the JVM. Off the top of my head it doesn't even start marking >> a method as hot until around 10,000 iterations (at which point it'll start >> to do the first stage of optimisations). If you're below that threshold >> you're dealing with pure interpreter performance. > > Ew, it's even longer than PyPy :-) > > In the PyPy case, the number 2000 is particularly bad, because the JIT > starts after 1039 iterations. It also adds a few extra paths > afterwards, starting maybe around ~400-500 extra iterations (as a mean > value). Each time, the JIT produces more machine code > and there is a relatively important pause. So 2000 is close to the > worst case: even running 2000 purely-interpreted iterations would be > faster. > > > A bientôt, > > Armin. > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed Armin, those are "2000 iterations of a benchmark" and not "2000 iterations of a loop". A lot of those are pypy benchmarks, just run longer ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Tool to run Python microbenchmarks
On Fri, Feb 12, 2016 at 4:58 PM, Victor Stinner
wrote:
> Hi,
>
> 2016-02-12 16:42 GMT+01:00 Maciej Fijalkowski :
>> timeit does two really terrible things - uses min(time) and disables
>> the garbage collector, which makes it completely unreliable.
>
> Can you please elaborate why using min(times) is a bad idea?
>
> I'm also using min() in my tool, I expect that it helps to ignore the
> sporadic peeks when the system is unstable.
>
> Victor
Yes, it also helps to ignore systematic peaks that will happen
randomly (due to cache alignment, memory ordering, dicts etc.). Some
operations are really random that you should not ignore. E.g. if you
have:
l.append('a') in a loop, you gonna ignore all the places that resize loop.
I'll look for a reference
___
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Tool to run Python microbenchmarks
Hi Victor timeit does two really terrible things - uses min(time) and disables the garbage collector, which makes it completely unreliable. On Thu, Feb 11, 2016 at 11:39 PM, Victor Stinner wrote: > Hi, > > To run "micro"-benchmarks on "micro"-optimizations, I started to use > timeit, but in my experience timeit it far from reliable. > > When I say micro: I'm talking about a test which takes less than 1000 > ns, sometimes even a few nanoseconds! > > You always have to run the same micro-benchmark when timeit *at least* > 5 times to find the "real" "minimum" runtime. > > That's why I wrote my own tool to run microbenchmarks: > https://bitbucket.org/haypo/misc/src/tip/python/benchmark.py > > Yury suggested me to add this tool to the Python benchmark project. > I'm ok with that, but only if we rename it to "microbench.py" :-) I > wrote this tool to compare micro-optimizations with a long list of > very simple tests. The result is written into a file. Then you can > compare two files and compare more files, and maybe even compare > multiple files to a "reference". It "hides" difference smaller than 5% > to ignore the noise. > > The main feature is benchmark.py is that it calibrates the benchmark > using time to choose the number of runs and number of loops. I > proposed a similar idea for perf.py: > https://bugs.python.org/issue26275 > > What do you think? Would this tool be useful? > > Victor > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Should we change what benchmarks we have?
I presume you looked at the pypy benchmark suite, which contains a large collection of library-based benchmarks. You can endlessly argue whether it's "macro enough", but it does cover some usages of various libraries as submitted/written with help from lib authors (sympy, twisted, various templating engines, sqlalchemy ORM, etc.) as well as interesting python programs that are CPU intensive found on the interwebs. On Fri, Feb 12, 2016 at 1:31 PM, Antoine Pitrou wrote: > On Thu, 11 Feb 2016 18:36:33 + > Brett Cannon wrote: >> Are we happy with the current benchmarks? Are there some we want to drop? >> How about add? Do we want to have explanations as to why each benchmark is >> included? > > There are no real explanations except the provenance of said benchmarks: > - the benchmarks suite was originally developed for Unladen Swallow > - some benchmarks were taken and adapted from the "Great Computer > Language Shootout" (which I think is a poor source of benchmarks) > - some benchmarks have been added for specific concerns that may not be > of enough interest in general (for example micro-benchmarks of > methods calls, or benchmarks of json / pickle performance) > >> A better balance of micro vs. macro benchmarks (and probably >> matching groups)? > > Easier said than done :-) Macro-benchmarks are harder to write, > especially with the constraints that 1) runtimes should be short enough > for convenient use 2) performance numbers should be stable enough > accross runs. > > Regards > > Antoine. > > > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] People interested in reworking the benchmark suite in 2016?
count me in (even if I don't make it to pycon) On Sun, Jan 3, 2016 at 9:57 PM, Brett Cannon wrote: > With the planned move to GitHub, there is an opportunity to try and rework > the set of benchmarks -- and anything else -- in 2016 by starting a new > benchmark repo from scratch. E.g., modern numeric benchmarks, long-running > benchmarks that warm up JITs, using pip with pegged bugfix versions so we > stop shipping library code with the benchmarks, etc. We could also > standardize results output -- e.g. should we just make everything run under > codespeed? -- so that the benchmarks are easy to run locally for one-off > results as well as continuous benchmarking for trend details with a common > benchmark driver? > > Would people be interested and motivated enough in getting representatives > from the various Python implementations together at PyCon and have a BoF to > discuss what we want from a proper, unified, baseline benchmark suite and > see if we can pull one together -- or at least start one -- in 2016? > > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed > ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Proposing Tornado benchmark
that patch is against what? anyway, I can incorporate it in PyPy repo, thanks On Sat, Oct 12, 2013 at 9:19 PM, Antoine Pitrou wrote: > > Hello, > > In http://bugs.python.org/issue19236 I've proposed a patch for a > very simple Tornado-based benchmark of an HTTP server and client. > > Regards > > Antoine. > > > ___ > Speed mailing list > Speed@python.org > https://mail.python.org/mailman/listinfo/speed ___ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
Re: [Speed] Status of speed.python.org?
On Sun, Oct 21, 2012 at 2:06 PM, Carsten Senger wrote: > > > Am 21.10.2012 11:08, schrieb Maciej Fijalkowski: >> On Sat, Oct 20, 2012 at 1:21 PM, Carsten Senger wrote: >>> Am 19.10.2012 21:38, schrieb Miquel Torres: >>>> Right. On the webapp (Codespeed) side of things I am willing to help >>>> in anything you need. >>>> The blocker has been mostly the benchmark runner, AFAIK. >>> >>> Some month ago I worked on the benchmark runner and the build slave >>> helpers in the pypy repository. They can run the test suite for both >>> pypy and cpython. It will need no or minor tweeks to use python3. What >>> was missing back then, and IIRC planned to be discussed at PyCo, was the >>> port of the benchmarks to python 3. >>> >>> Now it can be set up on speed.p.o (with minor adjustments if necessary). >>> I'd do work on that too. >>> >>> ..Carsten >>> >> >> FYI the running on python 2 never worked (because of option passing). >> It roughly shows up how people care (or how much they know). I suppose >> we can declare "running benchmarks on python 2" a completely pointless >> exercise. > > > I know I cared. It worked with my patches that where merged by you. They > removed the hard coded configuration from the build factories and the > runner. I stopped then cause nobody was interested to collect data for > 2.7 which would not receive notable changes. Same applied to historical > data for python 2. Pointless describes that very well, but for different > reasons. > > There where no python 3 benchmarks and the how and when seemed more like > a political discussion to me. > > > ..Carsten I'm sorry, my mail was definitely too harsh. For what is worth, there are python 3 benchmarks now. Maybe we should somehow restart it? Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] Status of speed.python.org?
On Sat, Oct 20, 2012 at 1:21 PM, Carsten Senger wrote: > Am 19.10.2012 21:38, schrieb Miquel Torres: >> Right. On the webapp (Codespeed) side of things I am willing to help >> in anything you need. >> The blocker has been mostly the benchmark runner, AFAIK. > > Some month ago I worked on the benchmark runner and the build slave > helpers in the pypy repository. They can run the test suite for both > pypy and cpython. It will need no or minor tweeks to use python3. What > was missing back then, and IIRC planned to be discussed at PyCo, was the > port of the benchmarks to python 3. > > Now it can be set up on speed.p.o (with minor adjustments if necessary). > I'd do work on that too. > > ..Carsten > FYI the running on python 2 never worked (because of option passing). It roughly shows up how people care (or how much they know). I suppose we can declare "running benchmarks on python 2" a completely pointless exercise. ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] Status of speed.python.org?
On Sat, Oct 20, 2012 at 12:08 AM, Brett Cannon wrote: > > > On Fri, Oct 19, 2012 at 3:38 PM, Miquel Torres wrote: >> >> Right. On the webapp (Codespeed) side of things I am willing to help >> in anything you need. >> The blocker has been mostly the benchmark runner, AFAIK. >> > > And how specifically is that a blocker so we can work on eliminating those > issues? > > -Brett Log in, setup a buildbot, make sure it's connected to the python buildmaster and it's running. Before that go to Noah and ask for an account. Then set up a codespeed instance somewhere (not sure where) and connect the two together. ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] Status of speed.python.org?
On Fri, Oct 19, 2012 at 12:46 PM, Jesse Noller wrote: > > > On Oct 19, 2012, at 6:25 AM, Maciej Fijalkowski wrote: > >> On Thu, Oct 18, 2012 at 10:40 PM, Victor Stinner >> wrote: >>> Hi, >>> >>> What is the status of speed.python.org? Where are the benchmarks? Does >>> anyone try to setup something to run regulary benchmarks and display >>> data on web pages? >>> >>> How can I help? >>> >>> Victor >> >> Hi Victor. >> >> The status is "noone cares". > > Hi. I care. Other people do too. Maybe you don't - that's ok. The problem is > lack of time / project planning. Ok, to be precise "noone cares enough to make things happen", this is a fact. I actually care to some extend and I'm willing to help with stuff, however, there must be someone who cares more on the python core development team to make the exercise not-completely-pointless. Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] Status of speed.python.org?
On Thu, Oct 18, 2012 at 10:40 PM, Victor Stinner wrote: > Hi, > > What is the status of speed.python.org? Where are the benchmarks? Does > anyone try to setup something to run regulary benchmarks and display > data on web pages? > > How can I help? > > Victor Hi Victor. The status is "noone cares". Brett ported a bunch of py3k benchmarks, so I suppose this is a very good start. Someone has to add a buildslave there (preferably for cpython buildbots) that runs runner.py from US and uploads the info to codespeed. Codespeed can be also set up somewhere (I did, it's not the end of the world). Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] slow benchmark
On Mon, Oct 1, 2012 at 2:04 AM, Antoine Pitrou wrote: > On Sun, 30 Sep 2012 19:58:02 -0400 > Brett Cannon wrote: >> On Sun, Sep 30, 2012 at 7:21 PM, Antoine Pitrou wrote: >> >> > >> > Hello, >> > >> > The hexiom benchmark is very slow. Is there a reason it's included >> > there? >> > >> >> Already been asked and answered: >> http://mail.python.org/pipermail/speed/2012-September/000209.html > > I didn't realize when reading this discussion that hexiom2 was *that* > slow. I don't think a benchmark taking 100+ seconds to run *in fast > mode* has a place in the benchmark suite. PyPy can maintain their own > benchmarks in their source tree, like CPython does. > > Regards > > Antoine. I strongly disagree. There are quite a few slow benchmarks that are very useful, like pypy translation toolchain. How about you skip this one in fast mode? ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
[Speed] Speed and disk space
Hello everyone. I would like to complain that speed has still no disk space, despite having a huge empty disk space on no partition chilling there. Last time I complained was July 13th and it's a little over 2 months by now. Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] standalone PyPy benchmarks ported
On Mon, Sep 17, 2012 at 8:17 PM, Brett Cannon wrote: > > > On Mon, Sep 17, 2012 at 11:36 AM, Maciej Fijalkowski > wrote: >> >> On Mon, Sep 17, 2012 at 5:00 PM, Brett Cannon wrote: >> > >> > >> > On Sun, Sep 16, 2012 at 10:54 AM, Maciej Fijalkowski >> > wrote: >> >> >> >> On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon wrote: >> >> > Quick question about the hexiom2 benchmark: what does it measure? It >> >> > is >> >> > by >> >> > far the slowest benchmark I ported, and considering it isn't a >> >> > real-world >> >> > app benchmark I want to make sure the slowness of it is worth it. >> >> > Otherwise >> >> > I would rather drop it since having something run 1/25 as many >> >> > iterations >> >> > compared to the other simple benchmarks seems to water down its >> >> > robustness. >> >> >> >> It's a puzzle solver. It got included because PyPy 1.9 got slower than >> >> 1.8 on this particular benchmark that people were actually running >> >> somewhere, so it has *some* value. >> > >> > >> > Fair enough. Just wanted to make sure that it was worth having a slow >> > execution over. >> > >> >> >> >> I wonder, does adding a fixed >> >> random number seed help the distribution? >> > >> > >> > Fix how? hexiom2 doesn't use a random value for anything. >> >> Ok, then please explain why having 1/25th of iterations kill robustness? > > > Less iterations to help smooth out any bumps in the measurements. E.g 4 > iterations compared to 100 doesn't lead to as even of a measurement. I mean > you would hope because the benchmark goes for so long that it would just > level out within a single run instead of needing multiple runs to get the > same evening out. Yes precisely :) I think the term "iterations" is a bit overloaded. The "stability" is more important. > > -Brett > >> >> >> > >> > -Brett >> > >> >> >> >> >> >> > >> >> > >> >> > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski >> >> > >> >> > wrote: >> >> >> >> >> >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon >> >> >> wrote: >> >> >> > So I managed to get the following benchmarks moved into the >> >> >> > unladen >> >> >> > repo >> >> >> > (not pushed yet until I figure out some reasonable scaling values >> >> >> > as >> >> >> > some >> >> >> > finish probably too fast and others go for a while): >> >> >> > >> >> >> > chaos >> >> >> > fannkuch >> >> >> > meteor-contest (renamed meteor_contest) >> >> >> > spectral-norm (renamed spectral_norm) >> >> >> > telco >> >> >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this >> >> >> > benchmark) >> >> >> > go >> >> >> > hexiom2 >> >> >> > json_bench (renamed json_dump_v2) >> >> >> > raytrace_simple (renamed raytrace) >> >> >> > >> >> >> > Most of the porting was range/xrange related. After that is was >> >> >> > str/unicode. >> >> >> > I also stopped having the benchmarks write out files as it was >> >> >> > always >> >> >> > to >> >> >> > verify results and not a core part of the benchmark. >> >> >> > >> >> >> > That leaves us with the benchmarks that rely on third-party >> >> >> > projects. >> >> >> > The >> >> >> > chameleon benchmark can probably be ported as chameleon has a >> >> >> > version >> >> >> > released running on Python 3. But django and html5lib have only >> >> >> > in-development versions that support Python 3. If we want to pull >> >> >> > in >> >> >> > the >> >> >> > tip >> >> >> > of their repos then those benchmarks can also be ported now rather >> >> >> > than >> >> >> > later. People have opinions on in-dev code vs. released for >> >> >> > benchmarking? >> >> >> > >> >> >> > There is also the sphinx benchmark, but that requires getting >> >> >> > CPython's >> >> >> > docs >> >> >> > building under Python 3 (see http://bugs.python.org/issue10224). >> >> >> > >> >> >> > ___ >> >> >> > Speed mailing list >> >> >> > Speed@python.org >> >> >> > http://mail.python.org/mailman/listinfo/speed >> >> >> > >> >> >> >> >> >> great job! >> >> > >> >> > >> > >> > > > ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] standalone PyPy benchmarks ported
On Mon, Sep 17, 2012 at 5:00 PM, Brett Cannon wrote: > > > On Sun, Sep 16, 2012 at 10:54 AM, Maciej Fijalkowski > wrote: >> >> On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon wrote: >> > Quick question about the hexiom2 benchmark: what does it measure? It is >> > by >> > far the slowest benchmark I ported, and considering it isn't a >> > real-world >> > app benchmark I want to make sure the slowness of it is worth it. >> > Otherwise >> > I would rather drop it since having something run 1/25 as many >> > iterations >> > compared to the other simple benchmarks seems to water down its >> > robustness. >> >> It's a puzzle solver. It got included because PyPy 1.9 got slower than >> 1.8 on this particular benchmark that people were actually running >> somewhere, so it has *some* value. > > > Fair enough. Just wanted to make sure that it was worth having a slow > execution over. > >> >> I wonder, does adding a fixed >> random number seed help the distribution? > > > Fix how? hexiom2 doesn't use a random value for anything. Ok, then please explain why having 1/25th of iterations kill robustness? > > -Brett > >> >> >> > >> > >> > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski >> > wrote: >> >> >> >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon >> >> wrote: >> >> > So I managed to get the following benchmarks moved into the unladen >> >> > repo >> >> > (not pushed yet until I figure out some reasonable scaling values as >> >> > some >> >> > finish probably too fast and others go for a while): >> >> > >> >> > chaos >> >> > fannkuch >> >> > meteor-contest (renamed meteor_contest) >> >> > spectral-norm (renamed spectral_norm) >> >> > telco >> >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this >> >> > benchmark) >> >> > go >> >> > hexiom2 >> >> > json_bench (renamed json_dump_v2) >> >> > raytrace_simple (renamed raytrace) >> >> > >> >> > Most of the porting was range/xrange related. After that is was >> >> > str/unicode. >> >> > I also stopped having the benchmarks write out files as it was always >> >> > to >> >> > verify results and not a core part of the benchmark. >> >> > >> >> > That leaves us with the benchmarks that rely on third-party projects. >> >> > The >> >> > chameleon benchmark can probably be ported as chameleon has a version >> >> > released running on Python 3. But django and html5lib have only >> >> > in-development versions that support Python 3. If we want to pull in >> >> > the >> >> > tip >> >> > of their repos then those benchmarks can also be ported now rather >> >> > than >> >> > later. People have opinions on in-dev code vs. released for >> >> > benchmarking? >> >> > >> >> > There is also the sphinx benchmark, but that requires getting >> >> > CPython's >> >> > docs >> >> > building under Python 3 (see http://bugs.python.org/issue10224). >> >> > >> >> > ___ >> >> > Speed mailing list >> >> > Speed@python.org >> >> > http://mail.python.org/mailman/listinfo/speed >> >> > >> >> >> >> great job! >> > >> > > > ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] standalone PyPy benchmarks ported
On Sun, Sep 16, 2012 at 4:43 PM, Brett Cannon wrote: > Quick question about the hexiom2 benchmark: what does it measure? It is by > far the slowest benchmark I ported, and considering it isn't a real-world > app benchmark I want to make sure the slowness of it is worth it. Otherwise > I would rather drop it since having something run 1/25 as many iterations > compared to the other simple benchmarks seems to water down its robustness. It's a puzzle solver. It got included because PyPy 1.9 got slower than 1.8 on this particular benchmark that people were actually running somewhere, so it has *some* value. I wonder, does adding a fixed random number seed help the distribution? > > > On Fri, Sep 14, 2012 at 5:44 PM, Maciej Fijalkowski > wrote: >> >> On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon wrote: >> > So I managed to get the following benchmarks moved into the unladen repo >> > (not pushed yet until I figure out some reasonable scaling values as >> > some >> > finish probably too fast and others go for a while): >> > >> > chaos >> > fannkuch >> > meteor-contest (renamed meteor_contest) >> > spectral-norm (renamed spectral_norm) >> > telco >> > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this >> > benchmark) >> > go >> > hexiom2 >> > json_bench (renamed json_dump_v2) >> > raytrace_simple (renamed raytrace) >> > >> > Most of the porting was range/xrange related. After that is was >> > str/unicode. >> > I also stopped having the benchmarks write out files as it was always to >> > verify results and not a core part of the benchmark. >> > >> > That leaves us with the benchmarks that rely on third-party projects. >> > The >> > chameleon benchmark can probably be ported as chameleon has a version >> > released running on Python 3. But django and html5lib have only >> > in-development versions that support Python 3. If we want to pull in the >> > tip >> > of their repos then those benchmarks can also be ported now rather than >> > later. People have opinions on in-dev code vs. released for >> > benchmarking? >> > >> > There is also the sphinx benchmark, but that requires getting CPython's >> > docs >> > building under Python 3 (see http://bugs.python.org/issue10224). >> > >> > ___ >> > Speed mailing list >> > Speed@python.org >> > http://mail.python.org/mailman/listinfo/speed >> > >> >> great job! > > ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] standalone PyPy benchmarks ported
On Fri, Sep 14, 2012 at 10:19 PM, Brett Cannon wrote: > So I managed to get the following benchmarks moved into the unladen repo > (not pushed yet until I figure out some reasonable scaling values as some > finish probably too fast and others go for a while): > > chaos > fannkuch > meteor-contest (renamed meteor_contest) > spectral-norm (renamed spectral_norm) > telco > bm_mako (renamed bm_mako_v2; also pulled in mako 0.9.7 for this benchmark) > go > hexiom2 > json_bench (renamed json_dump_v2) > raytrace_simple (renamed raytrace) > > Most of the porting was range/xrange related. After that is was str/unicode. > I also stopped having the benchmarks write out files as it was always to > verify results and not a core part of the benchmark. > > That leaves us with the benchmarks that rely on third-party projects. The > chameleon benchmark can probably be ported as chameleon has a version > released running on Python 3. But django and html5lib have only > in-development versions that support Python 3. If we want to pull in the tip > of their repos then those benchmarks can also be ported now rather than > later. People have opinions on in-dev code vs. released for benchmarking? > > There is also the sphinx benchmark, but that requires getting CPython's docs > building under Python 3 (see http://bugs.python.org/issue10224). > > ___ > Speed mailing list > Speed@python.org > http://mail.python.org/mailman/listinfo/speed > great job! ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] What PyPy benchmarks are (un)important?
On Thu, Sep 13, 2012 at 12:35 AM, Brett Cannon wrote: > I went through the list of benchmarks that PyPy has to see which ones could > be ported to Python 3 now (others can be in the future but they depend on a > project who has not released an official version with python 3 support): > > ai > chaos > fannkuch > float > meteor-contest > nbody_modified > richards > spectral-norm > telco > > bm_chameleon* > bm_mako > go > hexiom2 > json_bench > pidigits > pyflate-fast > raytrace-simple > sphinx* > > The first grouping is the 20 shown on the speed.pypy.org homepage, the rest > are in the complete list. Anything with an asterisk has an external > dependency that is not already in the unladen benchmarks. > > Are the twenty shown on the homepage of speed.pypy.org in some way special, > or were they the first benchmarks that you were good/bad at, or what? Are > there any benchmarks here that are particularly good or bad? I'm trying to > prioritize what benchmarks I port so that if I hit a time crunch I got the > critical ones moved first. The 20 shown on the front page are the ones that we have full historical data, so we can compare. Others are simply newer. I don't think there is any priority associated, we should probably put the others on the first page despite not having full data. ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] Are benchmarks and libraries mutable?
On Sun, Sep 2, 2012 at 12:10 AM, Brett Cannon wrote: > > > On Sat, Sep 1, 2012 at 2:57 PM, Antoine Pitrou wrote: >> >> On Sat, 1 Sep 2012 13:21:36 -0400 >> Brett Cannon wrote: >> > >> > One is moving benchmarks from PyPy over to the unladen repo on >> > hg.python.org/benchmarks. But I wanted to first make sure people don't >> > view >> > the benchmarks as immutable (e.g. as Octane does: >> > https://developers.google.com/octane/faq). Since the benchmarks are >> > always >> > relative between two interpreters their immutability isn't critical >> > compared to if we were to report some overall score. But it also means >> > that >> > any changes made would throw off historical comparisons. For instance, >> > if I >> > take PyPy's Mako benchmark (which does a lot more work), should it be >> > named >> > mako_v2, or should we just replace mako wholesale? >> >> mako_v2 sounds fine to me. Mutating benchmarks makes things confusing: >> one person may report that interpreter A is faster than interpreter B >> on a given benchmark, and another person retort that no, interpreter B >> is faster than interpreter A. >> >> Besides, if you want to have useful timelines on speed.p.o, you >> definitely need stable benchmarks. >> >> > And the second is the same question for libraries. For instance, the >> > unladen benchmarks have Django 1.1a0 as the version which is rather >> > ancient. And with 1.5 coming out with provisional Python 3 support I >> > obviously would like to update it. But the same questions as with >> > benchmarks crops up in reference to immutability. >> >> django_v2 sounds fine too :) > > > True, but having to carry around multiple copies of libraries just becomes a > pain. You just kill django when you introduce django v2 (alternatively you remove the history and keep the name django). Historical outdated benchmarks are not as interesting. > >> >> >> > (e.g. I will have to probably update the 2.7 code to use >> > io.BytesIO instead of StringIO.StringIO to be on more equal footing). >> >> I disagree. If io.BytesIO is faster than StringIO.StringIO then it's >> normal for the benchmark results to reflect that (ditto if it's slower). >> >> > If we can't find a reasonable way to handle all of this then what I will >> > do >> > is branch the unladen benchmarks for 2.x/3.x benchmarking, and then >> > create >> > another branch of the benchmark suite to just be for Python 3.x so that >> > we >> > can start fresh with a new set of benchmarks that will never change >> > themselves for benchmarking Python 3 itself. >> >> Why not simply add Python 3-specific benchmarks to the mix? >> You can then create a "py3" benchmark suite in perf.py (and perhaps >> also a "py2" one). > > > To avoid historical baggage and to start from a clean slate. I don't > necessarily want to carry around Python 2 benchmarks forever. It's not a > massive concern, just a nicety. If you guys want to have any cooperation with us, you have to carry Python 2 benchmarks for indefinite amount of time. Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] Are benchmarks and libraries mutable?
On Sat, Sep 1, 2012 at 9:45 PM, Gregory P. Smith wrote: > > > On Sat, Sep 1, 2012 at 10:21 AM, Brett Cannon wrote: >> >> Now that I can run benchmarks against Python 2.7 and 3.3 simultaneously, >> I'm ready to start updating the benchmarks. This involves two parts. >> >> One is moving benchmarks from PyPy over to the unladen repo on >> hg.python.org/benchmarks. But I wanted to first make sure people don't view >> the benchmarks as immutable (e.g. as Octane does: >> https://developers.google.com/octane/faq). Since the benchmarks are always >> relative between two interpreters their immutability isn't critical compared >> to if we were to report some overall score. But it also means that any >> changes made would throw off historical comparisons. For instance, if I take >> PyPy's Mako benchmark (which does a lot more work), should it be named >> mako_v2, or should we just replace mako wholesale? > > > I dislike benchmark immutability. The rest of the world including your > local computing environment where benchmarks run continues to change around > benchmarks which really makes using historical benchmark data from a run on > an old version for comparison to a recent modern run pointless. So far we (pypy) managed to maintain enough of the environment under control to have meaningful historical data. We have the same machine and we monitor whether changes introduce something new or not. Of course ideally, it's impossible, but for real world what we're doing is good enough. Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] Are benchmarks and libraries mutable?
On Sat, Sep 1, 2012 at 7:21 PM, Brett Cannon wrote: > Now that I can run benchmarks against Python 2.7 and 3.3 simultaneously, I'm > ready to start updating the benchmarks. This involves two parts. > > One is moving benchmarks from PyPy over to the unladen repo on > hg.python.org/benchmarks. But I wanted to first make sure people don't view > the benchmarks as immutable (e.g. as Octane does: > https://developers.google.com/octane/faq). Since the benchmarks are always > relative between two interpreters their immutability isn't critical compared > to if we were to report some overall score. But it also means that any > changes made would throw off historical comparisons. For instance, if I take > PyPy's Mako benchmark (which does a lot more work), should it be named > mako_v2, or should we just replace mako wholesale? > > And the second is the same question for libraries. For instance, the unladen > benchmarks have Django 1.1a0 as the version which is rather ancient. And > with 1.5 coming out with provisional Python 3 support I obviously would like > to update it. But the same questions as with benchmarks crops up in > reference to immutability. Another thing is that 2to3 can't actually be > ported using 2to3 (http://bugs.python.org/issue15834) and so that itself > will require two versions -- a 2.x version (probably from Python 2.7's > stdlib) and a 3.x version (from the 3.2 stdlib) -- which already starts to > add interesting issues for me in terms of comparing performance (e.g. I will > have to probably update the 2.7 code to use io.BytesIO instead of > StringIO.StringIO to be on more equal footing). Similar thing goes for > html5lib which has developed its Python 3 support separately from its Python > 2 code. > > If we can't find a reasonable way to handle all of this then what I will do > is branch the unladen benchmarks for 2.x/3.x benchmarking, and then create > another branch of the benchmark suite to just be for Python 3.x so that we > can start fresh with a new set of benchmarks that will never change > themselves for benchmarking Python 3 itself. That would also mean we could > start of with whatever is needed from PyPy and unladen to have the optimal > benchmark runner for speed.python.org. > > ___ > Speed mailing list > Speed@python.org > http://mail.python.org/mailman/listinfo/speed > Ideally I would like benchmarks to be immutable (have _v is fine). However, updating libraries might not be immutable (after all, you're interested in *the speed of running django*), but maybe we should mark this somehow in the history so we don't compare apples to organges. Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] merging PyPy and Python benchmark suite
On Wed, Jul 25, 2012 at 8:16 PM, Brett Cannon wrote: > > > On Wed, Jul 25, 2012 at 6:54 AM, Maciej Fijalkowski > wrote: >> >> On Wed, Jul 25, 2012 at 5:12 AM, Alex Gaynor >> wrote: >> > >> > >> > On Tue, Jul 24, 2012 at 7:37 PM, Nick Coghlan >> > wrote: >> >> >> >> On Wed, Jul 25, 2012 at 9:51 AM, Brett Cannon wrote: >> >> > >> >> > >> >> > On Tue, Jul 24, 2012 at 5:38 PM, Nick Coghlan >> >> > wrote: >> >> >> >> >> >> Antoine's right on this one - just use and redistribute the upstream >> >> >> components under their existing licenses. CPython itself is >> >> >> different >> >> >> because the PSF has chosen to reserve relicensing privileges for >> >> >> that, >> >> >> which >> >> >> requires the extra permissions granted in the contributor agreement. >> >> > >> >> > >> >> > But I'm talking about the benchmarks themselves, not the wholesale >> >> > inclusion >> >> > of Mako, etc. (which I am not worried about since the code in the >> >> > dependencies is not edited). Can we move the PyPy benchmarks >> >> > themselves >> >> > (e.g. bm_mako.py that PyPy has) over to the PSF benchmarks without >> >> > getting >> >> > contributor agreements. >> >> >> >> The PyPy team need to put a clear license notice (similar to the one >> >> in the main pypy repo) on their benchmarks repo. But yes, I believe >> >> you're right that copying that code as it stands would technically be >> >> a copyright violation, even if the PyPy team intend for it to be >> >> allowed. >> >> >> >> If you're really concerned, check with Van first, but otherwise I'd >> >> just file a bug with the PyPy folks requesting that they clarify the >> >> licensing by adding a LICENSE file and in the meantime assume they >> >> intended for it to be covered by the MIT license, just like PyPy >> >> itself. >> >> >> >> The PSF license is necessary for CPython because of the long and >> >> complicated history of that code base. We can use simpler licenses for >> >> other stuff (like the benchmark suite) and just run with license in = >> >> license out rather than preserving the right for the PSF to change the >> >> license. >> >> >> >> Cheers, >> >> Nick. >> >> >> >> -- >> >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> >> ___ >> >> Speed mailing list >> >> Speed@python.org >> >> http://mail.python.org/mailman/listinfo/speed >> > >> > >> > First, I believe all the unalden swallow stuff (including the runner) is >> > under the PSF licence, though you'd have to check the repo for a license >> > file or bug Jeffrey and Collin. Someone (fijal) will add an MIT license >> > for >> > our half of the repo. >> > >> > >> > Alex >> >> Done. PyPy benchmarks are MIT > > > Great! Then I'm happy with moving PyPy benchmarks over wholesale. Are there > any benchmarks that are *really* good and are thus a priority to move, or > any that are just flat-out bad and I shouldn't bother moviing? Note that not all benchmarks run nightly. twisted_accept for example run out of TCP connections. benchmarks.py is your helper. We improved the US runner qutie significantly (the main runner.py file), mostly by improving reporting. So it can save a .json file or upload stuff to a codespeed instance. Other than that, they all measure something. It's really up to you to decide which ones measure "something significant". Of course for our purposes benchmarks which require large libs are more interesting than others, but they all do something interesting. We removed those that we consider completely uninteresting. ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] merging PyPy and Python benchmark suite
On Wed, Jul 25, 2012 at 5:12 AM, Alex Gaynor wrote: > > > On Tue, Jul 24, 2012 at 7:37 PM, Nick Coghlan wrote: >> >> On Wed, Jul 25, 2012 at 9:51 AM, Brett Cannon wrote: >> > >> > >> > On Tue, Jul 24, 2012 at 5:38 PM, Nick Coghlan >> > wrote: >> >> >> >> Antoine's right on this one - just use and redistribute the upstream >> >> components under their existing licenses. CPython itself is different >> >> because the PSF has chosen to reserve relicensing privileges for that, >> >> which >> >> requires the extra permissions granted in the contributor agreement. >> > >> > >> > But I'm talking about the benchmarks themselves, not the wholesale >> > inclusion >> > of Mako, etc. (which I am not worried about since the code in the >> > dependencies is not edited). Can we move the PyPy benchmarks themselves >> > (e.g. bm_mako.py that PyPy has) over to the PSF benchmarks without >> > getting >> > contributor agreements. >> >> The PyPy team need to put a clear license notice (similar to the one >> in the main pypy repo) on their benchmarks repo. But yes, I believe >> you're right that copying that code as it stands would technically be >> a copyright violation, even if the PyPy team intend for it to be >> allowed. >> >> If you're really concerned, check with Van first, but otherwise I'd >> just file a bug with the PyPy folks requesting that they clarify the >> licensing by adding a LICENSE file and in the meantime assume they >> intended for it to be covered by the MIT license, just like PyPy >> itself. >> >> The PSF license is necessary for CPython because of the long and >> complicated history of that code base. We can use simpler licenses for >> other stuff (like the benchmark suite) and just run with license in = >> license out rather than preserving the right for the PSF to change the >> license. >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia >> ___ >> Speed mailing list >> Speed@python.org >> http://mail.python.org/mailman/listinfo/speed > > > First, I believe all the unalden swallow stuff (including the runner) is > under the PSF licence, though you'd have to check the repo for a license > file or bug Jeffrey and Collin. Someone (fijal) will add an MIT license for > our half of the repo. > > > Alex Done. PyPy benchmarks are MIT ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] merging PyPy and Python benchmark suite
On Tue, Jul 24, 2012 at 7:35 PM, Alex Gaynor wrote: > > > On Tue, Jul 24, 2012 at 10:29 AM, Maciej Fijalkowski > wrote: >> >> On Tue, Jul 24, 2012 at 7:10 PM, Brett Cannon wrote: >> > >> > >> > On Mon, Jul 23, 2012 at 7:34 PM, Maciej Fijalkowski >> > wrote: >> >> >> >> On Mon, Jul 23, 2012 at 11:46 PM, Brett Cannon >> >> wrote: >> >> > >> >> > >> >> > On Mon, Jul 23, 2012 at 4:39 PM, Armin Rigo wrote: >> >> >> >> >> >> Hi Brett, >> >> >> >> >> >> On Mon, Jul 23, 2012 at 10:15 PM, Brett Cannon >> >> >> wrote: >> >> >> > That's what I'm trying to establish; how much have they diverged >> >> >> > and >> >> >> > if >> >> >> > I'm >> >> >> > looking in the proper place. >> >> >> >> >> >> bm_mako.py is not from Unladen Swallow; that's why it is in >> >> >> pypy/benchmarks/own/. In case of doubts, check it in the history of >> >> >> Hg. The PyPy version was added from virhilo, which seems to be the >> >> >> name of his author, on 2010-12-21, and was not changed at all since >> >> >> then. >> >> > >> >> > >> >> > OK. Maciej has always told me that a problem with the Unladen >> >> > benchmarks >> >> > was >> >> > that some of them had artificial loop unrolling, etc., so I had >> >> > assumed >> >> > you >> >> > had simply fixed those instances instead of creating entirely new >> >> > benchmarks. >> >> >> >> No we did not use those benchmarks. Those were mostly completely >> >> artificial microbenchmarks (call, call_method etc.). We decided we're >> >> not really that interested in microbenchmarks. >> >> >> >> > >> >> >> >> >> >> >> >> >> Hg tells me that there was no change at all in the 'unladen_swallow' >> >> >> subdirectory, apart from 'unladen_swallow/perf.py' and adding some >> >> >> __init__.py somewhere. So at least these benchmarks did not receive >> >> >> any pypy-specific adapatations. If there are divergences, they come >> >> >> from changes done to the unladen-swallow benchmark suite after PyPy >> >> >> copied it on 2010-01-15. >> >> > >> >> > >> >> > I know that directory wasn't changed, but I also noticed that some >> >> > benchmarks had the same name, which is why I thought they were forked >> >> > versions of the same-named Unladen benchmarks. >> >> >> >> Not if they're in own/ directory. >> > >> > >> > OK, good to know. I realized I can't copy code wholesale from PyPy's >> > benchmark suite as I don't know the code's history and thus if the >> > contributor signed Python's contributor agreement. Can the people who >> > are >> > familiar with the code help move benchmarks over where the copyright >> > isn't >> > in question? >> > >> >> Can we find a home for benchmarks where we don't need everyone to sign >> the copyright agreement? >> >> Cheers, >> fijal >> >> ___ >> Speed mailing list >> Speed@python.org >> http://mail.python.org/mailman/listinfo/speed > > > It seems totally reasonable for them to be official and be under the PSF > license, and have copyright agreements signed. > > Alex > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > PyPy benchmark suite contains stuff from twisted, sympy, mako, tons of other libraries. I doubt we can get everyone to sign the contributor agreement. ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] merging PyPy and Python benchmark suite
On Tue, Jul 24, 2012 at 7:10 PM, Brett Cannon wrote: > > > On Mon, Jul 23, 2012 at 7:34 PM, Maciej Fijalkowski > wrote: >> >> On Mon, Jul 23, 2012 at 11:46 PM, Brett Cannon wrote: >> > >> > >> > On Mon, Jul 23, 2012 at 4:39 PM, Armin Rigo wrote: >> >> >> >> Hi Brett, >> >> >> >> On Mon, Jul 23, 2012 at 10:15 PM, Brett Cannon >> >> wrote: >> >> > That's what I'm trying to establish; how much have they diverged and >> >> > if >> >> > I'm >> >> > looking in the proper place. >> >> >> >> bm_mako.py is not from Unladen Swallow; that's why it is in >> >> pypy/benchmarks/own/. In case of doubts, check it in the history of >> >> Hg. The PyPy version was added from virhilo, which seems to be the >> >> name of his author, on 2010-12-21, and was not changed at all since >> >> then. >> > >> > >> > OK. Maciej has always told me that a problem with the Unladen benchmarks >> > was >> > that some of them had artificial loop unrolling, etc., so I had assumed >> > you >> > had simply fixed those instances instead of creating entirely new >> > benchmarks. >> >> No we did not use those benchmarks. Those were mostly completely >> artificial microbenchmarks (call, call_method etc.). We decided we're >> not really that interested in microbenchmarks. >> >> > >> >> >> >> >> >> Hg tells me that there was no change at all in the 'unladen_swallow' >> >> subdirectory, apart from 'unladen_swallow/perf.py' and adding some >> >> __init__.py somewhere. So at least these benchmarks did not receive >> >> any pypy-specific adapatations. If there are divergences, they come >> >> from changes done to the unladen-swallow benchmark suite after PyPy >> >> copied it on 2010-01-15. >> > >> > >> > I know that directory wasn't changed, but I also noticed that some >> > benchmarks had the same name, which is why I thought they were forked >> > versions of the same-named Unladen benchmarks. >> >> Not if they're in own/ directory. > > > OK, good to know. I realized I can't copy code wholesale from PyPy's > benchmark suite as I don't know the code's history and thus if the > contributor signed Python's contributor agreement. Can the people who are > familiar with the code help move benchmarks over where the copyright isn't > in question? > Can we find a home for benchmarks where we don't need everyone to sign the copyright agreement? Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] merging PyPy and Python benchmark suite
On Mon, Jul 23, 2012 at 11:53 PM, Antoine Pitrou wrote: > On Mon, 23 Jul 2012 17:46:29 -0400 > Brett Cannon wrote: >> >> OK. Maciej has always told me that a problem with the Unladen benchmarks >> was that some of them had artificial loop unrolling, etc., so I had assumed >> you had simply fixed those instances instead of creating entirely new >> benchmarks. > > Is artificial unrolling a real problem or does it simply make > maintenance more difficult? > > Regards > > Antoine. The benchmark alex linked to is completely silly. It ends up having a very weird profile on PyPy - it takes forever to warm up the JIT, but once it's warm, it figures out the code does not actually do anything and removes it almost completely. This is also very sensitive to heuristics taken in the JIT (the warm up time mostly), so we decided it's really so artificial we don't want to be too concerned in PyPy about it. Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] merging PyPy and Python benchmark suite
On Mon, Jul 23, 2012 at 11:46 PM, Brett Cannon wrote: > > > On Mon, Jul 23, 2012 at 4:39 PM, Armin Rigo wrote: >> >> Hi Brett, >> >> On Mon, Jul 23, 2012 at 10:15 PM, Brett Cannon wrote: >> > That's what I'm trying to establish; how much have they diverged and if >> > I'm >> > looking in the proper place. >> >> bm_mako.py is not from Unladen Swallow; that's why it is in >> pypy/benchmarks/own/. In case of doubts, check it in the history of >> Hg. The PyPy version was added from virhilo, which seems to be the >> name of his author, on 2010-12-21, and was not changed at all since >> then. > > > OK. Maciej has always told me that a problem with the Unladen benchmarks was > that some of them had artificial loop unrolling, etc., so I had assumed you > had simply fixed those instances instead of creating entirely new > benchmarks. No we did not use those benchmarks. Those were mostly completely artificial microbenchmarks (call, call_method etc.). We decided we're not really that interested in microbenchmarks. > >> >> >> Hg tells me that there was no change at all in the 'unladen_swallow' >> subdirectory, apart from 'unladen_swallow/perf.py' and adding some >> __init__.py somewhere. So at least these benchmarks did not receive >> any pypy-specific adapatations. If there are divergences, they come >> from changes done to the unladen-swallow benchmark suite after PyPy >> copied it on 2010-01-15. > > > I know that directory wasn't changed, but I also noticed that some > benchmarks had the same name, which is why I thought they were forked > versions of the same-named Unladen benchmarks. Not if they're in own/ directory. Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] more space on speed.python.org
On Fri, Jul 13, 2012 at 10:41 AM, Noah Kantrowitz wrote: > ** Hmm, that might require some surgery, but I'll see what I can do > tomorrow. > > --Noah Thanks Noah. I hope you understand why just having a partition /blah won't help :) > > > Maciej Fijalkowski wrote: >> >> On Fri, Jul 13, 2012 at 4:12 AM, Jesse Noller wrote: >> >>> There's more than 20gb there. Run df an you can see the partition layout >>> Lance helped us setup >>> >>> On Jul 12, 2012, at 10:08 PM, Noah Kantrowitz >>> wrote: >>> >>> > >>> > On Jul 12, 2012, at 7:03 PM, Maciej Fijalkowski wrote: >>> > >>> >> Hi >>> >> >>> >> who do I contact to beg for more than 20G of space on speed? I'm sure >>> there is more unused, because you simply cannot buy such small drives these >>> days. >>> > >>> > Hmm, odd, let me take a look. >>> > >>> > --Noah >>> > >>> > ___ >>> > Speed mailing list >>> > Speed@python.org >>> > http://mail.python.org/mailman/listinfo/speed >>> >> >> Hi Jesse >> >> /dev/sda2 19G 16G 1.8G 90% / >> none 7.9G 196K 7.9G 1% /dev >> none 7.9G 0 7.9G 0% /dev/shm >> none 7.9G 56K 7.9G 1% /var/run >> none 7.9G 0 7.9G 0% /var/lock >> /dev/sda1 184M 29M 147M 17% /boot >> >> I can be enlighntened on the special (none) stuff, but to me it looks >> like 19G on / and 184M on /boot >> >> Noah - I'm afraid "everywhere" is the answer :( I need more space in >> /home, /tmp and /usr >> >> Cheers, >> faijl >> > ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] more space on speed.python.org
On Fri, Jul 13, 2012 at 4:12 AM, Jesse Noller wrote: > There's more than 20gb there. Run df an you can see the partition layout > Lance helped us setup > > On Jul 12, 2012, at 10:08 PM, Noah Kantrowitz wrote: > > > > > On Jul 12, 2012, at 7:03 PM, Maciej Fijalkowski wrote: > > > >> Hi > >> > >> who do I contact to beg for more than 20G of space on speed? I'm sure > there is more unused, because you simply cannot buy such small drives these > days. > > > > Hmm, odd, let me take a look. > > > > --Noah > > > > ___ > > Speed mailing list > > Speed@python.org > > http://mail.python.org/mailman/listinfo/speed > Hi Jesse /dev/sda2 19G 16G 1.8G 90% / none 7.9G 196K 7.9G 1% /dev none 7.9G 0 7.9G 0% /dev/shm none 7.9G 56K 7.9G 1% /var/run none 7.9G 0 7.9G 0% /var/lock /dev/sda1 184M 29M 147M 17% /boot I can be enlighntened on the special (none) stuff, but to me it looks like 19G on / and 184M on /boot Noah - I'm afraid "everywhere" is the answer :( I need more space in /home, /tmp and /usr Cheers, faijl ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
[Speed] more space on speed.python.org
Hi who do I contact to beg for more than 20G of space on speed? I'm sure there is more unused, because you simply cannot buy such small drives these days. Cheers, fijal ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
Re: [Speed] Spare hands
On Tue, Mar 27, 2012 at 12:28 AM, Corbin Simpson wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Hey guys! I work across the hall from where speed.python.org is > racked. Is there anything I can do to help? Sure. Get in touch with me during slightly more european hours :) > > ~ C. > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.11 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iQEcBAEBAgAGBQJPcO2AAAoJEHdtwiIXd1rTKpgH/R7ihQ+te8qsU/GewUf7C2Fd > Wd30BMqn/rYBKWC6XmhfnCaYLXnVq3VOoLfGUVCVCAXaSApS54bkhworI3gynl5M > h8cYVT03S1IM5ozTPZ84ZQ4nxooo6riWOUuGkRc1oyYmN5NaFpPJcUSR71bD+0Bv > 0F6OjO6sadv5R7u4baIG/sEV0yRPCCSO/dto6M3KAlBMv2zhJNFmo8nHROvByFAa > mKR/5ge9QjQVx0dkmuYTPAbm34e2PQZohpab4HUCBVKjnElvKCE07wulHzNSeVkP > Mob4RJ9yAzyEYhwctjMdmRceOppNUU79z2klD0nSMNlpJTb3WHZ7h5PBU5GljNA= > =5weK > -END PGP SIGNATURE- > ___ > Speed mailing list > Speed@python.org > http://mail.python.org/mailman/listinfo/speed ___ Speed mailing list Speed@python.org http://mail.python.org/mailman/listinfo/speed
