Re: [Speed] steps to get pypy benchmarks running

[Brett Cannon]

On Sun, 25 Mar 2018 at 07:37 Matti Picus  wrote:

> On 20/03/18 17:31, Matti Picus wrote:
> > On 14/02/18 20:18, Nick Coghlan wrote:
> >> On 14 February 2018 at 07:52, Mark Shannon  wrote:
> >>> Hi,
> >>>
> >>> On 13/02/18 14:27, Matti Picus wrote:
>  I have begun to dive into the performance/perf code. My goal is to get
>  pypy benchmarks running on http://speed.python.org. Since PyPy has
>  a JIT,
>  the benchmark runs must have a warmup stage.
> >>> Why?
> >>> The other interpreters don't get an arbitrary chunk of time for
> >>> free, so
> >>> neither should PyPy. Warmup in an inherent cost of dynamic
> >>> optimisers. The
> >>> benefits should outweigh the costs, but the costs shouldn't be ignored.
> >> For speed.python.org purposes, that would likely be most usefully
> >> reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
> >> the former runs under the same conditions as CPython, while the latter
> >> is given the benefit of warming up the JIT first).
> >>
> >> Only reporting the former would miss the point of PyPy's main use case
> >> (i.e. long lived processes), while only reporting the latter would
> >> miss one of the main answers to "Why hasn't everyone already switched
> >> to PyPy for all their Python needs?" (i.e. when the app doesn't run
> >> long enough to pay back the increased start-up overhead).
> >>
> >> Cheers,
> >> Nick.
> > So would it be reasonable as a first step to get the PyPy runner(s)
> > into operation by modifying the nightly runs to download from the
> > latest nightly builds [1], [2]?
> > We can deal with reporting cold/warm statistics later. As people have
> > said, they are really two orthogonal issues.
> >
> > [1]
> > http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2
> > for python 2.7
> > [2]
> > http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2
> > for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)
> >
> > Matti
>
> No responses, maybe I asked the wrong question.
>

I think the people who have traditionally maintained speed.python.org are
just not available to answer the question, not that it was the wrong
question.


> I would be willing to issue a pull request to get PyPy runners into
> operation on "the beast" so it can report results to speed.python.org.
> Which repo holds the code that stages `performance` runs and reports to
> speed.pypy.org?
>

Unfortunately I don't know.

-Brett


> Matti
> ___
> Speed mailing list
> Speed@python.org
> https://mail.python.org/mailman/listinfo/speed
>
___
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed

Re: [Speed] steps to get pypy benchmarks running

[Matti Picus]

On 14/02/18 20:18, Nick Coghlan wrote:

On 14 February 2018 at 07:52, Mark Shannon  wrote:

Hi,

On 13/02/18 14:27, Matti Picus wrote:

I have begun to dive into the performance/perf code. My goal is to get
pypy benchmarks running on http://speed.python.org. Since PyPy has a JIT,
the benchmark runs must have a warmup stage.

Why?
The other interpreters don't get an arbitrary chunk of time for free, so
neither should PyPy. Warmup in an inherent cost of dynamic optimisers. The
benefits should outweigh the costs, but the costs shouldn't be ignored.

For speed.python.org purposes, that would likely be most usefully
reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
the former runs under the same conditions as CPython, while the latter
is given the benefit of warming up the JIT first).

Only reporting the former would miss the point of PyPy's main use case
(i.e. long lived processes), while only reporting the latter would
miss one of the main answers to "Why hasn't everyone already switched
to PyPy for all their Python needs?" (i.e. when the app doesn't run
long enough to pay back the increased start-up overhead).

Cheers,
Nick.
So would it be reasonable as a first step to get the PyPy runner(s) into 
operation by modifying the nightly runs to download from the latest 
nightly builds [1], [2]?
We can deal with reporting cold/warm statistics later. As people have 
said, they are really two orthogonal issues.


[1] 
http://buildbot.pypy.org/nightly/trunk/pypy-c-jit-latest-linux64.tar.bz2 
for python 2.7
[2] 
http://buildbot.pypy.org/nightly/py3.5/pypy-c-jit-latest-linux64.tar.bz2 
for python 3.5 (latest released pypy3 version, python 3.6 is still alpha)


Matti
___
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed

Re: [Speed] steps to get pypy benchmarks running

[Nick Coghlan]

On 14 February 2018 at 07:52, Mark Shannon  wrote:
> Hi,
>
> On 13/02/18 14:27, Matti Picus wrote:
>>
>> I have begun to dive into the performance/perf code. My goal is to get
>> pypy benchmarks running on http://speed.python.org. Since PyPy has a JIT,
>> the benchmark runs must have a warmup stage.
>
>
> Why?
> The other interpreters don't get an arbitrary chunk of time for free, so
> neither should PyPy. Warmup in an inherent cost of dynamic optimisers. The
> benefits should outweigh the costs, but the costs shouldn't be ignored.

For speed.python.org purposes, that would likely be most usefully
reported as separate "PyPy (cold)" and "PyPy (warm)" results (where
the former runs under the same conditions as CPython, while the latter
is given the benefit of warming up the JIT first).

Only reporting the former would miss the point of PyPy's main use case
(i.e. long lived processes), while only reporting the latter would
miss one of the main answers to "Why hasn't everyone already switched
to PyPy for all their Python needs?" (i.e. when the app doesn't run
long enough to pay back the increased start-up overhead).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed

Re: [Speed] steps to get pypy benchmarks running

[Mark Shannon]

Hi,

On 13/02/18 14:27, Matti Picus wrote:
I have begun to dive into the performance/perf code. My goal is to get 
pypy benchmarks running on http://speed.python.org. Since PyPy has a 
JIT, the benchmark runs must have a warmup stage.


Why?
The other interpreters don't get an arbitrary chunk of time for free, so 
neither should PyPy. Warmup in an inherent cost of dynamic optimisers. 
The benefits should outweigh the costs, but the costs shouldn't be ignored.


 There are some
first-cut warmup values hardcoded inside a few of the benchmarks. I 
think I would prefer to use a different mechanism - a separate 
calibrated data file alongside the performance benchmarks. We could 
start off with a rough guess for the benchmarks and get the system up 
and running, and then calibrate the warmups hopefully finding some 
statistical basis for the values.


Assuming the idea of an external data file is acceptable, I have begun 
diving into the code, trying to figure out the interaction between the 
performance package and the perf package. It seems that running 
"pyperformance run -b telco --warmups 10" does not work, the 
pyperformance cli runner accepts only a subset of the perf Runner 
command line options.Shouldn't the performance.cli parse_args() start 
with the perf._runner.py parser?

Would a pull request along those lines be a worthwhile goal?

Thanks,
Matti

Cheers,
Mark.
___
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed