On Mon, 2 May 2016 at 15:18 Kevin Modzelewski <[email protected]> wrote:
> I'm definitely interested and willing to clean up + contribute our
> benchmarks.
>
> On a side note, I'm a bit skeptical that there can be a single benchmark
> suite that satisfies everyone. I would imagine that there will still be
> projects with specific use-cases they prioritize (such as Pyston with
> webserver workloads), or that have some idea that their users will be
> "non-representative" in some way. One example of that is the emphasis on
> warmup vs steady-state performance, which can be reflected in different
> measurement methodologies -- I don't think there's a single right answer to
> the question "how much does warmup matter".
>
Totally agree. I think the general thinking is to at have a central
repository and a flexible enough benchmark runner that people can benchmark
whatever they find important to them. That way if e.g. Pyston adds nice web
server benchmarks other implementations can use them or users can decide
that's a workload they care about and make an informed decision of what
Python implementations may work for them (before testing their own workload
:).
-Brett
>
> But anyway, I'm still definitely +1 on the idea of merging all the
> benchmarks together, and I think that that will be better than the current
> situation. I'm imagining that we can at least have a common language for
> discussing these things ("Pyston prefers to use the flags `--webserver
> --include-warmup`"). I also see quite a few blog posts / academic papers
> on Python performance that seem to get led astray by the confusing
> benchmark situation, and I think having a blessed set of benchmarks (even
> if different people use them in different ways) would still be a huge step
> forward.
>
> kmod
>
> On Mon, May 2, 2016 at 9:25 AM, Brett Cannon <[email protected]> wrote:
>
>>
>>
>> On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <[email protected]> wrote:
>>
>>> On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <[email protected]>
>>> wrote:
>>> > On Wed, 13 Apr 2016 20:57:35 +0200
>>> > Maciej Fijalkowski <[email protected]>
>>> > wrote:
>>> >> Hi
>>> >>
>>> >> I have a radical idea: to take a pypy benchmark suite, update the
>>> >> libraries to newer ones and replace python benchmarks with that. The
>>> >> main reason being that pypy has a much better coverage of things that
>>> >> are not microbenchmarks, the list (in json):
>>> >
>>> > So why not consolidate all benchmarks together, instead of throwing
>>> > away work already done?
>>> >
>>> > Regards
>>> >
>>> > Antoine.
>>>
>>> Yeah, you can call it that too.
>>>
>>
>> I also reached out to Pyston at https://gitter.im/dropbox/pyston over
>> the weekend to see if they would want to participate as well.
>>
>> So are we actually going to try and make this happen? I guess we should
>> get people to vote on whether they like the idea enough before we hash out
>> how we want to structure the new repository and benchmark suite.
>>
>> I'm +1 on the idea, but I currently don't have the time to help beyond
>> helping drive the email conversation.
>>
>> _______________________________________________
>> Speed mailing list
>> [email protected]
>> https://mail.python.org/mailman/listinfo/speed
>>
>>
_______________________________________________
Speed mailing list
[email protected]
https://mail.python.org/mailman/listinfo/speed
