Hi
I'm willing to put some work after I'm back from holiday (mid-May)
On Tue, May 3, 2016 at 12:24 AM, Brett Cannon <br...@python.org> wrote:
>
>
> On Mon, 2 May 2016 at 15:18 Kevin Modzelewski <k...@dropbox.com> wrote:
>>
>> I'm definitely interested and willing to clean up + contribute our
>> benchmarks.
>>
>> On a side note, I'm a bit skeptical that there can be a single benchmark
>> suite that satisfies everyone. I would imagine that there will still be
>> projects with specific use-cases they prioritize (such as Pyston with
>> webserver workloads), or that have some idea that their users will be
>> "non-representative" in some way. One example of that is the emphasis on
>> warmup vs steady-state performance, which can be reflected in different
>> measurement methodologies -- I don't think there's a single right answer to
>> the question "how much does warmup matter".
>
>
> Totally agree. I think the general thinking is to at have a central
> repository and a flexible enough benchmark runner that people can benchmark
> whatever they find important to them. That way if e.g. Pyston adds nice web
> server benchmarks other implementations can use them or users can decide
> that's a workload they care about and make an informed decision of what
> Python implementations may work for them (before testing their own workload
> :).
>
> -Brett
>
>>
>>
>> But anyway, I'm still definitely +1 on the idea of merging all the
>> benchmarks together, and I think that that will be better than the current
>> situation. I'm imagining that we can at least have a common language for
>> discussing these things ("Pyston prefers to use the flags `--webserver
>> --include-warmup`"). I also see quite a few blog posts / academic papers on
>> Python performance that seem to get led astray by the confusing benchmark
>> situation, and I think having a blessed set of benchmarks (even if different
>> people use them in different ways) would still be a huge step forward.
>>
>> kmod
>>
>> On Mon, May 2, 2016 at 9:25 AM, Brett Cannon <br...@python.org> wrote:
>>>
>>>
>>>
>>> On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <fij...@gmail.com> wrote:
>>>>
>>>> On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <solip...@pitrou.net>
>>>> wrote:
>>>> > On Wed, 13 Apr 2016 20:57:35 +0200
>>>> > Maciej Fijalkowski <fij...@gmail.com>
>>>> > wrote:
>>>> >> Hi
>>>> >>
>>>> >> I have a radical idea: to take a pypy benchmark suite, update the
>>>> >> libraries to newer ones and replace python benchmarks with that. The
>>>> >> main reason being that pypy has a much better coverage of things that
>>>> >> are not microbenchmarks, the list (in json):
>>>> >
>>>> > So why not consolidate all benchmarks together, instead of throwing
>>>> > away work already done?
>>>> >
>>>> > Regards
>>>> >
>>>> > Antoine.
>>>>
>>>> Yeah, you can call it that too.
>>>
>>>
>>> I also reached out to Pyston at https://gitter.im/dropbox/pyston over the
>>> weekend to see if they would want to participate as well.
>>>
>>> So are we actually going to try and make this happen? I guess we should
>>> get people to vote on whether they like the idea enough before we hash out
>>> how we want to structure the new repository and benchmark suite.
>>>
>>> I'm +1 on the idea, but I currently don't have the time to help beyond
>>> helping drive the email conversation.
>>>
>>> _______________________________________________
>>> Speed mailing list
>>> Speed@python.org
>>> https://mail.python.org/mailman/listinfo/speed
>>>
>
_______________________________________________
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed
