I'm definitely interested and willing to clean up + contribute our
benchmarks.
On a side note, I'm a bit skeptical that there can be a single benchmark
suite that satisfies everyone. I would imagine that there will still be
projects with specific use-cases they prioritize (such as Pyston with
webserver workloads), or that have some idea that their users will be
"non-representative" in some way. One example of that is the emphasis on
warmup vs steady-state performance, which can be reflected in different
measurement methodologies -- I don't think there's a single right answer to
the question "how much does warmup matter".
But anyway, I'm still definitely +1 on the idea of merging all the
benchmarks together, and I think that that will be better than the current
situation. I'm imagining that we can at least have a common language for
discussing these things ("Pyston prefers to use the flags `--webserver
--include-warmup`"). I also see quite a few blog posts / academic papers
on Python performance that seem to get led astray by the confusing
benchmark situation, and I think having a blessed set of benchmarks (even
if different people use them in different ways) would still be a huge step
forward.
kmod
On Mon, May 2, 2016 at 9:25 AM, Brett Cannon <br...@python.org> wrote:
>
>
> On Thu, 14 Apr 2016 at 10:50 Maciej Fijalkowski <fij...@gmail.com> wrote:
>
>> On Thu, Apr 14, 2016 at 7:05 PM, Antoine Pitrou <solip...@pitrou.net>
>> wrote:
>> > On Wed, 13 Apr 2016 20:57:35 +0200
>> > Maciej Fijalkowski <fij...@gmail.com>
>> > wrote:
>> >> Hi
>> >>
>> >> I have a radical idea: to take a pypy benchmark suite, update the
>> >> libraries to newer ones and replace python benchmarks with that. The
>> >> main reason being that pypy has a much better coverage of things that
>> >> are not microbenchmarks, the list (in json):
>> >
>> > So why not consolidate all benchmarks together, instead of throwing
>> > away work already done?
>> >
>> > Regards
>> >
>> > Antoine.
>>
>> Yeah, you can call it that too.
>>
>
> I also reached out to Pyston at https://gitter.im/dropbox/pyston over the
> weekend to see if they would want to participate as well.
>
> So are we actually going to try and make this happen? I guess we should
> get people to vote on whether they like the idea enough before we hash out
> how we want to structure the new repository and benchmark suite.
>
> I'm +1 on the idea, but I currently don't have the time to help beyond
> helping drive the email conversation.
>
> _______________________________________________
> Speed mailing list
> Speed@python.org
> https://mail.python.org/mailman/listinfo/speed
>
>
_______________________________________________
Speed mailing list
Speed@python.org
https://mail.python.org/mailman/listinfo/speed
