On Mar 6, 3:27 am, lahike...@gmail.com wrote:
> I appreciate this, as i've been wanting to see some benchmarks between
> the implementations for a long time.  Of course, as a C advocate (i'm
> the author of protobuf-c), I'm hoping (and frankly expecting) that
> it'll win in the size AND speed category.
>
> I also like the style of separating the test data (.proto and sample
> data) from the test itself (like serialize v deserialize).
>
> Maybe we could have a standard across platforms for a benchmark utilty
> program, e.g.
>   benchmark-program TASK N_ITERATIONS PROTO_FILE DATA_FILE

Being able to specify an individual task (instead of running all 6
that I currently support) would indeed be very useful. I was doing
some optimisation yesterday (improved the C# deserialization by
10-20%!) and ended up commenting out the tasks I didn't want.
Specifying them on the command line is clearly nicer :)

On the other hand, by the time you've got all of that several times
for different variations, it becomes quite verbose. I'm also wary of
putting the "N_ITERATIONS" bit in - without wishing to blow my own
trumpet, I like the fact that the framework does it for you, making
sure you get a reasonable sample. Again though, that's a bit of a pain
when it comes to optimisation where you want a short run to just see
whether it looks like it's helping.

Maybe we need a sort of pseudo language for benchmarking:

proto = foo.proto
type = GoogleSpeed.Message1
data = message1.dat
run deserialize-byte-array
run deserialize-memory-stream
iterations = 100000
run serialize-memory-stream
iterations = 0 // back to auto-detect
run serialize-byte-array

etc

Then just pass in a file with that stuff - or even allow the user to
type it into stdin if they really want :)
We could *possibly* even allow "javatype = ..." "dotnettype=..." etc
and then share "standard" benchmarking files.
Is this all going a bit over the top, perhaps?

Jon


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to