On Mar 6, 3:27 am, [email protected] wrote: > I appreciate this, as i've been wanting to see some benchmarks between > the implementations for a long time. Of course, as a C advocate (i'm > the author of protobuf-c), I'm hoping (and frankly expecting) that > it'll win in the size AND speed category. > > I also like the style of separating the test data (.proto and sample > data) from the test itself (like serialize v deserialize). > > Maybe we could have a standard across platforms for a benchmark utilty > program, e.g. > benchmark-program TASK N_ITERATIONS PROTO_FILE DATA_FILE
Being able to specify an individual task (instead of running all 6 that I currently support) would indeed be very useful. I was doing some optimisation yesterday (improved the C# deserialization by 10-20%!) and ended up commenting out the tasks I didn't want. Specifying them on the command line is clearly nicer :) On the other hand, by the time you've got all of that several times for different variations, it becomes quite verbose. I'm also wary of putting the "N_ITERATIONS" bit in - without wishing to blow my own trumpet, I like the fact that the framework does it for you, making sure you get a reasonable sample. Again though, that's a bit of a pain when it comes to optimisation where you want a short run to just see whether it looks like it's helping. Maybe we need a sort of pseudo language for benchmarking: proto = foo.proto type = GoogleSpeed.Message1 data = message1.dat run deserialize-byte-array run deserialize-memory-stream iterations = 100000 run serialize-memory-stream iterations = 0 // back to auto-detect run serialize-byte-array etc Then just pass in a file with that stuff - or even allow the user to type it into stdin if they really want :) We could *possibly* even allow "javatype = ..." "dotnettype=..." etc and then share "standard" benchmarking files. Is this all going a bit over the top, perhaps? Jon --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~----------~----~----~----~------~----~------~--~---
