I appreciate this, as i've been wanting to see some benchmarks between
the implementations for a long time.  Of course, as a C advocate (i'm
the author of protobuf-c), I'm hoping (and frankly expecting) that
it'll win in the size AND speed category.

I also like the style of separating the test data (.proto and sample
data) from the test itself (like serialize v deserialize).

Maybe we could have a standard across platforms for a benchmark utilty
program, e.g.
  benchmark-program TASK N_ITERATIONS PROTO_FILE DATA_FILE

Where task is "deserialize" and "serialize" initially, but maybe
there's more variations.

On Mar 5, 3:07 am, Jon Skeet <sk...@pobox.com> wrote:
> I've just committed the first pass of a benchmarking tool to trunk.
> Everything is under "benchmarks" - see the readme.txt for usage
> guidelines.
>
> I have a bit of tidying up to do before the committed C# version is
> *exactly* equivalent (I've changed my build file around a bit) but
> I've got some initial results. There are two .proto files,
> representing the same messages (Message1 and Message2) but with one
> optimized for speed and the other for size. (This is a great case
> where a command-line option to override the optimization flag would be
> handy :) There are two data files, one for each message. The benchmark
> results below are running each version with each message file, with
> the Java code and my current C# code.
>
> I haven't optimised with a profiler very recently - I suspect there
> are some improvements which could be made by skipping the null
> handling when merging/parsing (as it should be unnecessary). I didn't
> use any particular options when running the Java version (1.6.0_11-
> b03) so I'm sure there are tweaks to be made there too.
>
> Java:
> Benchmarking benchmarks.GoogleSize$SizeMessage1 with file ../
> google_message1.dat
> Serialize to byte string: 1077894 iterations in 30.03s; 7.80469MB/s
> Serialize to byte array: 1044398 iterations in 28.86s; 7.86873MB/s
> Serialize to memory stream: 981322 iterations in 30.172s; 7.0720015MB/
> s
> Deserialize from byte string: 1378737 iterations in 30.048s;
> 9.977018MB/s
> Deserialize from byte array: 1374400 iterations in 30.581s; 9.772291MB/
> s
> Deserialize from memory stream: 1212133 iterations in 30.478s;
> 8.647663MB/s
>
> Benchmarking benchmarks.GoogleSpeed$SpeedMessage1 with file ../
> google_message1.dat
> Serialize to byte string: 8876207 iterations in 30.358s; 63.575417MB/s
> Serialize to byte array: 8977534 iterations in 30.044s; 64.9732MB/s
> Serialize to memory stream: 5715348 iterations in 30.485s; 40.765373MB/
> s
> Deserialize from byte string: 9295886 iterations in 30.159s;
> 67.020676MB/s
> Deserialize from byte array: 9003228 iterations in 29.266s; 66.89133MB/
> s
> Deserialize from memory stream: 5888670 iterations in 30.466s;
> 42.027805MB/s
>
> Benchmarking benchmarks.GoogleSize$SizeMessage2 with file ../
> google_message2.dat
> Serialize to byte string: 2598 iterations in 29.922s; 7.002691MB/s
> Serialize to byte array: 2602 iterations in 30.545s; 6.8704243MB/s
> Serialize to memory stream: 2412 iterations in 30.257s; 6.429362MB/s
> Deserialize from byte string: 1831 iterations in 30.37s; 4.8625035MB/s
> Deserialize from byte array: 1853 iterations in 30.383s; 4.918823MB/s
> Deserialize from memory stream: 1839 iterations in 30.754s;
> 4.8227696MB/s
>
> Benchmarking benchmarks.GoogleSpeed$SpeedMessage2 with file ../
> google_message2.dat
> Serialize to byte string: 18033 iterations in 30.088s; 48.33827MB/s
> Serialize to byte array: 17808 iterations in 30.264s; 47.457542MB/s
> Serialize to memory stream: 16569 iterations in 30.037s; 44.48936MB/s
> Deserialize from byte string: 13837 iterations in 29.609s; 37.690735MB/
> s
> Deserialize from byte array: 14163 iterations in 30.208s; 37.813744MB/
> s
> Deserialize from memory stream: 12945 iterations in 28.333s;
> 36.84902MB/s
>
> C#:
> Benchmarking
> Google.ProtocolBuffers.ProtoBench.SizeMessage1,BenchmarkTypes with
> file google_message1.dat
> Serialize to byte string: 1929013 iterations in 30.191s; 13.893MB/s
> Serialize to byte array: 1939375 iterations in 30.179s; 13.973MB/s
> Serialize to memory stream: 1759325 iterations in 29.956s; 12.770MB/s
> Deserialize from byte string: 1876302 iterations in 28.483s; 14.323MB/
> s
> Deserialize from byte array: 1995484 iterations in 30.554s; 14.201MB/s
> Deserialize from memory stream: 1831091 iterations in 30.731s;
> 12.956MB/s
>
> Benchmarking
> Google.ProtocolBuffers.ProtoBench.SpeedMessage1,BenchmarkTypes with
> file google_message1.dat
> Serialize to byte string: 17117282 iterations in 30.787s; 120.893MB/s
> Serialize to byte array: 17551170 iterations in 30.469s; 125.251MB/s
> Serialize to memory stream: 10226358 iterations in 28.105s; 79.117MB/s
> Deserialize from byte string: 12752420 iterations in 29.695s; 93.379MB/
> s
> Deserialize from byte array: 12713411 iterations in 29.833s; 92.661MB/
> s
> Deserialize from memory stream: 8628148 iterations in 39.201s;
> 47.859MB/s
>
> Benchmarking
> Google.ProtocolBuffers.ProtoBench.SizeMessage2,BenchmarkTypes with
> file google_message2.dat
> Serialize to byte string: 4735 iterations in 29.045s; 13.148MB/s
> Serialize to byte array: 4873 iterations in 30.436s; 12.913MB/s
> Serialize to memory stream: 4417 iterations in 30.081s; 11.843MB/s
> Deserialize from byte string: 3119 iterations in 30.270s; 8.310MB/s
> Deserialize from byte array: 3123 iterations in 30.153s; 8.353MB/s
> Deserialize from memory stream: 3088 iterations in 30.299s; 8.220MB/s
>
> Benchmarking
> Google.ProtocolBuffers.ProtoBench.SpeedMessage2,BenchmarkTypes with
> file google_message2.dat
> Serialize to byte string: 32605 iterations in 30.184s; 87.122MB/s
> Serialize to byte array: 32445 iterations in 30.360s; 86.191MB/s
> Serialize to memory stream: 19121 iterations in 30.093s; 51.245MB/s
> Deserialize from byte string: 16237 iterations in 28.680s; 45.660MB/s
> Deserialize from byte array: 17019 iterations in 30.160s; 45.512MB/s
> Deserialize from memory stream: 16830 iterations in 29.851s; 45.472MB/
> s
>
> This is only an initial cut, but I wanted to get it out into the
> community quickly rather than polishing build scripts etc. You
> currently have to build by hand, but there are instructions in the
> readme.
>
> Thanks,
> Jon
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en
-~----------~----~----~----~------~----~------~--~---

Reply via email to