I appreciate this, as i've been wanting to see some benchmarks between the implementations for a long time. Of course, as a C advocate (i'm the author of protobuf-c), I'm hoping (and frankly expecting) that it'll win in the size AND speed category.
I also like the style of separating the test data (.proto and sample data) from the test itself (like serialize v deserialize). Maybe we could have a standard across platforms for a benchmark utilty program, e.g. benchmark-program TASK N_ITERATIONS PROTO_FILE DATA_FILE Where task is "deserialize" and "serialize" initially, but maybe there's more variations. On Mar 5, 3:07 am, Jon Skeet <sk...@pobox.com> wrote: > I've just committed the first pass of a benchmarking tool to trunk. > Everything is under "benchmarks" - see the readme.txt for usage > guidelines. > > I have a bit of tidying up to do before the committed C# version is > *exactly* equivalent (I've changed my build file around a bit) but > I've got some initial results. There are two .proto files, > representing the same messages (Message1 and Message2) but with one > optimized for speed and the other for size. (This is a great case > where a command-line option to override the optimization flag would be > handy :) There are two data files, one for each message. The benchmark > results below are running each version with each message file, with > the Java code and my current C# code. > > I haven't optimised with a profiler very recently - I suspect there > are some improvements which could be made by skipping the null > handling when merging/parsing (as it should be unnecessary). I didn't > use any particular options when running the Java version (1.6.0_11- > b03) so I'm sure there are tweaks to be made there too. > > Java: > Benchmarking benchmarks.GoogleSize$SizeMessage1 with file ../ > google_message1.dat > Serialize to byte string: 1077894 iterations in 30.03s; 7.80469MB/s > Serialize to byte array: 1044398 iterations in 28.86s; 7.86873MB/s > Serialize to memory stream: 981322 iterations in 30.172s; 7.0720015MB/ > s > Deserialize from byte string: 1378737 iterations in 30.048s; > 9.977018MB/s > Deserialize from byte array: 1374400 iterations in 30.581s; 9.772291MB/ > s > Deserialize from memory stream: 1212133 iterations in 30.478s; > 8.647663MB/s > > Benchmarking benchmarks.GoogleSpeed$SpeedMessage1 with file ../ > google_message1.dat > Serialize to byte string: 8876207 iterations in 30.358s; 63.575417MB/s > Serialize to byte array: 8977534 iterations in 30.044s; 64.9732MB/s > Serialize to memory stream: 5715348 iterations in 30.485s; 40.765373MB/ > s > Deserialize from byte string: 9295886 iterations in 30.159s; > 67.020676MB/s > Deserialize from byte array: 9003228 iterations in 29.266s; 66.89133MB/ > s > Deserialize from memory stream: 5888670 iterations in 30.466s; > 42.027805MB/s > > Benchmarking benchmarks.GoogleSize$SizeMessage2 with file ../ > google_message2.dat > Serialize to byte string: 2598 iterations in 29.922s; 7.002691MB/s > Serialize to byte array: 2602 iterations in 30.545s; 6.8704243MB/s > Serialize to memory stream: 2412 iterations in 30.257s; 6.429362MB/s > Deserialize from byte string: 1831 iterations in 30.37s; 4.8625035MB/s > Deserialize from byte array: 1853 iterations in 30.383s; 4.918823MB/s > Deserialize from memory stream: 1839 iterations in 30.754s; > 4.8227696MB/s > > Benchmarking benchmarks.GoogleSpeed$SpeedMessage2 with file ../ > google_message2.dat > Serialize to byte string: 18033 iterations in 30.088s; 48.33827MB/s > Serialize to byte array: 17808 iterations in 30.264s; 47.457542MB/s > Serialize to memory stream: 16569 iterations in 30.037s; 44.48936MB/s > Deserialize from byte string: 13837 iterations in 29.609s; 37.690735MB/ > s > Deserialize from byte array: 14163 iterations in 30.208s; 37.813744MB/ > s > Deserialize from memory stream: 12945 iterations in 28.333s; > 36.84902MB/s > > C#: > Benchmarking > Google.ProtocolBuffers.ProtoBench.SizeMessage1,BenchmarkTypes with > file google_message1.dat > Serialize to byte string: 1929013 iterations in 30.191s; 13.893MB/s > Serialize to byte array: 1939375 iterations in 30.179s; 13.973MB/s > Serialize to memory stream: 1759325 iterations in 29.956s; 12.770MB/s > Deserialize from byte string: 1876302 iterations in 28.483s; 14.323MB/ > s > Deserialize from byte array: 1995484 iterations in 30.554s; 14.201MB/s > Deserialize from memory stream: 1831091 iterations in 30.731s; > 12.956MB/s > > Benchmarking > Google.ProtocolBuffers.ProtoBench.SpeedMessage1,BenchmarkTypes with > file google_message1.dat > Serialize to byte string: 17117282 iterations in 30.787s; 120.893MB/s > Serialize to byte array: 17551170 iterations in 30.469s; 125.251MB/s > Serialize to memory stream: 10226358 iterations in 28.105s; 79.117MB/s > Deserialize from byte string: 12752420 iterations in 29.695s; 93.379MB/ > s > Deserialize from byte array: 12713411 iterations in 29.833s; 92.661MB/ > s > Deserialize from memory stream: 8628148 iterations in 39.201s; > 47.859MB/s > > Benchmarking > Google.ProtocolBuffers.ProtoBench.SizeMessage2,BenchmarkTypes with > file google_message2.dat > Serialize to byte string: 4735 iterations in 29.045s; 13.148MB/s > Serialize to byte array: 4873 iterations in 30.436s; 12.913MB/s > Serialize to memory stream: 4417 iterations in 30.081s; 11.843MB/s > Deserialize from byte string: 3119 iterations in 30.270s; 8.310MB/s > Deserialize from byte array: 3123 iterations in 30.153s; 8.353MB/s > Deserialize from memory stream: 3088 iterations in 30.299s; 8.220MB/s > > Benchmarking > Google.ProtocolBuffers.ProtoBench.SpeedMessage2,BenchmarkTypes with > file google_message2.dat > Serialize to byte string: 32605 iterations in 30.184s; 87.122MB/s > Serialize to byte array: 32445 iterations in 30.360s; 86.191MB/s > Serialize to memory stream: 19121 iterations in 30.093s; 51.245MB/s > Deserialize from byte string: 16237 iterations in 28.680s; 45.660MB/s > Deserialize from byte array: 17019 iterations in 30.160s; 45.512MB/s > Deserialize from memory stream: 16830 iterations in 29.851s; 45.472MB/ > s > > This is only an initial cut, but I wanted to get it out into the > community quickly rather than polishing build scripts etc. You > currently have to build by hand, but there are instructions in the > readme. > > Thanks, > Jon --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~----------~----~----~----~------~----~------~--~---