On Mon, Mar 14, 2011 at 10:34, ksamdev <samvel.khalat...@gmail.com> wrote:
> Thanks for a quick reply.
> Honestly, I fill a set of histograms for each event. I've added this feature
> only recently and have a version of the code without histograms.
> Here is the same performance measurement without histograms:
> Read ProtoBuf
> Processed events: 1000000
> real 0m2.510s
> user 0m2.105s
> sys 0m0.298s
> READING (MULTITHREADS)
> process files
> init threads
> start threads
> run threads
> Thread read 1000000 events
> real 0m2.358s
> user 0m2.085s
> sys 0m0.236s
> Again, the same situation.
> My file is 384MB. I've already tested the use case with files above 1GB. It
> turs out that ProtoBuf has a "int" limitation on file size. Anyway, I am a
> way below the limit. The messages are pretty short (~400B).
Wait, where do you get the 1GB limit ? There is no real limit on the
file size (in fact, the protocol buffer library does not do anything
with files directly). (and: don't try to load everything in memory at
once as a gigantic repeated message, that will give you a lot of
Another thing that could be limiting your throughput in your library
is memory allocation: the system libraries are often not very good in
handling threads well; I am usually using tcmalloc (
http://code.google.com/p/google-perftools/ ) which is handling that
case pretty well.
Also: if you read a lot of messages, it is a good idea to Clear() and
re-use a message instead of allocating a new one (protocol buffers
attempt to re-use memory internally).
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to email@example.com.
> To unsubscribe from this group, send email to
> For more options, visit this group at
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to firstname.lastname@example.org.
To unsubscribe from this group, send email to
For more options, visit this group at