On Mon, Mar 14, 2011 at 10:08, ksamdev <samvel.khalat...@gmail.com> wrote: > Hi, > I have a large set of files with a number of the same type messages saved. > My code reads messages in a sequence from these files one after another. > I've measured time (with terminal "time" command) of running the code, and > get something like: > READING > ======= > Read ProtoBuf > Processed events: 50000000 > real 7m2.146s > user 5m25.545s > sys 0m31.959s > Then I've adjusted the code to read the files in threads (8 threads on 8 > cores machine). The reading procedure is independent and put into separate > class. Therefore each thread is really independent of the others. > Nevertheless, the time measurement is: > READING (MULTITHREADS) > ===================== > Thread read 6000000 events > Thread read 6000000 events > Thread read 6000000 events > Thread read 6000000 events > Thread read 6000000 events > Thread read 6000000 events > Thread read 7000000 events > Thread read 7000000 events > real 5m3.808s > user 5m42.301s > sys 0m35.221s > As you may see, the "user" as well as "real" time is pretty much the same. > So, it seems that there is some internal locks done somewhere. I only use > locks between threads and class, that creates and manages threads. The locks > are used only when thread finishes reading the file(s). > Does ProtoBuf use some sort of generic static/singleton functions/objects > that are used to de-serialize messages and therefore lock when accessed form > different threads?
No, there is no locking between threads being done. And Kenton would certainly be opposed to Singletons :) But did you check if you run into file IO limits ? How big are your files ? 6 Million protocol buffers in 300 seconds is 50 microseconds, that is very slow, or you have huge data. Are you doing other things, besides parsing in this benchmark ? > If so, is there a way to suppress this and get truly > independent messages parsing? > thanks. > P.S. My code can be browsed on github: http://goo.gl/DXCCF . The reading of > messages is done by: http://goo.gl/OsHV9 > The code uses ROOT framework (root.cern.ch) if one wants to compile it. > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to protobuf@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
