On Thu, Mar 20, 2014 at 3:28 AM, <[email protected]> wrote: > Hello, > > I have a message type like this: > > message SearchResponse { > enum Status { > OK = 200; > BAD_REQUEST = 400; > REQUEST_TIMEOUT = 408; > INTERNAL_SERVER_ERROR = 500; > } > required Status status = 1; > > message Result { > required int32 docid = 1; > optional float score = 2; > } > repeated Result result = 2; > } > > Now it it possible that the result repeats 5.000.000 times or more. At the > moment I'm working with about 2.100.000 repeats and the ParseFromCodedStream > function takes about 25 seconds. > > My Code looks like: > > google::protobuf::io::ArrayInputStream arrayIn(buffRecive, recived); > google::protobuf::io::CodedInputStream codedIn(&arrayIn); > google::protobuf::io::CodedInputStream::Limit msgLimit = > codedIn.PushLimit(recived); > srchResp.ParseFromCodedStream(&codedIn); > codedIn.PopLimit(msgLimit); > > The parsing and everything works great, but the performance is the problem. > The message with 2.100.000 repeats has around 2.600.000 bytes. > Is there a way to improve the performance?
If that message structure can be changed, I'd suggest the following: repeated int64 docid = 2 [packed=true]; repeated float score = 3 [packed=true]; Use the low bit of docid to indicate whether there's a score (hence the int32 -> int64 change... if your docid's fit in 31 bits, no need for that). So your encoding algo would be like add_docid(docid << 1 | !!has_score) if (has_score) add_score(score) And your reading algo would be the inverse. Unfortunately this means there's no easy way to seek to a particular doc id and get its score, so some post-processing on the receiver end will be necessary. > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
