Status: New
Owner: liuj...@google.com
Labels: Type-Defect Priority-Medium

New issue 341 by eyal.far...@gmail.com: proposed optimization for parsing of packed repeated fields
http://code.google.com/p/protobuf/issues/detail?id=341

What steps will reproduce the problem?
1.Create a message type with a packed repeated field of int32
2.Compile with the protoc compiler...
3.Parse a message with about 300-400 integers (in the packed field)


What is the expected output? What do you see instead?
It works, but could be faster

What version of the product are you using? On what operating system?
2.4.1 on win7 (x64)


Please provide any additional information below.

I've already posted this in the GPB group (https://groups.google.com/forum/#!topic/protobuf/P0GBgD74v08), and was suggested to open an issue.

basically I have a process that spends about half of its time parsing a protobuf message with a large repeated field of integers (300-400 items). profiling this process showed that ‘ReadPackedPrimitive’ dominated the parsing, a closer look at the code revealed that it calls ‘BytesUntilLimit()’ proportionally to the number of items in the repeated field, this method evaluates a condition and several arithmetic operations which seems harmless but become annoying when executed repeatedly.

I initially patched my generated code to use a local function instead of ‘ReadPackedPrimitive’, the basic idea was to check the size of the available buffer before resorting to ‘BytesUntilLimit()’, this buffer is usually large enough to hold the next item (actually, in my case the entire message feats in buffer). The result of this patch was a 10-15% increase in throughput of my process, and 20% increase when the process was 'crippled' to do only protobuf parsing (no farther processing).

I'm attaching a patch to wire_format_lite_inl.h, this patch is made on top of V2.4.1

I'd appreciate any feed-back, and be more than happy if it's accepted.

Eyal.


Attachments:
        protobuff.packed_rfield.diff  1.6 KB

--
You received this message because you are subscribed to the Google Groups "Protocol 
Buffers" group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Reply via email to