Re: [protobuf] Improve message parsing speed
On Thu, Feb 21, 2013 at 12:25 AM, Feng Xiao xiaof...@google.com wrote: On Thu, Feb 21, 2013 at 12:11 AM, Michael Grove m...@clarkparsia.comwrote: I am using protobuf for the wire format of a protocol I'm working on as a replacement to JSON. The original protobuf messages were not much more than JSON as protobuf; my protobuf message just contained the same fields w/ the same format as the JSON structure. This worked fine, but the payloads tended to be the same or larger than their JSON equivalent. I tried using the union types technique, specifically with extensions as outlined in the docs [1], and this worked very well wrt to compression, the resulting messages were much smaller than the previous approach. However, the parsing of the smaller messages far outweighs the advantage of less IO. You mean parsing protobufs performs worse than parsing JSON? For the nest structured based on extensions as described in the techniques sections of the protobuf docs, throughput it about the same. I assume that means parsing is slower because I'm sending fewer bytes over the wire. My original attempt at a protobuf based format was the fastest option, but it tended to be the most bytes sent over the wire, often more than the raw data I was sending. When I run a simple profiling example, the top 10-15 hot spots are all parsing of the messages. The top ten most expensive methods are as follows: MessageType1$Builder.mergeFrom MessageType2$Builder.mergeFrom MessageType1.getDescriptor() MessageType1$Builder.getDescriptorForType MessageType3$Builder.mergeFrom MessageType2.getDescriptor MessageType2$Builder.getDescriptorForType MessageType1$Builder.create MessageType1$Builder.buildPartial MessageType3.isInitialized The organization is pretty straightforward, MessageType3 contains a repeated list of MessageType2. MessageType2 has three required fields of type MessageType1. MessageType1 has a single required value, which is an enum. The value of the enum defines which of the extensions, again as shown in [1], are present on the message. There are a total of 6 possible extensions to MessageType1, each of which is a single primitive value, such as an int or a string. There tends to be no more than 3 of the 6 possible extensions used at any give time. The top two mergeFrom hot spots take ~32% of execution time, the test is the transmission of 1.85M objects of MessageType2 from client to server. These are bundled in roughly 64k chunks, using 58 top level MessageType3 objects. You can try the new parser API introduced in 2.5.0rc1, i.e., use MessageType3.parseFrom() instead of the Builder API to parse the message. Another option is to simplify the message structure. Instead of nesting many small MessageType2 in MessageType3, you can simply put the repeated extensions in MessageType3. This sounds good, I will try both of these options. Is 2.5.0rc1 fairly stable? Thanks. Michael Obviously all of the hot spot methods are auto-generated (Java). There might be some hand changes I could make to that code, but if I ever re-generate, then i'd lose that work. I am wondering if there are any tricks or changes that could be made to improve the parse time of the messages? Thanks. Michael [1] https://developers.google.com/protocol-buffers/docs/techniques -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Improve message parsing speed
On Thu, Feb 21, 2013 at 8:37 PM, Mike Grove m...@clarkparsia.com wrote: On Thu, Feb 21, 2013 at 12:25 AM, Feng Xiao xiaof...@google.com wrote: On Thu, Feb 21, 2013 at 12:11 AM, Michael Grove m...@clarkparsia.comwrote: I am using protobuf for the wire format of a protocol I'm working on as a replacement to JSON. The original protobuf messages were not much more than JSON as protobuf; my protobuf message just contained the same fields w/ the same format as the JSON structure. This worked fine, but the payloads tended to be the same or larger than their JSON equivalent. I tried using the union types technique, specifically with extensions as outlined in the docs [1], and this worked very well wrt to compression, the resulting messages were much smaller than the previous approach. However, the parsing of the smaller messages far outweighs the advantage of less IO. You mean parsing protobufs performs worse than parsing JSON? For the nest structured based on extensions as described in the techniques sections of the protobuf docs, throughput it about the same. I assume that means parsing is slower because I'm sending fewer bytes over the wire. My original attempt at a protobuf based format was the fastest option, but it tended to be the most bytes sent over the wire, often more than the raw data I was sending. When I run a simple profiling example, the top 10-15 hot spots are all parsing of the messages. The top ten most expensive methods are as follows: MessageType1$Builder.mergeFrom MessageType2$Builder.mergeFrom MessageType1.getDescriptor() MessageType1$Builder.getDescriptorForType MessageType3$Builder.mergeFrom MessageType2.getDescriptor MessageType2$Builder.getDescriptorForType MessageType1$Builder.create MessageType1$Builder.buildPartial MessageType3.isInitialized The organization is pretty straightforward, MessageType3 contains a repeated list of MessageType2. MessageType2 has three required fields of type MessageType1. MessageType1 has a single required value, which is an enum. The value of the enum defines which of the extensions, again as shown in [1], are present on the message. There are a total of 6 possible extensions to MessageType1, each of which is a single primitive value, such as an int or a string. There tends to be no more than 3 of the 6 possible extensions used at any give time. The top two mergeFrom hot spots take ~32% of execution time, the test is the transmission of 1.85M objects of MessageType2 from client to server. These are bundled in roughly 64k chunks, using 58 top level MessageType3 objects. You can try the new parser API introduced in 2.5.0rc1, i.e., use MessageType3.parseFrom() instead of the Builder API to parse the message. Another option is to simplify the message structure. Instead of nesting many small MessageType2 in MessageType3, you can simply put the repeated extensions in MessageType3. This sounds good, I will try both of these options. Is 2.5.0rc1 fairly stable? Yes, no big changes made since then. Thanks. Michael Obviously all of the hot spot methods are auto-generated (Java). There might be some hand changes I could make to that code, but if I ever re-generate, then i'd lose that work. I am wondering if there are any tricks or changes that could be made to improve the parse time of the messages? Thanks. Michael [1] https://developers.google.com/protocol-buffers/docs/techniques -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Improve message parsing speed
On Thu, Feb 21, 2013 at 9:02 AM, Feng Xiao xiaof...@google.com wrote: On Thu, Feb 21, 2013 at 8:37 PM, Mike Grove m...@clarkparsia.com wrote: On Thu, Feb 21, 2013 at 12:25 AM, Feng Xiao xiaof...@google.com wrote: On Thu, Feb 21, 2013 at 12:11 AM, Michael Grove m...@clarkparsia.comwrote: I am using protobuf for the wire format of a protocol I'm working on as a replacement to JSON. The original protobuf messages were not much more than JSON as protobuf; my protobuf message just contained the same fields w/ the same format as the JSON structure. This worked fine, but the payloads tended to be the same or larger than their JSON equivalent. I tried using the union types technique, specifically with extensions as outlined in the docs [1], and this worked very well wrt to compression, the resulting messages were much smaller than the previous approach. However, the parsing of the smaller messages far outweighs the advantage of less IO. You mean parsing protobufs performs worse than parsing JSON? For the nest structured based on extensions as described in the techniques sections of the protobuf docs, throughput it about the same. I assume that means parsing is slower because I'm sending fewer bytes over the wire. My original attempt at a protobuf based format was the fastest option, but it tended to be the most bytes sent over the wire, often more than the raw data I was sending. When I run a simple profiling example, the top 10-15 hot spots are all parsing of the messages. The top ten most expensive methods are as follows: MessageType1$Builder.mergeFrom MessageType2$Builder.mergeFrom MessageType1.getDescriptor() MessageType1$Builder.getDescriptorForType MessageType3$Builder.mergeFrom MessageType2.getDescriptor MessageType2$Builder.getDescriptorForType MessageType1$Builder.create MessageType1$Builder.buildPartial MessageType3.isInitialized The organization is pretty straightforward, MessageType3 contains a repeated list of MessageType2. MessageType2 has three required fields of type MessageType1. MessageType1 has a single required value, which is an enum. The value of the enum defines which of the extensions, again as shown in [1], are present on the message. There are a total of 6 possible extensions to MessageType1, each of which is a single primitive value, such as an int or a string. There tends to be no more than 3 of the 6 possible extensions used at any give time. The top two mergeFrom hot spots take ~32% of execution time, the test is the transmission of 1.85M objects of MessageType2 from client to server. These are bundled in roughly 64k chunks, using 58 top level MessageType3 objects. You can try the new parser API introduced in 2.5.0rc1, i.e., use MessageType3.parseFrom() instead of the Builder API to parse the message. Another option is to simplify the message structure. Instead of nesting many small MessageType2 in MessageType3, you can simply put the repeated extensions in MessageType3. This sounds good, I will try both of these options. Is 2.5.0rc1 fairly stable? Yes, no big changes made since then. 2.5.0rc1 did not work for me. For the messages in question, I changed from using mergeFrom to using parseFrom and I get 'Protocol message tag had invalid wire type.' errors when parsing the result. Did internal format of message change? I am using protobuf with Netty; there is a frame size that I must keep my protobuf payload within, and calling toByteArray after adding each MessageType2 to the MessageType3 builder is way too expensive. So I'm using CodedInputStream and toByteArray of MessageType2 directly to construct the serialized form of MessageType3. This way I can keep track of how many bytes i've written into the stream and can stop before exceeding the netty frame size. This is the only thing I can think of on my end that would cause parsing issues. Thanks. Michael Thanks. Michael Obviously all of the hot spot methods are auto-generated (Java). There might be some hand changes I could make to that code, but if I ever re-generate, then i'd lose that work. I am wondering if there are any tricks or changes that could be made to improve the parse time of the messages? Thanks. Michael [1] https://developers.google.com/protocol-buffers/docs/techniques -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to
[protobuf] Re: Issue 353 in protobuf: For widespread adoption, site should provide official java jar download
Comment #2 on issue 353 by kku...@countertack.com: For widespread adoption, site should provide official java jar download http://code.google.com/p/protobuf/issues/detail?id=353 Use maven as suggested in the README.TXT I just did this and it worked fine to build my own jar -rw-rw-r-- 1 kkuehl kkuehl 450509 Feb 21 15:02 protobuf-java-2.4.1.jar -- You received this message because this project is configured to send all issue notifications to this address. You may adjust your notification preferences at: https://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at http://groups.google.com/group/protobuf?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
