[protobuf] Re: Streaming protobuf
Hi Florian,
Thanks a lot for your quick response. It works like a charm.
Thanks,
Joan.
On Tuesday, May 9, 2023 at 12:08:44 PM UTC+2 Florian Enner wrote:
> You could manually iterate through the data and only check for that field,
> e.g., in Java (untested):
>
> import com.google.protobuf.CodedInputStream;
> import com.google.protobuf.WireFormat;
>
> private static boolean containsHotels(byte[] data) throws IOException {
> return containsField(CodedInputStream.newInstance(data), Message.
> HOTELS_FIELD_NUMBER);
> }
>
> private static boolean containsField(CodedInputStream input, int fieldNumber)
> throws IOException {
> while (!input.isAtEnd()) {
> int tag = input.readTag();
> if (WireFormat.getTagFieldNumber(tag) == fieldNumber) {
> return true;
> }
> input.skipField(tag);
> }
> return false;
> }
>
>
> On Tuesday, May 9, 2023 at 11:46:52 AM UTC+2 Joan Balagueró wrote:
>
>> Hello,
>>
>> I'm receiving protobuf messages from a client (I can't modify them in any
>> way) with a list of hotels. There are hotels in the 90% of the times, but I
>> need to differentitate when I receive hotels and when not.
>>
>> The proto file for "no hotels" is something like below (simplified):
>> 1: {
>> 1: {}
>> 2: {}
>> }
>>
>> Is there any way to check the "2:" element is empty without loading the
>> whole proto into memory?
>>
>> Thanks,
>>
>> Joan.
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/499e572f-ed3b-4e2f-88bc-1185549d10e1n%40googlegroups.com.
[protobuf] Re: Streaming protobuf
You could manually iterate through the data and only check for that field,
e.g., in Java (untested):
import com.google.protobuf.CodedInputStream;
import com.google.protobuf.WireFormat;
private static boolean containsHotels(byte[] data) throws IOException {
return containsField(CodedInputStream.newInstance(data), Message.
HOTELS_FIELD_NUMBER);
}
private static boolean containsField(CodedInputStream input, int fieldNumber)
throws IOException {
while (!input.isAtEnd()) {
int tag = input.readTag();
if (WireFormat.getTagFieldNumber(tag) == fieldNumber) {
return true;
}
input.skipField(tag);
}
return false;
}
On Tuesday, May 9, 2023 at 11:46:52 AM UTC+2 Joan Balagueró wrote:
> Hello,
>
> I'm receiving protobuf messages from a client (I can't modify them in any
> way) with a list of hotels. There are hotels in the 90% of the times, but I
> need to differentitate when I receive hotels and when not.
>
> The proto file for "no hotels" is something like below (simplified):
> 1: {
> 1: {}
> 2: {}
> }
>
> Is there any way to check the "2:" element is empty without loading the
> whole proto into memory?
>
> Thanks,
>
> Joan.
>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/protobuf/8582fd3a-e134-427c-9dcf-8a163aa198a7n%40googlegroups.com.
Re: [protobuf] Re: Streaming Serialization - Suggestion
On Wed, Mar 30, 2016 at 5:27 PM, Yoav H wrote: > I saw the start\end group but I couldn't find any information on those and > how to use them. > > Your point about skipping fields makes sense. > I think it is also solvable with applying the same idea of chunked > encoding, even on sub fields. > So instead of writing the full length of the child field, you allow the > serializer to write it in smaller chunks. > The deserializer can then just read the chunk markings and skip them. > A very basic serializer can put just one chunk (which will be equivalent > to the current implementation, plus one more zero marking at the end), but > it allows a more efficient serializer to stream data. > > Regarding adding something to the encoding spec, are you allowing proto2 > serializers to call into proto3 deserializers and vice versa? > I thought that if you have a protoX server, you expect clients to take the > protoX file and generate a client out of it, which will match that proto > version encoding. Isn't it the case? > Proto2 and proto3 are wire-compatible. We already have a lot of proto3 clients communicating with proto2 servers or vice versa. Like Josh mentioned, we can't change proto3's wire format now. > > Thanks, > Yoav. > > On Tuesday, March 29, 2016 at 5:06:46 PM UTC-7, Feng Xiao wrote: > >> >> >> On Mon, Mar 28, 2016 at 10:53 PM, Yoav H wrote: >> >>> They say on their website: "When evaluating new features, we look for >>> additions that are very widely useful or very simple". >>> What I'm suggesting here is both very useful (speeding up serialization >>> and eliminating memory duplication) and very simple (simple additions to >>> the encoding, no need to change the language). >>> So far, no response from the Google guys... >>> >> Actually there are already a "start embedding" tag and a "end embedding" >> tag in protobuf: >> https://developers.google.com/protocol-buffers/docs/encoding#structure >> >> 3 Start group groups (deprecated) >> 4 End group groups (deprecated) >> >> They are deprecated though. >> >> You mentioned it will be a performance gain, but what we experienced in >> google says otherwise. For example, in a lot places we are only interested >> in a few fields and want to skip through all other fields (if we are >> building a proxy, or the field is simply an unknown field). The start >> group/end group tag pair forces the parser to decode every single field in >> the a whole group even the whole group is to be ignored after parsing, and >> that's a very significant drawback. >> >> And adding a new wire tag type to protobuf is not a simple thing. >> Actually I don't think we have added any new wire type to protobuf before. >> There are a lot issues to consider. For example, isn't all code that switch >> on protobuf wire types now suddenly broken? if a new serializer uses the >> new wire type in its output, what will happen if the parsers can't >> understand it? >> >> Proto3 is already finalized and we will not add new wire types in proto3. >> Whether to add it in proto4 depends on whether we have a good use for it >> and whether we can mitigate the risks of rolling out a new wire type. >> >> >>> >>> >>> On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: This exact suggestion has been up for discussion long time ago(years?, before proto2?) When it comes to taking suggestions I'm only a 3rd party implementer but my understanding is that the design process of protocol buffers and its goals are internal to Google and they usually publish new versions of their code implementing new features before you can read about them in the documents. On Mar 27, 2016 5:31 AM, "Yoav H" wrote: > Any comment on this? > Will you consider this for proto3? > > On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: >> >> Hi, >> >> I have a suggestion fr improving the protobuf encoding. >> Is proto3 final? >> >> I like the simplicity of the encoding of protobuf. >> But I think it has one issue with serialization, using streams. >> The problem is with length delimited fields and the fact that they >> require knowing the length ahead of time. >> If we have a very long string, we need to encode the entire string >> before we know its length, so we basically duplicate the data in memory. >> Same is true for embedded messages, where we need to encode the >> entire embedded message before we can append it to the stream. >> >> I think there is a simple solution for both issues. >> >> For strings and byte arrays, a simple solution is to use "chunked >> encoding". >> Which means that the byte array is split into chunks and every chunk >> starts with the chunk length. End of array is indicated by length zero. >> >> For embedded messages, the solution is to have an "start embedding" >> tag and an "end embedding tag". >> Everythi
Re: [protobuf] Re: Streaming Serialization - Suggestion
Hi Yoav, Chunked encoding is definitely an interesting idea, and I can see the benefits you mentioned. However proto2 and proto3 are more or less frozen from a wire perspective. There are lots of existing clients out there already communicating with proto3, so we're not really at liberty to make any changes. Sorry about that. Best, Josh On Wednesday, March 30, 2016 at 5:27:53 PM UTC-7, Yoav H wrote: > > I saw the start\end group but I couldn't find any information on those and > how to use them. > > Your point about skipping fields makes sense. > I think it is also solvable with applying the same idea of chunked > encoding, even on sub fields. > So instead of writing the full length of the child field, you allow the > serializer to write it in smaller chunks. > The deserializer can then just read the chunk markings and skip them. > A very basic serializer can put just one chunk (which will be equivalent > to the current implementation, plus one more zero marking at the end), but > it allows a more efficient serializer to stream data. > > Regarding adding something to the encoding spec, are you allowing proto2 > serializers to call into proto3 deserializers and vice versa? > I thought that if you have a protoX server, you expect clients to take the > protoX file and generate a client out of it, which will match that proto > version encoding. Isn't it the case? > > Thanks, > Yoav. > > On Tuesday, March 29, 2016 at 5:06:46 PM UTC-7, Feng Xiao wrote: >> >> >> >> On Mon, Mar 28, 2016 at 10:53 PM, Yoav H wrote: >> >>> They say on their website: "When evaluating new features, we look for >>> additions that are very widely useful or very simple". >>> What I'm suggesting here is both very useful (speeding up serialization >>> and eliminating memory duplication) and very simple (simple additions to >>> the encoding, no need to change the language). >>> So far, no response from the Google guys... >>> >> Actually there are already a "start embedding" tag and a "end embedding" >> tag in protobuf: >> https://developers.google.com/protocol-buffers/docs/encoding#structure >> >> 3 Start group groups (deprecated) >> 4 End group groups (deprecated) >> >> They are deprecated though. >> >> You mentioned it will be a performance gain, but what we experienced in >> google says otherwise. For example, in a lot places we are only interested >> in a few fields and want to skip through all other fields (if we are >> building a proxy, or the field is simply an unknown field). The start >> group/end group tag pair forces the parser to decode every single field in >> the a whole group even the whole group is to be ignored after parsing, and >> that's a very significant drawback. >> >> And adding a new wire tag type to protobuf is not a simple thing. >> Actually I don't think we have added any new wire type to protobuf before. >> There are a lot issues to consider. For example, isn't all code that switch >> on protobuf wire types now suddenly broken? if a new serializer uses the >> new wire type in its output, what will happen if the parsers can't >> understand it? >> >> Proto3 is already finalized and we will not add new wire types in proto3. >> Whether to add it in proto4 depends on whether we have a good use for it >> and whether we can mitigate the risks of rolling out a new wire type. >> >> >>> >>> >>> On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: This exact suggestion has been up for discussion long time ago(years?, before proto2?) When it comes to taking suggestions I'm only a 3rd party implementer but my understanding is that the design process of protocol buffers and its goals are internal to Google and they usually publish new versions of their code implementing new features before you can read about them in the documents. On Mar 27, 2016 5:31 AM, "Yoav H" wrote: > Any comment on this? > Will you consider this for proto3? > > On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: >> >> Hi, >> >> I have a suggestion fr improving the protobuf encoding. >> Is proto3 final? >> >> I like the simplicity of the encoding of protobuf. >> But I think it has one issue with serialization, using streams. >> The problem is with length delimited fields and the fact that they >> require knowing the length ahead of time. >> If we have a very long string, we need to encode the entire string >> before we know its length, so we basically duplicate the data in memory. >> Same is true for embedded messages, where we need to encode the >> entire embedded message before we can append it to the stream. >> >> I think there is a simple solution for both issues. >> >> For strings and byte arrays, a simple solution is to use "chunked >> encoding". >> Which means that the byte array is split into chunks and every chun
Re: [protobuf] Re: Streaming Serialization - Suggestion
I saw the start\end group but I couldn't find any information on those and how to use them. Your point about skipping fields makes sense. I think it is also solvable with applying the same idea of chunked encoding, even on sub fields. So instead of writing the full length of the child field, you allow the serializer to write it in smaller chunks. The deserializer can then just read the chunk markings and skip them. A very basic serializer can put just one chunk (which will be equivalent to the current implementation, plus one more zero marking at the end), but it allows a more efficient serializer to stream data. Regarding adding something to the encoding spec, are you allowing proto2 serializers to call into proto3 deserializers and vice versa? I thought that if you have a protoX server, you expect clients to take the protoX file and generate a client out of it, which will match that proto version encoding. Isn't it the case? Thanks, Yoav. On Tuesday, March 29, 2016 at 5:06:46 PM UTC-7, Feng Xiao wrote: > > > > On Mon, Mar 28, 2016 at 10:53 PM, Yoav H > wrote: > >> They say on their website: "When evaluating new features, we look for >> additions that are very widely useful or very simple". >> What I'm suggesting here is both very useful (speeding up serialization >> and eliminating memory duplication) and very simple (simple additions to >> the encoding, no need to change the language). >> So far, no response from the Google guys... >> > Actually there are already a "start embedding" tag and a "end embedding" > tag in protobuf: > https://developers.google.com/protocol-buffers/docs/encoding#structure > > 3 Start group groups (deprecated) > 4 End group groups (deprecated) > > They are deprecated though. > > You mentioned it will be a performance gain, but what we experienced in > google says otherwise. For example, in a lot places we are only interested > in a few fields and want to skip through all other fields (if we are > building a proxy, or the field is simply an unknown field). The start > group/end group tag pair forces the parser to decode every single field in > the a whole group even the whole group is to be ignored after parsing, and > that's a very significant drawback. > > And adding a new wire tag type to protobuf is not a simple thing. Actually > I don't think we have added any new wire type to protobuf before. There are > a lot issues to consider. For example, isn't all code that switch on > protobuf wire types now suddenly broken? if a new serializer uses the new > wire type in its output, what will happen if the parsers can't understand > it? > > Proto3 is already finalized and we will not add new wire types in proto3. > Whether to add it in proto4 depends on whether we have a good use for it > and whether we can mitigate the risks of rolling out a new wire type. > > >> >> >> On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: >>> >>> This exact suggestion has been up for discussion long time ago(years?, >>> before proto2?) >>> >>> When it comes to taking suggestions I'm only a 3rd party implementer but >>> my understanding is that the design process of protocol buffers and its >>> goals are internal to Google and they usually publish new versions of their >>> code implementing new features before you can read about them in the >>> documents. >>> On Mar 27, 2016 5:31 AM, "Yoav H" wrote: >>> Any comment on this? Will you consider this for proto3? On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: > > Hi, > > I have a suggestion fr improving the protobuf encoding. > Is proto3 final? > > I like the simplicity of the encoding of protobuf. > But I think it has one issue with serialization, using streams. > The problem is with length delimited fields and the fact that they > require knowing the length ahead of time. > If we have a very long string, we need to encode the entire string > before we know its length, so we basically duplicate the data in memory. > Same is true for embedded messages, where we need to encode the entire > embedded message before we can append it to the stream. > > I think there is a simple solution for both issues. > > For strings and byte arrays, a simple solution is to use "chunked > encoding". > Which means that the byte array is split into chunks and every chunk > starts with the chunk length. End of array is indicated by length zero. > > For embedded messages, the solution is to have an "start embedding" > tag and an "end embedding tag". > Everything in between is the embedded message. > > By adding these two new features, serialization can be fully > streamable and there is no need to pre-serialize big chunks in memory > before writing them to the stream. > > Hope you'll find this suggestion useful and incorporate it into the > protocol. >
Re: [protobuf] Re: Streaming Serialization - Suggestion
On Wed, Mar 30, 2016 at 8:06 AM, 'Feng Xiao' via Protocol Buffers < protobuf@googlegroups.com> wrote: > > > On Mon, Mar 28, 2016 at 10:53 PM, Yoav H wrote: > >> They say on their website: "When evaluating new features, we look for >> additions that are very widely useful or very simple". >> What I'm suggesting here is both very useful (speeding up serialization >> and eliminating memory duplication) and very simple (simple additions to >> the encoding, no need to change the language). >> So far, no response from the Google guys... >> > Actually there are already a "start embedding" tag and a "end embedding" > tag in protobuf: > https://developers.google.com/protocol-buffers/docs/encoding#structure > > 3 Start group groups (deprecated) > 4 End group groups (deprecated) > > They are deprecated though. > > You mentioned it will be a performance gain, but what we experienced in > google says otherwise. For example, in a lot places we are only interested > in a few fields and want to skip through all other fields (if we are > building a proxy, or the field is simply an unknown field). The start > group/end group tag pair forces the parser to decode every single field in > the a whole group even the whole group is to be ignored after parsing, and > that's a very significant drawback. > This is definitely the use-case where delimiting makes perfect sense (proxy/middleware service that reads part of a message). The name 'protocol buffers' does kinda makes that use-case obvious. If using protobuf to simply serialize/deserialize, then start/end group would definitely benefit the streaming use-case. Shameless plug: https://github.com/protostuff/protostuff optimizes for the latter use-case and was mostly the reason it was created (java only though) > > And adding a new wire tag type to protobuf is not a simple thing. Actually > I don't think we have added any new wire type to protobuf before. There are > a lot issues to consider. For example, isn't all code that switch on > protobuf wire types now suddenly broken? if a new serializer uses the new > wire type in its output, what will happen if the parsers can't understand > it? > > Proto3 is already finalized and we will not add new wire types in proto3. > Whether to add it in proto4 depends on whether we have a good use for it > and whether we can mitigate the risks of rolling out a new wire type. > > >> >> >> On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: >>> >>> This exact suggestion has been up for discussion long time ago(years?, >>> before proto2?) >>> >>> When it comes to taking suggestions I'm only a 3rd party implementer but >>> my understanding is that the design process of protocol buffers and its >>> goals are internal to Google and they usually publish new versions of their >>> code implementing new features before you can read about them in the >>> documents. >>> On Mar 27, 2016 5:31 AM, "Yoav H" wrote: >>> Any comment on this? Will you consider this for proto3? On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: > > Hi, > > I have a suggestion fr improving the protobuf encoding. > Is proto3 final? > > I like the simplicity of the encoding of protobuf. > But I think it has one issue with serialization, using streams. > The problem is with length delimited fields and the fact that they > require knowing the length ahead of time. > If we have a very long string, we need to encode the entire string > before we know its length, so we basically duplicate the data in memory. > Same is true for embedded messages, where we need to encode the entire > embedded message before we can append it to the stream. > > I think there is a simple solution for both issues. > > For strings and byte arrays, a simple solution is to use "chunked > encoding". > Which means that the byte array is split into chunks and every chunk > starts with the chunk length. End of array is indicated by length zero. > > For embedded messages, the solution is to have an "start embedding" > tag and an "end embedding tag". > Everything in between is the embedded message. > > By adding these two new features, serialization can be fully > streamable and there is no need to pre-serialize big chunks in memory > before writing them to the stream. > > Hope you'll find this suggestion useful and incorporate it into the > protocol. > > Thanks, > Yoav. > > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com. To post to this group, send email to prot...@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout. >>> -
Re: [protobuf] Re: Streaming Serialization - Suggestion
On Mon, Mar 28, 2016 at 10:53 PM, Yoav H wrote: > They say on their website: "When evaluating new features, we look for > additions that are very widely useful or very simple". > What I'm suggesting here is both very useful (speeding up serialization > and eliminating memory duplication) and very simple (simple additions to > the encoding, no need to change the language). > So far, no response from the Google guys... > Actually there are already a "start embedding" tag and a "end embedding" tag in protobuf: https://developers.google.com/protocol-buffers/docs/encoding#structure 3 Start group groups (deprecated) 4 End group groups (deprecated) They are deprecated though. You mentioned it will be a performance gain, but what we experienced in google says otherwise. For example, in a lot places we are only interested in a few fields and want to skip through all other fields (if we are building a proxy, or the field is simply an unknown field). The start group/end group tag pair forces the parser to decode every single field in the a whole group even the whole group is to be ignored after parsing, and that's a very significant drawback. And adding a new wire tag type to protobuf is not a simple thing. Actually I don't think we have added any new wire type to protobuf before. There are a lot issues to consider. For example, isn't all code that switch on protobuf wire types now suddenly broken? if a new serializer uses the new wire type in its output, what will happen if the parsers can't understand it? Proto3 is already finalized and we will not add new wire types in proto3. Whether to add it in proto4 depends on whether we have a good use for it and whether we can mitigate the risks of rolling out a new wire type. > > > On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: >> >> This exact suggestion has been up for discussion long time ago(years?, >> before proto2?) >> >> When it comes to taking suggestions I'm only a 3rd party implementer but >> my understanding is that the design process of protocol buffers and its >> goals are internal to Google and they usually publish new versions of their >> code implementing new features before you can read about them in the >> documents. >> On Mar 27, 2016 5:31 AM, "Yoav H" wrote: >> >>> Any comment on this? >>> Will you consider this for proto3? >>> >>> On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: Hi, I have a suggestion fr improving the protobuf encoding. Is proto3 final? I like the simplicity of the encoding of protobuf. But I think it has one issue with serialization, using streams. The problem is with length delimited fields and the fact that they require knowing the length ahead of time. If we have a very long string, we need to encode the entire string before we know its length, so we basically duplicate the data in memory. Same is true for embedded messages, where we need to encode the entire embedded message before we can append it to the stream. I think there is a simple solution for both issues. For strings and byte arrays, a simple solution is to use "chunked encoding". Which means that the byte array is split into chunks and every chunk starts with the chunk length. End of array is indicated by length zero. For embedded messages, the solution is to have an "start embedding" tag and an "end embedding tag". Everything in between is the embedded message. By adding these two new features, serialization can be fully streamable and there is no need to pre-serialize big chunks in memory before writing them to the stream. Hope you'll find this suggestion useful and incorporate it into the protocol. Thanks, Yoav. -- >>> You received this message because you are subscribed to the Google >>> Groups "Protocol Buffers" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to protobuf+u...@googlegroups.com. >>> To post to this group, send email to prot...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/protobuf. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@g
Re: [protobuf] Re: Streaming Serialization - Suggestion
They say on their website: "When evaluating new features, we look for additions that are very widely useful or very simple". What I'm suggesting here is both very useful (speeding up serialization and eliminating memory duplication) and very simple (simple additions to the encoding, no need to change the language). So far, no response from the Google guys... On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: > > This exact suggestion has been up for discussion long time ago(years?, > before proto2?) > > When it comes to taking suggestions I'm only a 3rd party implementer but > my understanding is that the design process of protocol buffers and its > goals are internal to Google and they usually publish new versions of their > code implementing new features before you can read about them in the > documents. > On Mar 27, 2016 5:31 AM, "Yoav H" > > wrote: > >> Any comment on this? >> Will you consider this for proto3? >> >> On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: >>> >>> Hi, >>> >>> I have a suggestion fr improving the protobuf encoding. >>> Is proto3 final? >>> >>> I like the simplicity of the encoding of protobuf. >>> But I think it has one issue with serialization, using streams. >>> The problem is with length delimited fields and the fact that they >>> require knowing the length ahead of time. >>> If we have a very long string, we need to encode the entire string >>> before we know its length, so we basically duplicate the data in memory. >>> Same is true for embedded messages, where we need to encode the entire >>> embedded message before we can append it to the stream. >>> >>> I think there is a simple solution for both issues. >>> >>> For strings and byte arrays, a simple solution is to use "chunked >>> encoding". >>> Which means that the byte array is split into chunks and every chunk >>> starts with the chunk length. End of array is indicated by length zero. >>> >>> For embedded messages, the solution is to have an "start embedding" tag >>> and an "end embedding tag". >>> Everything in between is the embedded message. >>> >>> By adding these two new features, serialization can be fully streamable >>> and there is no need to pre-serialize big chunks in memory before writing >>> them to the stream. >>> >>> Hope you'll find this suggestion useful and incorporate it into the >>> protocol. >>> >>> Thanks, >>> Yoav. >>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to protobuf+u...@googlegroups.com . >> To post to this group, send email to prot...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
Re: [protobuf] Re: Streaming Serialization - Suggestion
This exact suggestion has been up for discussion long time ago(years?, before proto2?) When it comes to taking suggestions I'm only a 3rd party implementer but my understanding is that the design process of protocol buffers and its goals are internal to Google and they usually publish new versions of their code implementing new features before you can read about them in the documents. On Mar 27, 2016 5:31 AM, "Yoav H" wrote: > Any comment on this? > Will you consider this for proto3? > > On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: >> >> Hi, >> >> I have a suggestion fr improving the protobuf encoding. >> Is proto3 final? >> >> I like the simplicity of the encoding of protobuf. >> But I think it has one issue with serialization, using streams. >> The problem is with length delimited fields and the fact that they >> require knowing the length ahead of time. >> If we have a very long string, we need to encode the entire string before >> we know its length, so we basically duplicate the data in memory. >> Same is true for embedded messages, where we need to encode the entire >> embedded message before we can append it to the stream. >> >> I think there is a simple solution for both issues. >> >> For strings and byte arrays, a simple solution is to use "chunked >> encoding". >> Which means that the byte array is split into chunks and every chunk >> starts with the chunk length. End of array is indicated by length zero. >> >> For embedded messages, the solution is to have an "start embedding" tag >> and an "end embedding tag". >> Everything in between is the embedded message. >> >> By adding these two new features, serialization can be fully streamable >> and there is no need to pre-serialize big chunks in memory before writing >> them to the stream. >> >> Hope you'll find this suggestion useful and incorporate it into the >> protocol. >> >> Thanks, >> Yoav. >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to protobuf+unsubscr...@googlegroups.com. > To post to this group, send email to protobuf@googlegroups.com. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Streaming Serialization - Suggestion
Any comment on this? Will you consider this for proto3? On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: > > Hi, > > I have a suggestion fr improving the protobuf encoding. > Is proto3 final? > > I like the simplicity of the encoding of protobuf. > But I think it has one issue with serialization, using streams. > The problem is with length delimited fields and the fact that they require > knowing the length ahead of time. > If we have a very long string, we need to encode the entire string before > we know its length, so we basically duplicate the data in memory. > Same is true for embedded messages, where we need to encode the entire > embedded message before we can append it to the stream. > > I think there is a simple solution for both issues. > > For strings and byte arrays, a simple solution is to use "chunked > encoding". > Which means that the byte array is split into chunks and every chunk > starts with the chunk length. End of array is indicated by length zero. > > For embedded messages, the solution is to have an "start embedding" tag > and an "end embedding tag". > Everything in between is the embedded message. > > By adding these two new features, serialization can be fully streamable > and there is no need to pre-serialize big chunks in memory before writing > them to the stream. > > Hope you'll find this suggestion useful and incorporate it into the > protocol. > > Thanks, > Yoav. > > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscr...@googlegroups.com. To post to this group, send email to protobuf@googlegroups.com. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
[protobuf] Re: Streaming Dynamic Messages
Yes! That works, thanks. On Nov 9, 8:37 pm, Kenton Varda wrote: > You want: com.google.protobuf.Descriptors.FileDescriptor.buildFrom() > > > > On Mon, Nov 9, 2009 at 5:25 PM, Dan wrote: > > > Im trying to write the following client/server exchange using the Java > > Protobuf API: > > > 1) Client makes a request to the server > > 2) Server creates a series of Messages all of the same type to stream > > back to the client. The type of the messages will be different > > depending on the parameters of the client's request. > > 3) The Server streams the messages back to the client in the following > > format: > > > > > > > > > > > > > > > ... > > > 4) (This is where I'm stuck) The client makes a csv file from the > > results by reading the field names from the DescriptorProto to use for > > the header and iterates through the fields of each message to write > > the records. It looks like DynamicMessage should give me what I need > > but I can't figure out how to convert the DescriptorProto object to > > the Descriptor object. > > > Note: I know this approach of sending just the DescriptorProto for the > > messages cannot possibly work if the message has any fields that are > > of a Message Type that are not nested but for my use case I know that > > fields of the messages returned will always be a primative type. Even > > using the approach described in the techniques section of the wiki of > > sending an entire FileDescriptorSet I still have the same problem of > > converting the Proto objects to the proper framework objects. I've > > seen in previous posts how to do this using the C++ API but not the > > Java Implementation.- Hide quoted text - > > - Show quoted text - --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
[protobuf] Re: Streaming Dynamic Messages
You want: com.google.protobuf.Descriptors.FileDescriptor.buildFrom() On Mon, Nov 9, 2009 at 5:25 PM, Dan wrote: > > Im trying to write the following client/server exchange using the Java > Protobuf API: > > 1) Client makes a request to the server > 2) Server creates a series of Messages all of the same type to stream > back to the client. The type of the messages will be different > depending on the parameters of the client's request. > 3) The Server streams the messages back to the client in the following > format: > > > > > > > > ... > > 4) (This is where I'm stuck) The client makes a csv file from the > results by reading the field names from the DescriptorProto to use for > the header and iterates through the fields of each message to write > the records. It looks like DynamicMessage should give me what I need > but I can't figure out how to convert the DescriptorProto object to > the Descriptor object. > > Note: I know this approach of sending just the DescriptorProto for the > messages cannot possibly work if the message has any fields that are > of a Message Type that are not nested but for my use case I know that > fields of the messages returned will always be a primative type. Even > using the approach described in the techniques section of the wiki of > sending an entire FileDescriptorSet I still have the same problem of > converting the Proto objects to the proper framework objects. I've > seen in previous posts how to do this using the C++ API but not the > Java Implementation. > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: 'Streaming' messages (say over a socket)
Don't bother trying to implement your own std::ostream; it's a horrible
mess. Using CopyingOutputStreamAdaptor only requires that you implement the
CopyingOutputStream interface, which is extremely simple.
But yeah, since your messages are small, what you have now is fine.
On Wed, Jun 17, 2009 at 1:49 PM, Alex Black wrote:
> Thanks for pointing out CodedOuptutStream::Varint32Size(), I'll use that.
>
> My messages are lists of messages, and I am breaking them into batches of
> say 1,000 deliberately, so that each is a reasonable size that I can
> allocate a buffer for and send over the wire. In one scenario I have 100,000
> things to send, so I send them in batches, to avoid allocating memory for
> all 100,000 at once.
>
> m_Stream is a light wrapper around a socket, not a c++ stream, I need to
> get my head around how to implement a c++ style stream wrapper around a
> socket.
>
> - Alex
>
> --
> *From:* Kenton Varda [mailto:ken...@google.com]
> *Sent:* Wednesday, June 17, 2009 4:42 PM
> *To:* Alex Black
> *Cc:* Christopher Smith; Protocol Buffers
>
> *Subject:* Re: 'Streaming' messages (say over a socket)
>
> Mostly looks fine.
>
> Note that a varint can be up to 5 bytes. You should probably just use
> CodedOutputStream::Varint32Size() to compute the exact size so that you can
> allocate a buffer that is exactly large enough.
>
> Also note that if your message is large (say, 10k or more), allocating a
> single large buffer may make the memory allocator unhappy. I'm not sure
> what type m_Stream is, but you might consider wrapping it in a
> ZeroCopyOutputStream and wrapping that in a CodedOutputStream, then writing
> to that, so that it doesn't have to buffer the whole message all at once.
> Note that CopyingOutputStreamAdaptor makes it pretty easy to adapt
> traditional output streams to ZeroCopyOutputStream:
>
>
> http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.io.zero_copy_stream_impl.html#CopyingOutputStreamAdaptor
>
> On Tue, Jun 16, 2009 at 8:48 PM, Alex Black wrote:
>
>> Thanks, I got something working...how does this look? (ugly I'm sure...)
>>
>> On the C++ server side I am looping sending many messages (all of the same
>> type), and on the Java side I am looping parsing them out.
>>
>> C++ Server:
>>
>> int size = message.ByteSize();
>> EnsureBufferIsAtLeastSize(size + 4);
>>
>> char* pBuffer = (char*)
>> google::protobuf::io::CodedOutputStream::WriteVarint32ToArray(size,
>> (google::protobuf::uint8*) m_pBuffer);
>>
>> // Serialize the message
>> bool result = message.SerializeToArray(pBuffer, size);
>>
>> // Calculate how many bytes the 'size' took
>> int sizeSize = pBuffer - m_pBuffer;
>>
>> // Write the message to the stream
>> m_Stream.Write(m_pBuffer,size + sizeSize);
>>
>> Java client:
>>
>>com.google.protobuf.CodedInputStream stream =
>> com.google.protobuf.CodedInputStream.newInstance(url.openStream());
>>
>>while ( !stream.isAtEnd() )
>>{
>> Foor.Bar.Builder builder = Foo.Bar.newBuilder();
>> stream.readMessage(builder, null);
>>
>> Foo.Bar message = builder.build();
>>}
>>
>>
>>
>>
>> --
>> *From:* Christopher Smith [mailto:cbsm...@gmail.com]
>> *Sent:* Monday, June 15, 2009 2:58 PM
>> *To:* Alex Black
>> *Cc:* Protocol Buffers
>> *Subject:* Re: 'Streaming' messages (say over a socket)
>>
>> The normal way to do it is to send each Entity as a separate message.
>> CodedInput/OutputStream is handed for that kind of thing.
>>
>> --Chris
>>
>> On Sun, Jun 14, 2009 at 4:14 PM, Alex Black wrote:
>>
>>>
>>> Is there a way to start sending a message before its fully composed?
>>>
>>> Say we have messages like this:
>>>
>>> message Entity
>>> {
>>>required int32 id = 1;
>>>required string name = 2;
>>> }
>>>
>>> message Entities
>>> {
>>> repeated Entity entity = 1;
>>> }
>>>
>>> If we're sending a message Entities with 1,000 Entity objects in it,
>>> is there a way to avoid composing the entire message in memory,
>>> serializing it, and then sending it out?
>>>
>>> I'd like to avoid allocating RAM for the entire message, and just send
>>> it out as I compose it...
>>>
>>> thx,
>>>
>>> - Alex
>>> clear=all>
>>> --
>>> Chris
>>>
>>> >>>
>>
>>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
RE: 'Streaming' messages (say over a socket)
Thanks for pointing out CodedOuptutStream::Varint32Size(), I'll use
that.
My messages are lists of messages, and I am breaking them into batches
of say 1,000 deliberately, so that each is a reasonable size that I can
allocate a buffer for and send over the wire. In one scenario I have
100,000 things to send, so I send them in batches, to avoid allocating
memory for all 100,000 at once.
m_Stream is a light wrapper around a socket, not a c++ stream, I need to
get my head around how to implement a c++ style stream wrapper around a
socket.
- Alex
From: Kenton Varda [mailto:ken...@google.com]
Sent: Wednesday, June 17, 2009 4:42 PM
To: Alex Black
Cc: Christopher Smith; Protocol Buffers
Subject: Re: 'Streaming' messages (say over a socket)
Mostly looks fine.
Note that a varint can be up to 5 bytes. You should probably just use
CodedOutputStream::Varint32Size() to compute the exact size so that you
can allocate a buffer that is exactly large enough.
Also note that if your message is large (say, 10k or more), allocating a
single large buffer may make the memory allocator unhappy. I'm not sure
what type m_Stream is, but you might consider wrapping it in a
ZeroCopyOutputStream and wrapping that in a CodedOutputStream, then
writing to that, so that it doesn't have to buffer the whole message all
at once. Note that CopyingOutputStreamAdaptor makes it pretty easy to
adapt traditional output streams to ZeroCopyOutputStream:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.pr
otobuf.io.zero_copy_stream_impl.html#CopyingOutputStreamAdaptor
On Tue, Jun 16, 2009 at 8:48 PM, Alex Black wrote:
Thanks, I got something working...how does this look? (ugly I'm
sure...)
On the C++ server side I am looping sending many messages (all
of the same type), and on the Java side I am looping parsing them out.
C++ Server:
int size = message.ByteSize();
EnsureBufferIsAtLeastSize(size + 4);
char* pBuffer = (char*)
google::protobuf::io::CodedOutputStream::WriteVarint32ToArray(size,
(google::protobuf::uint8*) m_pBuffer);
// Serialize the message
bool result = message.SerializeToArray(pBuffer, size);
// Calculate how many bytes the 'size' took
int sizeSize = pBuffer - m_pBuffer;
// Write the message to the stream
m_Stream.Write(m_pBuffer,size + sizeSize);
Java client:
com.google.protobuf.CodedInputStream stream =
com.google.protobuf.CodedInputStream.newInstance(url.openStream());
while ( !stream.isAtEnd() )
{
Foor.Bar.Builder builder = Foo.Bar.newBuilder();
stream.readMessage(builder, null);
Foo.Bar message = builder.build();
}
From: Christopher Smith [mailto:cbsm...@gmail.com]
Sent: Monday, June 15, 2009 2:58 PM
To: Alex Black
Cc: Protocol Buffers
Subject: Re: 'Streaming' messages (say over a socket)
The normal way to do it is to send each Entity as a separate
message. CodedInput/OutputStream is handed for that kind of thing.
--Chris
On Sun, Jun 14, 2009 at 4:14 PM, Alex Black
wrote:
Is there a way to start sending a message before its
fully composed?
Say we have messages like this:
message Entity
{
required int32 id = 1;
required string name = 2;
}
message Entities
{
repeated Entity entity = 1;
}
If we're sending a message Entities with 1,000 Entity
objects in it,
is there a way to avoid composing the entire message in
memory,
serializing it, and then sending it out?
I'd like to avoid allocating RAM for the entire message,
and just send
it out as I compose it...
thx,
- Alex
clear=all>
--
Chris
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: 'Streaming' messages (say over a socket)
Mostly looks fine.
Note that a varint can be up to 5 bytes. You should probably just use
CodedOutputStream::Varint32Size() to compute the exact size so that you can
allocate a buffer that is exactly large enough.
Also note that if your message is large (say, 10k or more), allocating a
single large buffer may make the memory allocator unhappy. I'm not sure
what type m_Stream is, but you might consider wrapping it in a
ZeroCopyOutputStream and wrapping that in a CodedOutputStream, then writing
to that, so that it doesn't have to buffer the whole message all at once.
Note that CopyingOutputStreamAdaptor makes it pretty easy to adapt
traditional output streams to ZeroCopyOutputStream:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.io.zero_copy_stream_impl.html#CopyingOutputStreamAdaptor
On Tue, Jun 16, 2009 at 8:48 PM, Alex Black wrote:
> Thanks, I got something working...how does this look? (ugly I'm sure...)
>
> On the C++ server side I am looping sending many messages (all of the same
> type), and on the Java side I am looping parsing them out.
>
> C++ Server:
>
> int size = message.ByteSize();
> EnsureBufferIsAtLeastSize(size + 4);
>
> char* pBuffer = (char*)
> google::protobuf::io::CodedOutputStream::WriteVarint32ToArray(size,
> (google::protobuf::uint8*) m_pBuffer);
>
> // Serialize the message
> bool result = message.SerializeToArray(pBuffer, size);
>
> // Calculate how many bytes the 'size' took
> int sizeSize = pBuffer - m_pBuffer;
>
> // Write the message to the stream
> m_Stream.Write(m_pBuffer,size + sizeSize);
>
> Java client:
>
>com.google.protobuf.CodedInputStream stream =
> com.google.protobuf.CodedInputStream.newInstance(url.openStream());
>
>while ( !stream.isAtEnd() )
>{
> Foor.Bar.Builder builder = Foo.Bar.newBuilder();
> stream.readMessage(builder, null);
>
> Foo.Bar message = builder.build();
>}
>
>
>
>
> --
> *From:* Christopher Smith [mailto:cbsm...@gmail.com]
> *Sent:* Monday, June 15, 2009 2:58 PM
> *To:* Alex Black
> *Cc:* Protocol Buffers
> *Subject:* Re: 'Streaming' messages (say over a socket)
>
> The normal way to do it is to send each Entity as a separate message.
> CodedInput/OutputStream is handed for that kind of thing.
>
> --Chris
>
> On Sun, Jun 14, 2009 at 4:14 PM, Alex Black wrote:
>
>>
>> Is there a way to start sending a message before its fully composed?
>>
>> Say we have messages like this:
>>
>> message Entity
>> {
>>required int32 id = 1;
>>required string name = 2;
>> }
>>
>> message Entities
>> {
>> repeated Entity entity = 1;
>> }
>>
>> If we're sending a message Entities with 1,000 Entity objects in it,
>> is there a way to avoid composing the entire message in memory,
>> serializing it, and then sending it out?
>>
>> I'd like to avoid allocating RAM for the entire message, and just send
>> it out as I compose it...
>>
>> thx,
>>
>> - Alex
>> clear=all>
>> --
>> Chris
>>
>> >>
>
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
RE: 'Streaming' messages (say over a socket)
Thanks, I got something working...how does this look? (ugly I'm sure...)
On the C++ server side I am looping sending many messages (all of the
same type), and on the Java side I am looping parsing them out.
C++ Server:
int size = message.ByteSize();
EnsureBufferIsAtLeastSize(size + 4);
char* pBuffer = (char*)
google::protobuf::io::CodedOutputStream::WriteVarint32ToArray(size,
(google::protobuf::uint8*) m_pBuffer);
// Serialize the message
bool result = message.SerializeToArray(pBuffer, size);
// Calculate how many bytes the 'size' took
int sizeSize = pBuffer - m_pBuffer;
// Write the message to the stream
m_Stream.Write(m_pBuffer,size + sizeSize);
Java client:
com.google.protobuf.CodedInputStream stream =
com.google.protobuf.CodedInputStream.newInstance(url.openStream());
while ( !stream.isAtEnd() )
{
Foor.Bar.Builder builder = Foo.Bar.newBuilder();
stream.readMessage(builder, null);
Foo.Bar message = builder.build();
}
From: Christopher Smith [mailto:cbsm...@gmail.com]
Sent: Monday, June 15, 2009 2:58 PM
To: Alex Black
Cc: Protocol Buffers
Subject: Re: 'Streaming' messages (say over a socket)
The normal way to do it is to send each Entity as a separate message.
CodedInput/OutputStream is handed for that kind of thing.
--Chris
On Sun, Jun 14, 2009 at 4:14 PM, Alex Black wrote:
Is there a way to start sending a message before its fully
composed?
Say we have messages like this:
message Entity
{
required int32 id = 1;
required string name = 2;
}
message Entities
{
repeated Entity entity = 1;
}
If we're sending a message Entities with 1,000 Entity objects in
it,
is there a way to avoid composing the entire message in memory,
serializing it, and then sending it out?
I'd like to avoid allocating RAM for the entire message, and
just send
it out as I compose it...
thx,
- Alex
--
Chris
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: 'Streaming' messages (say over a socket)
http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming
On Sun, Jun 14, 2009 at 4:14 PM, Alex Black wrote:
>
> Is there a way to start sending a message before its fully composed?
>
> Say we have messages like this:
>
> message Entity
> {
>required int32 id = 1;
>required string name = 2;
> }
>
> message Entities
> {
> repeated Entity entity = 1;
> }
>
> If we're sending a message Entities with 1,000 Entity objects in it,
> is there a way to avoid composing the entire message in memory,
> serializing it, and then sending it out?
>
> I'd like to avoid allocating RAM for the entire message, and just send
> it out as I compose it...
>
> thx,
>
> - Alex
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: 'Streaming' messages (say over a socket)
The normal way to do it is to send each Entity as a separate message.
CodedInput/OutputStream is handed for that kind of thing.
--Chris
On Sun, Jun 14, 2009 at 4:14 PM, Alex Black wrote:
>
> Is there a way to start sending a message before its fully composed?
>
> Say we have messages like this:
>
> message Entity
> {
>required int32 id = 1;
>required string name = 2;
> }
>
> message Entities
> {
> repeated Entity entity = 1;
> }
>
> If we're sending a message Entities with 1,000 Entity objects in it,
> is there a way to avoid composing the entire message in memory,
> serializing it, and then sending it out?
>
> I'd like to avoid allocating RAM for the entire message, and just send
> it out as I compose it...
>
> thx,
>
> - Alex
> >
>
--
Chris
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Streaming different types of messages
On Fri, Mar 27, 2009 at 9:08 AM, Dave Bailey wrote: > I don't suppose there'd ever be a way to mark a set of fields as > mutually exclusive? Nope. I've tossed around the idea before but no one has had time to work on it. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Streaming different types of messages
achin...@gmail.com writes: > Thanks. Also how do I know the type of the message? One way would be > to check all optional fields (each represent a different type of > message) of the wrapper message and then pick the one which is not > null. Is that the only way? You can add a (required) field to indicate the intended contents of the message, as described at http://code.google.com/apis/protocolbuffers/docs/techniques.html#union Note that Protocol Buffers will not enforce the "business rule" that the message's declared type and actual content must match, but it is straightforward to create a wrapper that will do that. Michael Poole --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Streaming different types of messages
Thanks. Also how do I know the type of the message? One way would be to check all optional fields (each represent a different type of message) of the wrapper message and then pick the one which is not null. Is that the only way? On Mar 27, 12:08 pm, Dave Bailey wrote: > Kenton, > > I don't suppose there'd ever be a way to mark a set of fields as > mutually exclusive? > > -dave > > On Mar 27, 8:42 am, "Jon Skeet " > wrote: > > > On Mar 27, 1:32 pm, achin...@gmail.com wrote: > > > > If I understand correctly there is no good way to use proto buffers to > > > stream different types of messages, right? For example if my stream > > > has a mix of several messages of type m1 and m2, I will have to device > > > a scheme outside of proto buffers to separate it into 2 streams and > > > then pass it through parsers for each. > > > > In other words is there a way to do event based parsing using proto > > > buffers, or even a way to say don't parse a repetitive field unless > > > needed. > > > No, you don't have to do it into separate streams. Instead, stream a > > sequence of messages each of which has either an m1, or an m2, or an > > m3 etc. This basically ends up being (tag) (message) (tag) (message), > > where the tag is effectively identifying the type of message. > > > All you need to do is create the wrapper message, and the rest should > > work fine. > > > Jon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Streaming different types of messages
Kenton, I don't suppose there'd ever be a way to mark a set of fields as mutually exclusive? -dave On Mar 27, 8:42 am, "Jon Skeet " wrote: > On Mar 27, 1:32 pm, achin...@gmail.com wrote: > > > If I understand correctly there is no good way to use proto buffers to > > stream different types of messages, right? For example if my stream > > has a mix of several messages of type m1 and m2, I will have to device > > a scheme outside of proto buffers to separate it into 2 streams and > > then pass it through parsers for each. > > > In other words is there a way to do event based parsing using proto > > buffers, or even a way to say don't parse a repetitive field unless > > needed. > > No, you don't have to do it into separate streams. Instead, stream a > sequence of messages each of which has either an m1, or an m2, or an > m3 etc. This basically ends up being (tag) (message) (tag) (message), > where the tag is effectively identifying the type of message. > > All you need to do is create the wrapper message, and the rest should > work fine. > > Jon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Streaming different types of messages
On Mar 27, 1:32 pm, achin...@gmail.com wrote: > If I understand correctly there is no good way to use proto buffers to > stream different types of messages, right? For example if my stream > has a mix of several messages of type m1 and m2, I will have to device > a scheme outside of proto buffers to separate it into 2 streams and > then pass it through parsers for each. > > In other words is there a way to do event based parsing using proto > buffers, or even a way to say don't parse a repetitive field unless > needed. No, you don't have to do it into separate streams. Instead, stream a sequence of messages each of which has either an m1, or an m2, or an m3 etc. This basically ends up being (tag) (message) (tag) (message), where the tag is effectively identifying the type of message. All you need to do is create the wrapper message, and the rest should work fine. Jon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: [Fwd: Re: Streaming]
It's quite easy to write a helper function that reads/writes delimited
messages (delimited by size or by end tag).
For example, here's one for writing a length-delimited message:
bool WriteMessage(const Message& message, ZeroCopyOutputStream* output) {
CodedOutputStream coded_out(output);
return coded_out.WriteVarint32(message.ByteSize()) &&
message.SerializeWithCachedSizes(&coded_out);
}
and here's one for reading one message:
bool ReadMessage(ZeroCopyInputStream* input, Message* message) {
CodedInputStream coded_in(input);
uint32 size;
if (!coded_in.ReadVarint32(&size)) return false;
CodedInputStream::Limit limit = coded_in.PushLimit(size);
if (!message->ParseFromCodedStream(&coded_in)) return false;
if (!coded_in.ExpectAtEnd()) return false;
coded_in.PopLimit(limit);
return true;
}
(I haven't tested the above so it may contain minor errors.)
We could add these as methods of the Message class. Note, though, that for
many applications, this kind of streaming is too simplistic. For example,
the above will not allow you to efficiently seek to an arbitrary message in
the stream, since at the very least you have to read the sizes of all
messages before it to find it. It's also not very robust in the face of
data corruption -- if any of the sizes are corrupted, the whole stream is
unreadable. So, you may find you want to do something more complicated,
depending on your app. But, anything more complicated is really beyond the
scope of the protocol buffer library.
On Fri, Dec 5, 2008 at 8:27 AM, Shane Green <[EMAIL PROTECTED]>wrote:
>
> Thanks very much Jon (see below). You make good points and I like the
> approach the you describe. I am still thinking, however, that there is
> power in the ability for message instances to write and parse themselves
> from a stream.
>
> A message instance could be passed a stream object which chains back to
> the network connection from which bytes are being received. A stop flag
> based parsing mechanism could be passed this buffer object, and would
> handle reading the stream and initializing its properties, exiting when
> the serialization of that message instance stopped. At this point, a
> new message instance could be created, and the process repeated.
>
> The type of message doing the parsing could vary from message to
> message, even with the serializations being sent and received back to
> back. This mechanism would work regardless of field-types being
> streamed. A message type consisting solely of varint fields, whose
> length is determined while reading the varint's value, would support
> streaming no differently than any other message type.
>
> The solution also seems to support every requirement supported by the
> original buffer type. Messages serialized to a buffer, could just as
> easily be initialized from that buffer as they could from the string
> contained by the buffer.
>
> m1 = Message()
> buffer = Buffer()
> [...] (initialize instance vars)
> m1.SerializeToBuffer(buffer)
>
> m2 = Message()
> m2.ParseFromBuffer(buffer)
>
> Produces same result as:
>
> m2 = Message()
> bytes = m1.SerializeToString()
> m2.ParseFromString(bytes)
>
> The string-based parse would ignore the stop bit, parsing the entire
> string. The buffer-based parsing would stop parsing when the stop bit,
> producing the same result.
>
> Handling of concatenated serializations is supported through repeated
> calls to parse from buffer:
>
> m1 = Message()
> [...] (initialize instance vars)
> m2 = Message()
> [...] (initialize instance vars)
>
> buffer = Buffer()
> m1.SerializeToBuffer(buffer)
> m2.SerializeToBuffer(buffer)
>
> m3 = Message()
> m3.ParseFromBuffer(buffer)
> m3.ParseFromBuffer(buffer)
>
> Would produce same result as:
>
> m3 = Message()
> m3.ParseFromString(m1.SerializeToString() + m2.SerializeToString())
>
> As long as an unused, and never to be used, field number is used to
> generate the stop bit's key, then I don't believe there are any
> incompatibilities between buffer-based message marshalling and the
> existing string-based code. A very easy usage:
>
> # Sending side
> for message in messages:
> message.SerializeToBuffer(buffer)
>
> # Receiving side
> for msgtype in types:
> message = msgtype()
> message.ParseFromBuffer(buffer)
>
> Unless I've overlooked something, it seems like the stream based
> marshalling and unmarshalling is powerful, simple, and completely
> compatible with all existing code. But there is a very real chance I've
> overlooked something...
>
>
>
>
> - Shane
>
>
> Forwarded Message
> > From: Jon Skeet <[EMAIL PROTECTED]>
&
[Fwd: Re: Streaming]
Thanks very much Jon (see below). You make good points and I like the approach the you describe. I am still thinking, however, that there is power in the ability for message instances to write and parse themselves from a stream. A message instance could be passed a stream object which chains back to the network connection from which bytes are being received. A stop flag based parsing mechanism could be passed this buffer object, and would handle reading the stream and initializing its properties, exiting when the serialization of that message instance stopped. At this point, a new message instance could be created, and the process repeated. The type of message doing the parsing could vary from message to message, even with the serializations being sent and received back to back. This mechanism would work regardless of field-types being streamed. A message type consisting solely of varint fields, whose length is determined while reading the varint's value, would support streaming no differently than any other message type. The solution also seems to support every requirement supported by the original buffer type. Messages serialized to a buffer, could just as easily be initialized from that buffer as they could from the string contained by the buffer. m1 = Message() buffer = Buffer() [...] (initialize instance vars) m1.SerializeToBuffer(buffer) m2 = Message() m2.ParseFromBuffer(buffer) Produces same result as: m2 = Message() bytes = m1.SerializeToString() m2.ParseFromString(bytes) The string-based parse would ignore the stop bit, parsing the entire string. The buffer-based parsing would stop parsing when the stop bit, producing the same result. Handling of concatenated serializations is supported through repeated calls to parse from buffer: m1 = Message() [...] (initialize instance vars) m2 = Message() [...] (initialize instance vars) buffer = Buffer() m1.SerializeToBuffer(buffer) m2.SerializeToBuffer(buffer) m3 = Message() m3.ParseFromBuffer(buffer) m3.ParseFromBuffer(buffer) Would produce same result as: m3 = Message() m3.ParseFromString(m1.SerializeToString() + m2.SerializeToString()) As long as an unused, and never to be used, field number is used to generate the stop bit's key, then I don't believe there are any incompatibilities between buffer-based message marshalling and the existing string-based code. A very easy usage: # Sending side for message in messages: message.SerializeToBuffer(buffer) # Receiving side for msgtype in types: message = msgtype() message.ParseFromBuffer(buffer) Unless I've overlooked something, it seems like the stream based marshalling and unmarshalling is powerful, simple, and completely compatible with all existing code. But there is a very real chance I've overlooked something... - Shane Forwarded Message > From: Jon Skeet <[EMAIL PROTECTED]> > To: Shane Green <[EMAIL PROTECTED]> > Subject: Re: Streaming > Date: Fri, 5 Dec 2008 08:19:41 + > > 2008/12/5 Shane Green <[EMAIL PROTECTED]> > Thanks Jon. Those are good points. I rather liked the > self-delimiting > nature of fields, and thought this method would bring that > feature up to > the message level, without breaking any of the existing > capabilities. > So my goal was a message which could truly be streamed; > perhaps even > sent without knowing its own size up front. Perhaps I > overlooked > something? > > Currently the PB format requires that you know the size of each > submessage before you send it. You don't need to know the size of the > whole message, as it's assumed to be the entire size of the > datastream. It's unfortunate that you do need to provide the whole > message to the output stream though, unless you want to manually > serialize the individual fields. > > My goal was slightly different - I wanted to be able to stream a > sequence of messages. The most obvious use case (in my view) is a log. > Write out a massive log file as a sequence of entries, and you can > read it back in one at a time. It's not designed to help to stream a > single huge message though. > > Would you mind if I resent my questions to the group? I lack > confidence and wanted to make sure I wasn't overlooking > something > ridiculous, but am thinking that the exchange would be > informative. > > Absolutely. Feel free to quote anything I've written if you think it > helps. > > Also, how are you serializing and parsing messages as if they > are > repeated fields of a container message? Is there a fair bit > of parsing > or work being done outside the standard protocol-buffer APIs? &
Re: Streaming
On Dec 4, 10:00 pm, Shane Green <[EMAIL PROTECTED]> wrote: > Would the following two methods enable streaming of protocol buffer > messages: > > "SerializeToBuffer(buffer)" - which writes the serialized message to the > stream, plus a single field whose wire-type is 4 (End Group). > > "ParseFromBuffer(buffer)" - which parses data read from the buffer until > it reaches a standalone field of wire type 4. This field would be read > from the buffer and then discarded before returning. Rather than parse until a particular tag, it's simpler to serialize an object as if it were a repeated field of a container object. That way, apart from anything else, if you ever *do* write the container object, everything is already compatible. This is what I've done in my C# port (always using field number 1, although I may provide a way of writing/expecting a different field at some point) and it works well. Jon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Streaming
I have been thinking about the most efficient way of describing which message is being sent in a stream. All fields have a numeric key, so how about messages also having a (optional) numeric key too? Then the PB compiler could populate a global enumerator list of messages, which you could use in a message wrapper. Scott Stafford wrote: > You may want to consider the techniques described in the protobuf doc > under Techniques/Union Types here: > > http://code.google.com/apis/protocolbuffers/docs/techniques.html > > By using an enum/extensions, you can use protobuf for the header as > well as the payload. The wrapper message would represent the channel > number as enum or the extension field identifier. The advantages: > > * You can harness protobuf's forward/backward compatibility tricks to > modify the header > * You can easily use varint encoding for the channel number > > Scott > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Streaming
You may want to consider the techniques described in the protobuf doc
under Techniques/Union Types here:
http://code.google.com/apis/protocolbuffers/docs/techniques.html
By using an enum/extensions, you can use protobuf for the header as
well as the payload. The wrapper message would represent the channel
number as enum or the extension field identifier. The advantages:
* You can harness protobuf's forward/backward compatibility tricks to
modify the header
* You can easily use varint encoding for the channel number
Scott
On Nov 17, 12:00 pm, "Ivarsson, Magnus"
<[EMAIL PROTECTED]> wrote:
> Hi, we are assuming that the tcp connection gets us in sync so we
> skipped the start bit.
>
> On the other hand we needed to distinguish different channels (to be
> able to send several proto api:s to the same port) so our header looks
> like this:
>
> 1 byte 0x00 (to be used for version information if we change the header
> later on)
> 2 bytes in network byte order for the channel number
> 2 bytes in network byte order for the length
>
> The channel will probably raise a few eyebrows - the tcp port could do
> the job.. However when we go for ssl encryption and embedded devices I
> think one session should be enough.
>
> Please give us feedback if you don't (or do) like the solution.
>
> /Magnus
>
> -Original Message-
> From: protobuf@googlegroups.com [mailto:[EMAIL PROTECTED] On
>
> Behalf Of Stuart
> Sent: den 17 november 2008 16:23
> To: Protocol Buffers
> Subject: Streaming
>
> I have a c# server application with miltiple clients. I want to
> replace the existing protocol with PB.
>
> The clients connect, and remain connected prediodically passing data
> of differnet types.
>
> I have searched the discusstion group, and found a few postings
> relating to this issue, but nothing final, so before I start writing
> my own solution, I was wondering if anyone had any ideas?
>
> I'm thinking about doing something simple, like this:
>
> StartByte (constat value)
> Length of payload (compressed integer)
> PB data
> EndByte (constat value)
>
> To decode:
>
> while(true)
> {
> Look for StartByte
> Grab Length integer
> If EndByte is where it is expected
> : Decode
> }
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Streaming
On Nov 17, 3:23 pm, Stuart <[EMAIL PROTECTED]> wrote: > I have a c# server application with miltiple clients. I want to > replace the existing protocol with PB. > > The clients connect, and remain connected prediodically passing data > of differnet types. My C# PB implementation has a MessageStreamIterator and a MessageStreamWriter (or names along those lines - I don't have easy access to the code at this minute). The downside is that it will only stream messages of a single type - but if you create a type which contains a bunch of optional messages, one for each message type you want to transmit, that could act the same way. Jon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Streaming
I was thinking that a start and end byte would get over the possibility
of a bug causing loss of sync. Especially important, if third-party
implementation of the protocol are establishing connections.
Thinking about it. an end-byte is not necessary. The next start-byte
(or end of stream), can be used as verification.
For SSL/Authentication, I was thinking about having a shell-like
interface for the client/server to negotiate the protocol, send
username/password etc before binary transfer starts.
Ivarsson, Magnus wrote:
> Hi, we are assuming that the tcp connection gets us in sync so we
> skipped the start bit.
>
> On the other hand we needed to distinguish different channels (to be
> able to send several proto api:s to the same port) so our header looks
> like this:
>
> 1 byte 0x00 (to be used for version information if we change the header
> later on)
> 2 bytes in network byte order for the channel number
> 2 bytes in network byte order for the length
>
> The channel will probably raise a few eyebrows - the tcp port could do
> the job.. However when we go for ssl encryption and embedded devices I
> think one session should be enough.
>
> Please give us feedback if you don't (or do) like the solution.
>
> /Magnus
>
>
> -Original Message-
> From: protobuf@googlegroups.com [mailto:[EMAIL PROTECTED] On
> Behalf Of Stuart
> Sent: den 17 november 2008 16:23
> To: Protocol Buffers
> Subject: Streaming
>
>
> I have a c# server application with miltiple clients. I want to
> replace the existing protocol with PB.
>
> The clients connect, and remain connected prediodically passing data
> of differnet types.
>
> I have searched the discusstion group, and found a few postings
> relating to this issue, but nothing final, so before I start writing
> my own solution, I was wondering if anyone had any ideas?
>
> I'm thinking about doing something simple, like this:
>
> StartByte (constat value)
> Length of payload (compressed integer)
> PB data
> EndByte (constat value)
>
> To decode:
>
> while(true)
> {
> Look for StartByte
> Grab Length integer
> If EndByte is where it is expected
> : Decode
> }
>
>
>
>
>
>
> >
>
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
RE: Streaming
Hi, we are assuming that the tcp connection gets us in sync so we
skipped the start bit.
On the other hand we needed to distinguish different channels (to be
able to send several proto api:s to the same port) so our header looks
like this:
1 byte 0x00 (to be used for version information if we change the header
later on)
2 bytes in network byte order for the channel number
2 bytes in network byte order for the length
The channel will probably raise a few eyebrows - the tcp port could do
the job.. However when we go for ssl encryption and embedded devices I
think one session should be enough.
Please give us feedback if you don't (or do) like the solution.
/Magnus
-Original Message-
From: protobuf@googlegroups.com [mailto:[EMAIL PROTECTED] On
Behalf Of Stuart
Sent: den 17 november 2008 16:23
To: Protocol Buffers
Subject: Streaming
I have a c# server application with miltiple clients. I want to
replace the existing protocol with PB.
The clients connect, and remain connected prediodically passing data
of differnet types.
I have searched the discusstion group, and found a few postings
relating to this issue, but nothing final, so before I start writing
my own solution, I was wondering if anyone had any ideas?
I'm thinking about doing something simple, like this:
StartByte (constat value)
Length of payload (compressed integer)
PB data
EndByte (constat value)
To decode:
while(true)
{
Look for StartByte
Grab Length integer
If EndByte is where it is expected
: Decode
}
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Streaming
In general, protocol buffers can be used to encode individual chunks within a stream, but representing the entire stream as a protocol buffer won't work very well. On Thu, Oct 23, 2008 at 8:14 PM, sureshkk <[EMAIL PROTECTED]> wrote: > > Hi, > > I am looking at a use case where one needs to transfer streaming data > (video, file transfer etc..). Will PB support such a use case on its > own? > > Thanks & Best Regards, > Suresh > > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Streaming
On Oct 24, 4:14 am, sureshkk <[EMAIL PROTECTED]> wrote: > I am looking at a use case where one needs to transfer streaming data > (video, file transfer etc..). Will PB support such a use case on its > own? What I've done in the C# port is have the concept of a streaming writer and a streaming reader (iterator). The stream is exactly the same as if it were all part of one "container" protocol buffer with a repeated field (number 1). You can see the code at: http://github.com/jskeet/dotnet-protobufs/tree/master/src/ProtocolBuffers/MessageStreamIterator.cs and http://github.com/jskeet/dotnet-protobufs/tree/master/src/ProtocolBuffers/MessageStreamWriter.cs (The iterator is quite complicated due to generics stuff - the GetEnumerator method near the bottom is the critical loop.) The nice thing about this is that if you *have* a container element which is just a repeated field 1, you can use the two interchangably. I may add more options for flexibility (variable field number, skipping inappropriate fields etc) later, but this is a reasonable start. Hope that helps. Jon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
