Re: [protobuf] Where is Protocol Buffers wire level specification?

[Markus Schaber]

Hi, Jon,

Am Donnerstag, 7. Juni 2018 15:05:17 UTC+2 schrieb Jon Skeet:
>
> On Thursday, 7 June 2018 10:59:34 UTC+1, Markus Schaber wrote:
>>
>> This text is missing some interesting aspects, e. G. what happens when 
>> the field number needs more than 5 bits (I cannot imagine it's limited to 
>> 31 fields per message).
>>
>
> No, that's already described:
>
> > Each key in the streamed message is a varint with the value 
> (field_number << 3) | wire_type – in other words, the last three bits of 
> the number store the wire type.
>
> And slightly earlier there's a description of how varints are serialized. 
> So if you have a field number of (say) 1000 and a wire format of 1, that 
> tag would be serialized as the varint encoding of the number 8001, which is 
> (1000 << 3) | 1.
>

Yes, you're right - I was somehow reading this as a byte, not a varint... 
:-(

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Where is Protocol Buffers wire level specification?

[Jon Skeet]

On Thursday, 7 June 2018 10:59:34 UTC+1, Markus Schaber wrote:
>
> This text is missing some interesting aspects, e. G. what happens when the 
> field number needs more than 5 bits (I cannot imagine it's limited to 31 
> fields per message).
>

No, that's already described:

> Each key in the streamed message is a varint with the value (field_number 
<< 3) | wire_type – in other words, the last three bits of the number store 
the wire type.

And slightly earlier there's a description of how varints are serialized. 
So if you have a field number of (say) 1000 and a wire format of 1, that 
tag would be serialized as the varint encoding of the number 8001, which is 
(1000 << 3) | 1.

Jon

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Where is Protocol Buffers wire level specification?

[Markus Schaber]

Hi,

I'm also on the search for a "full" spec.

Am Dienstag, 9. Januar 2018 18:24:19 UTC+1 schrieb Josh Humphries:
>
> This describes the basic encoding of protobuf messages into bytes:
> https://developers.google.com/protocol-buffers/docs/encoding
>

This text is missing some interesting aspects, e. G. what happens when the 
field number needs more than 5 bits (I cannot imagine it's limited to 31 
fields per message).

So reverse engineering existing codec sources is the only way to get a spec 
for the binary encoding? (e. g. for developing our own language backend or 
an independent lightweight parser)

Thanks,
Markus

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Where is Protocol Buffers wire level specification?

[Sergei Gnezdov]

Thank you 

That is useful.  I thought that there is an official specification.

On Tuesday, January 9, 2018 at 9:24:19 AM UTC-8, Josh Humphries wrote:
>
> This describes the basic encoding of protobuf messages into bytes:
> https://developers.google.com/protocol-buffers/docs/encoding
>
> The format does not include a length header. If you have data that 
> concatenates protobuf messages, then each such message will typically be 
> prefixed by a varint-encoded length, to delimit multiple messages in a 
> single stream. (Though other usages of proto, such as gRPC, have different 
> framing mechanisms for concatenating multiple messages in a single stream.)
>
>
> 
> *Josh Humphries*
> jh...@bluegosling.com 
>
> On Tue, Jan 9, 2018 at 12:01 PM, Sergei Gnezdov  > wrote:
>
>> I see a user level spec but I don't see a spec that defines a binary 
>> format.
>>
>>
>> Thank you
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Protocol Buffers" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to protobuf+u...@googlegroups.com .
>> To post to this group, send email to prot...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/protobuf.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Where is Protocol Buffers wire level specification?

[Josh Humphries]

This describes the basic encoding of protobuf messages into bytes:
https://developers.google.com/protocol-buffers/docs/encoding

The format does not include a length header. If you have data that
concatenates protobuf messages, then each such message will typically be
prefixed by a varint-encoded length, to delimit multiple messages in a
single stream. (Though other usages of proto, such as gRPC, have different
framing mechanisms for concatenating multiple messages in a single stream.)



*Josh Humphries*
jh...@bluegosling.com

On Tue, Jan 9, 2018 at 12:01 PM, Sergei Gnezdov 
wrote:

> I see a user level spec but I don't see a spec that defines a binary
> format.
>
>
> Thank you
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[protobuf] Where is Protocol Buffers wire level specification?

[Sergei Gnezdov]

I see a user level spec but I don't see a spec that defines a binary format.


Thank you

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.