[protobuf] Re: Serialize with Length Prefix
First time poster and new to protobuf/protobuf-net so forgive me if this
has already been addressed. But I've searched quite a bit and cant find
anofficial answer.
If you want to use the fieldNumber argument in
your SerializeWithLengthPrefix/TryDeserializeWithLengthPrefix for message
type identification (or for whatever reason), all works great between two
C#/.NET apps. However if you have a c++ app using protobuf talking to a
C#/.NET app using protobuf-net, the fieldNumber argument will cause grief.
It took wireshark, plus diving into protobuf-net code and protobuf code to
find that the fieldNumber is encoded like (I assume) any other field within
a message. Whereas the c++ protobuf has no such notion (that I could find)
unless you want to use a CodedOutputSTream - which then has both a WriteTag
and WriteVarint32 (which WriteTag calls). However WriteTag/WriteVarint32
does NOT encode the same as the fieldNumber with SerializeWithLengthPrefix.
So... what do to?
Again, perhaps this has been addressed already but here's my solution (on
the c++ side):
#define PROTO_NET_TAG(x) (((x) 3) | (2 7))
...
google::protobuf::io::ZeroCopyOutputStream* raw_output = new
google::protobuf::io::StringOutputStream(str);
google::protobuf::io::CodedOutputStream * coded = new
google::protobuf::io::CodedOutputStream(raw_output);
coded-WriteTag(PROTO_NET_TAG(msgTag));
coded-WriteVarint32(msg-ByteSize());
msg-SerializeToCodedStream(coded);
...
On Monday, August 24, 2009 5:33:26 PM UTC-6, Peter Keen wrote:
I looked at the protobuf-net source a little bit and it looks like if
you write the length with WriteLittleEndian32 instead of WriteVarint32
you should be good to go, as long as you use PrefixStyle.Fixed32
within the C# call to DeserialzeWithLengthPrefix().
--Pete
On Mon, Aug 24, 2009 at 4:17 PM, Jay
Thomasjaydian...@gmail.comjavascript:
wrote:
This code ideally has to operate with C# code on the other side of the
socket that uses the DeserializeWithLengthPrefix() method call. The
message originates in C++ environment on a linux box and terminates in
C# environment on a windows PC. Why doesn't C++ support the same
methods as C#? The only document that I've seen that has list of
methods available is the message.h header file, installed into my
Linux usr directory.
On Aug 24, 4:05 pm, Peter Keen peter.k...@gmail.com wrote:
You sort of have to roll your own. In my project I'm doing something
like this:
coded_output-WriteVarint32(message-ByteSize());
message-SerializeToCodedStream(coded_output);
And then on the reading side:
uint32_t size;
if (! coded_input-ReadVarint32(size)) {
return NULL; // just an example. probably don't want to do this;
}
Message * m = new Message;
CodedInputStream::Limit limit = coded_input-PushLimit(size);
message-ParseFromCodedStream(coded_input);
coded_input-PopLimit(limit);
The biggest thing that helped me along was finding the Limit docs. I
couldn't figure out how to parse from a ZeroCopyInputStream without
consuming the entire stream, and then I ran across that and everything
became much more clear.
--Pete
On Mon, Aug 24, 2009 at 3:54 PM, Jay Thomasjaydianetho...@gmail.com
wrote:
Hello
I am looking for a way to serialize/deserialize with length prefix
under c++. The serialized bytes will sent to a TCP socket. I am
aware that C# has a method SerializeWithLengthPrefix() and
DeserializeWithLengthPrefix(). Are there any such analogous methods
for C++? Please point out any documentation that I may have missed
here.
Thanks for any assistance.
Jay
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: [protobuf] Re: Serialize with Length Prefix
this seems to be a resurrection of a protobuf-net specific discussion, so
let me jump in...
The SerializeWithLengthPrefix method, by default, aims to represent data in
a way that is a valid protobuf stream - in particular, as though it were
simply a member of a parent object or list. As such, it encodes a [
field-number and string marker ], then the length - as two separate
varints.
So assuming the data was stored in this way (the default), consuming 2
defaults should consume the header information - the second of the 2 values
being the length of the subsequent payload. If the data was stored with
different header-formats, then you may need to adjust - for example the
fixed32 obviously requires 4 bytes to be consumed.
Does that make sense? or have I missed the point of the question?
Marc
On 20 February 2013 13:32, Paul Shafer prabo...@gmail.com wrote:
First time poster and new to protobuf/protobuf-net so forgive me if this
has already been addressed. But I've searched quite a bit and cant find
anofficial answer.
If you want to use the fieldNumber argument in
your SerializeWithLengthPrefix/TryDeserializeWithLengthPrefix for message
type identification (or for whatever reason), all works great between two
C#/.NET apps. However if you have a c++ app using protobuf talking to a
C#/.NET app using protobuf-net, the fieldNumber argument will cause grief.
It took wireshark, plus diving into protobuf-net code and protobuf code to
find that the fieldNumber is encoded like (I assume) any other field within
a message. Whereas the c++ protobuf has no such notion (that I could find)
unless you want to use a CodedOutputSTream - which then has both a WriteTag
and WriteVarint32 (which WriteTag calls). However WriteTag/WriteVarint32
does NOT encode the same as the fieldNumber with SerializeWithLengthPrefix.
So... what do to?
Again, perhaps this has been addressed already but here's my solution (on
the c++ side):
#define PROTO_NET_TAG(x) (((x) 3) | (2 7))
...
google::protobuf::io::ZeroCopyOutputStream* raw_output = new
google::protobuf::io::StringOutputStream(str);
google::protobuf::io::CodedOutputStream * coded = new
google::protobuf::io::CodedOutputStream(raw_output);
coded-WriteTag(PROTO_NET_TAG(msgTag));
coded-WriteVarint32(msg-ByteSize());
msg-SerializeToCodedStream(coded);
...
On Monday, August 24, 2009 5:33:26 PM UTC-6, Peter Keen wrote:
I looked at the protobuf-net source a little bit and it looks like if
you write the length with WriteLittleEndian32 instead of WriteVarint32
you should be good to go, as long as you use PrefixStyle.Fixed32
within the C# call to DeserialzeWithLengthPrefix().
--Pete
On Mon, Aug 24, 2009 at 4:17 PM, Jay Thomasjaydian...@gmail.**com
wrote:
This code ideally has to operate with C# code on the other side of the
socket that uses the DeserializeWithLengthPrefix() method call. The
message originates in C++ environment on a linux box and terminates in
C# environment on a windows PC. Why doesn't C++ support the same
methods as C#? The only document that I've seen that has list of
methods available is the message.h header file, installed into my
Linux usr directory.
On Aug 24, 4:05 pm, Peter Keen peter.k...@gmail.com wrote:
You sort of have to roll your own. In my project I'm doing something
like this:
coded_output-WriteVarint32(**message-ByteSize());
message-**SerializeToCodedStream(coded_**output);
And then on the reading side:
uint32_t size;
if (! coded_input-ReadVarint32(**size)) {
return NULL; // just an example. probably don't want to do this;
}
Message * m = new Message;
CodedInputStream::Limit limit = coded_input-PushLimit(size);
message-ParseFromCodedStream(**coded_input);
coded_input-PopLimit(limit);
The biggest thing that helped me along was finding the Limit docs. I
couldn't figure out how to parse from a ZeroCopyInputStream without
consuming the entire stream, and then I ran across that and everything
became much more clear.
--Pete
On Mon, Aug 24, 2009 at 3:54 PM, Jay Thomasjaydianetho...@gmail.**com
wrote:
Hello
I am looking for a way to serialize/deserialize with length prefix
under c++. The serialized bytes will sent to a TCP socket. I am
aware that C# has a method SerializeWithLengthPrefix() and
DeserializeWithLengthPrefix(). Are there any such analogous methods
for C++? Please point out any documentation that I may have missed
here.
Thanks for any assistance.
Jay
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
--
Re: Serialize with Length Prefix
Just to offer my thanks as well, since I was offline at the time and unable to help. Regards, Marc Gravell (protobuf-net) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/protobuf?hl=en -~--~~~~--~~--~--~---
Re: Serialize with Length Prefix
You sort of have to roll your own. In my project I'm doing something like this:
coded_output-WriteVarint32(message-ByteSize());
message-SerializeToCodedStream(coded_output);
And then on the reading side:
uint32_t size;
if (! coded_input-ReadVarint32(size)) {
return NULL; // just an example. probably don't want to do this;
}
Message * m = new Message;
CodedInputStream::Limit limit = coded_input-PushLimit(size);
message-ParseFromCodedStream(coded_input);
coded_input-PopLimit(limit);
The biggest thing that helped me along was finding the Limit docs. I
couldn't figure out how to parse from a ZeroCopyInputStream without
consuming the entire stream, and then I ran across that and everything
became much more clear.
--Pete
On Mon, Aug 24, 2009 at 3:54 PM, Jay Thomasjaydianetho...@gmail.com wrote:
Hello
I am looking for a way to serialize/deserialize with length prefix
under c++. The serialized bytes will sent to a TCP socket. I am
aware that C# has a method SerializeWithLengthPrefix() and
DeserializeWithLengthPrefix(). Are there any such analogous methods
for C++? Please point out any documentation that I may have missed
here.
Thanks for any assistance.
Jay
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Serialize with Length Prefix
On Mon, Aug 24, 2009 at 4:17 PM, Jay Thomasjaydianetho...@gmail.com wrote:
This code ideally has to operate with C# code on the other side of the
socket that uses the DeserializeWithLengthPrefix() method call. The
message originates in C++ environment on a linux box and terminates in
C# environment on a windows PC.
I'm not familiar with the implementation of
DeserializeWithLengthPrefix(). The same idea should work, but you'll
have to go into the protobuf-net source to see what it uses to write
the length with. If it's not a Varint32 then just adjust it to
whatever it needs to be.
Why doesn't C++ support the same methods as C#?
They're different projects. protobuf-net has a different maintainer
than the protobuf project.
The only document that I've seen that has list of
methods available is the message.h header file, installed into my
Linux usr directory.
This is the documentation for the generated C++ code, which really
just deals with the fields that are generated by the protoc compliler:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp-generated.html
This is the documentation for the C++ library as a whole:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp/index.html
This is the documentation for the C++ version of Message:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.message.html
This is the documentation for the coded_stream stuff I was talking about:
http://code.google.com/apis/protocolbuffers/docs/reference/cpp/google.protobuf.io.coded_stream.html#CodedInputStream.Limit.details
--Pete
On Aug 24, 4:05 pm, Peter Keen peter.k...@gmail.com wrote:
You sort of have to roll your own. In my project I'm doing something like
this:
coded_output-WriteVarint32(message-ByteSize());
message-SerializeToCodedStream(coded_output);
And then on the reading side:
uint32_t size;
if (! coded_input-ReadVarint32(size)) {
return NULL; // just an example. probably don't want to do this;
}
Message * m = new Message;
CodedInputStream::Limit limit = coded_input-PushLimit(size);
message-ParseFromCodedStream(coded_input);
coded_input-PopLimit(limit);
The biggest thing that helped me along was finding the Limit docs. I
couldn't figure out how to parse from a ZeroCopyInputStream without
consuming the entire stream, and then I ran across that and everything
became much more clear.
--Pete
On Mon, Aug 24, 2009 at 3:54 PM, Jay Thomasjaydianetho...@gmail.com wrote:
Hello
I am looking for a way to serialize/deserialize with length prefix
under c++. The serialized bytes will sent to a TCP socket. I am
aware that C# has a method SerializeWithLengthPrefix() and
DeserializeWithLengthPrefix(). Are there any such analogous methods
for C++? Please point out any documentation that I may have missed
here.
Thanks for any assistance.
Jay
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Serialize with Length Prefix
I looked at the protobuf-net source a little bit and it looks like if
you write the length with WriteLittleEndian32 instead of WriteVarint32
you should be good to go, as long as you use PrefixStyle.Fixed32
within the C# call to DeserialzeWithLengthPrefix().
--Pete
On Mon, Aug 24, 2009 at 4:17 PM, Jay Thomasjaydianetho...@gmail.com wrote:
This code ideally has to operate with C# code on the other side of the
socket that uses the DeserializeWithLengthPrefix() method call. The
message originates in C++ environment on a linux box and terminates in
C# environment on a windows PC. Why doesn't C++ support the same
methods as C#? The only document that I've seen that has list of
methods available is the message.h header file, installed into my
Linux usr directory.
On Aug 24, 4:05 pm, Peter Keen peter.k...@gmail.com wrote:
You sort of have to roll your own. In my project I'm doing something like
this:
coded_output-WriteVarint32(message-ByteSize());
message-SerializeToCodedStream(coded_output);
And then on the reading side:
uint32_t size;
if (! coded_input-ReadVarint32(size)) {
return NULL; // just an example. probably don't want to do this;
}
Message * m = new Message;
CodedInputStream::Limit limit = coded_input-PushLimit(size);
message-ParseFromCodedStream(coded_input);
coded_input-PopLimit(limit);
The biggest thing that helped me along was finding the Limit docs. I
couldn't figure out how to parse from a ZeroCopyInputStream without
consuming the entire stream, and then I ran across that and everything
became much more clear.
--Pete
On Mon, Aug 24, 2009 at 3:54 PM, Jay Thomasjaydianetho...@gmail.com wrote:
Hello
I am looking for a way to serialize/deserialize with length prefix
under c++. The serialized bytes will sent to a TCP socket. I am
aware that C# has a method SerializeWithLengthPrefix() and
DeserializeWithLengthPrefix(). Are there any such analogous methods
for C++? Please point out any documentation that I may have missed
here.
Thanks for any assistance.
Jay
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
Re: Serialize with Length Prefix
cool. thanks Pete. your help is greatly appreciated!
On Aug 24, 4:33 pm, Peter Keen peter.k...@gmail.com wrote:
I looked at the protobuf-net source a little bit and it looks like if
you write the length with WriteLittleEndian32 instead of WriteVarint32
you should be good to go, as long as you use PrefixStyle.Fixed32
within the C# call to DeserialzeWithLengthPrefix().
--Pete
On Mon, Aug 24, 2009 at 4:17 PM, Jay Thomasjaydianetho...@gmail.com wrote:
This code ideally has to operate with C# code on the other side of the
socket that uses the DeserializeWithLengthPrefix() method call. The
message originates in C++ environment on a linux box and terminates in
C# environment on a windows PC. Why doesn't C++ support the same
methods as C#? The only document that I've seen that has list of
methods available is the message.h header file, installed into my
Linux usr directory.
On Aug 24, 4:05 pm, Peter Keen peter.k...@gmail.com wrote:
You sort of have to roll your own. In my project I'm doing something like
this:
coded_output-WriteVarint32(message-ByteSize());
message-SerializeToCodedStream(coded_output);
And then on the reading side:
uint32_t size;
if (! coded_input-ReadVarint32(size)) {
return NULL; // just an example. probably don't want to do this;
}
Message * m = new Message;
CodedInputStream::Limit limit = coded_input-PushLimit(size);
message-ParseFromCodedStream(coded_input);
coded_input-PopLimit(limit);
The biggest thing that helped me along was finding the Limit docs. I
couldn't figure out how to parse from a ZeroCopyInputStream without
consuming the entire stream, and then I ran across that and everything
became much more clear.
--Pete
On Mon, Aug 24, 2009 at 3:54 PM, Jay Thomasjaydianetho...@gmail.com
wrote:
Hello
I am looking for a way to serialize/deserialize with length prefix
under c++. The serialized bytes will sent to a TCP socket. I am
aware that C# has a method SerializeWithLengthPrefix() and
DeserializeWithLengthPrefix(). Are there any such analogous methods
for C++? Please point out any documentation that I may have missed
here.
Thanks for any assistance.
Jay
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en
-~--~~~~--~~--~--~---
