Re: [protobuf] protocol buffer within a protocol buffer from C++ to Java
On Mon, Oct 25, 2010 at 18:45, Paul wrote:
> Hi,
> I am trying to serialize a protocol buffers message into a string that
> is contained in another protocol buffer. I am doing the serialization
> on the C++ side.
>
> // This is the outer protocol buffer
> message Measurement {
> required string id = 1;
> optional string meas_rec_str = 2;
> }
> Measurement m;
>
> // This is the protocol buffer message I want to serialize into a
> string and store inside meas_rec.
> message MeasRec {
> optional int id = 1;
> }
>
> To serialize the MeasRec message, I am doing:
> MeasRec meas_rec;
> meas_rec.set_id(100);
> char* str_buf;
> str_buf = new char[meas_rec.ByteSize()];
> string str = str_buf;
> meas_rec.SerializeToString(&str);
> m.set_meas_rec_str(str);
>
> However, when I do this, I encounter the following error on the C++
> side:
>
> libprotobuf ERROR google/protobuf/wire_format.cc:1059] Encountered
> string containing invalid UTF-8 data while serializing protocol
> buffer. Strings must contain only UTF-8; use the 'bytes' type for raw
> bytes.
>
> how do I serialize meas_rec into "bytes" instead?
'bytes' will be the same std::string type in C++. The only differnce
is that it is not checked for UTF8-ness.
>
> Also, once I have it on the Java side, how do I deserialize?
> would I use parseFrom(byte[] data) ?
yes.
>
> Thanks,
> Paul
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] protocol buffer within a protocol buffer from C++ to Java
Hi,
I am trying to serialize a protocol buffers message into a string that
is contained in another protocol buffer. I am doing the serialization
on the C++ side.
// This is the outer protocol buffer
message Measurement {
required string id = 1;
optional string meas_rec_str = 2;
}
Measurement m;
// This is the protocol buffer message I want to serialize into a
string and store inside meas_rec.
message MeasRec {
optional int id = 1;
}
To serialize the MeasRec message, I am doing:
MeasRec meas_rec;
meas_rec.set_id(100);
char* str_buf;
str_buf = new char[meas_rec.ByteSize()];
string str = str_buf;
meas_rec.SerializeToString(&str);
m.set_meas_rec_str(str);
However, when I do this, I encounter the following error on the C++
side:
libprotobuf ERROR google/protobuf/wire_format.cc:1059] Encountered
string containing invalid UTF-8 data while serializing protocol
buffer. Strings must contain only UTF-8; use the 'bytes' type for raw
bytes.
how do I serialize meas_rec into "bytes" instead?
Also, once I have it on the Java side, how do I deserialize?
would I use parseFrom(byte[] data) ?
Thanks,
Paul
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: outer classname for C++
ok thanks. On Oct 25, 12:49 pm, Daniel Wright wrote: > No -- in C++ the message classes are placed directly in a namespace named > after the package, so there's no "outer" class. > > On Mon, Oct 25, 2010 at 11:42 AM, Paul wrote: > > Hi, > > > In the example of the .proto definition in Java, there is a > > > option java_outer_classname = "AddressBookProtos"; > > > Is there an equivalent statement for C++? > > > Thanks, > > Paul > > > -- > > You received this message because you are subscribed to the Google Groups > > "Protocol Buffers" group. > > To post to this group, send email to proto...@googlegroups.com. > > To unsubscribe from this group, send email to > > protobuf+unsubscr...@googlegroups.com > > . > > For more options, visit this group at > >http://groups.google.com/group/protobuf?hl=en. -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Possible Memory Leaks
On Mon, Oct 25, 2010 at 17:46, rthompson.dtisoft@gmail.com wrote: > I ran Valgrind on our application and found quite a few "possibly > lost" bytes traces. Here is an example of one > > ==1473== at 0x40263A0: operator new(unsigned int) > (vg_replace_malloc.c:214) > ==1473== by 0x4370EB8: > google::protobuf::DescriptorPool::DescriptorPool(google::protobuf::DescriptorDatabase*, > google::protobuf::DescriptorPool::ErrorCollector*) (descriptor.cc:768) > ==1473== by 0x43C8B35: > google::protobuf::compiler::Importer::Importer(google::protobuf::compiler::SourceTree*, > google::protobuf::compiler::MultiFileErrorCollector*) (importer.cc: > 188) Make sure to call ShutdownProtobufLibrary() to de-allocate memory allocated in global constructors. There are no known leaks (I use it daily with tons of data and protobufs never had been an issue wrt. leaks). -h > > and I have confirmed that I have freed the Importer(). > > Is this something to worry about, or can I count on google::protobuf > not having any memory leaks? > > Thanks in advance! > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Possible Memory Leaks
I ran Valgrind on our application and found quite a few "possibly lost" bytes traces. Here is an example of one ==1473==at 0x40263A0: operator new(unsigned int) (vg_replace_malloc.c:214) ==1473==by 0x4370EB8: google::protobuf::DescriptorPool::DescriptorPool(google::protobuf::DescriptorDatabase*, google::protobuf::DescriptorPool::ErrorCollector*) (descriptor.cc:768) ==1473==by 0x43C8B35: google::protobuf::compiler::Importer::Importer(google::protobuf::compiler::SourceTree*, google::protobuf::compiler::MultiFileErrorCollector*) (importer.cc: 188) and I have confirmed that I have freed the Importer(). Is this something to worry about, or can I count on google::protobuf not having any memory leaks? Thanks in advance! -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: omitting tag numbers
On Mon, Oct 25, 2010 at 16:11, Henner Zeller
wrote:
> On Mon, Oct 25, 2010 at 16:10, maninder batth
> wrote:
>> I disagree. You could encode field name in the binary. Then at de-
>> serialization, you can read the field descriptor and reconstruct the
>> field. There is absolutely no need for tags. They are indeed
>> cumbersome.
>
> If you include the field name, then your throw out part of the
> advantages of protocol buffers out of the window: speed and compact
> binary encoding.
.. and you would never be able to rename fields either.
>
>>
>> On Oct 22, 6:02 pm, Henner Zeller
>> wrote:
>>> On Fri, Oct 22, 2010 at 15:01, Paul wrote:
>>> > Hi,
>>>
>>> > This may seem like a basic question, but I find having to label
>>> > the .proto file with unique tag numbers for each field a little
>>> > cumbersome, especially if there are a lot of fields.
>>>
>>> > message Person {
>>> > required string name = 1;
>>> > required int32 id = 2;
>>> > optional string email = 3;
>>> > }
>>>
>>> > Can I define a .proto file without the tag numbers, like so?
>>>
>>> > message Person {
>>> > required string name;
>>> > required int32 id;
>>> > optional string email;
>>> > }
>>>
>>> No.
>>>
>>> The reason for this explicit definition is that the protocol buffer is
>>> 'future compatible': fields written with a particular tag will always
>>> be written with that tag. Consider you want to re-structure the fields
>>> in your proto buffer to say (Id, name, email) ... then they would get
>>> a different 'automatic' tag assigned and you wouldn't be able to read
>>> files written with older binaries. If the tags are assigned, then
>>> re-arranging fields in the file does not matter.
>>>
>>> -h
>>>
>>>
>>>
>>>
>>>
>>> > Thanks,
>>> > Paul
>>>
>>> > --
>>> > You received this message because you are subscribed to the Google Groups
>>> > "Protocol Buffers" group.
>>> > To post to this group, send email to proto...@googlegroups.com.
>>> > To unsubscribe from this group, send email to
>>> > protobuf+unsubscr...@googlegroups.com.
>>> > For more options, visit this group
>>> > athttp://groups.google.com/group/protobuf?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to proto...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> protobuf+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] Re: omitting tag numbers
On Mon, Oct 25, 2010 at 16:10, maninder batth wrote:
> I disagree. You could encode field name in the binary. Then at de-
> serialization, you can read the field descriptor and reconstruct the
> field. There is absolutely no need for tags. They are indeed
> cumbersome.
If you include the field name, then your throw out part of the
advantages of protocol buffers out of the window: speed and compact
binary encoding.
>
> On Oct 22, 6:02 pm, Henner Zeller
> wrote:
>> On Fri, Oct 22, 2010 at 15:01, Paul wrote:
>> > Hi,
>>
>> > This may seem like a basic question, but I find having to label
>> > the .proto file with unique tag numbers for each field a little
>> > cumbersome, especially if there are a lot of fields.
>>
>> > message Person {
>> > required string name = 1;
>> > required int32 id = 2;
>> > optional string email = 3;
>> > }
>>
>> > Can I define a .proto file without the tag numbers, like so?
>>
>> > message Person {
>> > required string name;
>> > required int32 id;
>> > optional string email;
>> > }
>>
>> No.
>>
>> The reason for this explicit definition is that the protocol buffer is
>> 'future compatible': fields written with a particular tag will always
>> be written with that tag. Consider you want to re-structure the fields
>> in your proto buffer to say (Id, name, email) ... then they would get
>> a different 'automatic' tag assigned and you wouldn't be able to read
>> files written with older binaries. If the tags are assigned, then
>> re-arranging fields in the file does not matter.
>>
>> -h
>>
>>
>>
>>
>>
>> > Thanks,
>> > Paul
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Protocol Buffers" group.
>> > To post to this group, send email to proto...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > protobuf+unsubscr...@googlegroups.com.
>> > For more options, visit this group
>> > athttp://groups.google.com/group/protobuf?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: omitting tag numbers
I disagree. You could encode field name in the binary. Then at de-
serialization, you can read the field descriptor and reconstruct the
field. There is absolutely no need for tags. They are indeed
cumbersome.
On Oct 22, 6:02 pm, Henner Zeller
wrote:
> On Fri, Oct 22, 2010 at 15:01, Paul wrote:
> > Hi,
>
> > This may seem like a basic question, but I find having to label
> > the .proto file with unique tag numbers for each field a little
> > cumbersome, especially if there are a lot of fields.
>
> > message Person {
> > required string name = 1;
> > required int32 id = 2;
> > optional string email = 3;
> > }
>
> > Can I define a .proto file without the tag numbers, like so?
>
> > message Person {
> > required string name;
> > required int32 id;
> > optional string email;
> > }
>
> No.
>
> The reason for this explicit definition is that the protocol buffer is
> 'future compatible': fields written with a particular tag will always
> be written with that tag. Consider you want to re-structure the fields
> in your proto buffer to say (Id, name, email) ... then they would get
> a different 'automatic' tag assigned and you wouldn't be able to read
> files written with older binaries. If the tags are assigned, then
> re-arranging fields in the file does not matter.
>
> -h
>
>
>
>
>
> > Thanks,
> > Paul
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com.
> > For more options, visit this group
> > athttp://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 228 in protobuf: Python reading text_format misses empty extension field
Comment #1 on issue 228 by palotai@gmail.com: Python reading text_format misses empty extension field http://code.google.com/p/protobuf/issues/detail?id=228 Or maybe text_format.py:211 could be un-indented? -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Issue 228 in protobuf: Python reading text_format misses empty extension field
Status: New Owner: ken...@google.com Labels: Type-Defect Priority-Medium New issue 228 by palotai@gmail.com: Python reading text_format misses empty extension field http://code.google.com/p/protobuf/issues/detail?id=228 What steps will reproduce the problem? 1. Extend a message A with an empty extension message E (see http://groups.google.com/group/protobuf/browse_thread/thread/26aef51ce964ac09/d719210277a16df0) 2. Write A as protobuf in text format, A having the extension set. 3. Read the buffer written in (2) using google.protobuf.text_format in python What is the expected output? What do you see instead? The message A should contain the empty extension. But the extension is missing. Worked around by adding "sub_message.SetInParent" in text_format.py line 208. What version of the product are you using? On what operating system? protobuf-2.3.0 linux/python. Please provide any additional information below. python, extension, empty -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] ParseFromArray -- in Java
On Oct 25, 2010, at 16:52 , ury wrote: i.e. Does the Java implementation has the Clear() method ? No, the Java implementation has immutable objects, so this is generally not possible. A new object must be created for each item. Immutable objects have benefits like being thread safe (see http://www.javapractices.com/topic/TopicAction.do?Id=29) That said, I think you *might* be able to hack something like this using the Builder object. I would be interested to know if you try this, and if it has any performance benefits. Evan -- Evan Jones http://evanjones.ca/ -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] ParseFromArray -- in Java
Might as well use a new object. Creating a new object is if anything likely less overhead. --Chris On Oct 25, 2010, at 1:52 PM, ury wrote: > Hi! > > Is it possible to reuse the same Protobuf object when unserializing a > large number of objects in Java using the equivalent of the C++ > ParseFromArray ? i.e. Does the Java implementation has the Clear() > method ? > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] ParseFromArray -- in Java
Hi! Is it possible to reuse the same Protobuf object when unserializing a large number of objects in Java using the equivalent of the C++ ParseFromArray ? i.e. Does the Java implementation has the Clear() method ? Thanks! -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Extensions Management
We have a geographically distributed team and instant communications
are not possible. Given this constraint in mind, i am wondering how to
manage extensions. To clarify my question, consider a message like
message Request {
extensions 1 to 100;
}
Different developers would be working on extending Request. For
example developer A uses the above request in a Client.proto file,
such as
extend Request {
optional int64 clientId = 1;
}
Developer B, uses the above request in a Location.proto file such as
extend Request {
optional int64 locationId = 1;
}
Question is how can developer A and B, extend the message Request,
without worrying about extension slots being already occupied by some
other message?
As shown above, both developers reserved the slot 1, with different Id
types.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] outer classname for C++
No -- in C++ the message classes are placed directly in a namespace named after the package, so there's no "outer" class. On Mon, Oct 25, 2010 at 11:42 AM, Paul wrote: > Hi, > > In the example of the .proto definition in Java, there is a > > option java_outer_classname = "AddressBookProtos"; > > Is there an equivalent statement for C++? > > Thanks, > Paul > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] outer classname for C++
Hi, In the example of the .proto definition in Java, there is a option java_outer_classname = "AddressBookProtos"; Is there an equivalent statement for C++? Thanks, Paul -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Message missing required fields exception when parsing a message that has required fields defaulted
I've gotten this message for reasons other than the field not being
set. make sure that you have the number of bytes on the delimiters
set correctly on the C++ side. also, maybe stringProperty should be
string Property (two words).
On Oct 22, 9:12 am, locky wrote:
> I have a question regarding required string properties and default
> values.
>
> Consider the following message definition.
>
> message StringTestProto {
> required stringProperty = 1;
>
> }
>
> I have a C++/Java setup where C++ app is producing a message and a
> Java app is consuming it.
>
> If the stringProperty value is set to an empty string I get the
> following exception when trying to parse the
> message once I receive it in the the Java app.
>
> com.google.protobuf.InvalidProtocolBufferException: Message missing
> required fields:
>
> The message is being read as follows:
>
> StringTestProto proto = StringTestProto.parseFrom(data);
>
> From what I can see the issue is as follows:
> As the stringProperty value is the same as the default (an empty
> string) it is not sent over the wire. When the generated Java code
> tries to build the proto from a byte[ ] (in the buildFrom method), a
> check is done to see if the proto is initialised, this check sees if
> the required fields have been set. As the stringProperty was never
> sent, the proto does not think it has been set (as it never was) and
> therefore the exception is thrown.
>
> So, how are you supposed to decode a byte[] that contains required
> properties, that may have default values?
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: omitting tag numbers
ok that makes sense. thanks!
On Oct 22, 4:02 pm, Henner Zeller
wrote:
> On Fri, Oct 22, 2010 at 15:01, Paul wrote:
> > Hi,
>
> > This may seem like a basic question, but I find having to label
> > the .proto file with unique tag numbers for each field a little
> > cumbersome, especially if there are a lot of fields.
>
> > message Person {
> > required string name = 1;
> > required int32 id = 2;
> > optional string email = 3;
> > }
>
> > Can I define a .proto file without the tag numbers, like so?
>
> > message Person {
> > required string name;
> > required int32 id;
> > optional string email;
> > }
>
> No.
>
> The reason for this explicit definition is that the protocol buffer is
> 'future compatible': fields written with a particular tag will always
> be written with that tag. Consider you want to re-structure the fields
> in your proto buffer to say (Id, name, email) ... then they would get
> a different 'automatic' tag assigned and you wouldn't be able to read
> files written with older binaries. If the tags are assigned, then
> re-arranging fields in the file does not matter.
>
> -h
>
>
>
> > Thanks,
> > Paul
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com.
> > For more options, visit this group
> > athttp://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
