Re: [protobuf] Performance for Messages containing big (1...100MByte) binary blobs

[Henner Zeller]

On 11 January 2018 at 04:20, Mario Emmenlauer  wrote:
>
> Hi,
>
> On 11.01.2018 13:13, rschoe@gmail.com wrote:
>> Hi,
>> How do Protocol Buffers (C++) perform when messages contain big binary 
>> blobs? With big I mean 1 ... 100MByte.
>
> My impression was that its not very suitable for that. Its a while
> ago that I checked but there was no plain binary encoding back then.
> To send binary files, I encoded them in some ASCII encoding (Base64
> I think) and sent them as strings. The encoding takes some CPU, and
> the data size goes up.

Don't do this, this is slow. You just want to store the plain binary
message in a bytes datatype. This is fast and cheap.

>
> I'd be happy to learn otherwise?! And possibly things have changed in
> the meantime...
>
> Cheers,
>
> Mario Emmenlauer
>
>
> --
> BioDataAnalysis GmbH, Mario Emmenlauer  Tel. Buero: +49-89-74677203
> Balanstr. 43   mailto: memmenlauer * biodataanalysis.de
> D-81669 München  http://www.biodataanalysis.de/
>
> --
> You received this message because you are subscribed to the Google Groups 
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] "out of memory" issue after integrating Protobuf in my project

[Henner Zeller]

On 1 January 2018 at 01:48, Krishna Patsariya  wrote:
> Hi All,
>
> I am facing one strange issue after integrating Protobuf in my project.
>
> "cc1: out of memory allocating 3355443200 bytes after a total of 585728
> bytes".

It might be useful to include what files you compiled that made the
compiler hungry for memory. What compiler are you using. On what
system ?

-h

>
> without protobuf I can compile my code without any issue. If any one
> encountered same situation kindly help me in this.
>
>
> Thanks,
> Krishna
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Why doesn't C++ libpropobuf accept defult values?

[Henner Zeller]

Required really means that you need to store values in there.

Default only makes sense for the optional fields (because only in the
optional fields, it is possible to _not_ set a value, so in that case,
you get the default). That answers your question in which cases you
use defaults.

So sounds like in your application you really mean to use optional
fields (which is the better design choice anyway: you should leave the
data validation to the application layer, not the data holder layer).

Cheers,
 H.


On 1 July 2017 at 18:27, hce h  wrote:
> Hi,
>
> In my process of response message, if there is an error, the process would
> be aborted and send back an error response message without filling the
> required variables. To avoid the libpropobuf throws system exception for
> missing required variables, I put the default values like following example
> but that did not work, the libprotobbuf still threw an exception, could
> anyone explain why? I am running protobuf in Debian Jessie. Of cause, I
> could  change the required to optional, but I want to know what the default
> values for?
>
> message SearchResponse {
> required string query = 1 [default = ""];
> required int32 page_number = 2 [default = 0];
>
>   required int32 result_per_page = 3 [default = 0];
>  :
> }
>
> Thank you.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] How to serialize/deserialize bytes data?

[Henner Zeller]

On 22 December 2016 at 14:50,   wrote:
> Thanks for the reply.
>
> My question here is: I use protobuffer to store a bytes data, how can I
> retrieve it back?

Symmetric to the SerializeToArray(), use ParseFromArray() or
ParseFromString() to parse a serialized version of a protocol buffer
back.

> In my previous example, just offset 2 from the protobuffer
> data? However, the size field may vary, right?

That code does not make any sense in the real world: you are starting
with a protocol buffer, then serialize the resulting string in a new
protocol buffer. I suspect that was only for testing something ?

>
> On Thursday, December 22, 2016 at 2:13:14 PM UTC-8, Adam Cozzette wrote:
>>
>> It looks like you are building up a protocol buffer containing a
>> serialized protocol buffer, which itself contains a serialized protocol
>> buffer, which in turn contains a serialized protocol buffer, etc. Each level
>> of nesting requires an additional two bytes: one byte for the tag number and
>> a second byte for the length of the byte string that follows.
>>
>> For example, the first time you serialize it you need six bytes:
>> [ tag (one byte) ] [ size = 4 (one byte) ] [ ... 4 bytes of data ]
>>
>> Then the second serialization looks like this (8 bytes total):
>>
>> [ tag (one byte) ] [ size = 6 (one byte) ] [ ... the six bytes from above
>> ]
>>
>> See here for more information about the wire format.
>>
>> On Thu, Dec 22, 2016 at 1:52 PM,  wrote:
>>>
>>> I defined a simple bytes message below
>>> message MfStream {
>>>   bytes message=1;
>>> }
>>>
>>> and run a test below, in which I use for a loop to keep
>>> serialize/deserialize the bytes message.
>>>
>>> char buf[1024];
>>> int tmp = 1;
>>> MfStream testMsg;
>>> testMsg.set_message(, sizeof(tmp));
>>> for (int i=0; i<5; i++) {
>>> int size = testMsg.ByteSize();
>>> testMsg.SerializeToArray(buf, sizeof(buf));
>>> cout << "i=" << i << ", size=" << size << endl;
>>> testMsg.set_message(buf, size);
>>> }
>>>
>>>
>>> The results below shows that the message size keeps increasing by 2 every
>>> loop. Why? How should I serialize/deserialize bytes data?
>>>
>>> i=0, size=6
>>> i=1, size=8
>>> i=2, size=10
>>> i=3, size=12
>>> i=4, size=14
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Protocol Buffers" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to protobuf+u...@googlegroups.com.
>>> To post to this group, send email to prot...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/protobuf.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at https://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] (objective-c) How to trigger code re-generation

[Henner Zeller]

Isn't this what Makefiles are for ?

On 9 November 2015 at 09:14, Rob Cecil  wrote:
> I'm using Cocoapods and my initial project build succeeds. Steps
>
> 1. Created .podspec referencing my .protos and output directory for
> generated objective-c code.
> 2. Created podfile
> 3. Ran pod install.
> 4. Success, open xcworkspace.
>
> Now when I try to either:
>
> 1. Update a .proto OR
> 2. touch podspec
>
> And run "pod update"
>
> I never see the codegen happen.
>
> What are the steps/best practice to continously refine .proto definitions
> and regenerate code?
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to protobuf+unsubscr...@googlegroups.com.
> To post to this group, send email to protobuf@googlegroups.com.
> Visit this group at http://groups.google.com/group/protobuf.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] file transfer using protobuf

[Henner Zeller]

On 21 May 2015 at 02:45, navin kurle kurlenavi...@gmail.com wrote:

 Hi,

 How I can remove or delete file using google protobuf?


Well, you need to write your protocol and code that does that. Protobuf
 allows you to write the protocol efficiently and platform independently -
but you have to write the code for whatever you want to do. Be it making
coffee, deleting files or blinking LEDs.

-h


 Regards,
 Navin


 On Wednesday, December 10, 2014 at 5:03:38 AM UTC+5:30, Kaustubh Deshmukh
 wrote:

 Hi Henner,

 That worked for me. Thank you for the reply.

 On Monday, October 7, 2013 10:59:21 AM UTC-7, Henner Zeller wrote:

 On 5 October 2013 19:43, Kaustubh Deshmukh kaud...@gmail.com wrote:

 Can I transfer file using google protobuf?


 The 'bytes' type can store arbitrary content.

 If you want to transfer huge files, they you might want to design a
 protocol araound it by sending chunks that each
 are less than 2GBytes, possibly much smaller.

 message FileChunk {
optional int64 offset = 1;
optional bytes data = 2;
 }

 Note, protocol buffers are just providing the encoding of your data, the
 network protocol you still have to do yourself.

 -h



 If yes, how?

 Thank you for short guidance and help.

 Regards,

 Kaustubh

 --
 You received this message because you are subscribed to the Google
 Groups Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to protobuf+u...@googlegroups.com.
 To post to this group, send email to prot...@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/groups/opt_out.


  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] protoc generator : The code of constructor ... is exceeding the 65535 bytes limit

[Henner Zeller]

On 5 January 2015 at 05:09, ravi kiran tunuguntla ravionl...@gmail.com wrote:
 protoc generated code could not be compiled. (as the constructor code has
 exceeded 65535 bytes limit). What can be done to work around this?

Use less fields :)

Your protocol buffer looks pretty degenerated using all these fields.
Java classes have a limit in size,
so you'll run into the problem at one point or another.
You might consider putting these attributes in a repeated field.
Or use extensions to better reflect generic data.

It will make your code using the protocol buffer much more readable as well.

-h


 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Partially serialize/deserialize protocol buffers

[Henner Zeller]

On 29 December 2014 at 10:53, Li Yan magma917...@gmail.com wrote:
 Hey guys,

 From https://developers.google.com/protocol-buffers/docs/encoding, I
 basically understand how to encode protocol buffers: the keys and values are
 concatenated into a byte stream. My question is whether there exists API to
 partially serialize/deserialize protocol buffers? I mean add serialized
 field at binary head, and only deserialize the first n fields in the binary
 protocol buffers.

You could use the CodedInputStream to read one field at a time.

-h



 Thanks,
 Li

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] zeromq with protobuf segmentation fault while parsing in c++

[Henner Zeller]

On 30 June 2014 00:41, Nayab Rasul rasulnra...@gmail.com wrote:

 I am using zeromq with protobuf to send/recieve messages but code was
 crashing on receiver end while *ParseFromString *with Segmentation fault
 error.

 Scan is my message and i has float data types as repeated filed.


 *sender.cpp   // part of code sending*

 *Scan proto_ls_msg;*

 *proto_ls_msg.set_angle_min(0.0);*

 *proto_ls_msg.set_angle_max(180.5);*

 *std::string ls_msg_str;*

 *proto_ls_msg.SerializeToString(ls_msg_str);*

 *zmq::message_t request (ls_msg_str.size());*

 *memcpy (request.data(), ls_msg_str.c_str(),ls_msg_str.size());*

 *socket.send (request);*


 *collector.cpp **// part of code recieving*

 *zmq::message_t recieved;*

 *socket.recv (recieved);*

 *std::string ls_msg_str((char*)recieved.data(),recieved.size()); *

 *Scan *pb_laser_msg_rcv;*

 *pb_laser_msg_rcv-ParseFromString(ls_msg_str); // --  Segmentation fault
 here*


no wonder, pb_laser_msg_rcv is an uninitizliaed pointer.

Do

Scan *b_laser_msg_rcv = new Scan();
pb_laser_msg_rcv-...

(or allocate it on the stack or elsewhere).


 Is there any example code for sending/recieving with repeated float fields.

 thank you.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Deprecated field annotations in gcc

[Henner Zeller]

On 30 May 2014 14:38, 'Feng Xiao' via Protocol Buffers
protobuf@googlegroups.com wrote:
 On Thu, May 29, 2014 at 5:23 PM, Priyendra Deshwal desh...@thoughtspot.com
 wrote:

 Hey guys,

 It seems like the deprecated field annotations in gcc are no longer
 emitted by the protobuf compiler and the following commit removed this
 functionality.


 https://code.google.com/p/protobuf/source/diff?spec=svn425r=425format=sidepath=/trunk/src/google/protobuf/generated_message_util.h

 Was this an intentional change?

 Yes. The deprecation warning is deemed not useful and thus removed.

Deprecation _is_ useful and it should be possible to generate warnings
for that (it is not used within Google, thus it was not noticed that
this somewhat cripples the experience).

Note, PROTOBUF_DEPRECATED is still generated in the pb.h headers. You
can define that to be  __attribute__ ((deprecated)); and should get
deprecation warnings.

However, I think this change might actually be gone too far, I have to
check - because with that change, code within the generated *.pb.cc
file might now return warnings, which is not helpful. I'll check.

-h




 Regards,
 -- Priyendra

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] support for bit fields?

[Henner Zeller]

On 19 May 2014 17:43, cinyoung hur hur...@gmail.com wrote:

 Hello,

 I would like to apply bit fields.
 For example,

 struct{
   unsigned int version : 1;
   unsigned int type: 1;} status;


 OR is there other tips to handle these examples?


There are no bit-fields provided in the proto definition language. if you
need these kind of things because you are conscious about the size on the
wire or stored size on disk, you need to use one of the fixed32 or fixed64
types and do it yourself.

If you are worried about the size in memory, I think the code generator
already attempts to use this to compact a set of boolean values in this way
in languages that support it (not 100% sure, would need to look at the code
again).

-h



  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] C++ ParseFromCodedStream takes too long

[Henner Zeller]

On 20 March 2014 06:10,  christian.kilpatrick.1...@gmail.com wrote:
 Oh yes you are right. There was a mistake in my calculation.

 Well I'll try without the submessage. The count of docid and of score is
 always the same. So there is no Problem for me.

 But it's sad that java takes around 2 seconds to send the Message to the
 server, recieve the Response and display the response.
 And C++ takes 20-25 seconds only for this function call
 srchResp.ParseFromCodedStream(codedIn); :(
 I don't think that changing the message will get me around 20 seconds :( Too
 bad.

If you man many things that need to be read into the repeated field,
maybe your memory allocator is running into trouble when resizing the
message all the time ? Since the repeated field does not know
beforehand how many objects are coming, it is forced to re-allocate
while parsing.

Just re-use the message object, as it can make use of the memory
already allocated. Create one C++ object, then use that to parse from
stream. When you're done with it,
re-use the same object for subsequent parse.
I think it is possible to somehow tell the repeated field beforehand
to reserve some capacity (similar to a vectorT), but don't know that
right now.

-h


 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

Re: [protobuf] Can I flush all protocol buffers using a static class method?

[Henner Zeller]

On 24 January 2014 07:05, Alex Wakefield forg...@gmail.com wrote:

 I would like to flush all data stored in the protocol buffers to disk, but
 I don't have a handle to the individual buffer objects.


 Is there a way to flush all of the buffers?


Can you be a bit more concrete ? What do you want to achieve ? What is 'all
of the buffers' ?
It sounds like you have some singleton proto buffers and want to magically
store them to disk somewhere ? I guess you'll have to implement that on
your own (_and_ rethink your design).

-h



 Thanks
 Alex

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] it would be cool if there are some functions that parse pb from (or serailize pb to) a vector of buffers

[Henner Zeller]

On 11 December 2013 18:45, wilesd...@gmail.com wrote:

 when recv data from tcp socket, we don't know how long the packet will
 be. Generally, I would pre-alloc(for the sake of reusing) a vector of
 buffers which each buffer has the same size. The problem is that it's hard
 to parse pb from the vector of buffers.


Can't you just write a very simple implementation of ZeroCopyInputStream
that returns a new buffer in each Next() call ?



  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] file transfer using protobuf

[Henner Zeller]

On 5 October 2013 19:43, Kaustubh Deshmukh kaudes...@gmail.com wrote:

 Can I transfer file using google protobuf?


The 'bytes' type can store arbitrary content.

If you want to transfer huge files, they you might want to design a
protocol araound it by sending chunks that each
are less than 2GBytes, possibly much smaller.

message FileChunk {
   optional int64 offset = 1;
   optional bytes data = 2;
}

Note, protocol buffers are just providing the encoding of your data, the
network protocol you still have to do yourself.

-h



 If yes, how?

 Thank you for short guidance and help.

 Regards,

 Kaustubh

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] example of enum to string.

[Henner Zeller]

On 7 October 2013 08:58, amirk kleen.a...@gmail.com wrote:

 Hi,

 I am new to prtobuf, and I am having issues to enum to string conversion
 in C++.
 I see that there is genereated code for it (see below) but when I try to
 call these functions I am getting a segmentation fault.


What is your stacktrace ?


 Can someone please post a C++ based example.


Don't let the subscribers of this list make the work for you when it is not
even clear yet what fails.
Create a minimal example that fails for you (protocol buffer, C++ program),
post it to the list and people will
gladly help figure out what is broken.

-h


 Thanks,
 Amir
 this is the code generated for an enum called XXX
 inline const ::std::xxx XXX_XXXIdentifier_Name(XXX_XXXIdentifier value) {
   return ::google::protobuf::internal::NameOfEnum(
 XXX_XXXIdentifier_descriptor(), value);
 }
 inline bool XXX_XXXIdentifier_Parse(
 const ::std::xxx name, XXX_XXXIdentifier* value) {
   return ::google::protobuf::internal::ParseNamedEnumXXX_XXXIdentifier(
 XXX_XXXIdentifier_descriptor(), name, value);
 }


  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf.
 For more options, visit https://groups.google.com/groups/opt_out.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] usage of 'null' as a variable name in protobuf-2.5.0

[Henner Zeller]

On 20 May 2013 09:55, Feng Xiao xiaof...@google.com wrote:




 On Thu, May 16, 2013 at 10:58 PM, daniel.ng1...@gmail.com wrote:

 Version 2.5.0 has the following line in a couple of places in
 repeated_field.h:

 MessageLite* null = NULL;


 My gnu-based compiler doesn't like this as it thinks 'null' is a constant.

 If I rename 'null' as 'narl' then it's OK.

 Is 'null' a valid variable name?

 I believe it is. You are probably using some library which defines null as
 a macro.


Still, it might not be a good idea to use such a name. I could imagine
translating this proto buffer into other languages might create trouble.

-h






  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.
 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Re: Why to reinvent the wheel ?

[Henner Zeller]

If you look at the complicated way CORBA works and the simplicity and of
Protobufs, then you know. You assume infinite resources in companies (and
hence assume assume politicial statement behind protobufs), but in reality
it is about getting things done. CORBA is a non-starter.


On 21 March 2013 09:04, Vic Devin vfn...@gmail.com wrote:

 This thread seems to be a bit old but anyway this topic became suddenly
 important for me since I start to hear the Protobuf new magic word.

 Now I was a bit surprised to discover that it is actually the same idea as
 CORBA!

 So the question asked at the beginning of this thread „Why to reinvent
 the wheel?“ is appropriate.

 Why Google need to reinvent the wheel if technologies were available?

 Now I have my own answer: companies dont trust anyone but themselves, they
 aspire to be the world domination force, they dont care about existing
 standards and they replace them with their own!

 Next question is: do I want to live in such a world where there is no
 trust and collaboration between people for the common good? My answer is
 NO, which seems to go against the current „politically correct“ way of life
 of having to be a liberal capitalist „competitive“, against the competitors
 (as if they were your worst enemies)!

 I have the feeling that the computer industry could be much more efficient
 if there were better standards and improving them instead of reinventing
 the wheel by private companies!

 So when company X will finally be dead, it will bring down also the
 companies which relied on their wheels, unless of course they will be quick
 to replace them!

 I understand that long-lasting standards drag with them old technologies
 and way of thinking, but why couldn’t they make a new CORBA standard to be
 the same as Protobuf if this is so good, but still call it CORBA so that
 everyone knows which wheel you are talking about, and that it doesnt belong
 to any particular company but goes to the benefit of us all!?!?

 Sure also Microsoft will tell you that you dont need to reinvent the
 wheel, but provided you use their wheel! ;)

 ** **http://dotnet.dzone.com/articles/don%E2%80%99t-reinvent-wheel-part-1

 **So as I see it this is not so much a technological motivation, rather a
 political / economical one. **

 **
 **
 On Thursday, November 11, 2010 3:19:18 PM UTC+2, Oliver wrote:

 On Wed, Nov 10, 2010 at 3:21 AM, Kalki70 kalk...@gmail.com wrote:

  Maybe the ASN.1 compiler that you used used too many memory
  allocations or was not too fast. There are some very good, like from
  OSS Novalka.

 I've used both OSS Nokalva's ASN.1 to Java compiler and protobuf in
 anger. protobuf is at least as fast, provides a better API (especially
 if you want to do any reflection), and is less buggy than OSS's
 product. Being able to build protobuf from source makes our build
 process a lot simpler, too.

 We actually use both in our system - OSS when we must talk ASN.1 in
 external protocols, and protobuf for our internal protocols where we
 are not implementing to an external specification.

 I think that protobuf's simplicity is a large part of why its
 implementation is better than the various ASN.1 products out there.
 ASN.1 seems to be the Ada of protocol description languages, really..

 Oliver

  --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To unsubscribe from this group and stop receiving emails from it, send an
 email to protobuf+unsubscr...@googlegroups.com.

 To post to this group, send email to protobuf@googlegroups.com.
 Visit this group at http://groups.google.com/group/protobuf?hl=en.
 For more options, visit https://groups.google.com/groups/opt_out.




-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to protobuf+unsubscr...@googlegroups.com.
To post to this group, send email to protobuf@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Re: [protobuf] Re: Protocol buffer performance degrading with time

[Henner Zeller]

On 11 January 2013 03:12,  debop...@2pirad.com wrote:
 Hi Xiao Feng,


 The line which is getting slower with time is highlighted in yellow, below.



 On Friday, January 11, 2013 4:31:46 PM UTC+5:30, Feng Xiao wrote:


 [adding back the group]

 On Fri, Jan 11, 2013 at 5:53 PM, Debopam Ghoshal deb...@2pirad.com
 wrote:

 Hi Xiao Feng, please find below the relevant sections of my applications
 code:

 This is the point where Apache MINA receives the request from the client.
 The MultiRequest is a generated class from the protobuf compiler. The .proto
 file and the java source code is attached for your reference.

 public class ProtocolBufferDecoder extends CumulativeProtocolDecoder {
 .

 @Override
 protected boolean doDecode(IoSession session, IoBuffer in,
 ProtocolDecoderOutput out)
 throws Exception {
 if (!in.prefixedDataAvailable(4)) {
 return false;
 }
 double randomId = Math.random();
 session.setAttribute(requestId, randomId);

 // read the bytes
 in.rewind();
 byte[] bytes = new byte[in.getInt()];
 in.get(bytes);

 // parse the bytes
 LOGGER.trace(creating request object from bytes for request id:
 {}, randomId);
 MultiRequest request = MultiRequest.parseFrom(bytes);

 Is it this line of code that you found gets slower over time? I see no
 reason why it should behave like that unless bytes is getting larger.


 LOGGER.debug(MultiRequest[{}] created successfully for request
 id: {},
 randomId);

 // send the request
 out.write(request);
 LOGGER.trace(sending request to io handler for request id: {},
 randomId);
 return true;
 }
 }


 After the Multirequest object is created, it is passed on to the MINA IO
 Handler in the following section:

 public class SocketIoHandler extends IoHandlerAdapter {
 .

 public void messageReceived(IoSession session, Object message)
 throws Exception {
 // decode requests
 LOGGER.trace(starting to decode request with id:
 {},session.getAttribute(requestId));
 ListRequest requestList = requestCodec.decodeAll(message);
 LOGGER.trace(finished decoding request with id: {},

So if decodeAll() is getting slower, what is the implementation of
decodeAll() ? You left it out in the post.
Something returning a List _is_ suspicious in particular if the list
returned is big or if the implementation has to do some expensive
discovery what the message it gets (here the parameter is just an
Object ?) actually is.

 session.getAttribute(requestId));
 .
 }
 .
 }

 The SimpleProtocolBufferDecoder (attached) decodes the MultiRequest and
 returns a list of request objects.


 Thanks  Best Regards,
 Debopam

 Cell: +919830410041
 --
   -|-
 --o-0-(.)-0-o--
 visit us at www.2pirad.com


 On Fri, Jan 11, 2013 at 3:06 PM, Feng Xiao xiao...@google.com wrote:

 Can you show us how the decoding code looks like? I.e., the range of
 code that you found is slowing over time.


 On Friday, January 11, 2013 4:45:40 PM UTC+8, debo...@2pirad.com wrote:

 Thanks for your suggestions. Please find my response below:


 On Friday, January 11, 2013 1:16:19 PM UTC+5:30, Дмитрий Дьяченко
 wrote:

 if i undestand correctly, You have a looong-run application.

 Yes, you are right, my application is a really long running one. It
 is expected to run 24 x 7 x 365.


 - at start You have 2 ms cycle request-response

 The actual request response cycle is a bit more than 2 ms, because
 after the request is recieved by the application (#9 in my list), it does 
 a
 few things, which may take longer. But, the time taken to decode the
 protobuf request is 2 ms at start.


 - after a week You have 200 ms cycle

 Right, the time taken to decode gradually increases and becomes 200 ms.
 If left running for longer, it even reached 1 sec in 15 days time.


 - Your app is correct and only place where problem may lay is
 protobuf?

 From the logs which are there in my application, I have seen that only
 the time taken to decode is gradually increasing. I have also done some
 memory and thread profiling of my application, but did not find any
 irregularities (like increase in memory/threads)



 can You do the following?
 -- run 'latencytop' at start and after a week?
 -- run 'oprofile' at start and after a week?

 My application is running on IBM AIX server (using IBM JVM). Are the
 above tools available for AIX?


 -- may be its reasonable to writeout times from crititical points of
 Ypur app? smth like: before req? after req? before response - after 
 response

 I have logs specific for this purpose which are there at the start and
 end of these points. Like, before decode and after decode, etc. From these
 logs I found out the point which was gradually slowing down.


 from my [very limited] 

Re: [protobuf] Re: Protocol buffer performance degrading with time

[Henner Zeller]

On 10 January 2013 23:30,  debop...@2pirad.com wrote:
 Because, to process the request object (in #7), the methods which are
 present in the google protobuf generated file are called. The protobuf
 request object contains multiple requests, and the time taken to extract
 these (requests) is gradually increasing with time.

If you think about it, it is pretty unlikely that the generated code
will get 'slower' over time.
Typically, I have seen JVMs slow down because they get busy with
garbage collecting. So this kind of behavior might be related to
memory allocation in the system and subsequently slower garbage
collection (from your description it sounds like  you're using Java).
Garbage collection that kicks in in the middle of processing will make
things slower. If you see such huge degradation of performance, then
doing a memory profiling will bring you do some good insight what is
happening. The JVM has some built-in diagnosis as well that you can
use initially (such as percentage of time it spends garbage
collecting).

Of course, it could be as something else, like increasing size of the
data you parse with the protocol buffers ? Or maybe somewhere in the
IO sub-system a buffer is not cleared and you parse a  huge chunk of
byte-array ? You could probably see that by adding a debugging log
that shows you the byte-buffer size before you call parseFrom().

Hope this helps,
 -h

 I think this is protobuf problem because these classes where generated by
 the protobuf compiler (from the .proto file).


 On Friday, January 11, 2013 12:41:35 PM UTC+5:30, Feng Xiao wrote:

 What makes you think it's protobuf's problem?

 On Friday, January 11, 2013 2:50:54 PM UTC+8, debo...@2pirad.com wrote:

 My application uses Google Protocol Buffer with Apache MINA. There are
 clients who connect to the MINA Connector, send request to the server and
 then wait for the response from the server. The flow of control is as
 follows:
 1. Client sends request
 2. Request received at MINA filter (Class extends
 CumulativeProtocolDecoder)
 3. doDecode() method is called
 4. A request object (generated from *.proto file) is created
 using the RequestObject.parseFrom(bytes)
 5. The request is passed on to the IOHandler (Class
 extends IoHandlerAdapter)
 6. messageReceived() method is called
 7. In this method, the request object (from #6)
 is processed to create the list of requests which has been sent by the
 client.
 8. At this point, we have noticed that the time
 taken to process the request object (#7) is gradually degrading with time.
 From a initial period of around 2 ms, the time period is going up to 200 ms
 in just 8 days of continuous usage. And this value gradually increases with
 time.
 9. The request list is processed in the
 application
   10. The response object is created
   11. The response is sent to the client.

 Any suggestion would be highly appreciated.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/protobuf/-/IRKxKX0EKRUJ.

 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] command not found

[Henner Zeller]

On 10 November 2012 20:29, Mohammed mr.behbo...@gmail.com wrote:

 I followed the instructions in 
 https://developers.google.com/protocol-buffers/docs/cpptutorial;
 and at the final step when I typed sudo ./main massage.bin I was having
 this error
 command not found


Why do you want to run it as root, using 'sudo'  ?

Some unices say something along the lines of 'command not found' when they
don't find shared libraries. What happens if you say
  ldd ./main
Does it list all shared libraries, or does it say things such as 'not
found' ?

-h


 someone help me!

 massage.bin is a binary file that I made to first run. is this correct or
 not?

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To view this discussion on the web visit
 https://groups.google.com/d/msg/protobuf/-/p7Ddw0LWB9cJ.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Tool to convert binary back to string representation

[Henner Zeller]

On 5 October 2012 12:28, JonathonS thejunk...@gmail.com wrote:
 Hi

 Is there a tool that will convert the serialized binary to its string
 representation so that I can see all the fields and their values?

There is the TextFormat class that allows you to print the contents of
a protocol buffer in a readable form
https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.text_format

-h


 Thanks,
 J

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Odd problem with protobuf implementation in C++

[Henner Zeller]

On Sat, Apr 7, 2012 at 21:18, G. geula.vainap...@mediamind.com wrote:
 Hi Hans,

 In my proto file, there are two structures, Auction and Bid. The
 Auction structure behaves normally - setting the fields works
 correctly. I am also using the pre-defined realtime-bidding.proto
 file, all of which structures are working fine. It is only the Bid
 structure that acts broken, and I can't see why - because there is
 nothing special about its definition.

 I am using the protoc.exe compiler that came with the protobuf 2.4.1
 installation package for windows, and my IDE is visual studio 2010
 professional, everything is compiled in x64.

One idea I can think of is that you forgot to allocate the Bid
structure and your compiler sets unallocated pointers to some default
value. Or Bid is already freed (a common technique in memory
allocators is to fill the deleted value with some pattern in debug
mode to detect this early). Setting values in your case then
accidentally does not crash but modifies the data structure the
garbage pointer happens to point to.

How do you get the instance of Bid ? It is a bit hard to debug your
code without seeing a self-contained snippet that exposes the problem.
In your example, you're using b.set_ Is 'b' a reference in this
case ? Could it be that you set values on a reference to a temporary
that already is gone ?
Try to run your program with valgrind maybe to find a broken allocation.

(Note it is very very unlikely that the protobuf C++ implementation
has such a bug that is undetected, so something else in your program
is going wrong.)

-h


 G.

 On Apr 5, 7:54 pm, Henner Zeller henner.zel...@googlemail.com wrote:
 On Thu, Apr 5, 2012 at 01:03, G. geula.vainap...@mediamind.com wrot Hi,



  Thanks for responding. This is exactly the way I access the fields.

  Code snippet:
  ...
         b.set_billableseat(Mediamind);
         b.set_category(5);
         b.set_clickthroughurl(http://www.ynet.co.il;);
         b.set_creativeid(ab15a);
  ...
  I walked through this with the debugger. When billableseat is set,
  Clickthroughurl and creativeId both become Mediamind as well. Same
  with clickthroughUrl and creativeId - setting each one of them turns
  all 3 fields the same value. Only setting Category didn't affect
  anything except category.

 What compiler are you using ?

 Protocol buffers definitely don't behave this way, so only a very
 broken compiler or something peculiar in your memory management or
 other setup can cause this.

 (IIRC, default values in protobufs share a const instance (in this
 case: the empty string) for memory reasons, so you would see the same
 address for the default values, but this is of course disengaged
 whenever a value is set.)

 If you just create a simple protocol buffer with two string fields and
 a simple main() function: do you get the same behavior ?

 -h







  Thanks,

  G.

  On Apr 4, 7:45 pm, Jeremiah Jordan jeremiah.jor...@gmail.com wrote:
  How are you setting the data?
  You should be using something like:
  bid.set_HtmlSnippet(Stuff);
  and
  std::string html = bid.HtmlSnippet();

  See:https://developers.google.com/protocol-buffers/docs/reference/cpp/goo...

  On Wednesday, April 4, 2012 7:25:15 AM UTC-5, G. wrote:

   Hi all,

   I am using protobuf 2.4.1, and I encountered a weird issue:

   I created the following .proto file:

   message Auction {
           // Bid request id
           required bytes Id = 1;
           optional bytes Ip = 2;
           required int32 adId = 3;
           required int32 adHeight = 4;
           required int32 adWidth = 5;
           optional string domain = 6;
           optional string country = 7;
           optional string region = 8;
           required string exchangeUserId = 9;
           optional string pageUrl = 10;
           optional int32 publisherId = 11;
           optional int32 timezoneOffset = 12;
           optional string userAgent = 13;
           required string identifier = 14;
   }

   message Bid {
           // Bid request Id
           required bytes Id = 1;
           required int32 processingTime = 2;
           required int32 adId = 3;
           required float bid = 4;
           required int32 advertiserId = 5;
           required string creativeId = 6;
           required string billableSeat = 7;
           required int32 category = 8;
           required int32 vendorType = 9;
           required int32 strategyId = 10;
           required string clickthroughUrl = 11;
           required string HtmlSnippet = 12;
   }

   It compiles fine with protoc.exe.

   However, when I tried assigning the fields, I noticed the following
   phenomenon: the fields id, billableseat and htmlsnippet in Bid
   structure share the same address! When one is assigned, so are the
   other two.

   What am I doing wrong? Has anyone encountered such a thing before?

   Thanks,

   G.

  --
  You received this message because you are subscribed to the Google Groups 
  Protocol Buffers

Re: [protobuf] Re: Odd problem with protobuf implementation in C++

[Henner Zeller]

On Thu, Apr 5, 2012 at 01:03, G. geula.vainap...@mediamind.com wrot Hi,

 Thanks for responding. This is exactly the way I access the fields.

 Code snippet:
 ...
        b.set_billableseat(Mediamind);
        b.set_category(5);
        b.set_clickthroughurl(http://www.ynet.co.il;);
        b.set_creativeid(ab15a);
 ...
 I walked through this with the debugger. When billableseat is set,
 Clickthroughurl and creativeId both become Mediamind as well. Same
 with clickthroughUrl and creativeId - setting each one of them turns
 all 3 fields the same value. Only setting Category didn't affect
 anything except category.

What compiler are you using ?

Protocol buffers definitely don't behave this way, so only a very
broken compiler or something peculiar in your memory management or
other setup can cause this.

(IIRC, default values in protobufs share a const instance (in this
case: the empty string) for memory reasons, so you would see the same
address for the default values, but this is of course disengaged
whenever a value is set.)

If you just create a simple protocol buffer with two string fields and
a simple main() function: do you get the same behavior ?

-h

 Thanks,

 G.



 On Apr 4, 7:45 pm, Jeremiah Jordan jeremiah.jor...@gmail.com wrote:
 How are you setting the data?
 You should be using something like:
 bid.set_HtmlSnippet(Stuff);
 and
 std::string html = bid.HtmlSnippet();

 See:https://developers.google.com/protocol-buffers/docs/reference/cpp/goo...







 On Wednesday, April 4, 2012 7:25:15 AM UTC-5, G. wrote:

  Hi all,

  I am using protobuf 2.4.1, and I encountered a weird issue:

  I created the following .proto file:

  message Auction {
          // Bid request id
          required bytes Id = 1;
          optional bytes Ip = 2;
          required int32 adId = 3;
          required int32 adHeight = 4;
          required int32 adWidth = 5;
          optional string domain = 6;
          optional string country = 7;
          optional string region = 8;
          required string exchangeUserId = 9;
          optional string pageUrl = 10;
          optional int32 publisherId = 11;
          optional int32 timezoneOffset = 12;
          optional string userAgent = 13;
          required string identifier = 14;
  }

  message Bid {
          // Bid request Id
          required bytes Id = 1;
          required int32 processingTime = 2;
          required int32 adId = 3;
          required float bid = 4;
          required int32 advertiserId = 5;
          required string creativeId = 6;
          required string billableSeat = 7;
          required int32 category = 8;
          required int32 vendorType = 9;
          required int32 strategyId = 10;
          required string clickthroughUrl = 11;
          required string HtmlSnippet = 12;
  }

  It compiles fine with protoc.exe.

  However, when I tried assigning the fields, I noticed the following
  phenomenon: the fields id, billableseat and htmlsnippet in Bid
  structure share the same address! When one is assigned, so are the
  other two.

  What am I doing wrong? Has anyone encountered such a thing before?

  Thanks,

  G.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Memory leaks in VS2008 using 2.4.1

[Henner Zeller]

On Sun, Mar 25, 2012 at 06:55, Eric Holtman e...@holtmans.com wrote:
 I've tried building the library as static.   I've tried building the
 library as dll.  I've even gone so far as starting from scratch and
 building my own project, and including all the .cc and .h files.

 I cannot make the memory leaks on program exit go away.

 I am definitely calling         google::protobuf::ShutdownProtobufLibrary ();

 I have traced through, and it *looks* like it's getting to the
 OnShutdown functions for my .proto generated message files, and
 calling delete.  I am definitely calling Shutdown *before* the leak
 checker runs.

 And yet, I still get reported memory leaks.

 I don't really want to ignore the leaks, because sooner or later,
 that's going to hide a real error.

 Am I not doing something correctly?   Is there any way to fix this.

Can you drill down _what_ object is leaking ? In particular, did you
narrow down that it is indeed protobufs that cause the leak ?
You need to find the leaking object first, otherwise it is a bit hard
do diagnose what is going on.

-h


 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] simple '.proto' structure

[Henner Zeller]

On Thu, Feb 9, 2012 at 03:33, MohanR radhakrishnan.mo...@gmail.com wrote:
 Hi,
         I tried to write and read a simple '.proto' structure but
 when I read it back I get an exception. Is there something wrong with
 the way I read ?


 Thanks.

 --proto-

 package message;

 message Load {

  enum LoadType {
    HIGH = 0;
    MEDIUM = 1;
    LOW = 2;
  }
  message LoadBalance {
    optional LoadType type = 2 [default = MEDIUM];
  }
  optional LoadBalance loadbalancer = 1;
 }

 -Write---

        Message.Load message =
 Message.Load.newBuilder().setLoadbalancer(

 Message.Load.LoadBalance.newBuilder().setType( 
 Message.Load.LoadType.HIGH)).build();

uhm, where do you set the value ? I'd expected something like
SerializeToString()

 Read--

            Message.Load message = Message.Load.parseFrom( value );

 ---Exception

 InvalidProtocolBufferExceptionProtocol message contained an invalid
 tag (zero).
 com.google.protobuf.InvalidProtocolBufferException: Protocol message
 contained an invalid tag (zero).
        at
 com.google.protobuf.InvalidProtocolBufferException.invalidTag(InvalidProtocolBufferException.java:
 68)
        at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:
 108)
        at message.Message$Load$Builder.mergeFrom(Message.java:723)

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: suggestions on improving the performance?

[Henner Zeller]

On Fri, Jan 13, 2012 at 11:22, Daniel Wright dwri...@google.com wrote:
 It's extremely unlikely that text parsing is faster than binary parsing on
 pretty much any message.  My guess is that there's something wrong in the
 way you're reading the binary file -- e.g. no buffering, or possibly a bug
 where you hand the protobuf library multiple messages concatenated together.

In particular, the
   object type, object, object type object ..
doesn't seem to include headers that describe the length of the
following message, but such a separator is needed.
( http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming )

  It'd be easier to comment if you post the code.

 Cheers
 Daniel


 On Fri, Jan 13, 2012 at 1:22 AM, alok alok.jad...@gmail.com wrote:

 any suggestions? experiences?

 regards,
 Alok

 On Jan 11, 1:16 pm, alok alok.jad...@gmail.com wrote:
  my point is ..should i have one message something like
 
  Message Record{
    required HeaderMessage header;
    optional TradeMessage trade;
    repeated QuoteMessage quotes; // 0 or more
    repeated CustomMessage customs; // 0 or more
 
  }
 
  or rather should i keep my file plain as
  object type, object, objecttype, object
  without worrying about the concept of a record.
 
  Each message in file is usually header + any 1 type of message (trade,
  quote or custom) ..  and mostly only 1 quote or custom message not
  more.
 
  what would be faster to decode?
 
  Regards,
  Alok
 
  On Jan 11, 12:41 pm, alok alok.jad...@gmail.com wrote:
 
 
 
 
 
 
 
   Hi everyone,
 
   My program is taking more time to read binary files than the text
   files. I think the issue is with the structure of the binary files
   that i have designed. (Or could it be possible that binary decoding is
   slower than text files parsing? ).
 
   Data file is a large text file with 1 record per row. upto 1.2 GB.
   Binary file is around 900 MB.
 
   **
    - Text file reading takes 3 minutes to read the file.
    - Binary file reading takes 5 minutes.
 
   I saw a very strange behavior.
    - Just to see how long it takes to skim through binary file, i
   started reading header on each message which holds the length of the
   message and then skipped that many bytes using the Skip() function of
   coded_input object. After making this change, i was expecting that
   reading through file should take less time, but it took more than 10
   minutes. Is skipping not same as adding n bytes to the file pointer?
   is it slower to skip the object than read it?
 
   Are their any guidelines on how the structure should be designed to
   get the best performance?
 
   My current structure looks as below
 
   message HeaderMessage {
     required double timestamp = 1;
     required string ric_code = 2;
     required int32 count = 3;
     required int32 total_message_size = 4;
 
   }
 
   message QuoteMessage {
           enum Side {
       ASK = 0;
       BID = 1;
     }
     required Side type = 1;
           required int32 level = 2;
           optional double price = 3;
           optional int64 size = 4;
           optional int32 count = 5;
           optional HeaderMessage header = 6;
 
   }
 
   message CustomMessage {
           required string field_name = 1;
           required double value = 2;
           optional HeaderMessage header = 3;
 
   }
 
   message TradeMessage {
           optional double price = 1;
           optional int64 size = 2;
           optional int64 AccumulatedVolume = 3;
           optional HeaderMessage header = 4;
 
   }
 
   Binary file format is
   object type, object, object type object ...
 
   1st object of a record holds header with n number of objects in that
   record. next n-1 objects will not hold header since they all belong to
   same record (same update time).
   now n+1th object belongs to the new record and it will hold header for
   next record.
 
   Any advices?
 
   Regards,
   Alok

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] tcp get packet

[Henner Zeller]

On Tue, Jan 3, 2012 at 18:37, calvin zhu calvinmy0...@gmail.com wrote:
 client send a proto packet to server by tcp, and how do the server
 certain the packet size?
 make a packet head?

If you want to make sure that you got the whole packet, you need to
add a header with the size, yes.
A simple way is to serialize the content to a string first, determine
the size and send that size at the
very beginning. This has as well the advantage, that you can send
multiple messages, each separated with
a size-header on the same tcp connection.
This is essentially described in
  http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming

Easiest is to just send a fixed 32 bit value. A bit more fancy would
be to use a varint (
http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints
).

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] encode negative int32 puzzle

[Henner Zeller]

On Sat, Dec 3, 2011 at 20:07, 萌 饶 raome...@gmail.com wrote:
 I have a puzzle for a long time that why a negative int32 must be
 encoded into 10 bytes instead of 5, 5 bytes of encoded var int has
 enough bits(5*7=35) to represent a 32 bit value, and packet decoder
 know it's a int32 so it extract 32 bits outof 5 bytes to restore the
 int32 precisely, even if it's negtive.So why must protobuf encode
 negative int32 to 10 bytes? Thank you.

The wiretype is not specified to int32 or int64, it is only specified
to be a variable encoded integer. This way, things are upwards
compatible if you change an int32 value to int64 later.

But because of that, the varint encoding always encodes the value as
int64 value; which requires 10 bytes for full encoding.

If you have negative values, you should use the zigzag encoded values
(sint32, sint64).

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] c server - java client

[Henner Zeller]

On Thu, Sep 29, 2011 at 05:34, Dominique Volery d.vol...@gmail.com wrote:
 hi,

 i'm using gpb for a client server application. on the server is a c
 gpb library in use and on the client the messages were generated by
 java.

Which implementation do you use ? I only know the Google
implementation - so I have to
guess wrt. the C implementation. Or do you use the C++ version ?
Also, some information is missing: how do you transfer the message
between client/server ?

So this answer is mostly general but covers typical problems.

 receiving large messages on the server resolves allways in an error
 message data too short after length-prefix of 13590511
 small messages are not a problem. so my question is: why i have this
 problem only when client and server have different api. c client to c
 server works perfect.
 does java not set a length when the prefix WIRETYPE_LENGTH_DELIMITED
 is set? because the c server looks for the base7 coded length when
 receiving a message with this identifier.

Often a problem in C is to use strlen() or other functions that assume
that strings are \0 terminated. The protocol buffer serialization
format in general can contain any character, so this would not work.
Could there be
something like this going on ?
Also, how is your data packaged when sending over the wire?
If your data is sufficiently large, then a single read from the socket
might not return all the data you need.

In a typical client/server wire protocol, you'd transfer the length of
the serialized message in the first bytes followed by the message.
On the receiving side, you read the length and then keep reading until
you've everything you need.

Other than covering these questions, the only reliable way is to do a
debug dump of what the java service is sending and what the c service
is receiving, to see what is different.
If it is the same: manually look through the data to see how things are encoded.

Hope this helps,
  -henner

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] messages of different types Serialized to string.

[Henner Zeller]

On Thu, Sep 22, 2011 at 03:32, Suraj surajn.v...@gmail.com wrote:
 Hi,

 Say I have two messages a and b of types A and B respectively.

 Can I serialize them to the same string?
 Something like..
 a.SerializeToString(str);
 b.SerializeTostring(str_new);
 str.append(str_new);

 Then while parsing, I will do,
 a.ParseFromString(str)
 and b.ParseFromString(str).

 I have tried this and it is working. But want to confirm whether it's
 a correct thing to do or not?

What you get is that message a and b both have the same content, more
precisely, the merged content of both the original messages ( see
http://code.google.com/apis/protocolbuffers/docs/encoding.html#optional
for merging ).

Not sure if this is what you want.

If you want to serialize multiple different messages in one string,
you need to put them in some kind of container format, e.g.
length1message1length2message2  this has been discussed on
this
list several times. You also find documentation in 'Techniques'
http://code.google.com/apis/protocolbuffers/docs/techniques.html

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Repeated fields support efficient random access?

[Henner Zeller]

On Sun, Aug 21, 2011 at 22:24, Tommy Li tommy...@ucla.edu wrote:
 Hi, are repeated fields supposed to support efficient random access?

In the in-memory representation of the protocol buffer, they are
essentially stored in an array. So copying it to an array wouldn't buy
you anything

 Or should we copy the data to an Array before we start doing random
 access on it?

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Proto Buf License missing

[Henner Zeller]

On Wed, Aug 17, 2011 at 15:50, Ali Asghari bob9...@gmail.com wrote:
 Hi, I would love to use your code, however the license for them
 appears to be missing. The link provided points to a license
 generation tool, however I doubt our legal department will accept this
 as it does not directly mention the license granter or product. Does
 anyone know where I can get a more complete license from?\

Download the code
  http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.bz2
Unpack. In the toplevel directory there is COPYING.txt.

Also, in each header or *.cc file, you find the license.

-h


 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Is the Protocol Buffers serialization format standardized?

[Henner Zeller]

On Fri, Jul 15, 2011 at 08:19, jl jlaben...@gmail.com wrote:
 We are wondering if the serialized data format used by Protocol
 Buffers is standardized or in some way set in stone? In other words,
 should we have any expectation that it will change in the future and
 we won't be able to read the serialized data back into a Message
 class?

 For example, we receive protobuf messages and archive them to disk in
 their serialized format using
 google::protobuf::Message::SerializeToFileDescriptor() or
 SerializeToOstream() and later, maybe 5 years from now using a newer
 version of Protocol Buffers, try to read them back using
 google::protobuf::Message::ParseFromFileDescriptor() or
 ParseFromIstream(). Can I expect this to work?

 I could not find any mention about this on the Protobuf website or by
 Googling, but perhaps I missed it. Any links to any official
 documentation saying either yes or no would be greatly appreciated.

Almost everything that Google stores on disk is in protocol buffer
format. That would be a huge legacy to abandon with an incompatible
protocol buffer change.
So, even though there is no official 'standard document' (yet?), you
can bet on the fact that stuff written now will be readable in 5 or 20
years from now (if your storage medium survives it :) ).

-h


 Thanks and regards,
    John

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Is message rename binary compatible?

[Henner Zeller]

On Tue, May 17, 2011 at 07:46, Imagination's End eda...@disemia.com wrote:
 Say I have a message used like this:

 message old_name { ... }

 message wrapper {
  repeated old_name items = 3;
 }

 I want to rename old_name to new_name. Is this binary compatible? That
 is, will users of the previous definition be able to read the new
 definition (assuming I leave the fields as is).  In particular will
 the items be accessible correctly?

Yes. The binary only contains the tag number to identify the fields
(in this case 3), so you're free to rename the fields and message
names.


 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Zero tag number in enum data type

[Henner Zeller]

On Mon, Apr 18, 2011 at 22:28, Canggih cangca...@gmail.com wrote:
 hello, i just wanna share.

 In Protobuf developer guide, in the sample proto file, there is a
 PhoneType enum data type, consist of MOBILE, HOME, and WORK. MOBILE
 use '0' as its tag number, and so on.

This is the integer value of the enumeration value, not to be confused
with a tag number you give to identify fields.

 Meanwhile, i'm using Protobuf plugin for Netbeans, for my project.
 When i was try to compile a proto file, there was an error. The error
 was in enum data type. Compiler couldn't compile enum data that had
 '0' tag number. Then, when I change to '1', it can run successfully.

If you mean tag number, then yes, tag numbers cannot be zero but have
to start with 1. Enumeration _values_ can be 0.

-h



 Any explanations?

 Thanks in advance
 Canggih

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] UnmodifiableLazyStringList is not Serializable

[Henner Zeller]

On Wed, Apr 13, 2011 at 06:45, J.S. goo...@juergenschmied.de wrote:
 Hi!

 Passing a buffer with a repeated String from one EJB to another, I
 get:

  javax.ejb.EJBException: java.rmi.MarshalException: CORBA BAD_PARAM
 1398079494 Maybe; nested exception is:
        java.io.NotSerializableException: WARNING: IOP0016: Class
 com.google.protobuf.UnmodifiableLazyStringList is not Serializable

Protocol buffers are objects to serialize things, but it doesn't
really make sense to use Java serialization to serialize their holder
objects. What you actually want to do is to use the ProtocolBuffer
functionality to serialize it to an byte array and use that to
transfer.

If you would try to Java serialize the protocol buffer objects, you
would loose all the functionality you were probably considering using
Protocol buffers in the first place: version independent, fast and
platform independent serialization. The object layout might change
between each protocol buffer library version or you might just change
the proto definition - all of which will create trouble if you use
Java serialization; the protocol buffer binary format, however, is
stable (and compact, and fast, and platform independent ...)

So in this case it was actually good that the
UnmodifiableLazyStringList was not Java serializable - it helped you
find an improper use.
I am not sure what the maintainers think, but I wouldn't add Java
'Serializable' to all classes used within the generated objects -
because then these accidental wrong uses would just go unnoticed.

Henner

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: UnmodifiableLazyStringList is not Serializable

[Henner Zeller]

On Wed, Apr 13, 2011 at 16:01, Ben Wright compuware...@gmail.com wrote:
 I agree with J.S. on this one - there are many situations in Java EE
 environments where Serializable is checked or java serialization
 used when it's not simple or feasible to leverage protobuf
 serialization.  Most of these situations are invm / in memory
 transfers.  Sometimes java serialization is used to clone objects or
 to safely pass them between ClassLoader scopes.

 I've run into this limitation with JBoss ESB and passing protocol
 buffer objects between services through the InVM message transport.

It is a couple of years that I've worked with Java, so my
serialization fu might not be up-to-par. But would the problem be
solved if the ProtocolBuffer object implements Externalizable ? That
way it can be serialized via the Java VM, but it would use the
protobuf specific serialization.

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] DebugString causing a crash

[Henner Zeller]

On Mon, Apr 4, 2011 at 00:26, des fitzgerald
d...@intrepid-geophysics.com wrote:
 Hi, below is more orless the same example in the on-line documentation

 the message Foo example turned into a cppunit_test

 I thought the DebugString method would always work, based upon what I
 have read.
 This crashes? any guidance?

Where does it crash (stacktrace?). What compiler are you using ?

 thanks
 des

 static std::string data; // Will store a serialized version of the
 message.
 void ProtocolsTestCase::testExerciseParsing() {
        testCreateSerializedMessage();

        // parse from string
        testParseFromString();

        //
        //      testDynamicParseFromString();
 }
 // get on top of dynamic creation of a protobuf message
 void ProtocolsTestCase::testCreateSerializedMessage() {
        //       Create a message and serialize it.
        Foo foo;
        foo.set_text(Hello World!);
        foo.add_numbers(1);
        foo.add_numbers(5);
        foo.add_numbers(42);

        foo.SerializeToString(data);
 }
 void ProtocolsTestCase::testParseFromString() {
        //       Parse the serialized message and check that it contains the
 correct data.
        Foo foo;
        foo.ParseFromString(data);

        CPPUNIT_ASSERT(foo.text() == Hello World!);
        CPPUNIT_ASSERT(foo.numbers_size() == 3);
        CPPUNIT_ASSERT(foo.numbers(0) == 1);
        CPPUNIT_ASSERT(foo.numbers(1) == 5);
        CPPUNIT_ASSERT(foo.numbers(2) == 42);
    std::string out = foo.DebugString();
    lprintf(%s\n, out.c_str());
 }

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Error while de-serializing

[Henner Zeller]

On Tue, Mar 22, 2011 at 10:53, nidheeshdas nidheesh...@gmail.com wrote:

 I've a bool and a string in the proto file compiled to cpp both
 required. when i send them with empty string the bool is also
 getting affected.

 does protobuf mishandles the bools ?


What do you mean with 'affected' ? This is not very clear.

Can you give a small, self contained code example that shows what you're
trying to do ?



 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Inheritance..

[Henner Zeller]

On Thu, Mar 17, 2011 at 02:41, ctapobep stanislav.bashkirt...@gmail.com wrote:
 So pity protobuf doesn't support inheritance, only because of this I
 can't use it. Any forecasts, are creators going to add this feature?

There are good reasons not to support inheritance and it would be a
design problem if it was; so it is likely not supported.

Typically if someone has the reflex to use inheritance, they actually
mean to do something else. So what is the exact use case you want to
address ?

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] ProtoBuf and Multi-Threads

[Henner Zeller]

On Tue, Mar 15, 2011 at 06:05, ksamdev samvel.khalat...@gmail.com wrote:
 I like the interest in the topic.
 I've put 1GB to emphasize that the use case is safe. In fact, I save
 messages in file in next way:
 XYXYXYXYXY.
 where X is the size of the message and Y is the message itself. Each message
 is read in the loop and overwritten. Clearly, I do not read the whole file
 (N GB's) into memory at once.
 Now, with this technique, I can generate files with size larger than 2^31 (~
 2GB). The file is successfully written. Consider the case with 5 GB file.
 Unfortunately, whenever I start reading this 5 GB's file, ProtoBuf fails
 after 2^31 bytes are read. Of course, I have to push the limit of read bytes
 with:

 CodedInputStream::SetTotalBytesLimit(int, int)
 Pay attention at the arguments type: int . I suppose ProtoBuf uses bytes
 read counter or some internal file read position pointer that is also int
 and therefore fails whenever reading procedure passes the 2^31 threshold.

You should just create a new CodedInputStream on the stack for each
message, that way you don't run into this limit and can read files as
large as you want.
(CodedInputStream is cheap to create, so it shouldn't influence your
performance numbers).

 Thanks for the link to perftools. Like you mentioned, I reuse the message in
 my code. Therefore there is no overhead.
 I guess, the problem was in the way I measured execution time. My command
 looked like:
 time executable args  echo -  time executable args
 So, I've cut it into 3 pieces and time, that is shown on the screen, start
 make sense:
 time executable args
 echo --
 time executable args

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] ProtoBuf and Multi-Threads

[Henner Zeller]

On Tue, Mar 15, 2011 at 10:09, Samvel Khalatyan
samvel.khalat...@gmail.com wrote:
 Thanks for the clarification. Does it mean that my code should look like the
 example below?
 fstream in(10_GB_file.pb, ios::in | ios::binary);
 bool continue_reading = true
 while(continue_reading)
 {
   ::google::protobuf::io::IstreamInputStream raw_in(in);

The ::google::protobuf::io::IstreamInputStream raw_in can probably be
out of the loop. I haven't looked at the code closely, but I think it
is not limited (except an int64 offset, but that is right now big
enough...).

Just try it.

   ::google::protobuf::io::CodedInputStream coded_in(raw_in);

... and this one should be in the loop, yes.

   // Read message and use it if read was successfull
   if (! /* are there more messages left? */)
     continue_reading = false;
 }
 On Tue, Mar 15, 2011 at 11:48 AM, Henner Zeller
 henner.zel...@googlemail.com wrote:

 On Tue, Mar 15, 2011 at 06:05, ksamdev samvel.khalat...@gmail.com wrote:
  I like the interest in the topic.
  I've put 1GB to emphasize that the use case is safe. In fact, I save
  messages in file in next way:
  XYXYXYXYXY.
  where X is the size of the message and Y is the message itself. Each
  message
  is read in the loop and overwritten. Clearly, I do not read the whole
  file
  (N GB's) into memory at once.
  Now, with this technique, I can generate files with size larger than
  2^31 (~
  2GB). The file is successfully written. Consider the case with 5 GB
  file.
  Unfortunately, whenever I start reading this 5 GB's file, ProtoBuf fails
  after 2^31 bytes are read. Of course, I have to push the limit of read
  bytes
  with:
 
  CodedInputStream::SetTotalBytesLimit(int, int)
  Pay attention at the arguments type: int . I suppose ProtoBuf uses bytes
  read counter or some internal file read position pointer that is also
  int
  and therefore fails whenever reading procedure passes the 2^31
  threshold.

 You should just create a new CodedInputStream on the stack for each
 message, that way you don't run into this limit and can read files as
 large as you want.
 (CodedInputStream is cheap to create, so it shouldn't influence your
 performance numbers).

  Thanks for the link to perftools. Like you mentioned, I reuse the
  message in
  my code. Therefore there is no overhead.
  I guess, the problem was in the way I measured execution time. My
  command
  looked like:
  time executable args  echo -  time executable args
  So, I've cut it into 3 pieces and time, that is shown on the screen,
  start
  make sense:
  time executable args
  echo --
  time executable args
 
  --
  You received this message because you are subscribed to the Google
  Groups
  Protocol Buffers group.
  To post to this group, send email to protobuf@googlegroups.com.
  To unsubscribe from this group, send email to
  protobuf+unsubscr...@googlegroups.com.
  For more options, visit this group at
  http://groups.google.com/group/protobuf?hl=en.
 



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] ProtoBuf and Multi-Threads

[Henner Zeller]

On Mon, Mar 14, 2011 at 10:08, ksamdev samvel.khalat...@gmail.com wrote:
 Hi,
 I have a large set of files with a number of the same type messages saved.
 My code reads messages in a sequence from these files one after another.
 I've measured time (with terminal time command) of running the code, and
 get something like:
 READING
 ===
 Read ProtoBuf
 Processed events: 5000
 real 7m2.146s
 user 5m25.545s
 sys 0m31.959s
 Then I've adjusted the code to read the files in threads (8 threads on 8
 cores machine). The reading procedure is independent and put into separate
 class. Therefore each thread is really independent of the others.
 Nevertheless, the time measurement is:
 READING (MULTITHREADS)
 =
 Thread read 600 events
 Thread read 600 events
 Thread read 600 events
 Thread read 600 events
 Thread read 600 events
 Thread read 600 events
 Thread read 700 events
 Thread read 700 events
 real 5m3.808s
 user 5m42.301s
 sys 0m35.221s
 As you may see, the user as well as real time is pretty much the same.
 So, it seems that there is some internal locks done somewhere. I only use
 locks between threads and class, that creates and manages threads. The locks
 are used only when thread finishes reading the file(s).
 Does ProtoBuf use some sort of generic static/singleton functions/objects
 that are used to de-serialize messages and therefore lock when accessed form
 different threads?

No, there is no locking between threads being done. And Kenton would
certainly be opposed to Singletons :)

But did you check if you run into file IO limits ? How big are your
files ? 6 Million protocol buffers in 300 seconds is 50 microseconds,
that is very slow, or you have huge data. Are you doing other things,
besides parsing in this benchmark ?

 If so, is there a way to suppress this and get truly
 independent messages parsing?
 thanks.
 P.S. My code can be browsed on github: http://goo.gl/DXCCF . The reading of
 messages is done by: http://goo.gl/OsHV9
 The code uses ROOT framework (root.cern.ch) if one wants to compile it.

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] ProtoBuf and Multi-Threads

[Henner Zeller]

On Mon, Mar 14, 2011 at 10:34, ksamdev samvel.khalat...@gmail.com wrote:
 Thanks for a quick reply.
 Honestly, I fill a set of histograms for each event. I've added this feature
 only recently and have a version of the code without histograms.
 Here is the same performance measurement without histograms:
 READING
 ===
 Read ProtoBuf
 Processed events: 100
 real 0m2.510s
 user 0m2.105s
 sys 0m0.298s
 ---===---
 READING (MULTITHREADS)
 =
 process files
 init threads
 start threads
 run threads
 Thread read 100 events
 real 0m2.358s
 user 0m2.085s
 sys 0m0.236s
 Again, the same situation.
 My file is 384MB. I've already tested the use case with files above 1GB. It
 turs out that ProtoBuf has a int limitation on file size. Anyway, I am a
 way below the limit. The messages are pretty short (~400B).

Wait, where do you get the 1GB limit ? There is no real limit on the
file size (in fact, the protocol buffer library does not do anything
with files directly). (and: don't try to load everything in memory at
once as a gigantic repeated message, that will give you a lot of
malloc overhead).

Another thing that could be limiting your throughput in your library
is memory allocation: the system libraries are often not very good in
handling threads well; I am usually using tcmalloc (
http://code.google.com/p/google-perftools/ ) which is handling that
case pretty well.

Also: if you read a lot of messages, it is a good idea to Clear() and
re-use a message instead of allocating a new one (protocol buffers
attempt to re-use memory internally).

-h


 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Store number of messages

[Henner Zeller]

Varint does an encoding whose length changes with the size of the
number - hence the name. So it can use between 1 and 10 bytes. So when
the value _events_written changes, re-writing the same number at the
beginning of the file will use a different number of bytes.

If you want to write a number that changes like this to a particular
position, you need to use a fixed size number encoding (e.g. fixed32
or fixed64) to do so.

On Fri, Mar 4, 2011 at 16:22, ksamdev samvel.khalat...@gmail.com wrote:
 I have an application where number of messages is a priory unknown. I can
 open output file and write 0 in the beginning:
 Writer::Writer(const string filename):
     _output(filename.c_str(), ios::out | ios::trunc | ios::binary),
     _events_written(0)
 {
     _raw_out.reset(new
 ::google::protobuf::io::OstreamOutputStream(_output));
     _coded_out.reset(new
 ::google::protobuf::io::CodedOutputStream(_raw_out.get()));
     _coded_out-WriteVarint32(_events_written);

At this time, _events_written is zero, which, encoded, uses a
different length in the file ...

 }
 and then in the destructor reset the position in the ofstream to the
 beginning to write the number of the events:
 Writer::~Writer()
 {
     _coded_out.reset();
     _raw_out.reset();
     _output.seekp(0);
     // Small trick to save number of the events
     //
     _raw_out.reset(new
 ::google::protobuf::io::OstreamOutputStream(_output));
     _coded_out.reset(new
 ::google::protobuf::io::CodedOutputStream(_raw_out.get()));
     _coded_out-WriteVarint32(_events_written);

... than here: _events_written is a larger number, encoded with more
bytes, so it will overwrite parts of the following data ..

     _coded_out.reset();
     _raw_out.reset();
     _output.close();
 }
 Then read the beginning of the file with number of the events:
 Reader::Reader(const string filename):
     _input(filename.c_str(), ios::in | ios::binary),
     _is_good(true),
     _events_written(0)
 {
     _raw_in.reset(new ::google::protobuf::io::IstreamInputStream(_input));
     _coded_in.reset(new
 ::google::protobuf::io::CodedInputStream(_raw_in.get()));
     _coded_in-ReadVarint32(_events_written);
 }
 Everything seems fine. Then events can be read one by one like:
 bool Reader::read(Event event)
 {
     event.Clear();
     uint32_t message_size;
     if (!_coded_in-ReadVarint32(message_size))

... so here you have a number, whose beginning is partially
overwritten by the end of the re-written events list - the reading
fails.

     {
         _is_good = false;
         return false;
     }
     if (0  message_size)
     {
         string message;
         if (!_coded_in-ReadString(message, message_size) ||
             !event.ParseFromString(message))
             return false;
     }
     return true;
 }
 Unfortunately, ReadVarint32 fails for some reason in the Reader::read(...)
 method.
 The code works fine in case I do not use seekp in the Writer::~Writer().
 What is the proper way to seek to the beginning of the file and store number
 of entries?

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] field of type timestamp or date with protobuf message

[Henner Zeller]

On Sat, Feb 12, 2011 at 09:13, Marco@worldcorp mmistr...@gmail.com wrote:
 HI all
  is it possible to add a field of type timestamp , or date, to a
 protobuf message..

There is no 'default' way. The reason for that is that protocol
buffers are platform independent, and there is no real agreeable way
to represent timestamps on all platforms; there are seconds,
milliseconds and microseconds since 1970 around; yet other system
represent times in even higher resolution since a different epoch
start.

Also, having protocol buffers handle timestamps might actually be too
generic; suppose you want to be sure to be interpretable in a
sufficient high resolution and protocol buffers choose to encode
microseconds since, say, 1970. Then this will encode in huge numbers (
= more wire bytes) - but OTOH, this is not really what is needed in
all contexts.

So - best is to have the application choose the time resolution that
makes sense depending on the application and deal with it in an
application dependent manner (e.g. I store seconds, milliseconds or
microseconds since 1970 GMT or Julian dates in protocol buffers to
transfer times in different contexts).

-h


 i have quickly read the guide, and i saw no trace of timestamps or
 fields of type date.

 could anyone assist?

 w/kindest regards
  marco

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to protobuf@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: New protobuf feature proposal: Generated classes for streaming / visitors

[Henner Zeller]

On Tue, Feb 1, 2011 at 23:02, fpmc f...@google.com wrote:
 Your proposal has one VisitXXX function for each repeated type.  How
 does it handle a message with two repeated fields of the same type?

I guess the naming is confusing in the example. The Visit is per
field-name; but since the typed is named the same as the field in this
example, it is confusing.


 On Feb 2, 2:30 pm, Kenton Varda ken...@google.com wrote:
 On Tue, Feb 1, 2011 at 3:17 PM, Jason Hsueh jas...@google.com wrote:
  Conceptually this sounds great, the big question to me is whether this
  should be implemented as an option in the compiler or as a separate plugin.
  I haven't taken a thorough look at the patch, but I'd guess it adds a 
  decent
  amount to the core code generator. I have a preference for the plugin
  approach, but of course I'm primarily an internal protobuf user, so I'm
  willing to be convinced otherwise :-) Would using a plugin, possibly even
  shipped with the standard implementation, make this feature too 
  inconvenient
  to use? Or is there enough demand for this that it warrants implementing as
  an option?

 First of all, note that this feature is off by default.  You have to turn it
 on with the generate_visitors message-level option.  The only new code added
 to the base library is a couple templates in WireFormatLite, which are of
 course never instantiated if you don't generate visitor code.

 There are a few reasons I prefer to make this part of the base code
 generator:

 - If you look at the patch, you'll see that the code generation for the two
 Guide classes actually shares a lot with the code generation for
 MergeFromCodedStream and SerializeWithCachedSizes.  To make this a plugin,
 either we'd have to expose parts of the C++ code generator internals
 publicly (eww) or we'd have to reproduce a lot of code (also eww).

 - The Reader and Writer classes directly use WireFormatLite, which is a
 private interface.

 - It seems clear that this feature is widely desired by open source users.
  We're not talking about a niche use case here.

  Regarding the proposed interfaces: I can imagine some applications where
  the const refs passed to the visitor methods may be too restrictive - the
  user may instead want to take ownership of the object. e.g., suppose the
  stream is a series of requests, and each of the visitor handlers needs to
  start some asynchronous work. It would be good to hear if users have use
  cases that don't quite fit into this model (or at least if the existing use
  cases will work).

 Interesting point.  In the Reader case, it's creating new objects, so in
 theory it ought to be able to hand off ownership to the Visitor it calls.
  But, the Walker is walking an existing object and thus clearly cannot give
 up ownership.  It seems clear that some use cases need const references,
 which means that the only way we could support ownership passing is by
 adding another parallel set of methods.  I suppose they could have default
 implementations that delegate to the const reference versions, in which case
 only people who wanted to optimize for them would need to override them.
  But I'd like to see that this is really desired first -- it's easy enough
 to add later.

 Also note that my code currently doesn't reuse message objects, but
 improving it to do so would be straightforward.  A Reader could allocate one
 object of each sub-message type for reuse.  But, it seems like that wouldn't
 play well with ownership-passing.









  On Tue, Feb 1, 2011 at 10:45 AM, Kenton Varda ken...@google.com wrote:

  Hello open source protobuf users,

  *Background*

  Probably the biggest deficiency in the open source protocol buffers
  libraries today is a lack of built-in support for handling streams of
  messages.  True, it's not too hard for users to support it manually, by
  prefixing each message with its size as described here:

 http://code.google.com/apis/protocolbuffers/docs/techniques.html#stre...

  However, this is awkward, and typically requires users to reach into the
  low-level CodedInputStream/CodedOutputStream classes and do a lot of work
  manually.  Furthermore, many users want to handle streams
  of heterogeneous message types.  We tell them to wrap their messages in an
  outer type using the union pattern:

   http://code.google.com/apis/protocolbuffers/docs/techniques.html#union

  But this is kind of ugly and has unnecessary overhead.

  These problems never really came up in our internal usage, because inside
  Google we have an RPC system and other utility code which builds on top of
  protocol buffers and provides appropriate abstraction. While we'd like to
  open source this code, a lot of it is large, somewhat messy, and highly
  interdependent with unrelated parts of our environment, and no one has had
  the time to rewrite it all cleanly (as we did with protocol buffers 
  itself).

  *Proposed solution:  Generated Visitors*

  I've been wanting to fix this for some 

Re: [protobuf] C# client to Java based server

[Henner Zeller]

On Tue, Dec 28, 2010 at 15:14, Kenton Varda ken...@google.com wrote:
 Protocol buffers itself has no built-in RPC implementation.  You have to
 find an RPC implementation that supports whatever languages you are
 interested in, or write your own.  It's not too hard to write a simple RPC
 implementation given protocol buffers as a base.  Sending protobufs over
 HTTP is a popular solution that lets you leverage existing infrastructure.

Good place to start looking for implementations is
  http://code.google.com/p/protobuf/wiki/ThirdPartyAddOns


 On Sun, Dec 26, 2010 at 11:55 PM, Tommy tommyhan...@gmail.com wrote:

 Hello,

    I originally was using JAX-WS to communicate between client and
 server. This
   works well when my client is C# and server is Java based service.

    I have been asked to speed up the performance using Protocol
 Buffers to do
   binary serialization instead of text-based serialization. Knowing
 that I can no longer
   use Soap and had to figure out another transport mechanism.

    Is RPC the best way? Can I use RPC when my client and server are
 using
   C# and Java respectively? I noticed the example of CXF-protobuf
 link that
   shows how you can make the connection between Java and Java using
 RPC,
    but nothing about if the Client is C#.

 thanks,
   Tommy

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] image

[Henner Zeller]

On Thu, Dec 16, 2010 at 04:11, leo kmukes...@gmail.com wrote:
 I am planning to use 'Protocol Buffers' in one of my Java ME project
 which requires server to send catalogs (possibly with images). Can I
 send images (PNG format) using 'Protocol Buffers'?

With the type 'bytes', you can send arbitrary binary data.

The only limit is the 'reasonable size', i.e. your messages should
typically not be bigger than a couple of megabytes with an absolute
maximum of 2G ... but that will not be a problem for smallish images I
suppose.

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] fails to parse from string

[Henner Zeller]

On Wed, Nov 10, 2010 at 08:23, Brad Lira snmp.apa...@gmail.com wrote:
 client side:

 address_book.SerializeToString(mystr)
 strncpy(buf, mystr.c_str(), strlen(mystr.c_str()));

This is an error - you only copy to the first \0 byte (strlen looks
for a nul-terminated string) -- however, the string contains binary
data.

And: why do you copy the data in the first place to some buffer ? This
is an additional (potentially expensive) memory copy; why not use the
buffer directly from the string ?

sendto(socket, mystr.data(), mystr.size() ...)

 sendto(socket, buf, )

 server side:

 recvfrom(socket, buf, )
 mystr.assign(buf, strlen(buf));

 if (address_book.ParseFromString(mystr) == false)
 {
       print deserialization failed
 }

 I get deserialization failed all the time, but get the message correctly 
 though.

 Maybe this method is not the right way to send string across socket.
 I tried using SerializeToFileDescriptor(socket), that worked on the
 client side, but on the server side, i never get the message with UDP sockets.
 is there a better way of sending data across network?

 thanks,


 On Tue, Nov 9, 2010 at 5:43 PM, Kenton Varda ken...@google.com wrote:
 It sounds like you probably have extra bytes at the end of mystr which are
 not part of the protobuf.  The parser parses everything before those bytes
 just fine, but then chokes when it gets to the bytes it doesn't recognize.
  Please make sure you only pass in the exact bytes which came out of the
 serializer at the other end, no more, no less.

 On Tue, Nov 9, 2010 at 1:11 PM, Brad Lira snmp.apa...@gmail.com wrote:

 I have a c++ client/server application that sends messages using
 protocol buffers, however, at the server side, when i call

 address_book.ParseFromString(mystr)

 it returns false, but it actually gets the message correctly from
 client side, so i am not sure why it thinks that parsing has failed.
 any ideas?

 thanks,
 brad

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.




 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Swap() on Windows

[Henner Zeller]

Is this even a legal transformation that the MS compiler is doing
there ? Sounds like a overzealous-optimization compiler bug to me.

Did you try to manually remove the 'const' in the generated proto code
to see if that would fix it ?

On Wed, Nov 10, 2010 at 18:51, Anand Ganesh
anantharamangan...@gmail.com wrote:
 Hi Kenton, all,

 When you have a message which contains a string member variable, the
 variable is initialized to a default-value.

 So you have
 ::std::string *name_;
 static const ::std::string _default_name_;
 name_ = _default_name;

 Seems like since _default_name is declared 'const' the microsoft
 compiler is optimizing for cache localization and putting
 _default_name_ inside each individual object (like it would for a
 member variable). Hence _default_name_ has different addresses in two
 different objects. Functionally this is not a problem because there's
 no data-sharing (as in, one object modifying the static variable to a
 new value and others reading the new value from it) because it is
 declared const.

 What this is doing is that when you Swap() two objects with the 'name'
 member variable, where one object is at the default value but the
 other is not, the destructor causes a core-dump. Let me explain better
 using the following workflow:

 ObjectA:
 name_ = 0X123 
 (0x123) _default_name_ = 

 ObjectB:
 name_ = 0x456 
 (0x456) _default_name_ = 

 ObjectA.set_name(Foo);
 ObjectA:
 name_ = 0xABC Foo
 (0x123) _default_name_ = 

 /* Lets see what happens when you Swap() */
 ObjectB.Swap(ObjectA);
 ObjectA:
 name_ = 0x456 
 (0x123)_default_name_ = 

 ObjectB:
 name_ = 0xABC Foo
 (0x456) _default_name_ = 

 delete ObjectB; // good!
 delete ObjectA; // crash! Because name_ != _default_name_, delete
 name_ is invoked.

 This is not a problem on Linux because even though _default_name_ is a
 static const, it seems the Linux compiler is optimizing for space and
 adding it to a global space and all objects' name_ values point to the
 same address.

 Comments? Anything that can be done on Windows to fix this?

 Regards,
 Anand

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Why to reinvent the wheel ?

[Henner Zeller]

There are some standards that pack many different ways to put things
under one umbrella. This is because they have been decided by
committee with many different companies involved that all want to
bring in 'their way'. The multitude of ways to encode things with
ASN.1 (why there is more than one ?) or to choose between encoding
with tags or without (from what I gathered from this communication)
means that there are more ways than one to do things.

How worth is a standard of communication if there are a myriad ways to
do things ? You end up with implementations that only support part of
it and suddenly the standard becomes worthless because two different
implementations don't support the same options of doing things on both
sides.

This discussion reminds me of SOAP. In the beginning, there was
XML-RPC - extremely simple way to communicate using XML with some
small shortcomings but a developer could get started in seconds
reading a simple example communication.
Then standard committees came in and started developed SOAP: out came
a 'standard' that easily is printed 5000 pages with all different ways
to encode things in XML, different transport schemes etc. Same problem
for many years: many implementations that all couldn't speak to each
other. Complicated to use (Yes and I wrote a book about it - I'll
never touch SOAP again).
It got a bit better but people moved on and don't use SOAP anymore.

Protocol buffers are simple and there is only one way to do things.
Simplicity usually wins over developers. This is why they developed
Protocol Buffers at Google in the early 2000s. They're putting it out
here for others to use, but you don't have to.

On Tue, Nov 9, 2010 at 06:44, Kalki70 kalki...@gmail.com wrote:
 Oh, I just found out that you are the developer. It seems I am not the
 only one who thinks you reinvented the wheel :

 http://google-opensource.blogspot.com/2008/07/protocol-buffers-googles-data.html

 As someone mentioned there :

 The apparent complexity of ASN.1 is largely due to its flexibility -
 if you're using only the sort of functionality that pbuffer gives you,
 it would be pretty much the same, I would think.

 Luis.

 On Nov 9, 2:42 am, Kenton Varda ken...@google.com wrote:
 My understanding of ASN.1 is that it has no affordance for forwards- and
 backwards-compatibility, which is critical in distributed systems where the
 components are constantly changing.

 On Mon, Nov 8, 2010 at 7:34 PM, Kalki70 kalki...@gmail.com wrote:
  Hello,

  I just discovered this developers tool, and I can't understand why it
  was invented. Why didn't Google use ASN.1, which is standard and it is
  used for this, to make a language, platform independent description of
  data to be enconded later as XML, or different binary formats, that
  can be faster and more efficient?

  All this is like reinventing ASN.1

  For instance, the example shown on the web page :

  message Person {
   required string name = 1;
   required int32 id = 2;
   optional string email = 3;

   enum PhoneType {
     MOBILE = 0;
     HOME = 1;
     WORK = 2;
   }

   message PhoneNumber {
     required string number = 1;
     optional PhoneType type = 2 [default = HOME];
   }

  In ASN.1, which is a standard used for over 20years, the same could be
  written similar to this (I haven't used it in a while, maybe I made
  some mistakes ):

  PhoneType ::= ENUMERATED { MOBILE, HOME, WORK }
  PhoneNumber ::= SEQUENCE
  {
     number [1] IA5String ,
     phone [2] PhoneType DEFAULT HOME
  }

  Person ::= SEQUENCE
  {
  name   [1] IA5String ,
  id          [2] INTEGER,
  email   [3] OCTET STRING OPTIONAL
  phone [4] SET OF PhoneNumber
  }

  Best regards,

  Luis

  --
  You received this message because you are subscribed to the Google Groups
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to
  protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
  .
  For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Read selective fields without deserializing the entire object?

[Henner Zeller]

On Fri, Oct 29, 2010 at 16:13, Ashwin Jayaprakash
ashwin.jayaprak...@gmail.com wrote:
 Is there a way to read selective fields without deserializing the
 entire object?

 My serialized message is stored in memory as a (Heap)ByteBuffer.

 I'm trying to find out if any of the (de-)serialization frameworks
 allow a way to read specific fields without having to deserialize the
 entire message. The message has almost 200 fields - mostly Strings and
 a few long and int fields. Deserialization is expensive and creates
 too much garbage considering the fact that I just need to read 1-2
 fields.

The fields are not 'indexed' in the beginning of the message so that
it is not simple to just jump to a particular field.

What you can do is to direclty use the CodedInputStream to parse
through the message, tag by tag and only read the tags you are
interested in. As soon as you have all fields together, you can stop
reading. So it is a bit manual work involved.
Also, if you want to be faster and skip bigger parts of the message,
make sure to create sub-messages within that message, maybe there is a
logical grouping of your fields ? You can skip a sub-message at once
if you don't need it instead of iterating through all fields in it,
which is faster.

H.




 Thanks in advance.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] including common message definitions across .proto files

[Henner Zeller]

On Tue, Oct 26, 2010 at 11:29, Paul mjpabl...@gmail.com wrote:
 Hi,
 Suppose I have two files, file1.proto and file2.proto, and I have a
 message defined in file1 that I want to use in file2.  How would I
 include the message from file1?  Is there an include statement I
 would use as in C or C++?

'import' is the thing you're looking for.

http://code.google.com/apis/protocolbuffers/docs/proto.html#other


  file1.proto 
 message M1 {
   optional string id = 1;
 }

  file2.proto 
 message M2 {
  optional string id = 1;
  optional M1 m1;   // how do I include this?
 }

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: omitting tag numbers

[Henner Zeller]

On Mon, Oct 25, 2010 at 16:10, maninder batth batth.manin...@gmail.com wrote:
 I disagree. You could encode field name in the binary. Then at de-
 serialization, you can read the field descriptor and reconstruct the
 field. There is absolutely no need for tags. They are indeed
 cumbersome.

If you include the field name, then your throw out part of the
advantages of protocol buffers out of the window: speed and compact
binary encoding.


 On Oct 22, 6:02 pm, Henner Zeller henner.zel...@googlemail.com
 wrote:
 On Fri, Oct 22, 2010 at 15:01, Paul mjpabl...@gmail.com wrote:
  Hi,

  This may seem like a basic question, but I find having to label
  the .proto file with unique tag numbers for each field a little
  cumbersome, especially if there are a lot of fields.

  message Person {
   required string name = 1;
   required int32 id = 2;
   optional string email = 3;
  }

  Can I define a .proto file without the tag numbers, like so?

  message Person {
   required string name;
   required int32 id;
   optional string email;
  }

 No.

 The reason for this explicit definition is that the protocol buffer is
 'future compatible': fields written with a particular tag will always
 be written with that tag. Consider you want to re-structure the fields
 in your proto buffer to say (Id, name, email) ... then they would get
 a different 'automatic' tag assigned and you wouldn't be able to read
 files written with older binaries. If the tags are assigned, then
 re-arranging fields in the file does not matter.

 -h





  Thanks,
  Paul

  --
  You received this message because you are subscribed to the Google Groups 
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to 
  protobuf+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/protobuf?hl=en.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: omitting tag numbers

[Henner Zeller]

On Mon, Oct 25, 2010 at 16:11, Henner Zeller
henner.zel...@googlemail.com wrote:
 On Mon, Oct 25, 2010 at 16:10, maninder batth batth.manin...@gmail.com 
 wrote:
 I disagree. You could encode field name in the binary. Then at de-
 serialization, you can read the field descriptor and reconstruct the
 field. There is absolutely no need for tags. They are indeed
 cumbersome.

 If you include the field name, then your throw out part of the
 advantages of protocol buffers out of the window: speed and compact
 binary encoding.

.. and you would never be able to rename fields either.



 On Oct 22, 6:02 pm, Henner Zeller henner.zel...@googlemail.com
 wrote:
 On Fri, Oct 22, 2010 at 15:01, Paul mjpabl...@gmail.com wrote:
  Hi,

  This may seem like a basic question, but I find having to label
  the .proto file with unique tag numbers for each field a little
  cumbersome, especially if there are a lot of fields.

  message Person {
   required string name = 1;
   required int32 id = 2;
   optional string email = 3;
  }

  Can I define a .proto file without the tag numbers, like so?

  message Person {
   required string name;
   required int32 id;
   optional string email;
  }

 No.

 The reason for this explicit definition is that the protocol buffer is
 'future compatible': fields written with a particular tag will always
 be written with that tag. Consider you want to re-structure the fields
 in your proto buffer to say (Id, name, email) ... then they would get
 a different 'automatic' tag assigned and you wouldn't be able to read
 files written with older binaries. If the tags are assigned, then
 re-arranging fields in the file does not matter.

 -h





  Thanks,
  Paul

  --
  You received this message because you are subscribed to the Google Groups 
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to 
  protobuf+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/protobuf?hl=en.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Possible Memory Leaks

[Henner Zeller]

On Mon, Oct 25, 2010 at 17:46, rthompson.dtisoft@gmail.com
rthompson.dtisoft@gmail.com wrote:
 I ran Valgrind on our application and found quite a few possibly
 lost bytes traces.  Here is an example of one

 ==1473==    at 0x40263A0: operator new(unsigned int)
 (vg_replace_malloc.c:214)
 ==1473==    by 0x4370EB8:
 google::protobuf::DescriptorPool::DescriptorPool(google::protobuf::DescriptorDatabase*,
 google::protobuf::DescriptorPool::ErrorCollector*) (descriptor.cc:768)
 ==1473==    by 0x43C8B35:
 google::protobuf::compiler::Importer::Importer(google::protobuf::compiler::SourceTree*,
 google::protobuf::compiler::MultiFileErrorCollector*) (importer.cc:
 188)

Make sure to call ShutdownProtobufLibrary() to de-allocate memory
allocated in global constructors.

There are no known leaks (I use it daily with tons of data and
protobufs never had been an issue wrt. leaks).

-h


 and I have confirmed that I have freed the Importer().

 Is this something to worry about, or can I count on google::protobuf
 not having any memory leaks?

 Thanks in advance!

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] protocol buffer within a protocol buffer from C++ to Java

[Henner Zeller]

On Mon, Oct 25, 2010 at 18:45, Paul mjpabl...@gmail.com wrote:
 Hi,
 I am trying to serialize a protocol buffers message into a string that
 is contained in another protocol buffer.  I am doing the serialization
 on the C++ side.

 // This is the outer protocol buffer
 message Measurement {
  required string id = 1;
  optional string meas_rec_str = 2;
 }
 Measurement m;

 // This is the protocol buffer message I want to serialize into a
 string and store inside meas_rec.
 message MeasRec {
  optional int id = 1;
 }

 To serialize the MeasRec message, I am doing:
 MeasRec meas_rec;
 meas_rec.set_id(100);
 char* str_buf;
 str_buf = new char[meas_rec.ByteSize()];
 string str = str_buf;
 meas_rec.SerializeToString(str);
 m.set_meas_rec_str(str);

 However, when I do this, I encounter the following error on the C++
 side:

 libprotobuf ERROR google/protobuf/wire_format.cc:1059] Encountered
 string containing invalid UTF-8 data while serializing protocol
 buffer. Strings must contain only UTF-8; use the 'bytes' type for raw
 bytes.

 how do I serialize meas_rec into bytes instead?

'bytes' will be the same std::string type in C++. The only differnce
is that it is not checked for UTF8-ness.


 Also, once I have it on the Java side, how do I deserialize?
 would I use parseFrom(byte[] data) ?

yes.


 Thanks,
 Paul

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] omitting tag numbers

[Henner Zeller]

On Fri, Oct 22, 2010 at 15:01, Paul mjpabl...@gmail.com wrote:
 Hi,

 This may seem like a basic question, but I find having to label
 the .proto file with unique tag numbers for each field a little
 cumbersome, especially if there are a lot of fields.

 message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
 }

 Can I define a .proto file without the tag numbers, like so?

 message Person {
  required string name;
  required int32 id;
  optional string email;
 }

No.

The reason for this explicit definition is that the protocol buffer is
'future compatible': fields written with a particular tag will always
be written with that tag. Consider you want to re-structure the fields
in your proto buffer to say (Id, name, email) ... then they would get
a different 'automatic' tag assigned and you wouldn't be able to read
files written with older binaries. If the tags are assigned, then
re-arranging fields in the file does not matter.

-h


 Thanks,
 Paul

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Do Protocol Buffers read entire collection of messages into memory from file?

[Henner Zeller]

On Tue, Oct 19, 2010 at 06:45, ksamdev ksam...@gmail.com wrote:
 Hi,

 I am wondering how do Protocol Buffers read input files? Is the entire
 file read into memory or some proxy technique is used and entries are
 read only when required?

If you have a sequence of messages you process then you'd put some
container around them in the file. A very simple scheme would be
  lengh-of-next-messagenext-message.
That way you can read the messages one by one. This has been discussed
several times on this list.
(You are free to hide this in some proxy technique implementation
though it will just complicate things without much gain).

The protocol buffer library tries to be as simple as possible in
providing the pure serialization functionality. It provides everything
to allow sending them over the wire or storing them in files, but you
actually would need to do that yourself (adding that to the core
protocol buffer library would be beyond the scope and you might
already have something you would like to store your data in, such as
Berkely DB for keyed data).


 This is a vital feature for large lists, say, some dataset with 10^9
 messages.

I regularly process datasets with more than 10^9 Protocol Buffer
messages and essentially store them the way I described above.
Depending on the content, it helps to use a compressing scheme on the
file level (Many people use GZip streams).
(For larger datasets it actually makes sense to add some sort of CRC
as disks have a noticeable error rate at that size).

-h

 Do Protocol Buffers use any additional archiving technique (zip, tar,
 etc.) to further compress saved information?

 sincerely, Sam.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Memory leak detected in protobuf

[Henner Zeller]

On Thu, Sep 2, 2010 at 08:03, Jean-Sebastien Stoezel
js.stoe...@gmail.com wrote:
 Hello,

 I am observing a memory leak after including the protobufs in my
 project. I would like to investigate whether this leak is due to this
 library or the use I make of it.

 I am using the protobufs (latest version) with VC++, running 3
 threads.
 2 server threads are pending, waiting for connections, another thread
 acts as client, connecting to the servers and sending messages to
 them.
 I tested this code without the protobufs and I did not notice an
 increase in memory usage over time. After adding the protobufs, I see
 a constant increase in memory usage.

 I instrumented the code with VC++ crtdbg library, the tool used to
 detect memory leaks. The tool is reporting memory leaks which I
 believe point to protobuf. I read the crtdbg documentation and if
 _CRTDBG_MAP_ALLOC is defined, the tool should report the name of the
 file where the leak happened. However no file name is provided by the
 tool, it could be because I statically link to the protobufs.

 Anyhow, from the server thread's context, protobufs are stored in a
 vector. A packet delimiter parses data from a socket. Each packet is
 parsed for a single protobuf message.

 Pseudo code:

 m_IncomingMessages is defined as
 std::vectorLFLComProtocol::LFLComProtocolMsg  m_IncomingMessages,
 where LFLComProtocol::LFLComProtocolMsg is a protobuf message.


 // adding a protobuf to the protobuf queue, after a packet of data was
 received
 // Add an element to the vector
 m_IncomingMessages.resize(m_IncomingMessages.size() + 1);
 // Parse the data from the new message
 m_IncomingMessages[m_IncomingMessages.size() -
 1].ParseFromArray(p_Data, dataSize);

 // consuming the protobufs, FIFO style
 m_IncomingMessages.erase(m_IncomingMessages.begin());

Can you create a compilable self contained C++ file including
proto-file (stripped down to the essentials) with the essence of your
code which would make it simpler (for you and people on the list) to
verify if there is a problem ?

Also, I would probably not use protocol buffers as values within
vectors (esp. if they're big - this advice is true for any bigger
object) because you implicitly might call copy constructors if you
resize() the vector and it has to realloc. Rather allocate protos on
the heap (new/delete). This allows for further improvements later,
like having a pool of pre-allocated protobuffers.

-h



 I have checked that the messages are properly queued and dequeued, the
 size of the vector does not increase and stay constant (usually 0
 since the message is consumed right away)

 At this point I am thinking that this way of erasing may be causing
 the leak. From the STL documentation I gather that the vector::erase
 method is supposed to call the destructor for each element... Is this
 good enough to free all the memory allocated for the message?

 Any other idea why this memory leak is happening?

 Jean-Sebastien


 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Status of protobufs

[Henner Zeller]

On Thu, Sep 2, 2010 at 09:59, Jean-Sebastien Stoezel
js.stoe...@gmail.com wrote:
 Hence I'm now using delimiters.

But make sure to use length delimiting: length + message. Don't feel
tempted to just delimit the message with 'special characters' because
of course they might be just part of a message.


 Thanks for the headsup.

 On Sep 2, 2010 11:53 AM, Daniel Wright dwri...@google.com wrote:

 See http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming
 Your solution of trying to parse after each byte received is not just slow,
 it's completely incorrect.  It's entirely possible, and quite likely, that
 if you break the encoded version of a message in half, each half will parse
 successfully on its own, but the contents of the message will be incorrect.

 On Thu, Sep 2, 2010 at 7:29 AM, Jean-Sebastien Stoezel
 js.stoe...@gmail.com wrote:   Hello,  ...

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] How to set a user defined data within a message

[Henner Zeller]

2010/8/19 Julian González julian@gmail.com:
 I have the following:

 message Configuration
 {
        required LogInfo logInfo = 1;
        required ApmCfg apmCfg = 2;
        repeated Rail rail = 3;
        required TriggerInfo triggerInfo = 4;
 }

 I have an object of type Configuration, but I do not see any setter
 for logInfo, apmCfg or triggerInfo or a way to access those members,
 just a way to read them.

which language ?
In C++ you can access the instances with the mutable_XX() accessors
that return a pointer to the object in question:

  config_object-mutable_loginfo()-set_field_in_loginf()


 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: iSeries port

[Henner Zeller]

On Mon, Aug 16, 2010 at 17:04, ury ury.se...@gmail.com wrote:
 Hi,

 It was EBCDIC vs. ASCII issues.

 If anyone is interested in iSeries/zSeries porting issues, or protobuf
 on EBCDIC, I'll be glad to help

In particular if you needed to modify the source and have a patch it
might be interesting to send that change upstream if it is not too
much hassle.


 Thanks


 On Aug 16, 2:18 pm, ury ury.se...@gmail.com wrote:
 Hi,

 I got the thing to compile and link on the iSeries. When I run the
 add_person  C++ example, I get

 libprotobuf ERROR /testc/protobuf/build/./src/google/protobuf/
 desc_datab.C:314] Invalid file descriptor data passed to
 EncodedDescriptorDatabase::Add().
 libprotobuf FATAL /testc/protobuf/build/./src/google/protobuf/desc.C:
 860] CHECK failed: generated_database_-Add(encoded_file_descriptor,
 size):

 Any idea what may cause this?

 Thanks

 On Aug 10, 3:52 pm, ury ury.se...@gmail.com wrote:

  Hi,

  I am trying to port the C++ protobuf runtime library to iSeries. (aka
  AS/400).
  Have anyone does this before ? I'd be glad to hear any advice.

  Thanks !

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Parsing problem using protobuf

[Henner Zeller]

On Wed, Aug 11, 2010 at 22:48, anup anup007s...@gmail.com wrote:
 Hi I am impkementing protobuffers in my Symbian application and during
 parsing I am sending a file to the proto parser but when I am
 collecting my attributes its giving garbage values along with the real
 data. I  dont know what may be the problem. Here are the functions
 which I used

 com::sidebar::protobuf::Executions* ie2;
 std::ifstream myfile;

 myfile.open (C:\\data\\file, std::ios::in);

 ie2-ParseFromIstream(myfile);

 if (execution.has_title())

Here you use execution, before you had ie2 as a pointer. You should
make sure that the examples you give are actually compiling, otherwise
it is really hard to figure out what the actual problem is.

 {

 const ::std::string titlevalue = execution.title();

 TPtrC8 ptr1(reinterpret_castconst TUint8* (titlevalue));

Looks like you want to access the string of the titlevalue. That is a
std::string, so you can't just cast that to a char*; you might want to
access the content of the stringt with titlevalue.c_str(), which
returns the nul-terminated string you seem to look for (
http://www.sgi.com/tech/stl/basic_string.html )



 iprotobuf.buffer_title.Copy(ptr1);


 I have attributes like content_type, title, artist,singer... but when
 i am collecting these informations after parsing I am getting garbage
 values when i debugged the code i checked the memory locations.I dont
 know why is these happening.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: What's the point of Protocol Buffers?

[Henner Zeller]

On Thu, Jul 22, 2010 at 07:50, Tim Acheson tim.ache...@gmail.com wrote:
 If you can show me a format which offers faster serialisation or
 deserialisation than JSON in a .NET application, I'd be impressed! :)

If the speed and size of JSON serialization is good enough for you,
stick with it.

 Although I haven't heard anybody experiencing problems with the
 performance of either direction in .NET with JSON or XML, the
 libraries provided by the framework offer excellent performance in
 both directions. :)

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Security Layer to make sure proto objects are not tampered when transferred over https

[Henner Zeller]

On Wed, Jul 21, 2010 at 03:32, Prakash Rao prakashrao1...@gmail.com wrote:
 Is there a way to add security layer to protocol buffer if I'm writing
 my proto objects to https output stream? In java we have sealed
 objects and we can have signature attached to it to make sure objects
 are not tampered on the receiving end. Is there something similar in
 protocol buffer as well?

 I’m just trying understand what all different options that protocol
 buffer provides as far security is concerned when compared to web
 services to make sure data is not tampered. Web services comes with
 good security models (such XML signature, SAML) to make sure data is
 not tampered.

Protocol buffers do the encoding. They expliclitly don't add
additional features that belong to the conceptual level of the
transport channel. Do one thing and do it right.

You can have this as simple as adding a HMAC to your message or use
channels that already provide that (like an SSL channel with the
appropriate options turned on). It would be just a confusing mess if
each encoding scheme for data would add yet another implementation of
what is already there (From your quote, apparently XML/SAML adds to
that mess, but it doesn't mean that it is smart to do).

It is sad that many designers of protocols mix up these layers.

-h


 Regards,
 Prakash

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Scalar types

[Henner Zeller]

2010/7/21 Julian González julian@gmail.com:
 Does protocol buffers support an scalar type of one byte or two byte
 length? lets say an int16, int8?

No. But note that integers are 'varint' encoded, so if you only give a
small value, it will only eat up a small amount of bits.
So a number in the range of 0..127 will actually be encoded as only
one byte etc.
Note that for instance int32 and int64 are actually encoded the same
on the wire for the same reason. This allows as well to do a backward
compatible change from int32 to int64 for a protocol field.

Details on the encoding:
http://code.google.com/apis/protocolbuffers/docs/encoding.html#varints

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: C++ syntax: how to set a singular enum field

[Henner Zeller]

On Fri, Jul 16, 2010 at 06:33, Evan Jones ev...@mit.edu wrote:
 On Jul 16, 2010, at 5:47 , mark.t.macdon...@googlemail.com wrote:

 if (stabiliser.retraction()==device::HOUSED) coutTrue\n;
 //but this doesn't compile
 stabiliser.set_retraction(device::RETRACTED);
 }

 See the generated .h file to see what might be going wrong. For enums, a
 .set_* method should be generated. Note that the .proto you sent doesn't
 have RETRACTED defined, so maybe that is your problem?

.. and given that you indeed have RETRACTED defined, look out for
global #define somewhere - they might be the source for hard-to-see
problems with ALL_UPPERCASE enumeration values.
Best to debug is to run your compiler in preprocessing mode and eyball
the output along the non-compiling lines.

(and I assume 'device' is indeed RR_system ? And 'stabiliser' is 'dev'
?... please don't try to partially redact the proto-buffer and code
because then it is really hard to find out what the problem is.
Provide a self contained example for these cases).

-h


 Evan

 --
 Evan Jones
 http://evanjones.ca/

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Design advice: variability (at runtime) on the number of fields in a message

[Henner Zeller]

On Wed, Jul 14, 2010 at 14:19, mark.t.macdon...@googlemail.com
mark.t.macdon...@googlemail.com wrote:
 Let's assume I'm designing a distributed (client-server) application.
 The server owns a bunch of parameters. The client (GUI) is to be
 periodically updated when any of their values change.

 I could:

 i) send a small wire message containing one field each time it changes
 ii) periodically send a big wire message containing only the fields
 which have changed
 iii) periodically send a big wire message containing all current
 values

 Which approach is best and why?

This is a broad question and entirely depends on your application, the
size of the data to be transferred, the frequency of changes, the
'cheapness' of network, latency, the number of fields...
Any of your options you mention above can be the best for your application.
You need to be clear what you want to optimize for and then implement that.

 Should optional fields only be added/removed at design-time, or is it
 ok to vary their presence at runtime too?

You can send additional fields over the wire that are not specified in
the (think extensions:
http://code.google.com/apis/protocolbuffers/docs/proto.html#extensions
)
But of course, your client/server need to be able to make sense out of it.

Best to start with is probably to write your proto buffer with a bunch
of optional fields and fill out the ones you're interested at the time
(so design time approach). Note, that only the optional fields that
you fill are actually transferred over the wire, so it is 'cheap' to
have many fields defined if you only send a few of them.


 How do I learn best practices for designing messages and exchange
 protocols?

Be clear about what you want to achieve and write down some parameter
you want to control (such as bandwidth). Then apply good judgement.
And over time, you collect best practices :)

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] hasXxx methods

[Henner Zeller]

Hi,
2010/6/22 Miguel Muñoz swingguy1...@yahoo.com:
 Comrades,
   I've been setting a value of a single optional field in a message with
 many optional fields. In my diagnostic code I'm calling the hasXxx() methods
 to find out which field is set, but it's telling me they're all set. Is this
 a bug or a bad design? I can't find any circumstance where the methods will
 return false.

The has_xxx methods should only return true if the optional field has
been set. This is well tested and should work. If you find it works
differently for you, can you provide a small, self-contained example
that exposes the problem ?

-H

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: question about SerializeToString output for cpp

[Henner Zeller]

I have no idea about the hadoop pipes, but wouldn't it be easier to
just add an implementation of the pipes to be length delimited instead
of trying to mundge the binary data you're sending ? You'd probably
loose a lot of speed if you would edit the binary data afterwards.
After all, this is what the available source code for hadoop is good for :)


On Fri, Jun 11, 2010 at 10:01, mistlike mistl...@gmail.com wrote:
 thanks, i try to use protocol buffer for hadoop c++ pipes, however, c+
 + pipes interface for hadoop may have problem because input for hadoop
 mapper must line by line with \n between every one item for mapper
 input.
 i try to make following change:

 data input for hadoop make following change,
 \n -- \\n
 \0 --\\0
 \\ --

 and while data transfer to mapper , then make reverse change, and use
 protocol buffer ParseFromArray to parse buffer.

 I am not sure is there other better solution for hadoop c++ pipes when
 use protocol buffer.


 On Jun 11, 11:54 pm, Jason Hsueh jas...@google.com wrote:
 Yes, '\0' may appear in the binary format. output-size() will return the
 correct result: C++ string can store null characters without any issue.
 However, strlen(output-c_str()) and other calls that assume null-terminated
 C strings will not.

 On Thu, Jun 10, 2010 at 11:30 PM, mistlike mistl...@gmail.com wrote:
  bool SerializeToString(string* output) const;: serializes the message
  and stores the bytes in the given string. Note that the bytes are
  binary, not text; we only use the string class as a convenient
  container.

  string output store binary data, and whether has \0 in output
  string?
  such as if i want to use output-size() or strlen(output-c_str())
  get its length.
  i see many \n in binary data, but not sure it whether has \0,
  since if has \0 may have some problem for string to store it.

  --
  You received this message because you are subscribed to the Google Groups
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to
  protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
  .
  For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Message was missing required fields

[Henner Zeller]

On Fri, Apr 16, 2010 at 11:05, SyRenity stas.os...@gmail.com wrote:
 Hi.

 I'm getting occasionally the following error below in my Java app:

 com.google.protobuf.InvalidProtocolBufferException: Message was
 missing required fields.  (Lite runtime could not determine which
 fields were missing).
        at
 com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:
 81)
        at classes.cameraInfoProto$camera
 $Builder.buildParsed(cameraInfoProto.java:242)
        at classes.cameraInfoProto$camera$Builder.access
 $11(cameraInfoProto.java:238)
        at classes.cameraInfoProto
 $camera.parseFrom(cameraInfoProto.java:133)
        at app.jSockets.FetcherSockets
 $ResponseThread.readMessage(FetcherSockets.java:386)
        at app.jSockets.FetcherSockets
 $ResponseThread.run(FetcherSockets.java:268)


 I double-checked my code, but I have only a single required field, and
 I'm always filling it up in the C++ app.

Just a wild guess: is it an enumeration field that you extended
recently ? If you set the new value in the C++ app, but the Java code
is still compiled with the old version that does not know about that
enumeration value, that non-matching enum field is stashed away in an
'unknown field' instead of ending up in the enum which the application
might not be able to handle - thus the required field is not set.
Could it be this ?

-h


 Any idea how to diagnose it?

 Thanks.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Protocol Buf compliation problem (Linux 2.4)

[Henner Zeller]

gcc 3.2.3 is a pretty ancient, seven years old compiler. Can't you
just install new version ?

On Tue, Apr 13, 2010 at 03:38, Nir Belinky belink...@gmail.com wrote:
 I downloadd protobuf 2.3.0 and tried to compile it on linux 2.4, I
 receive the following errors:

 OS:
 Linux version 2.4.21-4.ELsmp (bhcomp...@daffy.perf.redhat.com) (gcc
 version 3.2.3 20030502 (Red Hat Linux 3.2.3-20)) #1 SMP Fri Oct 3
 17:52:56 EDT 2003

 Compiler:
 Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.2.3/specs
 Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --
 infodir=/usr/share/info --enable-shared --enable-threads=posix --
 disable-checking --with-system-zlib --enable-__cxa_atexit --host=i386-
 redhat-linux
 Thread model: posix
 gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-20)


 Error Message:
 libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -pthread -Wall -Wwrite-
 strings -Woverloaded-virtual -Wno-sign-compare -O2 -g -DNDEBUG -MT
 extension_set.lo -MD -MP -MF .deps/extension_set.Tpo -c google/
 protobuf/extension_set.cc  -fPIC -DPIC -o .libs/extension_set.o
 google/protobuf/repeated_field.h: In member function `void
   google::protobuf::internal::RepeatedPtrFieldBase::MergeFrom(const
   google::protobuf::internal::RepeatedPtrFieldBase) [with
 TypeHandler =

 google::protobuf::RepeatedPtrFieldstd::string::TypeHandlerstd::string]':
 google/protobuf/repeated_field.h:845:   instantiated from `void
 google::protobuf::RepeatedPtrFieldElement::MergeFrom(const
 google::protobuf::RepeatedPtrFieldElement) [with Element =
 std::string]'
 google/protobuf/extension_set.cc:618:   instantiated from here
 google/protobuf/repeated_field.h:678: no matching function for call to
 `
   google::protobuf::internal::RepeatedPtrFieldBase::Get(int) const'
 make[2]: *** [extension_set.lo] Error 1
 make[2]: Leaving directory `/amd/galaxy/vol/vol0/users5/nirbe/
 cpprotobuf/Linux_2.4/protobuf-2.3.0/src'
 make[1]: *** [all-recursive] Error 1
 make[1]: Leaving directory `/amd/galaxy/vol/vol0/users5/nirbe/
 cpprotobuf/Linux_2.4/protobuf-2.3.0'
 make: *** [all] Error 2
 6.950u 0.460s 0:08.00 92.6%     0+0k 0+0io 15012pf+0w


 I think it is a gcc problem. Is there any patch for this?

 Thanks,
 Nir

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: compile issue with import statements having path info

[Henner Zeller]

On Tue, Apr 13, 2010 at 11:24, CB cn...@verizon.net wrote:

 Workarounds aside, there is still a bug to be fixed.

 If you wish to argue that the .proto files or the -I options I passed
 to proto are invalid, then protoc should have declared an error.

 If you wish to argue that the .proto files and the -I options I passed
 to proto are valid, then the c++ code emitted by protoc should have
 compiled without error.

 If the error was simply an issue of -I options passed to protoc/g++
 that would be one thing, but in this case, protoc generated a call to
 a function that doesn't even exist in the generated code set.  That's
 a major bug.

Providing a patch that does what you think it should do instead is
probably a good way to start the discussion and get it fixed.

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] i am not able to compile the files which includes import statements

[Henner Zeller]

On Fri, Apr 9, 2010 at 04:09, prashant waykar waykar.prash...@gmail.com wrote:
 Hello,
 can someone please help me out as i am new to PB.
 i am getting compile errors in the files which consists import
 statements
 Thanks in advance

You need to be a bit more specific. So if you mention errors, it would
be good if you paste them to the email so that we can have a look at
it.
So give the proto file you have and the error you see.

Might just be a wrong relative path or something.

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Steps for installing the java set of the compiler libraries

[Henner Zeller]

On Tue, Mar 30, 2010 at 08:16, Suvojit chanda.c...@gmail.com wrote:
 can anyone please help me with the set of instructions for installing
 the protobuf compiler compatible with the java version.I did go
 through the readme file, but does evrything needs to be done thru
 commmand line or is there some GUI version of it for installing the
 compiler.

What failed when you ran the command line ?

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Serializing nested messages

[Henner Zeller]

HI,
On Mon, Mar 29, 2010 at 23:20, Nader Salehi nadersale...@gmail.com wrote:


 On Mar 29, 10:59 pm, Henner Zeller henner.zel...@googlemail.com
 wrote:
 On Mon, Mar 29, 2010 at 22:53, Nader Salehi nadersale...@gmail.com wrote:
  These are valid points, but I need to work within the constraints that
  I have; I need to write serialized buffers into different files.  I
  might end up serializing the second time, but still would like to know
  if there is a way which is NOT a hack to avoid this.

 mmh, but what is wrong in having two files open, wrap them in streams
 and serialize a() and b() into these individually ? No way to hack
 around, two files you write to, minimal copying of data from a buffer
 to a file.
 So I guess you need to specify exactly what your constraints are.


 Let's say that you are dealing with a spaghetti code and want to deal
 minimal changes :)

 For that reason, I want to avoid wrapping the fds in streams.  Also,
 what I'm reading from your comment is that there is no explicit API to
 allow me to grab the enclosed message and that I'm better off
 serializing a second time.  If true, then it's a good enough take-away
 for me also.

Or to put it this way: I never had the need to :) I'd just go ahead
and serialize the messages individually. You will probably not have
any measurable performance impact compared to serializing everything
at once and then splitting things apart.
The added benefit is as well that you only need to allocate a smaller
buffer for your serialization.

-h


 Thanks,
 Nader

 -h



  Nader

  On Mar 29, 2:41 pm, Henner Zeller henner.zel...@googlemail.com
  wrote:
  On Mon, Mar 29, 2010 at 10:31, Nader Salehi nadersale...@gmail.com 
  wrote:
   Hi,

   In my code, I have a PB message which encompass other PB messages.
   For instance,

   Protocol Buffer File
   ===
   message A {
    required int32 a = 1;
   }

   message B {
    required int32 b = 1;
   }

   message C {
    required A a = 1;
    required B b = 2;
   }

   C++ File
   
   #include PB File.pb.h

   using namespace whatever

   int main ()
   {
    C c;
     ...
    char buff[MAX_SIZE]
    int len = c.SerializeToArray(buff, MAX_SIZE);
    write(fd, buff, len);
    ...
   }

   As part of the logic, I need to write into separate files the
   serialized equivalent of c.a and c.b.  Of course you can always call
   c.a().SerializeToArray() but that requires additional CPU time which I
   would like to avoid.  Is there a non-hackish way of getting the offset
   of c.a in the serialized buffer?  Can I use the offset and
   c.a().ByteSize() to write the message into the file descriptor?

  You might want to write the data directly to the stream with the
  available streams instead of copying into a buffer first.
  Second, I wouldn't worry at all about the required CPU serializing
  c.a() and c.b() individually; you should first measure if it actually
  makes a difference before thinking of optimizing it in a hackish way.

  -h

  --
  You received this message because you are subscribed to the Google Groups 
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to 
  protobuf+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/protobuf?hl=en.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Serializing nested messages

[Henner Zeller]

On Mon, Mar 29, 2010 at 10:31, Nader Salehi nadersale...@gmail.com wrote:
 Hi,

 In my code, I have a PB message which encompass other PB messages.
 For instance,

 Protocol Buffer File
 ===
 message A {
  required int32 a = 1;
 }

 message B {
  required int32 b = 1;
 }

 message C {
  required A a = 1;
  required B b = 2;
 }

 C++ File
 
 #include PB File.pb.h

 using namespace whatever

 int main ()
 {
  C c;
   ...
  char buff[MAX_SIZE]
  int len = c.SerializeToArray(buff, MAX_SIZE);
  write(fd, buff, len);
  ...
 }

 As part of the logic, I need to write into separate files the
 serialized equivalent of c.a and c.b.  Of course you can always call
 c.a().SerializeToArray() but that requires additional CPU time which I
 would like to avoid.  Is there a non-hackish way of getting the offset
 of c.a in the serialized buffer?  Can I use the offset and
 c.a().ByteSize() to write the message into the file descriptor?

You might want to write the data directly to the stream with the
available streams instead of copying into a buffer first.
Second, I wouldn't worry at all about the required CPU serializing
c.a() and c.b() individually; you should first measure if it actually
makes a difference before thinking of optimizing it in a hackish way.

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Serializing nested messages

[Henner Zeller]

On Mon, Mar 29, 2010 at 22:53, Nader Salehi nadersale...@gmail.com wrote:
 These are valid points, but I need to work within the constraints that
 I have; I need to write serialized buffers into different files.  I
 might end up serializing the second time, but still would like to know
 if there is a way which is NOT a hack to avoid this.

mmh, but what is wrong in having two files open, wrap them in streams
and serialize a() and b() into these individually ? No way to hack
around, two files you write to, minimal copying of data from a buffer
to a file.
So I guess you need to specify exactly what your constraints are.

-h


 Nader

 On Mar 29, 2:41 pm, Henner Zeller henner.zel...@googlemail.com
 wrote:
 On Mon, Mar 29, 2010 at 10:31, Nader Salehi nadersale...@gmail.com wrote:
  Hi,

  In my code, I have a PB message which encompass other PB messages.
  For instance,

  Protocol Buffer File
  ===
  message A {
   required int32 a = 1;
  }

  message B {
   required int32 b = 1;
  }

  message C {
   required A a = 1;
   required B b = 2;
  }

  C++ File
  
  #include PB File.pb.h

  using namespace whatever

  int main ()
  {
   C c;
    ...
   char buff[MAX_SIZE]
   int len = c.SerializeToArray(buff, MAX_SIZE);
   write(fd, buff, len);
   ...
  }

  As part of the logic, I need to write into separate files the
  serialized equivalent of c.a and c.b.  Of course you can always call
  c.a().SerializeToArray() but that requires additional CPU time which I
  would like to avoid.  Is there a non-hackish way of getting the offset
  of c.a in the serialized buffer?  Can I use the offset and
  c.a().ByteSize() to write the message into the file descriptor?

 You might want to write the data directly to the stream with the
 available streams instead of copying into a buffer first.
 Second, I wouldn't worry at all about the required CPU serializing
 c.a() and c.b() individually; you should first measure if it actually
 makes a difference before thinking of optimizing it in a hackish way.

 -h

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] int32 negative numbers

[Henner Zeller]

On Sun, Mar 21, 2010 at 08:05, Adam Kwintkiewicz
adam.kwintkiew...@gmail.com wrote:
 ... for a negative number, the resulting varint is always ten bytes long
 ...

Reason for that is the varint encoding: it only encodes the bits that
are set in an integer. For small positive values that results in a
more compact format. However, negative values always have the very
first bit set (the sign bit), so these values end up to be longer.

If you have values that are centering around zero but whose absolute
values usually don't use the full range, then the 'sint32' would be
probably a better encoding for you: it is done in 'zigzag'-encoding
that uses short encoding for small absolute values and longer for
larger absolute values.
If your numbers are big or pretty random, then you might consider fixed32.

Note however, that changing the type from int32 to sint32 or fixed32
are not compatible - so if you've already data stored that way or have
running services that talk RPC in that way, you need an upgrade path;
probably adding a new field with the new encoding, setting it in
parallel for some time (until all other users are gone). If you have
stored data the old way and don't want to recode than you've to
forever test - on reading - if the 'old' field exists and take that
value.

-h


 I didn't saw that part.
 Thanx

 2010/3/21 Evan Jones ev...@mit.edu

 On Mar 21, 2010, at 8:46 , adamdms wrote:

 I am wonder why int32 field (with negative value) has 10 bytes?
 10 - field No 2, wire type 0
 FD FF FF FF FF FF FF FF FF 01 - field value = -3

 Can someone explain it to me?

 See: http://code.google.com/apis/protocolbuffers/docs/encoding.html#types

 Hope this helps,

 Evan

 --
 Evan Jones
 http://evanjones.ca/


 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.


-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] do the protocol buffer support java generics?

[Henner Zeller]

Hi,
 do the protocol buffer support java generics? if support, how can I
 define the .proto file ?

Not sure what you mean here. Lists for instance you pass to the
builder or get from a proto buff object have the type of the content
as generic parameter.

There are no 'template parameters' for proto buffer message
definitions in .proto files if you mean that.
Can you expand a bit what you would like to do ?

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] signing protobuf messages

[Henner Zeller]

On Thu, Mar 18, 2010 at 10:36, maxwolf waxm...@gmail.com wrote:
 I wonder if protobuf messages are safe to be crypto signed?

If you just sign the content of a message, then this should be an
operation that should not require that a message is generated the same
for different implementations, right ? Meaning, you have some binary
encoded message generated by some implementation and its signature so
you can compare if that content is indeed signed by the owner. You
directly compare that binary encoding with that signature.

But looks like you're looking for a bit stronger guarantee: that you
can just operate only on the hash of some message and want that to be
identical for messages with the same content generated by different
implementations.

 More
 precisely - will certain message serialized from the same set of field
 values be exactly the same for every platform/language?

The encoding scheme does not enforce this per se: it is entirely valid
to send fields in a different order over the wire and thus have
equivalent messages whose binary encoding is different.
However, all current Google implementations actually encode the same
messages the same way - I guess too many people relied on being able
to reliably store hash values of messages (Kenton needs to confirm
this, but I am pretty sure).
With other words: there is no strong guarantee but in practice, it works :)

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] How to send classes defined in proto over a socket

[Henner Zeller]

You're trying to send the raw C++ memory representation of a protocol object
  send(socket, data_snd, sizeof(data_snd), 0)
That doesn't work. And is the reason why protocol buffers are there in
the first place: they provide serialization techniques.

You need to use the serialization methods that SerializeToStream or
SerializeToString that generate a platform independent string
serialization from the content of the protocol buffer. On the other
side deserialize with ParseFromString/ParseFromStream

  http://code.google.com/apis/protocolbuffers/docs/overview.html

-h
On Fri, Mar 12, 2010 at 09:29, mk apollo...@gmail.com wrote:
 Hi,

 I am trying to send a proto over a socket, but i am getting
 segmentation error.  Could someone please help and tell me what is
 wrong with this?

 file.proto
 message data{
        required string x1 = 1;
        required uint32 x2 = 2;
        required float x3 = 3;
 }

 client.cpp
 ...
    // class defined in proto
    data data_snd;
    data data_rec;

    char *y1 = operation1;
    uint32_t y2 = 123 ;
    float y3 = 3.14;

    // assigning data to send()
    data_snd.set_x1(y1);
    data_snd.set_x2(y2);
    data_snd.set_x3(y3);

    //sending data to the server
    if (send(socket, data_snd, sizeof(data_snd), 0)  0) {
       cerr  send() failed ;
       exit(1);
     }

     //receiving data from the client
     if (recv(socket, data_rcv, sizeof(data_rcv), 0)  0) {
        cerr  recv() failed;
        exit(1);
     }

     //printing received data
     cout  data_rec.x1()  \n;
     cout  data_rec.x2()  \n;
     cout  data_rec.x3()  \n;
 ...

 server.cpp
  ...

     //receiving data from the client
     if (recv(socket, data_rcv, sizeof(data_rcv), 0)  0) {
        cerr  recv() failed;
        exit(1);
     }

     //printing received data
     cout  data_rec.x1()  \n;
     cout  data_rec.x2()  \n;
     cout  data_rec.x3()  \n;

   // assigning data to send()
    data_snd.set_x1(data_rec.x1());
    data_snd.set_x2(data_rec.x2());
    data_snd.set_x3(data_rec.x3());

    //sending data to the server
    if (send(socket, data_snd, sizeof(data_snd), 0)  0) {
       cerr  send() failed ;
       exit(1);
     }
 ...

 Thanks for help and replies-

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: How to send classes defined in proto over a socket

[Henner Zeller]

On Fri, Mar 12, 2010 at 10:20, mk apollo...@gmail.com wrote:
 Thanks for reply !!

 Could you please give an example? I tried several ways from this
 morning but  as I am a new user of protbuf, I don't see what way to
 follow ... Thanks!

Have a look at the documentation, it is full of examples how to
serialize/deserialize.

-h



 On Mar 12, 6:38 pm, Kenton Varda ken...@google.com wrote:
 Wait.



 On Fri, Mar 12, 2010 at 9:29 AM, mk apollo...@gmail.com wrote:
  Hi,

  I am trying to send a proto over a socket, but i am getting
  segmentation error.  Could someone please help and tell me what is
  wrong with this?

  file.proto
  message data{
         required string x1 = 1;
         required uint32 x2 = 2;
         required float x3 = 3;
  }

  client.cpp
  ...
     // class defined in proto
     data data_snd;
     data data_rec;

     char *y1 = operation1;
     uint32_t y2 = 123 ;
     float y3 = 3.14;

     // assigning data to send()
     data_snd.set_x1(y1);
     data_snd.set_x2(y2);
     data_snd.set_x3(y3);

     //sending data to the server
     if (send(socket, data_snd, sizeof(data_snd), 0)  0) {

 You can't do that.

 You have to use one of the Serialize methods.

        cerr  send() failed ;
        exit(1);
      }

      //receiving data from the client
      if (recv(socket, data_rcv, sizeof(data_rcv), 0)  0) {
         cerr  recv() failed;
         exit(1);
      }

      //printing received data
      cout  data_rec.x1()  \n;
      cout  data_rec.x2()  \n;
      cout  data_rec.x3()  \n;
  ...

  server.cpp
   ...

      //receiving data from the client
      if (recv(socket, data_rcv, sizeof(data_rcv), 0)  0) {
         cerr  recv() failed;
         exit(1);
      }

      //printing received data
      cout  data_rec.x1()  \n;
      cout  data_rec.x2()  \n;
      cout  data_rec.x3()  \n;

    // assigning data to send()
     data_snd.set_x1(data_rec.x1());
     data_snd.set_x2(data_rec.x2());
     data_snd.set_x3(data_rec.x3());

     //sending data to the server
     if (send(socket, data_snd, sizeof(data_snd), 0)  0) {
        cerr  send() failed ;
        exit(1);
      }
  ...

  Thanks for help and replies-

  --
  You received this message because you are subscribed to the Google Groups
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to
  protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
  .
  For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Please help with serialization/deserialization of classes defined in .proto

[Henner Zeller]

Reconsider having a look at your code and ask yourself the following
questions
 - what are the input/output streams connected to. Is it the socket ?
 - what do you send over the socket ? Is it the data coming from a
SerializeTo*() method of the protocol buffer ?

Trial and Error programming just does not work. Please go through the whole
tutorial
First, try it out on your machine and understand each part of it. If you're
comfortable
Modifying and understanding it, go on and try to do the serialization over
the socket. I guess there is not much we can help here for now.

-h

On Fri, Mar 12, 2010 at 19:55, mk apollo...@gmail.com wrote:
 Thanks for reply
 As I said before, I don't have any experience in writing serialization/
 deserialization codes
 I read and followed the tutorial you referred to and still I am the
 error. Actually, the error is  when the server calls
 serialize(dataexge)

 Any help would be very appreciated.

 Thanks again!

 On Mar 13, 4:40 am, Kenton Varda ken...@google.com wrote:
 Please read the tutorial:

 http://code.google.com/apis/protocolbuffers/docs/cpptutorial.html

 On Fri, Mar 12, 2010 at 7:39 PM, mk apollo...@gmail.com wrote:
  Hi,

  Could someone please help me with serialization/deserialization
  classes defined in .proto (protobuf). here is an exp that I am trying
  to build (see server where I get error):

  file.proto

 message Data{
   required string x1 = 1;
   required uint32 x2 = 2;
   required float x3 = 3;
 }
 message DataExge {
   repeated Data data = 1;
 }

  client.cpp
 ...
 void serialize(const DataExge data_snd){
 try {
   ofstream ofs(DataExge);
   data_snd.SerializeToOstream(ofs);

 } catch(exception e) {
   cerr  serialize/exception:   e.what()  endl;
   exit(1);
 }
 }

 void deserialize(DataExge data_rec){
 try {
   ifstream ifs(DataExge);
   data_rec.ParseFromIstream(ifs);

 } catch(exception e) {
   cerr  deserialize/exception:   e.what()  endl;
   exit(1);
 }
 }

 int main(){
 ...
 DataExge dataexge;
 Data *dat = dataexge.add_data();

 char *y1 = operation1;
 uint32_t y2 = 123 ;
 float y3 = 3.14;

 // assigning data to send()
 dat-set_set_x1(y1);
 dat-set_set_x2(y2);
 dat-set_set_x3(y3);

 //sending data to the client
 serialize(dataexge);

 if (send(socket, dataexge, sizeof(dataexge), 0)  0) {
cerr  send() failed ;
exit(1);
  }

  //receiving data from the server
  deserialize(dataexge);

  if (recv(socket, dataexge, sizeof(dataexge), 0)  0) {
 cerr  recv() failed;
 exit(1);
  }

  //printing received data
  cout  dat-x1()  \n;
  cout  dat-x2()  \n;
  cout  dat-x3()  \n;
 ...
 }

  server.cpp
 ...
 void serialize(const DataExge data_snd){
 try {
   ofstream ofs(DataExge);
   data_snd.SerializeToOstream(ofs);

 } catch(exception e) {
   cerr  serialize/exception:   e.what()  endl;
   exit(1);
 }
 }

 void deserialize(DataExge data_rec){
 try {
   ifstream ifs(DataExge);
   data_rec.ParseFromIstream(ifs);

 } catch(exception e) {
   cerr  deserialize/exception:   e.what()  endl;
   exit(1);
 }
 }

 int main(){
 ...
 DataExge dataexge;
 Data *dat = dataexge.add_data();

 //receiving data from the client
 deserialize(dataexge);

 if (recv(socket, dataexge, sizeof(dataexge), 0)  0) {
cerr  recv() failed;
exit(1);
 }

 //printing received data
 cout  dat-x1()  \n;
 cout  dat-x2()  \n;
 cout  dat-x3()  \n;

 // assigning data to send()
 dat-set_set_x1(operation2);
 dat-set_set_x2(dat-x2() + 1);
 dat-set_set_x3(dat-x3() + 1.1);

 //sending data to the client
 serialize(dataexge);   //... I am getting error at this line ...

 if (send(socket, dataexge, sizeof(dataexge), 0)  0) {
cerr  send() failed ;
exit(1);
 }
 ...
 }

  Thanks for your help and replies -

  --
  You received this message because you are subscribed to the Google
Groups
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to
  protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
protobuf%2bunsubscr...@googlegroups.comprotobuf%252bunsubscr...@googlegroups.com

  .
  For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.

 --
 You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, 

Re: [protobuf] Fudge proto project

[Henner Zeller]

On Sun, Feb 28, 2010 at 09:13, Adewale Oshineye adew...@gmail.com wrote:
 Hi all,
 Have you seen this: http://www.fudgemsg.org/display/FDG/Fudge+Proto
 which claims to be compatible with the syntax of protobufs but use
 different on-disk and on-the-wire representations?

Looks like they fell into the trap of wanting to provide
self-description (with a pretty high overhead per message) and, worse,
message inheritance.

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Invitation to connect on LinkedIn

[Henner Zeller]

On Mon, Feb 22, 2010 at 17:00, Dan Homerick danhomer...@gmail.com wrote:
 Yes sir, as you can see from my LinkedIn profile, I know C++, Java, Python
 and Ruby. Why, just last weekend I went to Ruby's house for a BBQ and pool
 party...

Fair enough. I like to hang out at Java's place for a good cup of coffee.

-h

 On Mon, Feb 22, 2010 at 4:28 PM, Henner Zeller
 henner.zel...@googlemail.com wrote:

 Wow, that elevates protocol buffers to be a real person ;)

 --
 You received this message because you are subscribed to the Google Groups
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Sorting protocol buffer messages.

[Henner Zeller]

Some random idea: repeated fields support stl-style iterators; should
be possible to use regular std::sort() on it with a custom functor.
You probably can hide a lot of code complexity by writing such
functors and avoiding runtime complexity by keeping stuff sorted,
indeed.

On Sun, Feb 14, 2010 at 11:35, J Jack jjjac...@yahoo.com wrote:
 Hello,

 We're using protocol buffers extensively within our application, in
 what I believe is a somewhat strange manner.

 We define a message type, that essentially looks like a struct of an
 id, possibly some data, and a type, and add this include the struct
 within the message (i.e, infinite recursability),  to give us a tree-
 like structure.

 We add nodes with some data, as well as nodes to those nodes with some
 other data (specifically related to those nodes), etc.

 Quite often, we haveto merge two of these structures based on
 arbitrary rules (for instance, add this to container X, but only if it
 doesn't contain Y, or contain something already), etc.
 Right now, we're essentially iterating through both containers at the
 top level matching up ids, etc, and then recursing into each sub-node
 based on the id, this works fine, but, the code isn't as nice as it
 could be.

 What I'm wondering is if there's any way to store the messages sorted
 in some manner within each record (by the id), this would allow us to
 just do a binary search on merges, turning time into (I believe) log
 N, drastically speeding up both searches, and merges.

 I understand that protocol buffers itself offers no such facility, but
 I'm wondering if implementing one on top is advisable? Essentially I
 guess I'd haveto rebuild the structure everytime I do a merge (in
 order to keep it sorted). Each structure individually is fairly small
 (~1kb), but, we deal with billions of these structures on a daily
 basis(merging, searching through them, etc), I'm worried having to
 keep them always sorted (and hence re-building the structure on each
 merge) will cause memory fragmentation in the long run.

 On the top level (i.e, right below the root), we have maybe 10-15
 nodes, each of which contain up to 1000 items, (a random amount of
 these will be nodes that contain subnodes too).

 Another idea I had was to store the entire thing as a hashmap of
 hashmaps, but, right now, the coding/overhead for it doesn't seem to
 be worth it.

 And finally, is there some way Swap could be used to maintain a sorted
 structure efficiently?

 Any responses appreciated,

 Thanks!

 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.



-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] PB parsing with empty/reset values

[Henner Zeller]

Hi,
On Fri, Feb 5, 2010 at 08:43, Hershiv Haria darkmaste...@gmail.com wrote:
 Hi all,

 I have an odd issue at the moment. I am sending a PB from an android
 app to an app engine server, using Java. My app is being used to store
 files online, and I use the PB to send the file name, destination, and
 file data (as a string).

Probably unrelated, but if you have binary file content you might
consider using the type 'bytes' instead of 'string'.

 The problem is that when I send it from the android app, it is fine,
 but attempting to parse it on the server gives my ints as 0 and my
 strings as null. HOWEVER, if I use the same PBs in a test project, it
 works fine.

 Any ideas why this would happen?

This is not really enough information to know what is going on. How do
you transfer the data ? Do you have authentication in place that maybe
fails so that you don't get the data to deserialize ? How does the
receiving side know the length of the data it receives (did you check
that it actually gets the number of bytes you expected to be sent) ?
How do you know that sending it from the android app succeeds ?

-h

-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] C++ Parsing and de-serialization

[Henner Zeller]

Hi,
On Sun, Jan 17, 2010 at 13:21, Johan freddielunchb...@gmail.com wrote:
 Hi,

 I read data messages that comes in on a TCP/IP socket.
 These message can be of different types example Person, Project etc,
 that I have defined and the corresponding .proto files.

 I can do:

    string s((char*)message,  strlen(message));
    Person p;
    p.ParseFromString(s))

Be careful here: the buffer can contain '\0' characters so strlen
would return a shorter length. You need to transfer by some other
means the length of the message.


 and then access the object Person.

 However, I would like to have it more generic.
 I cannot know from the beginning if the incoming message is a Person
 or a Project or whatever.
 Also, I am not so interested in the access methods, as I plan to loop
 though the Fields using reflection to the decode the object.

 Is there a way to decode the 'message' into a generic object (i.e a
 Message) that I can then inspect using reflection?

 I can accept to link in the different proto buffers at compile time.

 I hope this makes sense.. and thanks for a great protocol !


 BR
 johan












 --
 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.




-- 
You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] How to assign a singel nested object value in C++?

[Henner Zeller]

On Tue, Dec 29, 2009 at 11:11, Nigel Pickard pickard.ni...@gmail.com wrote:
 I'm having a problem working out how to assign a nested object's value
 in C++ (my Java version works fine).  E.g. in the proto file for my
 car object I define:

  import mycustomengine.proto;
  message MyCar {
  optional string name = 1;
  optional string model = 2;
  optional MyCustomEngine customengine  = 3;

 Note I have assigned there to only be 0 or 1 MyCustomEngines to a Car
 object.  Now in my code, I've already created an instance of
 MyCustomEngine, e.g.:

  my::objects::MyCustomEngine myCustomEngine;
  myCustomEngine.set_name(hey, this is my custom engine!);

 Now I create my Car object

  my::objects::MyCar myCar;
  myCar.set_name(Ford);
  myCar.set_model(Fiesta);

 But now I'm stuck on how to assign the myCustomEngine instance to the
 myCar instance?  There appears to be no method such as
 myCar.set_customengine(myCustomEngine).  So any ideas on how to do

it is something along the lines of
myCar.mutable_customengine()-CopyFrom(myCustomEngine).

Since this copies data, you might want to consider setting things
directly in the customengine in the first place - if this is time
critical.

void InitCustomEngine(MyCustomEngine *engine) {
   engine-set_name(My cool custom engine);
}

...
myCar.set_name(foo);
InitCustomEngine(myCar.mutable_customengine());

 this?

 --

 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.




--

You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: protoc plugin compiler extension framework

[Henner Zeller]

Hi,
On Tue, Dec 22, 2009 at 06:42, Christopher Piggott cpigg...@gmail.com wrote:
 Hmm maybe I can use the UninterpretedOption message to do this.
 Would something like this work?

 message ChrisMessage {
  option javadoc = This is an object representing Chris's Message;
  repeated int32 field1 = 1 [javadoc=This is a javadoc for field 1];
  repeated int32 field2 = 2 [javadoc=This is a javadoc for field 2];
 }


This javadoc option looks too much of a clutter to me and decreases
the readability of the proto file.

I'd rather have the protoc compiler understand regular JavaDoc like
comments that make it into the meta data - from there the
corresponding language specific comments can be generated.

My suggestion is to support two types of comments. One in front of the
field or message, potentially multiline and starting with
slash-doublestar, much like JavaDoc

  /** This is some field doc
   *  that could potentially span multiple
   *  lines
   */
 repeated int32 field1 = 1;

.. and a end-of-line comment that is on the same line of a field
declaration to save space:

  repeated int32 field2 = 2;  // some field doc describing field in same line.

(this is what is missing from standard JavaDoc and makes field
comments there 100% more space consuming )

Both of these can be unambiguously associated with the corresponding
fields. I have a low prioritized entry on my personal TODO list to add
at some point, but I am happy if someone else wants to work on this ;)

-h

--

You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

[protobuf] Thinking of implementing: extract documentation in .proto file and store in FileDescriptorProto

[Henner Zeller]

Hi,
Since this question came up earlier today and I have this anyway on my
TODO list, I think this is as well some nice side project for the
holidays I could work on ;)

Basically, there are two forms of comments typically found for
messages and fields: block comments in front of the declaration of the
message/field and a single end-of-line comment at the end of a field.
While often block comments are regarded as a multi-line /* ... */
block, I'd as well like to see a multi-line comment as multiple
consecutive lines of //-style comments:

/* some block
 * comment
 */
message Foo {

  /* some stray comment, not part of a field documentation */

  /*
   * some block comment
   */
  int32 some_field = 1;

  int32 some_other_field = 2;  // short comment.

  // Some stray comment, not part of a field. No documentation.

  // Some block comment
  // comprising of consecutive //-style comments
  // over multiple lines with no newline in-between
  int32 yet_another_field = 3;
}

There are several documentation styles out there such as JavaDoc or
Doxygen that require a particular start of a comment (like /** .. */
or /*! .. */ or ///... ). Is this a constraint we want to have or need
? I think this makes sense for these documentation tools as they are
designed for code that can have some arbitrary comments in-between.
The only requirement I'd propose is that there should be no empty line
between a block comment and the field/message it describes. This
enforces readability and prevents stray comments or file-header
comments being accidentally included in the documentation.

Thoughts ?

H.

--

You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Thinking of implementing: extract documentation in .proto file and store in FileDescriptorProto

[Henner Zeller]

Hi,
On Tue, Dec 22, 2009 at 15:00, Christopher Piggott cpigg...@gmail.com wrote:


 On Dec 22, 4:53 pm, Henner Zeller henner.zel...@googlemail.com
 wrote:
   /*
    * some block comment
    */
   int32 some_field = 1;
   int32 some_other_field = 2;  // short comment.

 I would be fine with that, but I also woudn't have a problem with you
 requiring everything be a block, because you can still do it on one
 line:

   /**
    * some block comment
    */
   int32 some_field = 1;
   int32 some_other_field = 2;  /** short comment */

 Notice I did keep the /** in there, because:

 Is this a constraint we want to have or need

 I think so.  I think it's helpful to say This comment is special.  I
 can see a good argument, though, that it's redundant - especially if
 pass documentation comments to generated code is a .proto file
 option.  I like /** something */ because it fits well with java and
 C/C++ (with Doxygen) and because I think the Python triple-quote is
 ugly.  If you really wanted // then I'd be happiest with ///

 Ultimately, though, the .proto is its own language, so decide upon
 whatever makes sense to you.  It shouldn't be overly cumbersome or
 ugly, and it should be reasonably easy for the .proto parsing code to
 handle (so you don't wind up hating me).

I don't see much gain in having to revisit all my existing protocol
buffer files to add the information that a comment is special ;) If I
commented a field, I probably meant to, uhm, comment it - so this is
what the protocol compiler get out of it.


 The only requirement I'd propose is that there should be no empty line
 between a block comment and the field/message it describes. This
 enforces readability and prevents stray comments or file-header
 comments being accidentally included in the documentation.

 I agree.


 There are some other things you didn't ask that are bothering me a
 little.  One has to do with fields.  The fields themselves, at least
 in java, are private, so documenting them in this way is not
 especially useful.  What you really want is to have these documents
 put something meaningful in the .hasSomething(), .getSomething
 (), .getSomethingCount(), etc. and in the builder, to .setSomething()
 and .addSomething(), and similar methods.

 How to make this work and look good is a real question in my mind.
 If you do:

 message Something {
   required int32 ageField = 1; /** Age of this human */
 }

 what you really would want for useful inline documentation (using
 javadoc as an exapmle, but same for Doxygen) would be something like

 /** Get ageField
   * @return Age of this human
   */
 public int hasAgeField() { ... }


 for the builder:

 /** Set ageField
  * @param value Age of this human
  */
 public void setAgeField(int value) { ... }

 and similar for lists etc.  The ageField part I grabbed from the
 field name, and the actual comment I applied to the @return and
 @param.

 I would have to take some time to think about how you would phrase
 this so that it makes sense for lists.

Yeah, haven't thought too much about the target documentation yet
which will be done in each code generator explicitly. But it would go
along the lines of what you suggest. Some heuristics will evolve there
I guess  (e.g. Using the first sentence as short documentation for the
field - some trick JavaDoc does).
The implementation would be two steps: first get the documentation in
the meta-data, then have the code generators generate the pretty
documentation

 This is kind of where I was going / what I was wishing for with regard
 to fields.  The decisions are a little more straight-forward when it's
 documentation for messages, as that documentation I would expect to
 more or less pass straight through unchanged, and use it to document
 the classes being generated.  (The fields are just more complicated,
 since the actual fields in the class are private).

 I'm not sure where you'd go with services - method calls I suppose,
 but for java/doxygen those would be the form @param @param ...
 @return.  I don't really know python/php so I'm not sure how this maps
 over to those languages.

 Is that helpful?

 --

 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.




--

You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Thinking of implementing: extract documentation in .proto file and store in FileDescriptorProto

[Henner Zeller]

.. alright, will spend some time tomorrow implementing things.

--

You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Java -using GPB -which type of stream should I be using over sockets?

[Henner Zeller]

On Wed, Dec 2, 2009 at 12:31, Nigel Pickard pickard.ni...@gmail.com wrote:
 Jason -do you mean any Java output stream type will work (as all Java
 IO stream types ultimately use bytes), and that the byte stream
 produced will have the bytes put in an accepted standard order by the
 writeTo method?  In other words, using the writeTo method for *any*
 Java output stream type will produce a series of bytes that will
 *always* be able to be read using the C++ equivalent of parseFrom on
 the C++ side?

Yes. Protocol Buffers are designed to be platform and language
independent so given a transparent way to transport raw bytes (any
output stream), serialization and deserialization between any
combination works as you expect. Since you asked: raw types such as
integers are serialized in a proto buffer specific way with a
consistent byte order independent of the platform.

-h



 On Dec 2, 1:20 pm, Jason Hsueh jas...@google.com wrote:
 Protocol buffers implement their own serialization routines: the
 writeTo(OutputStream output) method will write the binary format to the
 OutputStream. From a socket, just do:

 MyProtoBuf pb = new MyProtoBuf;
 ...
 pb.writeTo(socket.getOutputStream());

 On Wed, Dec 2, 2009 at 7:47 AM, Nigel Pickard pickard.ni...@gmail.comwrote:

  OK, so far I like GPB.  However, I have a very simple question, but
  one which I can't find an answer for:

  I have created a test Java application that uses ObjectInputStream and
  ObjectOutputStream over sockets.  No problems, it works! I use my GPB
  class writeTo and parseFrom to send and reconstitute my GPB class
  using the ObjectInput and Output Streams

  But: isn't GPB allowing for serialization using bytes?  Are Object I/O
  Streams the best streams to use?  Should I be using something else
  like Data I/O streams?  E.g. from JavaDocs, An ObjectOutputStream
  writes primitive data types and graphs of Java objects to an
  OutputStream -is there a better, more efficient I/O stream I should
  be using?

  My concern is that when it comes to attempting to open a socket to a C+
  + app, I assume the correct I/O stream type should be used.

  If anyone has a working example of best practice using Data I/O
  streams or a recommended stream type, that would be extremely useful
  for me (even better if someone has a simple Java to C++ using sockets
  example).

  Thanks.

  --

  You received this message because you are subscribed to the Google Groups
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to
  protobuf+unsubscr...@googlegroups.comprotobuf%2bunsubscr...@googlegroups.com
  .
  For more options, visit this group at
 http://groups.google.com/group/protobuf?hl=en.



 --

 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.




--

You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Java -using GPB -which type of stream should I be using over sockets?

[Henner Zeller]

Are your ObjectOutputStreams transparent ? Or do they prepend/append
things to the data ?

On Wed, Dec 2, 2009 at 13:11, Nigel Pickard pickard.ni...@gmail.com wrote:
 Oh, wait So I have a test Java app where one thread is running as
 a server, one as a client.

 I've been sending a GPB defined class instance between them, no
 problem when I use Object I/O streams (e.g. ObjectOutputStream on the
 server, ObjectInputStream on the client.  I've been calling writeTo
 and parseFrom on the GBP object successfully.

 If I understand the previous posts correctly it should not make a
 difference which outputstream or inputstream type I use, right?  The
 stream of whatever type it is should just be a series of bytes, able
 to be parsed by my GBP object  But if I change my
 ObjectOutputStream to DataOutputStream on my client (so I'm leaving my
 server still using ObjectInputStream) I get the following error:

 com.google.protobuf.InvalidProtocolBufferException: Protocol message
 end-group tag did not match expected tag.

 Wouldn't this suggest there is a difference (at least in Java)
 concerning which inputstream you use?

 --

 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.




--

You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Re: [protobuf] Re: Java -using GPB -which type of stream should I be using over sockets?

[Henner Zeller]

Hi,
On Wed, Dec 2, 2009 at 15:21, Nigel Pickard pickard.ni...@gmail.com wrote:
 Henner:

 I'm not sure what you mean by the datastream being transparent, but
 basically I'm sending a GPB instance from the server to the client.
 Initially I used ObjectOutputStream on the server, ObjectInputStream
 on the client.  I then changed the client to use DataInputStream
 (apologies, my previous post stated output when I meant input).

transparent as in: the same data is written out that you put in. If
you see a difference then this might be because the ObjectOutputStream
might add something to the data you provide, such as writing
delimiters or something. So I guess the simple write(byte[]) will not
actually write the content of the arrays but as well the length or
something (this is a wild guess - haven't looked at the sources, but
it would explain what you see).

Why do you use the ObjectOutputStream anyway instead of the simpler
OutputStream ?


 I'm just sending one instance each run for simplicity (my object is a
 simple one) -the exact same GPB instance that worked when I used
 Object IO streams on the client and server that worked -and then
 stopping.  The only thing that has changed has been the input stream
 type on the client.

 Basically, on the client side I called the GPB object's parseFrom
 method with the DataInputStream.as an argument and expected it to work
 but got that error message.




 On Dec 2, 4:17 pm, Henner Zeller henner.zel...@googlemail.com wrote:
 Are your ObjectOutputStreams transparent ? Or do they prepend/append
 things to the data ?

 On Wed, Dec 2, 2009 at 13:11, Nigel Pickard pickard.ni...@gmail.com wrote:
  Oh, wait So I have a test Java app where one thread is running as
  a server, one as a client.

  I've been sending a GPB defined class instance between them, no
  problem when I use Object I/O streams (e.g. ObjectOutputStream on the
  server, ObjectInputStream on the client.  I've been calling writeTo
  and parseFrom on the GBP object successfully.

  If I understand the previous posts correctly it should not make a
  difference which outputstream or inputstream type I use, right?  The
  stream of whatever type it is should just be a series of bytes, able
  to be parsed by my GBP object  But if I change my
  ObjectOutputStream to DataOutputStream on my client (so I'm leaving my
  server still using ObjectInputStream) I get the following error:

  com.google.protobuf.InvalidProtocolBufferException: Protocol message
  end-group tag did not match expected tag.

  Wouldn't this suggest there is a difference (at least in Java)
  concerning which inputstream you use?

  --

  You received this message because you are subscribed to the Google Groups 
  Protocol Buffers group.
  To post to this group, send email to proto...@googlegroups.com.
  To unsubscribe from this group, send email to 
  protobuf+unsubscr...@googlegroups.com.
  For more options, visit this group 
  athttp://groups.google.com/group/protobuf?hl=en.

 --

 You received this message because you are subscribed to the Google Groups 
 Protocol Buffers group.
 To post to this group, send email to proto...@googlegroups.com.
 To unsubscribe from this group, send email to 
 protobuf+unsubscr...@googlegroups.com.
 For more options, visit this group at 
 http://groups.google.com/group/protobuf?hl=en.




--

You received this message because you are subscribed to the Google Groups 
Protocol Buffers group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.