Re: [protobuf] Re: protoc generated .h file has errors in VS2008

2009-12-10 Thread Daniel Wright
Maybe ask the compiler for the pre-processed output (sorry, I don't know the VS2008 flag), and search through that for the first appearance of those identifiers. That'll tell you what header they come from. Adding a package should fix this kind of problem by putting all of the messages in a c++

Re: [protobuf] Re: Protocol Buffers using Lzip

2009-12-11 Thread Daniel Wright
As I'm sure you can imagine, we store a lot of data in protocol buffer format at Google, so we often want to store very large files with many serialized protocol buffers. The technique we use is to batch a bunch of records together, compress them and write the compressed block to the file (with

Re: [protobuf] PreProcessor for TextFormat files

2010-03-17 Thread Daniel Wright
You might want to consider something like GNU m4 as a preprocessor to your config files. I've never used it for proto files, but used it successfully for other things -- it lets you define macros and evaluate expressions. See http://en.wikipedia.org/wiki/M4_(computer_language) On Wed, Mar 17,

Re: [protobuf] File size of the serialized records

2010-03-22 Thread Daniel Wright
The most likely cause is a bug in your code where there's something you aren't clearing each time you write a record, so at each iteration in your loop, the record you're writing is getting bigger. Of course I can't say for sure without seeing the code. Daniel On Mon, Mar 22, 2010 at 1:13 PM,

Re: [protobuf] Tcl: decoding a serialized ProtoBuf

2010-05-26 Thread Daniel Wright
The wire format is documented in http://code.google.com/apis/protocolbuffers/docs/encoding.html Daniel On Wed, May 26, 2010 at 9:11 AM, nedbrek nedb...@yahoo.com wrote: Hello all, I am interested in decoding a ProtoBuf I read off the network (080010001800220100) in Tcl. Full PB support

Re: [protobuf] [protobuff] How to set a field of Message type? (C++)

2010-07-07 Thread Daniel Wright
In C++ you use the mutable_ accessor to set the value. So for example, you could do: my_message.mutable_auth_resp_msg()-set_foo(1); On Tue, Jul 6, 2010 at 3:01 PM, Maxim Leonovich lm.b...@gmail.com wrote: I have a protocol like that: message MSG { enum MessageType {

Re: [protobuf] Multiple messages

2010-07-23 Thread Daniel Wright
The standard solution for this is to length-delimit your messages in the file with CodedOutputStream. So you create a CodedOutputStream for the file (call it coded_output_stream), and then for each message, something like (untested): coded_output_stream.WriteVarint32(message.ByteSize());

Re: [protobuf] Encoding/Decoding of data - Question on CodedInputStream CodedOutputStream

2010-08-16 Thread Daniel Wright
I'm not completely sure I understand your question, but if you're asking about the difference between writeTo(OutputStream) and writeTo(CodedOutputStream), they're the same -- writeTo(OutputStream) just wraps the OutputStream in a CodedOutputStream and writes to that. Here's the code: public

Re: [protobuf] Performance analysis of RepeatedField versus generated class

2010-08-17 Thread Daniel Wright
How about this way-more-readable variant of option B: add_double_vector(deal_pb.bucket(i).bucketdouble().data(), deal_pb.bucket(i).bucketdouble().size()); This assumes that add_bouble_vector only needs a const pointer. If it needs a non-const pointer, add the

Re: [protobuf] Re: Status of protobufs

2010-09-02 Thread Daniel Wright
See http://code.google.com/apis/protocolbuffers/docs/techniques.html#streaming Your solution of trying to parse after each byte received is not just slow, it's completely incorrect. It's entirely possible, and quite likely, that if you break the encoded version of a message in half, each half

Re: [protobuf] doubt

2010-09-08 Thread Daniel Wright
They're effectively the same -- it's just a style question. If B only makes sense in the context of A, I'd go with the second version, otherwise I'd go with the first. But the generated code should be the same except for the name of B in the generated code. 2010/9/8 alf alberto@gmail.com

Re: [protobuf] What's the state of TextFormat?

2010-10-04 Thread Daniel Wright
TextFormat is used extensively within Google, often for exactly the purpose you describe. It should work well, though I'd recommend keeping the files in UTF8 to avoid localisation issues. On Mon, Oct 4, 2010 at 5:08 PM, Dan drtet...@gmail.com wrote: Hi there. I'm wondering what the state of

Re: [protobuf] serialize to a file using FileOutputStream

2010-10-11 Thread Daniel Wright
I think it's being buffered in the FileOutputStream -- you should be sure to delete the output streams (in the reverse order that you created them) before you close the file. On Mon, Oct 11, 2010 at 1:07 PM, Paul Yang mjpabl...@gmail.com wrote: Hi, I am new to protocol buffers, and I am trying

Re: [protobuf] outer classname for C++

2010-10-25 Thread Daniel Wright
No -- in C++ the message classes are placed directly in a namespace named after the package, so there's no outer class. On Mon, Oct 25, 2010 at 11:42 AM, Paul mjpabl...@gmail.com wrote: Hi, In the example of the .proto definition in Java, there is a option java_outer_classname =

Re: [protobuf] Message vs MessageLite

2010-10-26 Thread Daniel Wright
Yes -- the serialized format is identical. On Tue, Oct 26, 2010 at 2:56 PM, Alsmom2005 gundanu...@gmail.com wrote: Hi all, Is it ok if the serialization is made using libprotobuf library and the deserialization (on the other end) is made using code built with libprotobuf-lite library ? That

Re: [protobuf] Dynamic Message

2010-11-05 Thread Daniel Wright
This is exactly what extensions are for -- see http://code.google.com/apis/protocolbuffers/docs/proto.html#extensions It would look something like: message BaseMessage { required MsgType type = 1; extensions 100 to 10; } Then each module would have a message like: message Msg1 {

Re: [protobuf] Re: Dynamic Message

2010-11-05 Thread Daniel Wright
100 to 10;? On Nov 5, 3:21 pm, Daniel Wright dwri...@google.com wrote: This is exactly what extensions are for -- seehttp:// code.google.com/apis/protocolbuffers/docs/proto.html#extensions It would look something like: message BaseMessage { required MsgType type = 1

Re: [protobuf] Storing a Protocol Buffer inside another Protocol Buffer

2011-08-09 Thread Daniel Wright
Both methods that you mention will work and are fairly common. The first is generally preferred because you're less likely to have type errors, and debugging tools can show you more of RPCBuffer (e.g. the rpc library could print the DebugString of the RPCBuffer in debugging mode, showing exactly

Re: [protobuf] Storing protocol buffers: binary vs. text

2011-12-11 Thread Daniel Wright
The main concern with text format is that it doesn't have nearly as good backwards- and forwards- compatibility as the binary format. E.g. what happens if you release your program, and then in a future update want to remove or rename a field? The new binary format code would have no trouble

Re: [protobuf] Re: suggestions on improving the performance?

2012-01-13 Thread Daniel Wright
It's extremely unlikely that text parsing is faster than binary parsing on pretty much any message. My guess is that there's something wrong in the way you're reading the binary file -- e.g. no buffering, or possibly a bug where you hand the protobuf library multiple messages concatenated

Re: [protobuf] Re: suggestions on improving the performance?

2012-01-15 Thread Daniel Wright
; } /code On Jan 14, 3:37 am, Henner Zeller henner.zel...@googlemail.com wrote: On Fri, Jan 13, 2012 at 11:22, Daniel Wright dwri...@google.com wrote: It's extremely unlikely that text parsing is faster than binary parsing on pretty much any message. My guess is that there's something wrong

Re: [protobuf] incompatible type changes philosophy

2012-05-08 Thread Daniel Wright
On Tue, May 8, 2012 at 4:42 PM, Jeremy Stribling st...@nicira.com wrote: I'm working on a project to upgrade- and downgrade-proof a distributed system that uses protobufs to communicate data between instances of a C ++ program. I'm trying to cover all possible cases for data schema changes