[protobuf] using compression on protobuf messages

2010-06-22 Thread sheila miguez
I'm relatively inexperienced with this, and would appreciate advice and criticism. After investigating gzip as a possible algorithm to use with our messages, we want to try lzo. There's a java implementation of lzo being used by the hadoop community. Gzip has around 120 msec latency for medium

[protobuf] How do I make repeatable properties 'optional' in protobuf for c#

2010-06-22 Thread Manh Do
I am working a project where we need to determine if a protobuf list have is null or not in the protocol buffer. The problem is the generated c# automatically instantiates the list. IE. Protocol: message B { repeated A MyList = 0; } Generated C#: private readonly

Re: [protobuf] using compression on protobuf messages

2010-06-22 Thread Evan Jones
On Jun 22, 2010, at 13:54 , sheila miguez wrote: When I have a message to compress, I know the size of the byte array stream buffer to allocate. Then call the writeTo on it. Is there anything I should do other than this, given a message? writeTo should be pretty performant, yes? In unit test,

Re: [protobuf] using compression on protobuf messages

2010-06-22 Thread sheila miguez
On Tue, Jun 22, 2010 at 1:28 PM, Evan Jones ev...@mit.edu wrote: On Jun 22, 2010, at 13:54 , sheila miguez wrote: When I have a message to compress, I know the size of the byte array stream buffer to allocate. Then call the writeTo on it. Is there anything I should do other than this, given a

Re: [protobuf] using compression on protobuf messages

2010-06-22 Thread Evan Jones
On Jun 22, 2010, at 15:35 , sheila miguez wrote: I've got a servlet filter which wraps the HttpServletResponse. So, the servlet response's output stream, which is wrapped in a stream from the lzo library, is compressing data as it is getting written to. Ah, so the best case is probably

Re: [protobuf] How do I make repeatable properties 'optional' in protobuf for c#

2010-06-22 Thread Marc Gravell
Kenton is of course quite correct. There is no *native* distinction between a null list and an empty list in the raw protobuf data stream. There *are* some ways of working around that, to distinguish between the two, but not when using the code-generation from .proto. Marc On 22 June 2010 18:59,

[protobuf] Re: How do I make repeatable properties 'optional' in protobuf for c#

2010-06-22 Thread Manh Do
What are your suggestions? One option that I was thinking is to add a boolean field in the message that determines if the list is empty or not. What do you think? On Jun 22, 1:21 pm, Marc Gravell marc.grav...@gmail.com wrote: Kenton is of course quite correct. There is no *native* distinction

Re: [protobuf] Re: How do I make repeatable properties 'optional' in protobuf for c#

2010-06-22 Thread Marc Gravell
That was exactly my thought. Something *broadly* like (untested): [ProtoMember(1)] public ListFoo Foos {get;set;} [ProtoMember(2), DefaultValue(false)] private bool FoosHasValue { get { return Foos != null; } set { Foos = value ? ( Foos ?? new ListFoo() ) : null;

[protobuf] Re: using compression on protobuf messages

2010-06-22 Thread Alex Antonov
On Jun 22, 3:04 pm, Evan Jones ev...@mit.edu wrote: On Jun 22, 2010, at 15:35 , sheila miguez wrote: I've got a servlet filter which wraps the HttpServletResponse. So, the servlet response's output stream, which is wrapped in a stream from the lzo library, is compressing data as it is

Re: [protobuf] Re: using compression on protobuf messages

2010-06-22 Thread Evan Jones
Alex Antonov wrote: When I use .writeTo(...) and pass it a CompressionOutputStream as an input, it takes only 38,226,661 ns to compress 112,178 bytes. Wow! Glad to hear this helped so much. If you have a sequence of messages, you could try using a single CodedOutputStream. Something like:

[protobuf] hasXxx methods

2010-06-22 Thread Miguel Muñoz
Comrades, I've been setting a value of a single optional field in a message with many optional fields. In my diagnostic code I'm calling the hasXxx() methods to find out which field is set, but it's telling me they're all set. Is this a bug or a bad design? I can't find any circumstance

Re: [protobuf] hasXxx methods

2010-06-22 Thread Henner Zeller
Hi, 2010/6/22 Miguel Muñoz swingguy1...@yahoo.com: Comrades,   I've been setting a value of a single optional field in a message with many optional fields. In my diagnostic code I'm calling the hasXxx() methods to find out which field is set, but it's telling me they're all set. Is this a bug

[protobuf] Serializing large arrays in Java

2010-06-22 Thread Joshua Hartman
Hi, I'm trying to serialize a large primitive array of int in Java, and I'm testing using 1,000,000 elements. My .proto message is just a packed repeated field of type int32. To serialize, I just create a builder and loop through the array, adding elements one at a time. I took a peek at the