[protobuf] Message.Builder and sub-builders
Let's assume we have the following structure:
message Alpha {
message Beta {
optional int32 mercury = 1;
optional int32 venus = 2;
}
optional int32 earth = 1;
optional Beta mars = 2;
}
The Java generated code allows to get a sub-builder for mars field from a
builder for an Alpha message by calling
Alpha.newBuilder().getMarsBuilder(). This is nice because I can keep
builders for structured messages and update them as needed and build the
Alpha message as needed, but still hold on to the big builder which I plan
to update further.
But I do not think there is a way to do this in a generic way, using
Message.Builder interface. Is there any plan for something like this?
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To view this discussion on the web visit
https://groups.google.com/d/msg/protobuf/-/Y2ff3aqv72wJ.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] using GzipOutputStream increases size
Hi Guys,
I am working with protocol buffers and I have come across this weird
problem:
Objective : The objective was to write a few thousand messages after
zipping them in a file. I do it like this:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
This gives me a file of size 54k but when I don't use gzipstream and
directly write the encoded message in the file like below it gives me
a file of size 38k (54k)
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(out);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
The gzip'd file can be unzipped from command line using 'gzip -d
filename which indicates that it is a valid file and after unzipping
the file its size comes down to 38k
Also I tried using ZLIB format for the gzipoutputstream it gives me a
file of 42k again greater than 38k(size of uncompressed file). I also
tried gzipping the uncompressed file from command line and it gave me
a file of size 2.3 kB which was what I was expecting for gzip stream
too.
I am completely stumped here, instead of compressing the file it has
increased the size of the file.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
Re: [protobuf] using GzipOutputStream increases size
On Thu, Dec 1, 2011 at 3:22 PM, Vinay Bansal vinsalw...@gmail.com wrote:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
If you're doing that for every (small) message, then the compressor is
never going to have a good chunk of data to work with; the compression
dictionary will be reset for every message, plus a gzip header gets
written each time. (For comparison, try a command-line gzip of a file
containing only a single message - that's essentially what you're
doing here)
You want to open the file once, create one GzipOutputStream, then
write many messages to it.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: using GzipOutputStream increases size
Hey Oliver,
I figured it out just a minutes before your reply.
Thanks anyway
Cheers,
-Vinay
On Dec 1, 2:28 pm, Oliver Jowett oliver.jow...@gmail.com wrote:
On Thu, Dec 1, 2011 at 3:22 PM, Vinay Bansal vinsalw...@gmail.com wrote:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
If you're doing that for every (small) message, then the compressor is
never going to have a good chunk of data to work with; the compression
dictionary will be reset for every message, plus a gzip header gets
written each time. (For comparison, try a command-line gzip of a file
containing only a single message - that's essentially what you're
doing here)
You want to open the file once, create one GzipOutputStream, then
write many messages to it.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: using GzipOutputStream increases size
Hey,
I have run into one more problem.
Whenever I try to find out the number of bytes written by the streams,
CodedOutputStream.ByteSize() gives the right answer but
ZeroCopyOutputStream.ByteSize() always gives 0 and GzipOutputStream
always gives a fixed no. like 63356 and not the actual no. of bytes
written by the stream.
I would really appreciate the help.
Thanks
On Dec 1, 3:10 pm, Vinay Bansal vinsalw...@gmail.com wrote:
Hey Oliver,
I figured it out just a minutes before your reply.
Thanks anyway
Cheers,
-Vinay
On Dec 1, 2:28 pm, Oliver Jowett oliver.jow...@gmail.com wrote:
On Thu, Dec 1, 2011 at 3:22 PM, Vinay Bansal vinsalw...@gmail.com wrote:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
If you're doing that for every (small) message, then the compressor is
never going to have a good chunk of data to work with; the compression
dictionary will be reset for every message, plus a gzip header gets
written each time. (For comparison, try a command-line gzip of a file
containing only a single message - that's essentially what you're
doing here)
You want to open the file once, create one GzipOutputStream, then
write many messages to it.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Issue 351 in protobuf: Make protobuf_lite proto files not create any static initializers
Status: New Owner: liuj...@google.com Labels: Type-Defect Priority-Medium New issue 351 by tha...@chromium.org: Make protobuf_lite proto files not create any static initializers http://code.google.com/p/protobuf/issues/detail?id=351 We're trying to remove all static initialization from chromium: http://crbug.com/105626 This is made harder by protoc generating a static initializer for every protobuf, which is used to create the const reference returned by default_instance() (we use light protobufs for everything as far as I know, so we don't use the reflection bits). We don't call default_instance() in chromium, so it would be nice if we could get rid of the static initializers somehow. Have you thought about this, or any suggestions we might go about this? I'm willing to work on protoc to make this happen. -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: Issue 351 in protobuf: Make protobuf_lite proto files not create any static initializers
Issue 351: Make protobuf_lite proto files not create any static initializers http://code.google.com/p/protobuf/issues/detail?id=351 This issue is now blocking issue chromium:105626. See http://code.google.com/p/chromium/issues/detail?id=105626 -- You received this message because you are listed in the owner or CC fields of this issue, or because you starred this issue. You may adjust your issue notification preferences at: http://code.google.com/hosting/settings -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
