[protobuf] Getting fields from a generated Dynamic Message
Hey, I have generated DynamicMessage from available FileDescriptorSet, now I want to be able to get fields from this message. Based on the typename I know what kind of message this is, but I don't know how to get the value of any particular field. Available getter functions available in java library like getField(Descriptors.FieldDescriptor field) need a FieldDescriptor for the field, which I don't know how to generate. In C++ library there is a func called FindFieldByName(String) which returns the corresponding FieldDescriptor but I couldn't find a similar function for java library. Please help -- You received this message because you are subscribed to the Google Groups Protocol Buffers group. To post to this group, send email to protobuf@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: using GzipOutputStream increases size
Thanks Jason!!
On Dec 5, 11:59 pm, Jason Hsueh jas...@google.com wrote:
You should make sure to destroy the CodedOutputStream before getting sizes
from ZeroCopyOutputStream. CodedOutputStream effectively asks for a buffer
from ZeroCopyOutputStream, and ByteCount() reflects the total count that
CodedOutputStream has asked for. Upon destruction, CodedOutputStream may
return buffer space to the stream implementation with BackUp(). Likewise,
you need to flush the GzipOutputStream before checking your underlying
ZeroCopyOutputStream - same sort of story there.
On Thu, Dec 1, 2011 at 5:24 PM, Vinay Bansal vinsalw...@gmail.com wrote:
Hey,
I have run into one more problem.
Whenever I try to find out the number of bytes written by the streams,
CodedOutputStream.ByteSize() gives the right answer but
ZeroCopyOutputStream.ByteSize() always gives 0 and GzipOutputStream
always gives a fixed no. like 63356 and not the actual no. of bytes
written by the stream.
I would really appreciate the help.
Thanks
On Dec 1, 3:10 pm, Vinay Bansal vinsalw...@gmail.com wrote:
Hey Oliver,
I figured it out just a minutes before your reply.
Thanks anyway
Cheers,
-Vinay
On Dec 1, 2:28 pm, Oliver Jowett oliver.jow...@gmail.com wrote:
On Thu, Dec 1, 2011 at 3:22 PM, Vinay Bansal vinsalw...@gmail.com
wrote:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
If you're doing that for every (small) message, then the compressor is
never going to have a good chunk of data to work with; the compression
dictionary will be reset for every message, plus a gzip header gets
written each time. (For comparison, try a command-line gzip of a file
containing only a single message - that's essentially what you're
doing here)
You want to open the file once, create one GzipOutputStream, then
write many messages to it.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] using GzipOutputStream increases size
Hi Guys,
I am working with protocol buffers and I have come across this weird
problem:
Objective : The objective was to write a few thousand messages after
zipping them in a file. I do it like this:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
This gives me a file of size 54k but when I don't use gzipstream and
directly write the encoded message in the file like below it gives me
a file of size 38k (54k)
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(out);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
The gzip'd file can be unzipped from command line using 'gzip -d
filename which indicates that it is a valid file and after unzipping
the file its size comes down to 38k
Also I tried using ZLIB format for the gzipoutputstream it gives me a
file of 42k again greater than 38k(size of uncompressed file). I also
tried gzipping the uncompressed file from command line and it gave me
a file of size 2.3 kB which was what I was expecting for gzip stream
too.
I am completely stumped here, instead of compressing the file it has
increased the size of the file.
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: using GzipOutputStream increases size
Hey Oliver,
I figured it out just a minutes before your reply.
Thanks anyway
Cheers,
-Vinay
On Dec 1, 2:28 pm, Oliver Jowett oliver.jow...@gmail.com wrote:
On Thu, Dec 1, 2011 at 3:22 PM, Vinay Bansal vinsalw...@gmail.com wrote:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
If you're doing that for every (small) message, then the compressor is
never going to have a good chunk of data to work with; the compression
dictionary will be reset for every message, plus a gzip header gets
written each time. (For comparison, try a command-line gzip of a file
containing only a single message - that's essentially what you're
doing here)
You want to open the file once, create one GzipOutputStream, then
write many messages to it.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: using GzipOutputStream increases size
Hey,
I have run into one more problem.
Whenever I try to find out the number of bytes written by the streams,
CodedOutputStream.ByteSize() gives the right answer but
ZeroCopyOutputStream.ByteSize() always gives 0 and GzipOutputStream
always gives a fixed no. like 63356 and not the actual no. of bytes
written by the stream.
I would really appreciate the help.
Thanks
On Dec 1, 3:10 pm, Vinay Bansal vinsalw...@gmail.com wrote:
Hey Oliver,
I figured it out just a minutes before your reply.
Thanks anyway
Cheers,
-Vinay
On Dec 1, 2:28 pm, Oliver Jowett oliver.jow...@gmail.com wrote:
On Thu, Dec 1, 2011 at 3:22 PM, Vinay Bansal vinsalw...@gmail.com wrote:
void write() {
int fd = open(myfile, O_WRONLY), O_APPEND);
google::protobuf::io::ZeroCopyOutputStream *out = new
google::protobuf::io::FileOutputStream(fd);
google::protobuf::io::GzipOutputStream *gzipOut = new
google::protobuf::io::GzipOutputStream(out, options);
google::protobuf::io::CodedOutputStream *codedOut = new
google::protobuf::io::CodedOutputStream(gzipOut);
codedOut-WriteVarint32(message.ByteSize());
message.SerializeToCodedStream(codedOut);
close(fd);
}
If you're doing that for every (small) message, then the compressor is
never going to have a good chunk of data to work with; the compression
dictionary will be reset for every message, plus a gzip header gets
written each time. (For comparison, try a command-line gzip of a file
containing only a single message - that's essentially what you're
doing here)
You want to open the file once, create one GzipOutputStream, then
write many messages to it.
Oliver
--
You received this message because you are subscribed to the Google Groups
Protocol Buffers group.
To post to this group, send email to protobuf@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
