Re: [protobuf] Re: using compression on protobuf messages
Alex Antonov wrote:
When I use .writeTo(...) and pass it a CompressionOutputStream as an
input, it takes only 38,226,661 ns to compress 112,178 bytes.
Wow! Glad to hear this helped so much.
If you have a sequence of messages, you could try using a single
CodedOutputStream. Something like:
CodedOutputStream out = new CodedOutputStream(compressionStream);
for (msg : messages) {
msg.writeTo(out);
}
out.flush();
This should be slightly faster than using:
msg.writeTo(compressionStream);
because it avoids re-allocating the CodedOutputStream (and its internal
buffer). It should be quite a bit better for small messages.
Now I'm trying to figure out how I can speed up the decompression on
the receiving side.
What I have right now is:
* Take the CompressionInputStream, convert it into a byte[]
* Take the resulting byte[] and do .parseFrom(byte[])
This seems to be a faster route, then just
doing .parseFrom(CompressionInputStream).
Interesting. The only reason that I can think of which might make the
byte[] version faster is that maybe you use a big read, while
.parseFrom(InputStream) defaults to 4096 bytes. You could try editing
the source to make BUFFER_SIZE in CodedInputStream bigger, if you care.
The only thing I can think of is if you are reading a sequence of many
messages, you can again re-use a single CodedInputStream, although this
requires some work. Again, this will be better for small messages but
probably not large messages. This is trickier than re-using a single
CodedOutputStream. If you are interested, I can send the details about
what I have used. Although to be honest: I haven't tested it carefully
to see if this is *actually* faster than doing the simple thing such as
.parseFrom() and friends.
Evan
--
Evan Jones
http://evanjones.ca/
--
You received this message because you are subscribed to the Google Groups "Protocol
Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
[protobuf] Re: using compression on protobuf messages
On Jun 22, 3:04 pm, Evan Jones wrote: > On Jun 22, 2010, at 15:35 , sheila miguez wrote: > > > I've got a servlet filter which wraps the HttpServletResponse. So, the > > servlet response's output stream, which is wrapped in a stream from > > the lzo library, is compressing data as it is getting written to. > > Ah, so the best case is probably message.writeTo(servletOutputStream) > If you are writing multiple messages, you'll probably want to > explicitly create a single CodedOutputStream to write all of them. > > If you experiment with this and find something different, I would be > interested to know. > > Evan > > -- > Evan Joneshttp://evanjones.ca/ Evan, thatnks for the great tip about using CodedOutputStream for the writing of protobuf into a compressible stream. When calling a .toByteArray() method and then writing it to a CompressionOutputStream, it takes 396,078,181 ns to compress 112,178 bytes. When I use .writeTo(...) and pass it a CompressionOutputStream as an input, it takes only 38,226,661 ns to compress 112,178 bytes. So its almost a 10x reduction, which is GREAT!!! Now I'm trying to figure out how I can speed up the decompression on the receiving side. What I have right now is: * Take the CompressionInputStream, convert it into a byte[] * Take the resulting byte[] and do .parseFrom(byte[]) This seems to be a faster route, then just doing .parseFrom(CompressionInputStream). 10,404 bytes of compressed payload takes 181,643,533 ns to decompress and convert into a proto message using .parseFrom(CompressionInputStream). 10,404 bytes of compressed payload takes only 155,264,303 ns, when first I create a byte[] and then pass it into .parseFrom(byte[]) on the message. Any thoughts on how this can be (if possible at all) made even faster?! :) Thanks, Alex -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
