On Tue, Jan 19, 2010 at 4:46 AM, duselbaer <ronny.spie...@googlemail.com>wrote:
> (and as the Java developers told me, reading a byte
> array with protobuf creates 3 copies of it in memory).
>
This is not true in the common case. However, it looks like there is a code
path which creates 3 copies but could fairly easily be improved. Remember
that if you run into a performance problem with protobufs, it's usually
because no one else has exercised that path before. Please do not be afraid
to send me patches to improve performance!
> So what would be a good solution to this?
>
Ideally:
1) Extend ByteString so that it can hold its data in multiple pieces instead
of one, big, flat array. Allocating gigantic flat arrays is dangerous so
should be avoided.
2) Improve the parsing code so that it can parse large ByteStrings without
making multiple copies of the data.
3) Do not count large ByteStrings against the 64MB message size limit.
Then you can use the "bytes" type to store blobs of hundreds of megs with no
problems. No new features are needed -- this is purely an optimization
problem.
For the first step above, I think ByteString should have an internal class
like:
private static interface Node {
int size();
byte get(int index);
// other stuff
}
There should then be two implementations of Node: One that is a flat array
and implemented as ByteString is currently, and one which is a concatenation
of some set of sub-nodes. The ByteString itself then just has a single
field which points to the root Node. Hopefully, this approach would allow
the JIT to inline calls to get() in the common case where ByteStrings are
composed of just one flat array, so that this case does not become
significantly slower than it is now. This approach also makes it easy to
implement concatenation in O(1) time and substring in O(log n).
-- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
