Good question. I think you could re-implement whatever those libraries we talked about are doing. Conceptually it is very simple, but I'm pretty sure that you must have a bunch of tricky things for performance to be careful. So a naive implementation (with memcpys) could be done in a day I think. A more solid implementation (no memcpys) might take another day. More optimizations: no clue.
But this is going to be very very boilerplate-like. It might be nice to have a code generator (something as simple as StringTemplate). It would save you a lot of time. Marc-André LAVERDIÈRE "Perseverance must finish its work so that you may be mature and complete, not lacking anything." -James 1:4 mlaverd.theunixplace.com/blog /"\ \ / ASCII Ribbon Campaign X against HTML e-mail / \ 2010/6/29 Kenton Varda <ken...@google.com>: > Ideally, we want to avoid dependencies if at all possible. The library is > already considered to be too big (motivating protobuf-lite, among other > things). A dependency could make things worse. > I wonder how much code it would take to implement an absolutely minimal set > of primitive-value ArrayLists, designed solely for use by protobufs. > And yeah, providing a way to call ensureCapacity() seems worthwhile. > > On Thu, Jun 24, 2010 at 11:05 AM, Josh Hartman <atomicfo...@gmail.com> > wrote: >> >> Using something like Fastutil or the collections library posted would >> definitely improve performance. Google guys/committers - what do you think >> about adding a dependency on these libraries? I don't mind contributing code >> for this. >> I still think a huge performance increase could be seen if we could >> preallocate the ArrayList in the Builder. Looking at generated code for a >> simple repeated int32 type, whenever an item is added to the list >> ArrayLists's append is called. If we exposed ArrayList's ensureCapacity >> method, we could reserve the capacity we needed and wouldn't have to >> reallocate objects and copy over as we build up the List. >> >> >> 2010/6/23 Marc-André Laverdière <marcandre.laverdi...@gmail.com> >>> >>> There is an Apache Commons project that provide the primitive lists: >>> http://commons.apache.org/primitives/ >>> >>> I think that those would do the heavy lifting for us :) >>> >>> Marc-André LAVERDIÈRE >>> "Perseverance must finish its work so that you may be mature and >>> complete, not lacking anything." -James 1:4 >>> mlaverd.theunixplace.com/blog >>> >>> /"\ >>> \ / ASCII Ribbon Campaign >>> X against HTML e-mail >>> / \ >>> >>> >>> >>> 2010/6/24 Kenton Varda <ken...@google.com>: >>> > This is a use case that is, unfortunately, not well-optimized by the >>> > Java >>> > implementation. What's needed is versions of ArrayList which store >>> > unboxed >>> > values. Unfortunately, a separate such implementation would be needed >>> > for >>> > every primitive field type, since Java generics work only with boxed >>> > types. >>> > It will be tedious, but you're welcome to work on it. >>> > >>> > On Tue, Jun 22, 2010 at 7:16 PM, Joshua Hartman <atomicfo...@gmail.com> >>> > wrote: >>> >> >>> >> Hi, >>> >> >>> >> I'm trying to serialize a large primitive array of int in Java, and >>> >> I'm testing using 1,000,000 elements. My .proto message is just a >>> >> packed repeated field of type int32. To serialize, I just create a >>> >> builder and loop through the array, adding elements one at a time. I >>> >> took a peek at the underlying implementation and it seems that it uses >>> >> an ArrayList. However, there's no way to reserve the size of the >>> >> ArrayList, so serializing is slower than it could be. In fact, it's >>> >> faster for me to just serialize the data an ObjectOutputStream to put >>> >> the data in a protocol buffer bytes message. >>> >> >>> >> Is there a way for me to do the serialization faster? I took a peek >>> >> through the API but didn't see anything obvious. I saw >>> >> CodedOutputStream as well, but I didn't understand the intent of that >>> >> class. Could anyone please explain what it's used for? >>> >> >>> >> Thanks, >>> >> Josh H >>> >> >>> >> -- >>> >> You received this message because you are subscribed to the Google >>> >> Groups >>> >> "Protocol Buffers" group. >>> >> To post to this group, send email to proto...@googlegroups.com. >>> >> To unsubscribe from this group, send email to >>> >> protobuf+unsubscr...@googlegroups.com. >>> >> For more options, visit this group at >>> >> http://groups.google.com/group/protobuf?hl=en. >>> >> >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> > Groups >>> > "Protocol Buffers" group. >>> > To post to this group, send email to proto...@googlegroups.com. >>> > To unsubscribe from this group, send email to >>> > protobuf+unsubscr...@googlegroups.com. >>> > For more options, visit this group at >>> > http://groups.google.com/group/protobuf?hl=en. >>> > >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Protocol Buffers" group. >>> To post to this group, send email to proto...@googlegroups.com. >>> To unsubscribe from this group, send email to >>> protobuf+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/protobuf?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To post to this group, send email to proto...@googlegroups.com. >> To unsubscribe from this group, send email to >> protobuf+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/protobuf?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To post to this group, send email to proto...@googlegroups.com. > To unsubscribe from this group, send email to > protobuf+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/protobuf?hl=en. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To post to this group, send email to proto...@googlegroups.com. To unsubscribe from this group, send email to protobuf+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
