I've started working on implementations for this. We can also keep the
current interface stable by having our custom IntList implement
List<Integer>. Unfortunately, users will have to pay the boxing penalty for
certain operations but we can make others a lot faster and save memory.
I've also added ensureCapacity. The code looks like this in
java_primitive_field.cc:
"public Builder ensure$capitalized_name$Capacity(int minCapacity) {\n"
" if (result.$name$_.isEmpty()) {\n"
" result.$name$_ = new java.util.ArrayList<$boxed_type$>(minCapacity);\n"
" } else {\n"
" ((java.util.ArrayList<$boxed_type$>)
result.$name$_).ensureCapacity(minCapacity);\n"
" }\n"
" return this;\n"
. Unfortunately, it's actually slower for me to use this than not. I've
played around for hours to get optimal GC settings since GC contributes a
significant portion of the execution time, but setting the capacity before
adding all the elements actually takes a tad longer. Does anyone have any
suggestions why this might be the case? I see nothing weird when poring over
the GC logs or anything. I'm on OSX 1.6.20 64bit sun JVM. I'm going to try
again on a windows VM when I can get the chance.
The relevant code I've been testing is at
http://sites.google.com/site/jhartmancode/files/SerializationTests2.zip
Josh Hartman
2010/6/28 Marc-André Laverdière <marcandre.laverdi...@gmail.com>
> Good question.
>
> I think you could re-implement whatever those libraries we talked
> about are doing. Conceptually it is very simple, but I'm pretty sure
> that you must have a bunch of tricky things for performance to be
> careful.
> So a naive implementation (with memcpys) could be done in a day I
> think. A more solid implementation (no memcpys) might take another
> day. More optimizations: no clue.
>
> But this is going to be very very boilerplate-like. It might be nice
> to have a code generator (something as simple as StringTemplate). It
> would save you a lot of time.
>
> Marc-André LAVERDIÈRE
> "Perseverance must finish its work so that you may be mature and
> complete, not lacking anything." -James 1:4
> mlaverd.theunixplace.com/blog
>
> /"\
> \ / ASCII Ribbon Campaign
> X against HTML e-mail
> / \
>
>
>
> 2010/6/29 Kenton Varda <ken...@google.com>:
> > Ideally, we want to avoid dependencies if at all possible. The library
> is
> > already considered to be too big (motivating protobuf-lite, among other
> > things). A dependency could make things worse.
> > I wonder how much code it would take to implement an absolutely minimal
> set
> > of primitive-value ArrayLists, designed solely for use by protobufs.
> > And yeah, providing a way to call ensureCapacity() seems worthwhile.
> >
> > On Thu, Jun 24, 2010 at 11:05 AM, Josh Hartman <atomicfo...@gmail.com>
> > wrote:
> >>
> >> Using something like Fastutil or the collections library posted would
> >> definitely improve performance. Google guys/committers - what do you
> think
> >> about adding a dependency on these libraries? I don't mind contributing
> code
> >> for this.
> >> I still think a huge performance increase could be seen if we could
> >> preallocate the ArrayList in the Builder. Looking at generated code for
> a
> >> simple repeated int32 type, whenever an item is added to the list
> >> ArrayLists's append is called. If we exposed ArrayList's ensureCapacity
> >> method, we could reserve the capacity we needed and wouldn't have to
> >> reallocate objects and copy over as we build up the List.
> >>
> >>
> >> 2010/6/23 Marc-André Laverdière <marcandre.laverdi...@gmail.com>
> >>>
> >>> There is an Apache Commons project that provide the primitive lists:
> >>> http://commons.apache.org/primitives/
> >>>
> >>> I think that those would do the heavy lifting for us :)
> >>>
> >>> Marc-André LAVERDIÈRE
> >>> "Perseverance must finish its work so that you may be mature and
> >>> complete, not lacking anything." -James 1:4
> >>> mlaverd.theunixplace.com/blog
> >>>
> >>> /"\
> >>> \ / ASCII Ribbon Campaign
> >>> X against HTML e-mail
> >>> / \
> >>>
> >>>
> >>>
> >>> 2010/6/24 Kenton Varda <ken...@google.com>:
> >>> > This is a use case that is, unfortunately, not well-optimized by the
> >>> > Java
> >>> > implementation. What's needed is versions of ArrayList which store
> >>> > unboxed
> >>> > values. Unfortunately, a separate such implementation would be
> needed
> >>> > for
> >>> > every primitive field type, since Java generics work only with boxed
> >>> > types.
> >>> > It will be tedious, but you're welcome to work on it.
> >>> >
> >>> > On Tue, Jun 22, 2010 at 7:16 PM, Joshua Hartman <
> atomicfo...@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> I'm trying to serialize a large primitive array of int in Java, and
> >>> >> I'm testing using 1,000,000 elements. My .proto message is just a
> >>> >> packed repeated field of type int32. To serialize, I just create a
> >>> >> builder and loop through the array, adding elements one at a time. I
> >>> >> took a peek at the underlying implementation and it seems that it
> uses
> >>> >> an ArrayList. However, there's no way to reserve the size of the
> >>> >> ArrayList, so serializing is slower than it could be. In fact, it's
> >>> >> faster for me to just serialize the data an ObjectOutputStream to
> put
> >>> >> the data in a protocol buffer bytes message.
> >>> >>
> >>> >> Is there a way for me to do the serialization faster? I took a peek
> >>> >> through the API but didn't see anything obvious. I saw
> >>> >> CodedOutputStream as well, but I didn't understand the intent of
> that
> >>> >> class. Could anyone please explain what it's used for?
> >>> >>
> >>> >> Thanks,
> >>> >> Josh H
> >>> >>
> >>> >> --
> >>> >> You received this message because you are subscribed to the Google
> >>> >> Groups
> >>> >> "Protocol Buffers" group.
> >>> >> To post to this group, send email to proto...@googlegroups.com.
> >>> >> To unsubscribe from this group, send email to
> >>> >> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> .
> >>> >> For more options, visit this group at
> >>> >> http://groups.google.com/group/protobuf?hl=en.
> >>> >>
> >>> >
> >>> > --
> >>> > You received this message because you are subscribed to the Google
> >>> > Groups
> >>> > "Protocol Buffers" group.
> >>> > To post to this group, send email to proto...@googlegroups.com.
> >>> > To unsubscribe from this group, send email to
> >>> > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> .
> >>> > For more options, visit this group at
> >>> > http://groups.google.com/group/protobuf?hl=en.
> >>> >
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> Groups
> >>> "Protocol Buffers" group.
> >>> To post to this group, send email to proto...@googlegroups.com.
> >>> To unsubscribe from this group, send email to
> >>> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> .
> >>> For more options, visit this group at
> >>> http://groups.google.com/group/protobuf?hl=en.
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "Protocol Buffers" group.
> >> To post to this group, send email to proto...@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> .
> >> For more options, visit this group at
> >> http://groups.google.com/group/protobuf?hl=en.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to proto...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> .
> > For more options, visit this group at
> > http://groups.google.com/group/protobuf?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com<protobuf%2bunsubscr...@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.
