If we add this support in the new version, we can afford to add a new
interface for the new primitive-enabled API.

But we probably will need to keep the old interface as wrapper or something...

I think that, for the sake of saving many hours, one could prototype
using the Apache stuff. If the benchmarks are good, then we can think
about porting it and tweaking to fit our needs better. That might be
more enjoyable than tweaking GC settings :)

Marc-André LAVERDIÈRE
"Perseverance must finish its work so that you may be mature and
complete, not lacking anything." -James 1:4
mlaverd.theunixplace.com/blog

 /"\
 \ /    ASCII Ribbon Campaign
  X      against HTML e-mail
 / \



2010/6/30 Josh Hartman <atomicfo...@gmail.com>:
> I've started working on implementations for this. We can also keep the
> current interface stable by having our custom IntList implement
> List<Integer>. Unfortunately, users will have to pay the boxing penalty for
> certain operations but we can make others a lot faster and save memory.
> I've also added ensureCapacity. The code looks like this in
> java_primitive_field.cc:
> "public Builder ensure$capitalized_name$Capacity(int minCapacity) {\n"
> "  if (result.$name$_.isEmpty()) {\n"
> "    result.$name$_ = new java.util.ArrayList<$boxed_type$>(minCapacity);\n"
> "  } else {\n"
> "    ((java.util.ArrayList<$boxed_type$>)
> result.$name$_).ensureCapacity(minCapacity);\n"
> "  }\n"
> "  return this;\n"
> . Unfortunately, it's actually slower for me to use this than not. I've
> played around for hours to get optimal GC settings since GC contributes a
> significant portion of the execution time, but setting the capacity before
> adding all the elements actually takes a tad longer. Does anyone have any
> suggestions why this might be the case? I see nothing weird when poring over
> the GC logs or anything. I'm on OSX 1.6.20 64bit sun JVM. I'm going to try
> again on a windows VM when I can get the chance.
> The relevant code I've been testing is
> at http://sites.google.com/site/jhartmancode/files/SerializationTests2.zip
> Josh Hartman
>
>
> 2010/6/28 Marc-André Laverdière <marcandre.laverdi...@gmail.com>
>>
>> Good question.
>>
>> I think you could re-implement whatever those libraries we talked
>> about are doing. Conceptually it is very simple, but I'm pretty sure
>> that you must have a bunch of tricky things for performance to be
>> careful.
>> So a naive implementation (with memcpys) could be done in a day I
>> think. A more solid implementation (no memcpys) might take another
>> day. More optimizations: no clue.
>>
>> But this is going to be very very boilerplate-like. It might be nice
>> to have a code generator (something as simple as StringTemplate). It
>> would save you a lot of time.
>>
>> Marc-André LAVERDIÈRE
>> "Perseverance must finish its work so that you may be mature and
>> complete, not lacking anything." -James 1:4
>> mlaverd.theunixplace.com/blog
>>
>>  /"\
>>  \ /    ASCII Ribbon Campaign
>>  X      against HTML e-mail
>>  / \
>>
>>
>>
>> 2010/6/29 Kenton Varda <ken...@google.com>:
>> > Ideally, we want to avoid dependencies if at all possible.  The library
>> > is
>> > already considered to be too big (motivating protobuf-lite, among other
>> > things).  A dependency could make things worse.
>> > I wonder how much code it would take to implement an absolutely minimal
>> > set
>> > of primitive-value ArrayLists, designed solely for use by protobufs.
>> > And yeah, providing a way to call ensureCapacity() seems worthwhile.
>> >
>> > On Thu, Jun 24, 2010 at 11:05 AM, Josh Hartman <atomicfo...@gmail.com>
>> > wrote:
>> >>
>> >> Using something like Fastutil or the collections library posted would
>> >> definitely improve performance. Google guys/committers - what do you
>> >> think
>> >> about adding a dependency on these libraries? I don't mind contributing
>> >> code
>> >> for this.
>> >> I still think a huge performance increase could be seen if we could
>> >> preallocate the ArrayList in the Builder. Looking at generated code for
>> >> a
>> >> simple repeated int32 type, whenever an item is added to the list
>> >> ArrayLists's append is called. If we exposed ArrayList's ensureCapacity
>> >> method, we could reserve the capacity we needed and wouldn't have to
>> >> reallocate objects and copy over as we build up the List.
>> >>
>> >>
>> >> 2010/6/23 Marc-André Laverdière <marcandre.laverdi...@gmail.com>
>> >>>
>> >>> There is an Apache Commons project that provide the primitive lists:
>> >>> http://commons.apache.org/primitives/
>> >>>
>> >>> I think that those would do the heavy lifting for us :)
>> >>>
>> >>> Marc-André LAVERDIÈRE
>> >>> "Perseverance must finish its work so that you may be mature and
>> >>> complete, not lacking anything." -James 1:4
>> >>> mlaverd.theunixplace.com/blog
>> >>>
>> >>>  /"\
>> >>>  \ /    ASCII Ribbon Campaign
>> >>>  X      against HTML e-mail
>> >>>  / \
>> >>>
>> >>>
>> >>>
>> >>> 2010/6/24 Kenton Varda <ken...@google.com>:
>> >>> > This is a use case that is, unfortunately, not well-optimized by the
>> >>> > Java
>> >>> > implementation.  What's needed is versions of ArrayList which store
>> >>> > unboxed
>> >>> > values.  Unfortunately, a separate such implementation would be
>> >>> > needed
>> >>> > for
>> >>> > every primitive field type, since Java generics work only with boxed
>> >>> > types.
>> >>> >  It will be tedious, but you're welcome to work on it.
>> >>> >
>> >>> > On Tue, Jun 22, 2010 at 7:16 PM, Joshua Hartman
>> >>> > <atomicfo...@gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> I'm trying to serialize a large primitive array of int in Java, and
>> >>> >> I'm testing using 1,000,000 elements. My .proto message is just a
>> >>> >> packed repeated field of type int32. To serialize, I just create a
>> >>> >> builder and loop through the array, adding elements one at a time.
>> >>> >> I
>> >>> >> took a peek at the underlying implementation and it seems that it
>> >>> >> uses
>> >>> >> an ArrayList. However, there's no way to reserve the size of the
>> >>> >> ArrayList, so serializing is slower than it could be. In fact, it's
>> >>> >> faster for me to just serialize the data an ObjectOutputStream to
>> >>> >> put
>> >>> >> the data in a protocol buffer bytes message.
>> >>> >>
>> >>> >> Is there a way for me to do the serialization faster? I took a peek
>> >>> >> through the API but didn't see anything obvious. I saw
>> >>> >> CodedOutputStream as well, but I didn't understand the intent of
>> >>> >> that
>> >>> >> class. Could anyone please explain what it's used for?
>> >>> >>
>> >>> >> Thanks,
>> >>> >> Josh H
>> >>> >>
>> >>> >> --
>> >>> >> You received this message because you are subscribed to the Google
>> >>> >> Groups
>> >>> >> "Protocol Buffers" group.
>> >>> >> To post to this group, send email to proto...@googlegroups.com.
>> >>> >> To unsubscribe from this group, send email to
>> >>> >> protobuf+unsubscr...@googlegroups.com.
>> >>> >> For more options, visit this group at
>> >>> >> http://groups.google.com/group/protobuf?hl=en.
>> >>> >>
>> >>> >
>> >>> > --
>> >>> > You received this message because you are subscribed to the Google
>> >>> > Groups
>> >>> > "Protocol Buffers" group.
>> >>> > To post to this group, send email to proto...@googlegroups.com.
>> >>> > To unsubscribe from this group, send email to
>> >>> > protobuf+unsubscr...@googlegroups.com.
>> >>> > For more options, visit this group at
>> >>> > http://groups.google.com/group/protobuf?hl=en.
>> >>> >
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "Protocol Buffers" group.
>> >>> To post to this group, send email to proto...@googlegroups.com.
>> >>> To unsubscribe from this group, send email to
>> >>> protobuf+unsubscr...@googlegroups.com.
>> >>> For more options, visit this group at
>> >>> http://groups.google.com/group/protobuf?hl=en.
>> >>>
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "Protocol Buffers" group.
>> >> To post to this group, send email to proto...@googlegroups.com.
>> >> To unsubscribe from this group, send email to
>> >> protobuf+unsubscr...@googlegroups.com.
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/protobuf?hl=en.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Protocol Buffers" group.
>> > To post to this group, send email to proto...@googlegroups.com.
>> > To unsubscribe from this group, send email to
>> > protobuf+unsubscr...@googlegroups.com.
>> > For more options, visit this group at
>> > http://groups.google.com/group/protobuf?hl=en.
>> >
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Protocol Buffers" group.
>> To post to this group, send email to proto...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> protobuf+unsubscr...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/protobuf?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Protocol Buffers" group.
> To post to this group, send email to proto...@googlegroups.com.
> To unsubscribe from this group, send email to
> protobuf+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/protobuf?hl=en.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Protocol Buffers" group.
To post to this group, send email to proto...@googlegroups.com.
To unsubscribe from this group, send email to 
protobuf+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/protobuf?hl=en.

Reply via email to