[
https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637508#comment-13637508
]
Leo Romanoff commented on AVRO-1282:
------------------------------------
@Doug: Thanks that you confirmed my measurements! 3x is nice indeed. On my
MacBook Air with i7 and 8 GB of memory I've seen even 4-5 speedups on the same
tests. So, it seems to be a big performance win in any case. Do you plan to
include it into the trunk? If so, when?
Coming back to the beginning of this thread:
Improvements we have seen so far are purely due to the optimization of accesses
to the fields of structures and avoiding reflection as much as possible.
Based on my experience with Kryo, it could be even more beneficial to use
Unsafe-based approach for Input/Output streams. But this would affect the
format of produced data, because Unsafe reads/writes primitive types using
their native size and native byte order. We touched a bit upon this at the
beginning, but may be it is time now to decide what to do with it? Do you see
any option to support it in Avro, e.g. by introducing a new format id or
something like this? or may be by introducing new type ids (something like
LITTLE_ENDIAN_INT, LITTLE_ENDIAN_LONG, LITTLE_ENDIAN_FLOAT,
LITTLE_ENDIAN_DOUBLE,etc and the same for BIG_ENDIAN). This way one could use
both encodings for different parts/fields of the same object grpah. What do you
think?
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports
> it
> -------------------------------------------------------------------------------
>
> Key: AVRO-1282
> URL: https://issues.apache.org/jira/browse/AVRO-1282
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.4
> Reporter: Leo Romanoff
> Priority: Minor
> Attachments: avro-1282-v1.patch, avro-1282-v2.patch,
> avro-1282-v3.patch, avro-1282-v4.patch, avro-1282-v5.patch, avro-1282-v6.patch
>
>
> Unsafe can be used to significantly speed up serialization process, if a JDK
> implementation supports java.misc.Unsafe properly. Most JDKs running on PCs
> support it. Some platforms like Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the
> reflection-based approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick
> input/output.
>
> 3) More over, Unsafe makes it possible to serialize to/deserialize from
> off-heap memory directly and very quickly, without any intermediate buffers
> allocated on heap. There is virtually no overhead compared to the usual byte
> arrays.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
