[
https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13610942#comment-13610942
]
Leo Romanoff commented on AVRO-1282:
------------------------------------
One thing I'd like to ask is the following:
I've implemented this kind of optimization for Kryo and the results are
impressive.
While doing this, I almost have rewritten most of the Input/Output classes from
Kryo. They now support Unsafe-based streams and contribute the most to the
speed ups. Since these IO classes are pretty self-contained and do not really
depend on Kryo, it could be interesting to reuse them for Avro (and may be for
other serialization frameworks).
What do you think about this idea?
Note:
One of the biggest wins for Kryo was Unsafe-based serialization of arrays. In
this mode, Kryo would use native byte order and native size of primitive types
(because this is what Unsafe supports), but it can serialize whole arrays of
primitive types extremely quickly. Overall, most of speed ups coming from
Unsafe-based streams are due to the usage of very efficient native byte order
operations by Unsafe. E.g. it can serialize primitive types as a whole and not
byte-by-byte.
Since Avro uses a platform-independent format, this feature cannot be used with
Avro then I guess. Or one could enable this mode optionally with a disclaimer
that is makes serialized representation non-portable between platforms with
different byte-orders.
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports
> it
> -------------------------------------------------------------------------------
>
> Key: AVRO-1282
> URL: https://issues.apache.org/jira/browse/AVRO-1282
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.4
> Reporter: Leo Romanoff
> Priority: Minor
>
> Unsafe can be used to significantly speed up serialization process, if a JDK
> implementation supports java.misc.Unsafe properly. Most JDKs running on PCs
> support it. Some platforms like Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the
> reflection-based approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick
> input/output.
>
> 3) More over, Unsafe makes it possible to serialize to/deserialize from
> off-heap memory directly and very quickly, without any intermediate buffers
> allocated on heap. There is virtually no overhead compared to the usual byte
> arrays.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
