[
https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638981#comment-13638981
]
Leo Romanoff commented on AVRO-1282:
------------------------------------
I did some experiments with Unsafe IO, i.e. I've created
UnsafeBufferedBinaryEncoder and UnsafeBinaryDecoder. Both are derived from the
corresponding classes without Unsafe prefix. The only thing I did so far is to
redefine the read/writeFloat and read/writeDouble to use Unsafe. Below you can
see the results:
Unsafe Binary Encoder/Decoder
test name time M entries/sec M bytes/sec bytes/cycle
FloatRead: 6184 ms 323,374 1293,497 1000000
FloatWrite: 6405 ms 312,232 1248,927 1000000
DoubleRead: 7424 ms 269,371 2154,968 2000000
DoubleWrite: 8221 ms 243,262 1946,097 2000000
Standard BinaryEncoder/Decoder:
test name time M entries/sec M bytes/sec bytes/cycle
FloatRead: 9181 ms 217,827 871,307 1000000
FloatWrite: 9963 ms 200,736 802,943 1000000
DoubleRead: 15589 ms 128,289 1026,316 2000000
DoubleWrite: 15583 ms 128,339 1026,710 2000000
So, it seems to be a win to have Unsafe-based IO streams.
>From experience with Kryo, one can get much better results for arrays of
>primitives - I've observed results like 20x or 40x compared to the standard
>version, which does not use Unsafe. I.e. Unsafe can efficiently write/read
>whole arrays of primitives in one go. The drawback of it: It uses native byte
>order and fixed size representations per element. You also cannot write
>anything between elements. It really simply dumps memory into the target
>buffer.
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports
> it
> -------------------------------------------------------------------------------
>
> Key: AVRO-1282
> URL: https://issues.apache.org/jira/browse/AVRO-1282
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.4
> Reporter: Leo Romanoff
> Priority: Minor
> Attachments: avro-1282-v1.patch, avro-1282-v2.patch,
> avro-1282-v3.patch, avro-1282-v4.patch, avro-1282-v5.patch, avro-1282-v6.patch
>
>
> Unsafe can be used to significantly speed up serialization process, if a JDK
> implementation supports java.misc.Unsafe properly. Most JDKs running on PCs
> support it. Some platforms like Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the
> reflection-based approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick
> input/output.
>
> 3) More over, Unsafe makes it possible to serialize to/deserialize from
> off-heap memory directly and very quickly, without any intermediate buffers
> allocated on heap. There is virtually no overhead compared to the usual byte
> arrays.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
