[
https://issues.apache.org/jira/browse/AVRO-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scott Carey updated AVRO-1282:
------------------------------
Attachment: AVRO-1282-s6.patch
This patch (AVRO-1282-s6.patch) includes the following:
* The Reflect API now uses sun.misc.Unsafe to do Field reflection, for
significantly improved performance. (Leo)
* The Reflect API avoids boxing for reading and writing primitive arrays.
(Leo, Scott)
* Resolution of whether it is safe to use Unsafe is done statically, only one
implementation is loaded. The Unsafe implementation is tested to ensure that
all features function properly at load time (for example, to handle Android or
other JVMs with partial Unsafe support). A unit test is added that uses a
classloader that fails to load Unsafe to test this. (Leo, Scott)
* EncoderFactory is fixed to properly configure the BlockingBinaryEncoder.
(Scott)
* Reflection now supports reading the Blocked encoding into native arrays.
(Scott)
* A dozen new tests added to Perf.java to cover the
ReflectDatum{Reader,Writer}, including with the blocked binary encoding. (Leo,
Scott)
* Additional unit tests in TestReflect to cover encoding and decoding primitive
arrays, including with blocked binary encoding. (Scott)
Performance on Reflect Perf.java tests is approximately 2.5x to 29x faster,
usually between 3x and 5x faster.
Before:
{noformat}
test name time M entries/sec M
bytes/sec bytes/cycle
ReflectRecordRead: 15067 ms 1.106 42.927
808498
ReflectRecordWrite: 10903 ms 1.529 59.319
808498
ReflectBigRecordRead: 19285 ms 0.519 31.832
767380
ReflectBigRecordWrite: 13958 ms 0.716 43.979
767380
ReflectFloatRead: 29594 ms 0.000 27.032
1000004
ReflectFloatWrite: 33327 ms 0.000 24.004
1000004
ReflectDoubleRead: 27442 ms 0.000 58.303
2000004
ReflectDoubleWrite: 31529 ms 0.000 50.746
2000004
ReflectIntArrayRead: 38088 ms 0.438 18.057
859709
ReflectIntArrayWrite: 23342 ms 0.714 29.464
859709
ReflectLongArrayRead: 18476 ms 0.451 34.869
805344
ReflectLongArrayWrite: 12715 ms 0.655 50.667
805344
ReflectDoubleArrayRead: 19411 ms 0.515 33.718
818144
ReflectDoubleArrayWrite: 13825 ms 0.723 47.340
818144
ReflectFloatArrayRead: 39502 ms 0.506 17.137
846172
ReflectFloatArrayWrite: 27492 ms 0.727 24.623
846172
ReflectNestedFloatArrayRead: 41225 ms 0.485 16.420
846172
ReflectNestedFloatArrayWrite: 30229 ms 0.662 22.393
846172
ReflectNestedObjectArrayRead: 31679 ms 0.126 16.291
645104
ReflectNestedObjectArrayWrite: 17206 ms 0.232 29.994
645104
ReflectNestedLargeFloatArrayRead: 33099 ms 0.101 26.282
1087381
ReflectNestedLargeFloatArrayWrite: 35159 ms 0.095 24.742
1087381
ReflectNestedLargeFloatArrayBlockedRead: 33326 ms 0.100 26.302
1095674
ReflectNestedLargeFloatArrayBlockedWrite: 36921 ms 0.090 23.741
1095674
{noformat}
After:
{noformat}
test name time M entries/sec M
bytes/sec bytes/cycle
ReflectRecordRead: 6058 ms 2.751 106.754
808498
ReflectRecordWrite: 3750 ms 4.444 172.470
808498
ReflectBigRecordRead: 6767 ms 1.478 90.709
767380
ReflectBigRecordWrite: 4433 ms 2.255 138.466
767380
ReflectFloatRead: 6155 ms 0.000 129.970
1000004
ReflectFloatWrite: 1083 ms 0.001 738.434
1000004
ReflectDoubleRead: 6610 ms 0.000 242.028
2000004
ReflectDoubleWrite: 1968 ms 0.000 812.864
2000004
ReflectIntArrayRead: 9462 ms 1.761 72.683
859709
ReflectIntArrayWrite: 2468 ms 6.751 278.584
859709
ReflectLongArrayRead: 5556 ms 1.500 115.941
805344
ReflectLongArrayWrite: 1851 ms 4.500 347.921
805344
ReflectDoubleArrayRead: 5755 ms 1.737 113.719
818144
ReflectDoubleArrayWrite: 1423 ms 7.027 459.900
818144
ReflectFloatArrayRead: 11373 ms 1.758 59.519
846172
ReflectFloatArrayWrite: 2267 ms 8.821 298.557
846172
ReflectNestedFloatArrayRead: 12626 ms 1.584 53.610
846172
ReflectNestedFloatArrayWrite: 4216 ms 4.743 160.542
846172
ReflectNestedObjectArrayRead: 10522 ms 0.380 49.046
645104
ReflectNestedObjectArrayWrite: 5565 ms 0.719 92.725
645104
ReflectNestedLargeFloatArrayRead: 8022 ms 0.415 108.436
1087381
ReflectNestedLargeFloatArrayWrite: 1788 ms 1.863 486.359
1087381
ReflectNestedLargeFloatArrayBlockedRead: 8098 ms 0.412 108.799
1101357
ReflectNestedLargeFloatArrayBlockedWrite: 7511 ms 0.444 117.295
1101357
{noformat}
> Make use of the sun.misc.Unsafe class during serialization if a JDK supports
> it
> -------------------------------------------------------------------------------
>
> Key: AVRO-1282
> URL: https://issues.apache.org/jira/browse/AVRO-1282
> Project: Avro
> Issue Type: Improvement
> Components: java
> Affects Versions: 1.7.4
> Reporter: Leo Romanoff
> Priority: Minor
> Attachments: AVRO-1282-s1.patch, AVRO-1282-s2.patch,
> AVRO-1282-s3.patch, AVRO-1282-s5.patch, AVRO-1282-s6.patch,
> avro-1282-v1.patch, avro-1282-v2.patch, avro-1282-v3.patch,
> avro-1282-v4.patch, avro-1282-v5.patch, avro-1282-v6.patch,
> avro-1282-v7.patch, avro-1282-v8.patch, AVRO-1282-v9.patch,
> TestUnsafeUtil.java
>
>
> Unsafe can be used to significantly speed up serialization process, if a JDK
> implementation supports java.misc.Unsafe properly. Most JDKs running on PCs
> support it. Some platforms like Android lack a proper support for Unsafe yet.
> There are two possibilities to use Unsafe for serialization:
> 1) Very quick access to the fields of objects. It is way faster than with the
> reflection-based approach using Field.get/set
> 2) Input and Output streams can be using Unsafe to perform very quick
> input/output.
>
> 3) More over, Unsafe makes it possible to serialize to/deserialize from
> off-heap memory directly and very quickly, without any intermediate buffers
> allocated on heap. There is virtually no overhead compared to the usual byte
> arrays.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
