[
https://issues.apache.org/jira/browse/SPARK-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265805#comment-15265805
]
Sandeep Singh commented on SPARK-928:
-------------------------------------
[~joshrosen] I've started working on it.
I tried benchmarking the difference between unsafe kryo and our current impl.
and then we can have a spark.kryo.useUnsafe flag as Matei has mentioned.
{code:title=Without Kryo UnSafe|borderStyle=solid}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
Serialize and then deserialize: Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------
primitive:Long 1 / 4 11223.1 0.1
1.0X
primitive:Double 1 / 1 19409.0 0.1
1.7X
Array:Long 38 / 49 412.4 2.4
0.0X
Array:Double 25 / 35 631.4 1.6
0.1X
Map of string->Double 2651 / 2766 5.9 168.6
0.0X
{code}
{code:title=With Kryo UnSafe|borderStyle=solid}
Java HotSpot(TM) 64-Bit Server VM 1.8.0_60-b27 on Mac OS X 10.11.4
Intel(R) Core(TM) i7-4770HQ CPU @ 2.20GHz
Serialize and then deserialize: Best/Avg Time(ms) Rate(M/s) Per
Row(ns) Relative
-------------------------------------------------------------------------------------------
primitive:Long 1 / 3 15872.0 0.1
1.0X
primitive:Double 1 / 1 17769.7 0.1
1.1X
Array:Long 24 / 42 642.3 1.6
0.0X
Array:Double 22 / 26 719.4 1.4
0.0X
Map of string->Double 2560 / 2582 6.1 162.8
0.0X
{code}
You can find the code for benchmarking here
(https://github.com/techaddict/spark/commit/46fa44141c849ca15bbd6136cea2fa52bd927da2),
very ugly right now but will improve it(add more benchmarks) before creating a
PR.
> Add support for Unsafe-based serializer in Kryo 2.22
> ----------------------------------------------------
>
> Key: SPARK-928
> URL: https://issues.apache.org/jira/browse/SPARK-928
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Matei Zaharia
> Priority: Minor
> Labels: starter
>
> This can reportedly be quite a bit faster, but it also requires Chill to
> update its Kryo dependency. Once that happens we should add a
> spark.kryo.useUnsafe flag.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
