[
https://issues.apache.org/jira/browse/SPARK-25839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-25839.
-------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 22855
[https://github.com/apache/spark/pull/22855]
> Implement use of KryoPool in KryoSerializer
> -------------------------------------------
>
> Key: SPARK-25839
> URL: https://issues.apache.org/jira/browse/SPARK-25839
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.0.2, 2.3.1, 2.3.2
> Reporter: Patrick Brown
> Assignee: Patrick Brown
> Priority: Minor
> Fix For: 3.0.0
>
>
> The current implementation of KryoSerializer does not use KryoPool, which is
> recommended by Kryo due to the creation of a Kryo instance being slow.
>
> The current implementation references the KryoSerializerInstance private
> variable cachedKryo as effectively being a pool of size 1. However (in my
> admittedly somewhat limited research) it seems that frequently (such as in
> the ClosureCleaner ensureSerializable method) a new instance of
> KryoSerializerInstance is created, which in turn forces a new instance of
> Kryo itself to be created, this instance is then dropped from scope, causing
> the "pool" not to be re-used.
>
> I have a small set of proposed changes we have been using on an internal
> production application (running 24x7 for 6+ months, processing 10k+ jobs a
> day) which implements using a KryoPool inside KryoSerializer which is then
> used by each KryoSerializerInstance to borrow a Kryo instance.
>
> I believe this is mainly a performance improvement for applications
> processing a large number of small jobs, where the cost of instantiating Kryo
> instances is a larger portion of execution time compared to larger jobs.
>
> I have discussed this proposed change in the dev mailing list and it was
> suggested I create this issue and a PR. It was also suggested I accompany
> that with some performance metrics, which it is my plan to do.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
