[
https://issues.apache.org/jira/browse/SPARK-28102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen resolved SPARK-28102.
--------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
Issue resolved by pull request 24905
[https://github.com/apache/spark/pull/24905]
> Failed LZ4 JNI initialization is repeatedly re-attempted, causing lock
> contention issues
> ----------------------------------------------------------------------------------------
>
> Key: SPARK-28102
> URL: https://issues.apache.org/jira/browse/SPARK-28102
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
> Fix For: 3.0.0
>
>
> Spark's use of {{lz4-java}} ends up calling {{LZ4Factory.fastestInstance}},
> which attempts to load JNI libraries and falls back on Java implementations
> in case the JNI library cannot be loaded or initialized.
> Unfortunately, the LZ4 library does not cache the fact that the JNI library
> failed to load, so every call to {{LZ4Factory.fastestInstance}} ends up
> repeating this failed load. These initialization attempts are performed in a
> static synchronized method, so exceptions from failures are thrown while
> holding a global lock and this causes lock-contention performance issues.
> To avoid this problem, we can cache the LZ4Factory ourselves (since it's
> thread-safe), thereby caching the outcome of the failed load and avoiding
> performance problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
