[
https://issues.apache.org/jira/browse/SPARK-28102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-28102:
------------------------------------
Assignee: Apache Spark (was: Josh Rosen)
> Add configuration for selecting LZ4 implementation (safe, unsafe, JNI)
> ----------------------------------------------------------------------
>
> Key: SPARK-28102
> URL: https://issues.apache.org/jira/browse/SPARK-28102
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Josh Rosen
> Assignee: Apache Spark
> Priority: Major
>
> Spark's use of {{lz4-java}} ends up calling {{Lz4Factory.fastestInstance}},
> which attempts to load JNI libraries and falls back on Java implementations
> in case the JNI library cannot be loaded or initialized.
> I run Spark in a configuration where the JNI libraries don't work, so I'd
> like to configure LZ4 to not even attempt to use JNI code: if the JNI library
> loads but cannot be initialized then the fallback code path involves catching
> an exception and this is slow because the exception is thrown under a static
> initializer lock (leading to significant lock contention because the filling
> of stacktraces is done while holding this lock).
> I propose to introduce a {{spark.io.compression.lz4.factory}} configuration
> for selecting the LZ4 implementation, allowing users to disable the use of
> the JNI library without having to recompile Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
