Josh Rosen created SPARK-28102:
----------------------------------
Summary: Add configuration for selecting LZ4 implementation (safe,
unsafe, JNI)
Key: SPARK-28102
URL: https://issues.apache.org/jira/browse/SPARK-28102
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.0.0
Reporter: Josh Rosen
Assignee: Josh Rosen
Spark's use of {{lz4-java}} ends up calling {{Lz4Factory.fastestInstance}},
which attempts to load JNI libraries and falls back on Java implementations in
case the JNI library cannot be loaded or initialized.
I run Spark in a configuration where the JNI libraries don't work, so I'd like
to configure LZ4 to not even attempt to use JNI code: if the JNI library loads
but cannot be initialized then the fallback code path involves catching an
exception and this is slow because the exception is thrown under a static
initializer lock (leading to significant lock contention because the filling of
stacktraces is done while holding this lock).
I propose to introduce a {{spark.io.compression.lz4.factory}} configuration for
selecting the LZ4 implementation, allowing users to disable the use of the JNI
library without having to recompile Spark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
