Josh Rosen created SPARK-28102: ---------------------------------- Summary: Add configuration for selecting LZ4 implementation (safe, unsafe, JNI) Key: SPARK-28102 URL: https://issues.apache.org/jira/browse/SPARK-28102 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.0.0 Reporter: Josh Rosen Assignee: Josh Rosen
Spark's use of {{lz4-java}} ends up calling {{Lz4Factory.fastestInstance}}, which attempts to load JNI libraries and falls back on Java implementations in case the JNI library cannot be loaded or initialized. I run Spark in a configuration where the JNI libraries don't work, so I'd like to configure LZ4 to not even attempt to use JNI code: if the JNI library loads but cannot be initialized then the fallback code path involves catching an exception and this is slow because the exception is thrown under a static initializer lock (leading to significant lock contention because the filling of stacktraces is done while holding this lock). I propose to introduce a {{spark.io.compression.lz4.factory}} configuration for selecting the LZ4 implementation, allowing users to disable the use of the JNI library without having to recompile Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org