replacement for SPARK_LIBRARY_PATH ?

[Eric Friedman]

I used to use SPARK_LIBRARY_PATH to specify the location of native libs
for lzo compression when using spark 0.9.0.

The references to that environment variable have disappeared from the docs
for
spark 1.0.1 and it's not clear how to specify the location for lzo.

Any guidance?

Re: replacement for SPARK_LIBRARY_PATH ?

[Zongheng Yang]

One way is to set this in your conf/spark-defaults.conf:

spark.executor.extraLibraryPath /path/to/native/lib

The key is documented here:
http://spark.apache.org/docs/latest/configuration.html

On Thu, Jul 17, 2014 at 1:25 PM, Eric Friedman
eric.d.fried...@gmail.com wrote:
 I used to use SPARK_LIBRARY_PATH to specify the location of native libs
 for lzo compression when using spark 0.9.0.

 The references to that environment variable have disappeared from the docs
 for
 spark 1.0.1 and it's not clear how to specify the location for lzo.

 Any guidance?

Re: replacement for SPARK_LIBRARY_PATH ?

[Koert Kuipers]

but be aware that spark-defaults.conf is only used if you use spark-submit
On Jul 17, 2014 4:29 PM, Zongheng Yang zonghen...@gmail.com wrote:

 One way is to set this in your conf/spark-defaults.conf:

 spark.executor.extraLibraryPath /path/to/native/lib

 The key is documented here:
 http://spark.apache.org/docs/latest/configuration.html

 On Thu, Jul 17, 2014 at 1:25 PM, Eric Friedman
 eric.d.fried...@gmail.com wrote:
  I used to use SPARK_LIBRARY_PATH to specify the location of native libs
  for lzo compression when using spark 0.9.0.
 
  The references to that environment variable have disappeared from the
 docs
  for
  spark 1.0.1 and it's not clear how to specify the location for lzo.
 
  Any guidance?