I need to set java.library.path to get access to some native code.
Following directions, I made a spark-env.sh:
#!/usr/bin/env bash
export
LD_LIBRARY_PATH="/usr/local/lib/libcdfNativeLibrary.so:/usr/local/lib/libcdf.so:${LD_LIBRARY_PATH}"
export SPARK_WORKER_OPTS=-Djava.library.path=/usr/local/lib
export SPARK_WORKER_MEMORY=2g
to no avail. (I tried both with and without exporting the environment).
Looking at how the worker actually starts up:
/usr/lib/jvm/default/bin/java -cp /home/spark/conf/:/home/spark/jars/*
-Xmx1024M -Dspark.driver.port=37219
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://
[email protected]:37219 --executor-id 113 --hostname 10.2.2.1
--cores 8 --app-id app-20171225145607-0003 --worker-url spark://
[email protected]:35449
It doesn't seem to take any options. I put an 'echo' in just to confirm
that spark-env.sh is getting invoked (and it is).
So, just out of curiosity, I tried to troubleshoot this:
spark@node2-1:~$ grep -R SPARK_WORKER_OPTS *
conf/spark-env.sh:export
SPARK_WORKER_OPTS=-Djava.library.path=/usr/local/lib
conf/spark-env.sh.template:# - SPARK_WORKER_OPTS, to set config
properties only for the worker (e.g. "-Dx=y")
The variable doesn't seem to get referenced anywhere in the spark
distribution. I checked a number of other options in spark-env.sh.template
and they didn't seem to be referenced either. I expected to find them in
various startup scripts.
I can probably "fix" my problem by hacking the lower-level startup scripts,
but first I'd like to inquire about what's going on here. How and where
are these variables actually used?
