Hi,
I'm confused about the -Dspark.local.dir and SPARK_WORKER_DIR(--work-dir).
What's the difference?
I have set -Dspark.local.dir for all my worker nodes but I'm still seeing
directories being created in /tmp when the job is running.
I have also tried setting -Dspark.local.dir when I run the
I'm not 100% sure but I think it goes
like this :
spark.local.dir can and should be set both on the executors and on
the driver (if the driver broadcast variables, the files will be
stored in this directory)
the SPARK_WORKER_DIR is where the
spark.local.dir can and should be set both on the executors and on the
driver (if the driver broadcast variables, the files will be stored in this
directory)
Do you mean the worker nodes?
Don’t think they are jetty connectors and the directories are empty:
spark.local.dir can and
should be set both on the executors and on the driver
(if the driver broadcast variables, the files will be
stored in this directory)