We use MapR Hadoop and I have configured mesos-0.18.1 and spark-1.0.1 to work together on top of the nodes running mapr hadoop. I would like to configure spark to access files from the mapr filesystem (maprfs://) and I'm starting with configuring the SPARK_EXECUTOR_URI environment variable in the spark-env.sh file to access the precompiled spark version for our environment. I have tried the following in spark-env.sh:
export SPARK_EXECUTOR_URI=maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz but when I submit a job from the spark-shell going throught the mesos master, I see this in my stderr log: WARNING: Logging before InitGoogleLogging() is written to STDERR I0819 11:09:59.404290 27027 fetcher.cpp:73] Fetching URI 'maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz' E0819 11:09:59.404392 27027 fetcher.cpp:142] A relative path was passed for the resource but the environment variable MESOS_FRAMEWORKS_HOME is not set. Please either specify this config option or avoid using a relative path Failed to fetch: maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz I the tried setting the MESOS_FRAMEWORKS_HOME environment variable in /etc/environment to MESOS_FRAMEWORKS_HOME=/usr/local/mesos-0.18.1 and now I see this in the stderr log: WARNING: Logging before InitGoogleLogging() is written to STDERR I0819 11:05:12.430522 4774 fetcher.cpp:73] Fetching URI 'maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz' I0819 11:05:12.430616 4774 fetcher.cpp:138] Prepended environment variable MESOS_FRAMEWORKS_HOME to relative path, making it: '/usr/local/mesos-0.18.1/maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz' I0819 11:05:12.430635 4774 fetcher.cpp:160] Copying resource from '/usr/local/mesos-0.18.1/maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz' to '/tmp/mesos/slaves/20140815-101817-3334820618-5050-32618-0/frameworks/20140819-101702-3334820618-5050-16778-0001/executors/20140815-101817-3334820618-5050-32618-0/runs/e56fffbe-942d-4b15-a798-a00401387927' cp: cannot stat `/usr/local/mesos-0.18.1/maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz': No such file or directory E0819 11:05:12.433964 4774 fetcher.cpp:165] Failed to copy '/usr/local/mesos-0.18.1/maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz' : Exit status 256 Failed to fetch: maprfs:///mapr/CLUSTER1/MAIN/tmp/spark-1.0.1-bin-mapr3.tgz Has any gotten Spark to interact successfully with maprfs?
