Hi, We are running a Spark Standalone cluster on EMR (note: not using YARN) and are trying to use S3 w/ EmrFS as our event logging directory.
We are having difficulties with a ClassNotFoundException on EmrFileSystem when we navigate to the event log screen. This is to be expected as the EmrFs jars are not on the classpath. But -- I have not been able to figure out a way to add additional classpath jars to the start-up of the Master daemon. SPARK_CLASSPATH has been deprecated, and looking around at spark-class, etc.. everything seems to be pretty locked down. Do I have to shove everything into the assembly jar? Am I missing a simple way to add classpath to the daemons? thanks, Michal
