Hi guys, I’m trying to deploy a custom application.conf that contains various app specific entries and config overrides for akka and spray.
The file is successfully loaded by the driver by submitting with —driver-class-path – at least for the application specific config. I haven’t yet managed to convince the executors to load and apply the akka and spray config, although it’s my understanding that Typesafe config will use any application.conf file on the class path and merge/update the reference.conf file from the uber-jar. So far I’ve tried: * Absolute path (file is present on all the nodes) specified as spark.executor.extraJavaOptions “-Dconfig.file=file:///path/to/application.conf” * Uploading file from submit (similar to how you enable custom logging) + custom executor class path: --files /path/to/application.conf and spark.executor.extraClassPath=./ None have worked so far. I’m using Spark streaming 1.5.1 and Yarn (hadoop 2.6). I’ve already searched the list for custom application.conf and haven’t seen any successful examples. Thanks! -adrian
