Hi, Is it possible to use spark.authenticate with shared secrets in mesos client mode? I can get the driver to start up and expect to use authenticated channels, but when the executors start up the SecurityManager outputs "authentication: disabled".
Looking through the CoarseGrainedSchelduerBackend code, it just creates a new SparkEnv (and doesn't look like it loads spark-defaults.conf) and attempts to fetch properties from the driver. But the driver is expecting an authentication challenge and bails when it realizes the executor is not attempting the challenge. Gist of some of the logs https://gist.github.com/kalvinnchau/53c60c91a0e1b47ad0afd9fe06d81101 Relevant code here, I don't see where the executor would load up auth settings and grab the shared secret. // Bootstrap to fetch the driver's Spark properties. val executorConf = new SparkConf val port = executorConf.getInt("spark.executor.port", 0) val fetcher = RpcEnv.create( "driverPropsFetcher", hostname, port, executorConf, new SecurityManager(executorConf), clientMode = true) val driver = fetcher.setupEndpointRefByURI(driverUrl) val cfg = driver.askSync[SparkAppConfig](RetrieveSparkAppConfig) val props = cfg.sparkProperties ++ Seq[(String, String)]((" spark.app.id", appId)) fetcher.shutdown() Is there some setting I'm missing to get the executor backend to pull in authentication settings? Thanks, Kalvin
