skonto commented on issue #23546: [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System URL: https://github.com/apache/spark/pull/23546#issuecomment-463340476 @vanzin I am calling you here because I saw you did the work with adding net auth secret support to the k8s side. The integration test `Run SparkRemoteFileTest using a remote data file` was failing above due to: ``` 2019-02-13 18:21:18 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.lang.IllegalArgumentException: A secret key must be specified via the spark.authenticate.secret config at org.apache.spark.SecurityManager.$anonfun$getSecretKey$6(SecurityManager.scala:328) at scala.Option.getOrElse(Option.scala:138) at org.apache.spark.SecurityManager.getSecretKey(SecurityManager.scala:327) at org.apache.spark.util.Utils$.constructURIForAuthentication(Utils.scala:378) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:658) at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:138)``` That is caused because early in spark-submit I use `downloadFileList` that calls doFetchFile at some point and that point the driver has not generated the key as secMgr is not initialized. I noticed that Security manager's `initializeAuth()` is called from SparkEnv set in SparkContext. The problem is that a few lines before my call to `downloadFileList` there are [several similar calls](https://github.com/apache/spark/blob/a829234df35c87c169425f2c79fd1963b5420888/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L367-L376) for client mode. I managed to work-around this by filtering out http(s) urls as I dont need to touch them but the other calls will probably fail... thoughts?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
