liupc edited a comment on issue #23786: [SPARK-26877]Support user-level app staging directory in yarn mode when spark.yarn… URL: https://github.com/apache/spark/pull/23786#issuecomment-464051617 @vanzin I test the config you provide, but it seems only get the current username on the OS, but not the correct kerberos name when hadoop security enabled. ``` 19/02/15 20:28:20 INFO yarn.Client: Uploading resource file:/tmp/spark-51e5acbe-0dee-49f3-a954-1de89b353c3f/__spark_libs__6081884678722264795.zip -> hdfs://zjyprc-hadoop/tmp/liupengcheng/staging/liupengcheng/.sparkStaging/application_1550196143681_22965/__spark_libs__6081884678722264795.zip ``` I think `UGI.getUserName` is ok, this is verified on our production cluster. What's more, this kind of method call is used many places in spark: https://github.com/apache/spark/blob/71170e74df5c7ec657f61154212d1dc2ba7d0613/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L796 In the code referred by above link, we can see that spark use this `UGI.getUserName` as the `SPARK_USER` in yarn mode.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
