Github user vanzin commented on a diff in the pull request:
    --- Diff: yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala ---
    @@ -1189,6 +1217,10 @@ private[spark] class Client(
     private object Client extends Logging {
       def main(argStrings: Array[String]) {
    +    mainWithEnv(argStrings, Map() ++ sys.env)
    +  }
    +  def mainWithEnv(argStrings: Array[String], env: Map[String, String]): 
Unit = {
    --- End diff --
    There's something that bothers me about this "env" argument. What is it 
supposed to be? It sounds too much like it should be a custom `sys.env` but the 
code that calls this tells me otherwise. Reading the rest of the code it seems 
you're using it both for `SparkConf` entries and as an override for `sys.env`; 
I think it would be better to have separate arguments.
    Also, I think it would be better to create an explicit interface for this. 
Like a trait that defines a `sparkMain` method that takes app args and 
Spark-specific args. Something like:
    trait SparkApp {
      this: Singleton =>
      def sparkMain(args: Array[String], conf: Map[String, String]): Int
    Thinking about the future, I think it would even be good to think about 
making that trait a public interface, and have `conf` be a `SparkConf` 
(although there are a few complications w.r.t. logging before that can happen). 
That would solve at least a couple of problems:
    - have an explicit interface for Spark apps instead of overloading Java's 
    - have an explicit exit code for Spark apps (see how messy that is with 
yarn-cluster mode currently)
    For your particular change, that trait can remain `private[spark]`, and if 
the class being run does not implement it, you can throw an exception if 
launching an app in-process, kinda like your current code. But I think having 
an explicit interface for this would make both your approach easier to 
understand, and easier to extend it to other cluster managers / apps in the 
    If adding the trait, you can also make the code in `SparkSubmit` simpler by 
having an implementation of the trait that wraps a regular app that just has a 
`main(String[])` method.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to