Github user tgravescs commented on the issue:
    I'm not worried about the multiple spark contexts, I say we just say this 
is cluster mode only since we don't support multiple spark context for 
    Thanks for pointing out the multiple apps launched though, I hadn't thought 
of that use case since we were thinking about oozie launcher which only 
launches one per process.
    Changing SparkSubmit to not use system properties looks like a pretty big 
task. We then need to figure out how things will get passed. I would think 
passing on the command line could get to big so then you are looking at writing 
a file and then with that you have to configure where that file goes, etc.  
Really the SparkSubmit front end is a bit clunky since its really like we have 
a Client calling another Client which then finally does the work.  Ideally we 
would get rid of that second wrapper.   Did you have ideas on this?
    one other option would be for us just to only support thread mode if 
launching a single app for cluster mode.  I'm not super fond of it as it could 
confuse users but it at least gets us more functionality then now and we can do 
the system properties change separate.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to