GitHub user andrewor14 opened a pull request:
https://github.com/apache/spark/pull/1538
[SPARK-2260] Fix standalone-cluster mode, which was broken
The main thing was that spark configs were not propagated to the driver,
and so applications that do not specify `master` or `appName` automatically
failed. This PR fixes that and a couple of miscellaneous related things.
One thing that still needs to be addressed is that the jars used must be
available on the driver node. In `standalone-cluster` mode, this effectively
means these jars must be available on all the worker machines, since the driver
is launched on one of them. The semantics here are not the same as
`yarn-cluster` mode, however, where all the relevant jars are uploaded to a
distributed cache automatically and shipped to the containers. This may or may
not be an issue, but is certainly worth a mention at the very least.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/andrewor14/spark standalone-cluster
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1538.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1538
----
commit fd9da51251de69d1ae776eaad6d666ff7970692a
Author: Andrew Or <[email protected]>
Date: 2014-07-22T21:10:49Z
Formatting changes (minor)
commit 855256edaacaaf3b1fafc5bcb2ba12243184c134
Author: Andrew Or <[email protected]>
Date: 2014-07-22T22:41:42Z
Fix standalone-cluster mode
The problem was that spark properties are not propagated to the driver.
The solution is simple: pass the properties as part of the driver
description, such that the command that launches the driver
automatically sets the spark properties as its java system properties,
which will then be loaded by SparkConf.
commit 7f854bc0cf83e04cce9314913547b21dbd203da1
Author: Andrew Or <[email protected]>
Date: 2014-07-23T01:51:53Z
Fix test
commit 6ceb14fb6abb28ca080eb95e936006c0d7746ba7
Author: Andrew Or <[email protected]>
Date: 2014-07-23T02:03:57Z
Allow relevant configs to propagate to standalone Driver
commit d7e27285ab961a2be71504c41d14c397def6817f
Author: Andrew Or <[email protected]>
Date: 2014-07-23T02:06:15Z
Avoid deprecation warning in standalone Client
commit c141a007bcdaa469a3394baa2571677231bf2056
Author: Andrew Or <[email protected]>
Date: 2014-07-23T02:26:56Z
Don't display "unknown app" on driver log pages
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
