GitHub user pgandhi999 opened a pull request:
https://github.com/apache/spark/pull/19047
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for
launching daemons like History Server
History Server Launch uses SparkClassCommandBuilder for launching the
server. It is observed that SPARK_CLASSPATH has been removed and deprecated.
For spark-submit this takes a different route and spark.driver.extraClasspath
takes care of specifying additional jars in the classpath that were previously
specified in the SPARK_CLASSPATH. Right now the only way specify the additional
jars for launching daemons such as history server is using SPARK_DIST_CLASSPATH
(https://spark.apache.org/docs/latest/hadoop-provided.html) but this I presume
is a distribution classpath. It would be nice to have a similar config like
spark.driver.extraClasspath for launching daemons similar to history server.
Added new environment variable SPARK_DAEMON_CLASSPATH to set classpath for
launching daemons. Tested and verified for History Server and Standalone Mode.
## How was this patch tested?
Initially, history server start script would fail for the reason being that
it could not find the required jars for launching the server in the java
classpath. Same was true for running Master and Worker in standalone mode. By
adding the environment variable SPARK_DAEMON_CLASSPATH to the java classpath,
both the daemons(History Server, Standalone daemons) are starting up and
running.
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pgandhi999/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19047.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19047
----
commit 172fc20898896058b7288360eb5292ed9df9d79c
Author: pgandhi <[email protected]>
Date: 2017-07-21T21:00:22Z
[SPARK-21503]: Fixed the issue
Added the case ExecutorLostFailure which was previously not there, thus,
the default case would be executed in which case, task would be marked as
completed.
commit 81422e0f634c0f06eb2ea29fba4281176a1ab528
Author: pgandhi <[email protected]>
Date: 2017-07-25T14:54:41Z
[SPARK-21503][UI]: Adding changes as per comments
commit 55c6c37d09b41ae6914edb5d067e7f2c252ac92a
Author: pgandhi999 <[email protected]>
Date: 2017-07-26T21:26:27Z
Merge pull request #1 from apache/master
Apache Spark Pull Request - July 26, 2017
commit f454c8933e07967548095e068063bd313ae4845c
Author: pgandhi <[email protected]>
Date: 2017-07-26T21:41:16Z
[SPARK-21541]: Spark Logs show incorrect job status for a job that does not
create SparkContext
Added a flag to check whether user has initialized Spark Context. If it is
true, then we let Application Master unregister with Resource Manager else we
do not.
commit 6b7d5c6e2565c7c4dd97f31fe404c59e73c7474c
Author: pgandhi <[email protected]>
Date: 2017-07-26T21:58:27Z
Revert "[SPARK-21541]: Spark Logs show incorrect job status for a job that
does not create SparkContext"
This reverts commit f454c8933e07967548095e068063bd313ae4845c.
"Merged another issue to this one by mistake"
commit bc4166490d2ff68898c00fae4c1ca1b8abe1e795
Author: pgandhi999 <[email protected]>
Date: 2017-07-28T15:24:55Z
Merge pull request #2 from apache/master
Spark - July 28, 2017
commit e46126fe0f3d8d6f92f7f51c30d8c2154bddc126
Author: pgandhi <[email protected]>
Date: 2017-07-28T16:08:08Z
[SPARK-21503]- Making Changes as per comments
[SPARK-21503]- Making Changes as per comments: Removed match case statement
and replaced it with an if clause.
commit 9b3cebc6b65d2da835f02efaa27015cfd1b0ccae
Author: pgandhi999 <[email protected]>
Date: 2017-08-01T13:58:12Z
Merge pull request #4 from apache/master
Spark - August 1, 2017
commit 7f03341093c843086920e8218463b5d2ba6e37d2
Author: pgandhi <[email protected]>
Date: 2017-08-01T15:52:13Z
[SPARK-21503]: Reverting Unit Test Code
[SPARK-21503]: Reverting Unit Test Code - Not needed.
commit 2d01cab45ae269db9044815970dd008c851a46cc
Author: pgandhi999 <[email protected]>
Date: 2017-08-24T21:59:52Z
Merge pull request #5 from apache/master
SPARK - August 24, 2017
commit eaf63e6bd4dddc726cf57fda080b9b5d6341e2f8
Author: pgandhi <[email protected]>
Date: 2017-08-24T22:03:29Z
[SPARK-21798]: No config to replace deprecated SPARK_CLASSPATH config for
launching daemons like History Server
Adding new env variable SPARK_DAEMON_CLASSPATH to set classpath for
launching daemons. Tested and verified for History Server and Standalone Mode.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
