GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/5297
[SAPRK-3591][YARN]fire and forget for YARN cluster mode
https://issues.apache.org/jira/browse/SPARK-3591
The output after this patch:
>doggie153:/opt/oss/spark-1.3.0-bin-hadoop2.4/bin # ./spark-submit --class
org.apache.spark.examples.SparkPi --master yarn-cluster
../lib/spark-examples*.jar
15/03/31 21:15:25 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
15/03/31 21:15:25 INFO RMProxy: Connecting to ResourceManager at
doggie153/10.177.112.153:8032
15/03/31 21:15:25 INFO Client: Requesting a new application from cluster
with 4 NodeManagers
15/03/31 21:15:25 INFO Client: Verifying our application has not requested
more than the maximum memory capability of the cluster (8192 MB per container)
15/03/31 21:15:25 INFO Client: Will allocate AM container, with 896 MB
memory including 384 MB overhead
15/03/31 21:15:25 INFO Client: Setting up container launch context for our
AM
15/03/31 21:15:25 INFO Client: Preparing resources for our AM container
15/03/31 21:15:26 INFO Client: Uploading resource
file:/opt/oss/spark-1.3.0-bin-hadoop2.4/lib/spark-assembly-1.4.0-SNAPSHOT-hadoop2.4.1.jar
->
hdfs://doggie153:9000/user/root/.sparkStaging/application_1427257505534_0016/spark-assembly-1.4.0-SNAPSHOT-hadoop2.4.1.jar
15/03/31 21:15:27 INFO Client: Uploading resource
file:/opt/oss/spark-1.3.0-bin-hadoop2.4/lib/spark-examples-1.3.0-hadoop2.4.0.jar
->
hdfs://doggie153:9000/user/root/.sparkStaging/application_1427257505534_0016/spark-examples-1.3.0-hadoop2.4.0.jar
15/03/31 21:15:28 INFO Client: Setting up the launch environment for our AM
container
15/03/31 21:15:28 INFO SecurityManager: Changing view acls to: root
15/03/31 21:15:28 INFO SecurityManager: Changing modify acls to: root
15/03/31 21:15:28 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(root); users with
modify permissions: Set(root)
15/03/31 21:15:28 INFO Client: Submitting application 16 to ResourceManager
15/03/31 21:15:28 INFO YarnClientImpl: Submitted application
application_1427257505534_0016
15/03/31 21:15:28 INFO Client: ... waiting before polling ResourceManager
for application state
15/03/31 21:15:33 INFO Client: ... polling ResourceManager for application
state
15/03/31 21:15:33 INFO Client: Application report for
application_1427257505534_0016 (state: RUNNING)
15/03/31 21:15:33 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: doggie157
ApplicationMaster RPC port: 0
queue: default
start time: 1427807728307
final status: UNDEFINED
tracking URL:
http://doggie153:8088/proxy/application_1427257505534_0016/
user: root
/cc @andrewor14
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/WangTaoTheTonic/spark SPARK-3591
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5297.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5297
----
commit 0cbdce8fb44644672886a0c608e0134b5e0263cb
Author: WangTaoTheTonic <[email protected]>
Date: 2015-03-31T13:08:41Z
fire and forget for YARN cluster mode
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
