GitHub user yoonlee95 opened a pull request:
https://github.com/apache/spark/pull/18978
[SPARK-21737][YARN]Create communication channel between arbitrary clients
and the Spark AM in YARN mode
## What changes were proposed in this pull request?
In this JIRA, I develop code to create a communication channel between
arbitrary clients and a Spark AM on YARN. This code can be utilized to send
commands such as getting status command, getting history info from the CLI,
killing the application and pushing new tokens.
Design Doc:
https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing
## How was this patch tested?
Tested by sending a Kill RPC message to an AM.
Worked for both sasl/auth layer
Code used to test the communication channel.
https://github.com/yoonlee95/spark/tree/authkill
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yoonlee95/spark clientToAMRpc
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18978.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18978
----
commit 14c680c6635402c35e3bed4a249936f54bceee82
Author: John Lee <[email protected]>
Date: 2017-08-09T19:59:59Z
patch applied with merge conflict fixed
commit af9d7e3b48e438c06ee6e948c65792aa59c95e4a
Author: John Lee <[email protected]>
Date: 2017-08-09T21:28:44Z
tested cluster & client mode
commit 26a92dc40a846e801646ac3a934faa7fc6894e0f
Author: John Lee <[email protected]>
Date: 2017-08-10T16:24:38Z
scala style fixed
commit f53848391b0506f2f75a9912256ee4d348857c28
Author: John Lee <[email protected]>
Date: 2017-08-10T17:19:47Z
removed out file
commit 19478d10d35c727e59a8b8548e149ade7a1c9af7
Author: John Lee <[email protected]>
Date: 2017-08-10T21:26:37Z
init striping
commit f829584b4270fcadf323c387fc846e2fa3b1bee9
Author: John Lee <[email protected]>
Date: 2017-08-16T19:29:20Z
added auth rpc layer support
commit ef5aec76019c74955af5c7b3ca081c920bd47ae8
Author: John Lee <[email protected]>
Date: 2017-08-17T16:32:27Z
auth rpc debugged
commit 8c46a0400f032f4e62e28a087fa7ca84635d9837
Author: John Lee <[email protected]>
Date: 2017-08-17T16:44:17Z
Merge branch 'master' of https://github.com/apache/spark into clientToAMRpc
Merged with master before making a pr
commit 96b06c3074f0758ef70b30b3f86dbae47dfc6605
Author: John Lee <[email protected]>
Date: 2017-08-17T18:02:00Z
made hadoopSeecurityUils Class
commit 7e6b0762f37d389864a2ebe62a461bb86247c7ec
Author: John Lee <[email protected]>
Date: 2017-08-17T18:51:50Z
style fixed
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
