[
https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198665#comment-16198665
]
Thomas Graves commented on SPARK-21737:
---------------------------------------
I haven't had time to think about it much since the pr discussion. My initial
thought was its a generic class/api but you still need resource manager
specific parts. I was thinking it would initially only be supported on yarn
until someone interested in supporting on the others looked at it. Meaning for
yarn you would still use the RM to register the endpoint. Without that you
need some other service discovery mechanism like you mention. I'm not a fan of
saving this information to hdfs.
Have you looked to see if mesos has a similar service?
> Create communication channel between arbitrary clients and the Spark AM in
> YARN mode
> ------------------------------------------------------------------------------------
>
> Key: SPARK-21737
> URL: https://issues.apache.org/jira/browse/SPARK-21737
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Jong Yoon Lee
> Priority: Minor
>
> In this JIRA, I develop code to create a communication channel between
> arbitrary clients and a Spark AM on YARN. This code can be utilized to send
> commands such as getting status command, getting history info from the CLI,
> killing the application and pushing new tokens.
> Design Doc:
> https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
