[
https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16198674#comment-16198674
]
Saisai Shao commented on SPARK-21737:
-------------------------------------
I was trying to understand how Spark communicate with Mesos, but my knowledge
of Mesos is quite poor, till now I cannot figure out a way to address this
problem in Mesos.
If we only restrict this problem to a Spark on YARN problem, then this is not
an issue any more. But my thinking is that only focusing on YARN makes this
channel not so useful.
Another way is that we build an interface to get endpoint address from cluster
manager, and only have a on-yarn implementation initially. Later on we can
support Standalone and Mesos if it is required, what do you think?
> Create communication channel between arbitrary clients and the Spark AM in
> YARN mode
> ------------------------------------------------------------------------------------
>
> Key: SPARK-21737
> URL: https://issues.apache.org/jira/browse/SPARK-21737
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Jong Yoon Lee
> Priority: Minor
>
> In this JIRA, I develop code to create a communication channel between
> arbitrary clients and a Spark AM on YARN. This code can be utilized to send
> commands such as getting status command, getting history info from the CLI,
> killing the application and pushing new tokens.
> Design Doc:
> https://docs.google.com/document/d/1QMbWhg13ocIoADywZQBRRVj-b9Zf8CnBrruP5JhcOOY/edit?usp=sharing
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
