[
https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727697#comment-14727697
]
Marcelo Vanzin commented on SPARK-10288:
----------------------------------------
So, the design document still spends more time talking about Mesos and
Standalone that about what really is being proposed here. It makes it sound
like this is a much bigger change than it really is: all you're doing, from a
quick look at the PR, is using a new client class that talks to YARN's REST
server instead of the usual RM RPC channel.
That being said, exactly because of that, it's still unclear to me what this is
achieving. Steve and you mention long haul submission, but your code deals
exclusively with the YARN part. What about all the code in Client.scala that
needs to talk to HDFS? Wouldn't that also be a concern in that case? If you
expect users to have a specific configuration for this to work (e.g. they
should configure HDFS in a certain way and the configuration on their
submitting node should be different from the configuration in their cluster),
then that needs to be, at the very least, documented.
And if you require a different configuration on the submitting node, then other
parts of Client.scala would break, because Spark localizes the configuration of
the submitting node, meaning the whole app would be using that configuration.
So at this point I'm still a little concerned about exactly what this is
fixing. "Long haul submission" involves more than just the RPC port of the RM.
> Add a rest client for Spark on Yarn
> -----------------------------------
>
> Key: SPARK-10288
> URL: https://issues.apache.org/jira/browse/SPARK-10288
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Reporter: Saisai Shao
>
> This is a proposal to add rest client for Spark on Yarn. Rest API offers a
> convenient addition to let user to submit application through rest client,
> people will easily achieve long haul submission, build their own submission
> gateway through rest client.
> Here is the design doc
> (https://docs.google.com/document/d/1m_P-4olXrp0tJ3kEOLZh1rwrjTfAat7P3fAVPR5GTmg/edit?usp=sharing).
> Currently I'm working on it, working branch is
> (https://github.com/jerryshao/apache-spark/tree/yarn-rest-support), the major
> part is already finished.
> Any comment is greatly appreciated, thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
