[
https://issues.apache.org/jira/browse/YARN-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786299#comment-13786299
]
Robert Joseph Evans commented on YARN-913:
------------------------------------------
Yes it does have plenty of races. I'll try to get some detailed designs up
shortly but at a high level the general idea is to have a restful web service.
For the most common use case there just needs to be two interfaces.
- Register/Monitor a Service
- Query for Services
Part of the reason we need the service registry is to securely verify that a
client is talking to the real service, and no one has grabbed the service's
port after it registered. To do that I want to have the concept of a verified
service. For that we would need an admin interface for adding, updating, and
removing verified services.
The registry would provide a number of pluggable ways for services to
authenticate. Part of adding a verified service would include indicating which
authentication models the service can use to register and which "users" are
allowed to register that service.
The registry could also act like a trusted Certificate Authority. Another part
of adding in a verified service would include indicating how clients could
verify they are talking to the true service. This could include just
publishing an application id so the client can go to the RM and get a
delegation token. Another option would be having the service generate a
public/private key pair. When the service registers it would get the private
key and the public key would be available through the discovery interface.
The plan is to also have the registry monitor the service similar to ZK. The
service would heartbeat in to the registry periodically (could be on the order
of mins depending on the service) after a certain period of time of inactivity
the service would be removed from the registry. Perhaps we should add in an
explicit unregister as well.
I want to make sure that the data model it is generic enough that we could
support something like a web service on the gird where each server can register
itself and all of them would show up in the registry, so a service could have
one or more servers that are a part of it, and each server could have some
separate metadata about it.
I also want to have a plug-in interface for discovery, so we could potentially
make the registry look like a DNS server or an SSL Certificate Authority which
would make compatibility with existing applications and clients a lot simpler.
> Add a way to register long-lived services in a YARN cluster
> -----------------------------------------------------------
>
> Key: YARN-913
> URL: https://issues.apache.org/jira/browse/YARN-913
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: api
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Robert Joseph Evans
>
> In a YARN cluster you can't predict where services will come up -or on what
> ports. The services need to work those things out as they come up and then
> publish them somewhere.
> Applications need to be able to find the service instance they are to bond to
> -and not any others in the cluster.
> Some kind of service registry -in the RM, in ZK, could do this. If the RM
> held the write access to the ZK nodes, it would be more secure than having
> apps register with ZK themselves.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
