[
https://issues.apache.org/jira/browse/TAJO-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909687#comment-13909687
]
Min Zhou edited comment on TAJO-611 at 2/23/14 8:32 PM:
--------------------------------------------------------
Hi Mafish,
Briefly, Apache Curator is based on zookeeper, which is not a single point. On
the contrary, service discovery is for solving the SPOF. Service discovery is
not for resource management, it is just used for finding each other services.
DNS might be one kind of the most well known discovery services by us. (see
http://tools.ietf.org/html/rfc6763 )
{quote}
A Service Discovery system provides a mechanism for:
Services to register their availability
Locating a single instance of a particular service
Notifying when the instances of a service change
{quote}
Actually, the write performance of zookeeper is not as good as we expected,
especially when the nodes grows. We couldn't update tons of records on zk.
The reason I referred TAJO-540 here is that I need service discovery to help me
find out all of the live task runners/workers. This job can be done by service
discovery, and by TajoMaster as well. We can keep both just through an
interface. let's say DiscoveryService. So no worry about introducing a new
dependency to the system.
was (Author: coderplay):
Hi Mafish,
Briefly, Apache Curator is based on zookeeper, which is not a single point. On
the contrary, service discovery is for solving the SPOF. Service discovery is
not for resource management, it is just used for finding each other services.
DNS might be one kind of the most discovery services be known by us. (see
http://tools.ietf.org/html/rfc6763 )
{quote}
A Service Discovery system provides a mechanism for:
Services to register their availability
Locating a single instance of a particular service
Notifying when the instances of a service change
{quote}
Actually, the write performance of zookeeper is not as good as we expected,
especially when the nodes grows. We couldn't update tons of records on zk.
The reason I referred TAJO-540 here is that I need service discovery to help me
find out all of the live task runners/workers. This job can be done by service
discovery, and by TajoMaster as well. We can keep both just through an
interface. let's say DiscoveryService. So no worry about introducing a new
dependency to the system.
> (Umbrella) Service Discovery
> -----------------------------
>
> Key: TAJO-611
> URL: https://issues.apache.org/jira/browse/TAJO-611
> Project: Tajo
> Issue Type: New Feature
> Affects Versions: 1.0-incubating
> Reporter: Min Zhou
> Assignee: Mafish
> Fix For: 1.0-incubating
>
>
> As we talked offline, high availability is one of our next goal. Service
> discovery can help us maintain health statuses for all daemons ( master and
> workers) . Meanwhile, those daemons can find each other easily. Furthermore,
> it's very useful for my current work TAJO-540 cuz it can randomly select
> nodes for tajo scheduler.
> One of the best candidates is Netflix Curator.
> http://curator.apache.org/curator-x-discovery/index.html
> I'd like to introduce Xuhui to help us with this issue. Xuhui was my
> colleague at Alibaba group. He was active in Hive community, one of his job
> is adding multi-distinct aggregation feature to Hive. Currently, he is a
> researcher work at Microsoft.
> [~mafish]
> Below is comment on this issue from Xuhui
> To my understanding, this feature is for high availability as well as high
> scalability. We don't need to provide all machine info for every service when
> tajo starts. Instead, we can dynamically register services to service
> discovery if necessary. Also, if some machine fails, it can be easily
> detected and replaced.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
