[
https://issues.apache.org/jira/browse/TAJO-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909676#comment-13909676
]
Mafish commented on TAJO-611:
-----------------------------
Thanks Hyunsik for adding me as contributor.
I'm think about resource management because I believe it's more nature to add
Service Discovery as part of resource management when services are treated as
one kind of resource. Now we have two options.
1. We think about Service Discovery and resource management separately. This is
easier for service discovery and can be done more quick if we use Curator to
implement this feature. The drawback is that we introduce another critical
service that may be single point of failure if all services depend on Service
Discovery to find other services.
2. We define services as a kind of resource and implement Service Discovery
within yarn. Eventually we'll user Yarn to manage our resources (actually we
have used it), this feature can be implemented within current framework and
will not introduce new components.
These are basic thinking and I'm not sure if Option 2 is practical. And If it
is, I prefer Option 2. How do you think?
> (Umbrella) Service Discovery
> -----------------------------
>
> Key: TAJO-611
> URL: https://issues.apache.org/jira/browse/TAJO-611
> Project: Tajo
> Issue Type: New Feature
> Affects Versions: 1.0-incubating
> Reporter: Min Zhou
> Assignee: Min Zhou
> Fix For: 1.0-incubating
>
>
> As we talked offline, high availability is one of our next goal. Service
> discovery can help us maintain health statuses for all daemons ( master and
> workers) . Meanwhile, those daemons can find each other easily. Furthermore,
> it's very useful for my current work TAJO-540 cuz it can randomly select
> nodes for tajo scheduler.
> One of the best candidates is Netflix Curator.
> http://curator.apache.org/curator-x-discovery/index.html
> I'd like to introduce Xuhui to help us with this issue. Xuhui was my
> colleague at Alibaba group. He was active in Hive community, one of his job
> is adding multi-distinct aggregation feature to Hive. Currently, he is a
> researcher work at Microsoft.
> [~mafish]
> Below is comment on this issue from Xuhui
> To my understanding, this feature is for high availability as well as high
> scalability. We don't need to provide all machine info for every service when
> tajo starts. Instead, we can dynamically register services to service
> discovery if necessary. Also, if some machine fails, it can be easily
> detected and replaced.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
