[
https://issues.apache.org/jira/browse/HELIX-128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kanak Biscuitwala updated HELIX-128:
------------------------------------
Description:
It's clearly important for members of a cluster to communicate with one another.
Currently all communications between controller, participant, and spectator
happens via zookeeper. This is important for systems where fault tolerance
cannot be compromised and consensus is necessary. But this also increases the
latency of communication quite a bit.
For some applications, it is important that communication between different
components is fast, it's ok to lose messages in some cases or if controller
fails after the message is sent. we can either use tcp or any other pub/sub
mechanism to make this as faster.
Furthermore, members of the cluster may want to transfer larger amounts of data
while still routing based on current Helix assignments. One way to do this is
to wrap the current messaging API, but use a different underlying framework.
Netty provides an efficient framework for intra-cluster communication and may
be a good fit here.
The big thing is that we don't want ZooKeeper to turn into a data store, nor do
we want it to be a place where there is high write traffic of any size. On top
of that, not all applications need consensus for every piece of communication.
This is why we need an alternative channel.
Programming language: Java
Experience required: Object-oriented programming, concurrency, possibly basic
networking
Helpful knowledge: Helix API, ZooKeeper, frameworks like Netty
was:
It's clearly important for members of a cluster to communicate with one another.
Currently all communications between controller, participant, and spectator
happens via zookeeper. This is important for systems where fault tolerance
cannot be compromised and consensus is necessary. But this also increases the
latency of communication quite a bit.
For some applications, it is important that communication between different
components is fast, it's ok to lose messages in some cases or if controller
fails after the message is sent. we can either use tcp or any other pub/sub
mechanism to make this as faster.
Furthermore, members of the cluster may want to transfer larger amounts of data
while still routing based on current Helix assignments. One way to do this is
to wrap the current messaging API, but use a different underlying framework.
Netty provides an efficient framework for intra-cluster communication and may
be a good fit here.
The big thing is that we don't want ZooKeeper to turn into a data store, nor do
we want it to be a place where there is high write traffic of any size. On top
of that, not all applications need consensus for every piece of communication.
This is why we need an alternative channel.
> Support multiple communication channels between controllers, participants,
> and spectators
> -----------------------------------------------------------------------------------------
>
> Key: HELIX-128
> URL: https://issues.apache.org/jira/browse/HELIX-128
> Project: Apache Helix
> Issue Type: Improvement
> Reporter: Kanak Biscuitwala
> Labels: Java, cluster, distributed-systems, gsoc2014, mentor,
> netty
>
> It's clearly important for members of a cluster to communicate with one
> another.
> Currently all communications between controller, participant, and spectator
> happens via zookeeper. This is important for systems where fault tolerance
> cannot be compromised and consensus is necessary. But this also increases the
> latency of communication quite a bit.
> For some applications, it is important that communication between different
> components is fast, it's ok to lose messages in some cases or if controller
> fails after the message is sent. we can either use tcp or any other pub/sub
> mechanism to make this as faster.
> Furthermore, members of the cluster may want to transfer larger amounts of
> data while still routing based on current Helix assignments. One way to do
> this is to wrap the current messaging API, but use a different underlying
> framework. Netty provides an efficient framework for intra-cluster
> communication and may be a good fit here.
> The big thing is that we don't want ZooKeeper to turn into a data store, nor
> do we want it to be a place where there is high write traffic of any size. On
> top of that, not all applications need consensus for every piece of
> communication. This is why we need an alternative channel.
> Programming language: Java
> Experience required: Object-oriented programming, concurrency, possibly basic
> networking
> Helpful knowledge: Helix API, ZooKeeper, frameworks like Netty
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
