hmm.... I thought Kafka was the natural selection for ZK. Is there a built-in one in ZK which we can use instead?
On Thu, Jul 24, 2014 at 11:44 AM, Lahiru Gunathilake <[email protected]> wrote: > No I was curious about the Kafka use-case here. > > > On Thu, Jul 24, 2014 at 11:39 AM, Saminda Wijeratne <[email protected]> > wrote: > >> That is what I meant. we use ZK messaging system to share the messages >> among the nodes. >> >> >> On Thu, Jul 24, 2014 at 10:49 AM, Lahiru Gunathilake <[email protected]> >> wrote: >> >>> Why not just use ZK ? >>> >>> >>> On Wed, Jul 23, 2014 at 3:51 PM, Saminda Wijeratne <[email protected]> >>> wrote: >>> >>>> Hi Devs, >>>> >>>> We currently (unofficially) use google guava eventbus[1] to propagate >>>> experiment/node/task/job status change events. The use of eventbus is >>>> hidden behind the class org.apache.airavata.common.utils.MonitorPublisher >>>> and we have not yet fully evaluated the said framework. All subscriber >>>> classes has to register with the MonitorPublisher in order to receive >>>> messages. This framework can be used to notify of any event occurred within >>>> Airavata components. >>>> >>>> However currently this framework does not work across JVMs. This is a >>>> potential issue since we can have multiple GFac instances and/or different >>>> thrift services hosted in different JVM instances. We are currently using >>>> Apache Zookeeper to manage FT and load balancing in Airavata. Combined with >>>> Apache Kafka, Zookeeper can help delegate solving the problem of sharing >>>> messages between nodes (JVMs) in a quick a simple implementation as >>>> follows, >>>> >>>> >>>> 1. Each JVM instance will have a MonitorPublisher instance which >>>> listeners in that JVM will subscribe to. >>>> 2. Each instance of MonitorPublisher instance will subscribe to >>>> Kafka messages >>>> 3. When a MonitorPublisher receives a message to publish it will >>>> also forward the message to Kafka. Kafka will then relay the message to >>>> other MonitorPublishers in other zookeeper nodes (other JVM instances). >>>> 4. When MonitorPublisher receives a message from kafka it'll >>>> publish the said message in the guava eventbus. >>>> >>>> However this is not efficient since it leads message flooding. We are >>>> thinking of going ahead of this until we find a better solution. wdyt? >>>> >>>> >>>> 1. http://code.google.com/p/guava-libraries/wiki/EventBusExplained >>>> >>> >>> >>> >>> -- >>> System Analyst Programmer >>> PTI Lab >>> Indiana University >>> >> >> > > > -- > System Analyst Programmer > PTI Lab > Indiana University >
