No I was curious about the Kafka use-case here.
On Thu, Jul 24, 2014 at 11:39 AM, Saminda Wijeratne <[email protected]> wrote: > That is what I meant. we use ZK messaging system to share the messages > among the nodes. > > > On Thu, Jul 24, 2014 at 10:49 AM, Lahiru Gunathilake <[email protected]> > wrote: > >> Why not just use ZK ? >> >> >> On Wed, Jul 23, 2014 at 3:51 PM, Saminda Wijeratne <[email protected]> >> wrote: >> >>> Hi Devs, >>> >>> We currently (unofficially) use google guava eventbus[1] to propagate >>> experiment/node/task/job status change events. The use of eventbus is >>> hidden behind the class org.apache.airavata.common.utils.MonitorPublisher >>> and we have not yet fully evaluated the said framework. All subscriber >>> classes has to register with the MonitorPublisher in order to receive >>> messages. This framework can be used to notify of any event occurred within >>> Airavata components. >>> >>> However currently this framework does not work across JVMs. This is a >>> potential issue since we can have multiple GFac instances and/or different >>> thrift services hosted in different JVM instances. We are currently using >>> Apache Zookeeper to manage FT and load balancing in Airavata. Combined with >>> Apache Kafka, Zookeeper can help delegate solving the problem of sharing >>> messages between nodes (JVMs) in a quick a simple implementation as follows, >>> >>> >>> 1. Each JVM instance will have a MonitorPublisher instance which >>> listeners in that JVM will subscribe to. >>> 2. Each instance of MonitorPublisher instance will subscribe to >>> Kafka messages >>> 3. When a MonitorPublisher receives a message to publish it will >>> also forward the message to Kafka. Kafka will then relay the message to >>> other MonitorPublishers in other zookeeper nodes (other JVM instances). >>> 4. When MonitorPublisher receives a message from kafka it'll publish >>> the said message in the guava eventbus. >>> >>> However this is not efficient since it leads message flooding. We are >>> thinking of going ahead of this until we find a better solution. wdyt? >>> >>> >>> 1. http://code.google.com/p/guava-libraries/wiki/EventBusExplained >>> >> >> >> >> -- >> System Analyst Programmer >> PTI Lab >> Indiana University >> > > -- System Analyst Programmer PTI Lab Indiana University
