@Apoorv, Do you have some architectural diagrams to proceed? If not, I would recommend sketching out some and sharing it with the group. We can then work on it to improve/enhance if needed.
Thanks and Regards, Gourav Shenoy From: "Pierce, Marlon" <[email protected]> Reply-To: <[email protected]> Date: Thursday, July 20, 2017 at 2:10 PM To: "[email protected]" <[email protected]> Subject: Re: Using Redis or Zookeeper for Email Monitoring Agreed. I think some architecture diagrams are in order. Marlon From: "Shenoy, Gourav Ganesh" <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Thursday, July 20, 2017 at 1:35 PM To: "[email protected]" <[email protected]> Subject: Re: Using Redis or Zookeeper for Email Monitoring Apoorv, Good thought about making the email monitoring service scalable and reliable. But I was wondering if you considered the Helix angle? Helix inherently uses Zookeeper to manage state and the cluster as a whole. You might want to consider leveraging some of Helix’s data handling capabilities. Nothing against Redis, Zookeeper looks like a definite candidate to me just because of Helix. Also, could you chalk out and share an architecture of how this email monitoring service would integrate with Airavata (assume a Helix architecture). Drawing an architecture would help you better understand the problem and even find smaller issues which you didn’t consider. Good work so far! Thanks and Regards, Gourav Shenoy From: Apoorv Palkar <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, July 19, 2017 at 1:59 PM To: "[email protected]" <[email protected]> Subject: Using Redis or Zookeeper for Email Monitoring Currently I'm trying to solve the problem of done emails coming before start emails and making the system scalable. Today, we have only one GFaC that handles the email monitoring system. As we are moving towards a microservices approach from the monolithic code we have, this email monitoring system also needs to adapt to these changes. Currently, in the gfac code, a concurrent hashmap is kept to keep track of start/end of emails using their respective experiment ID's. Instead of keeping the hashmap locally, we should keep it in a global state so in the future multiple GFaCs can handle the map. Supun has suggested to use Zookeeper for this as it has high avaliability and realibility. I was also thinking that since these experiment IDs are a key value pair, Redis would be a good option for such a use case. What do you guys think about each one. I understand airavata currently uses zookeeper, so development wise there seems to be an edge toward it. Would Redis be a good for such use case?
