Hi Marlon, Thank you for the comments :)
*Monitoring-direct communication-complex topology* Right now the way I have it in proposal is, the GFAC binds it selves with some queue. The producer sends a message to the broker with routing key (which the target identification module will determine). The broker's exchange routes the message on basis of the given routing key and delivers it to the apt queue. Now if that queue is bound to a GFAC, it will be delivered to it, if its bound to something else it will be delivered to that I understand what you are saying, but right now I made the solution according to the discussion we had that day. If we really want to implement what you are saying, we will have to be more concrete with what is actually required. - If we just need capability of having multiple consumers for a job status message, we could have a "topic" exchange setup, wherein all the consumers tell the exchange which topic (herein, it might be something like jobID or maybe something more generic depending on our requirement) they want to subscribe to. Then whenever a message arrives at exchange for a specific topic, it will be distributed to all its subscribers. - If we have a complex topology, wherein some consumers want all the messages and some only need on the basis of key and maybe there are some more who want on the basis of topic. RabbitMq has a feature of "exchange of exchange" bindings, where we can have different type of exchanges binded to an exchange in addition to the normal queues, so we can pretty much design whatever we can think of, but we will have to come up with exactly what we want. A simple solution to cope up with node failures that I have proposed in the proposal is delayed acknowledgement. If we have a "reliable" queue (its a feature in RabbitMQ, which allows setting up queues which survive server restarts) setup, the consumer fetches the message from this type of queue, but does not acknowledge it until its done processing the message. The way this will address our problem is, if the consumer crashes for some reason and restarts and fetches a message from its queue, it will get the same old message, because the message was never acknowledged and and hence not deleted from the queue. If the broker server restarts, the queue survives anyways. * Added the figures * Added new smaller milestones * Added a point under risk and mitigation, don't really have much to put in there I have the revised version ready and up for review. I have have submitted a pdf version of the same for now on gsoc portal. Thank you for taking the time out to review my proposal :) Best, Siddharth Jain On Thu, Mar 24, 2016 at 10:34 AM, Pierce, Marlon <[email protected]> wrote: > Hi Sidd, > > This is a well written proposal. Some comments: > > * The monitoring component doesn’t need to directly communicate with GFAC. > There can be multiple GFACs, so the original GFAC that submitted the > request may no longer be running. Also, other components may want to > receive updates, including external subscribers. GFAC subscribers can pick > up the message and act. There is the issue of managing race conditions > between multiple GFACs and other distributed computing issues. If a GFAC > instance picks up the message and wants to act on it but then becomes > unresponsive, how do we handle this situation? This is my opinion, anyway. > It is harder to implement than the direct approach. > > * You still need your high level architecture diagram and other figures > where you have placeholders currently. > > * You need 2 week deliverables in your timeline. Add discussion of risk > and risk mitigation, as I mentioned in my general email earlier today. > > Marlon > > > From: Siddharth Jain <[email protected]> > Reply-To: "[email protected]" <[email protected]> > Date: Wednesday, March 23, 2016 at 12:52 AM > To: "[email protected]" <[email protected]> > Subject: Re: [GSoC Proposal] Apache Airavata Monitoring Module > > Correction, the cwiki.apache.org link is: > > https://cwiki.apache.org/confluence/display/AIRAVATA/[GSoC+Proposal]+Apache+Airavata+Monitoring+Module > <https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGSoC+Proposal%5D+Apache+Airavata+Monitoring+Module> > > On Tue, Mar 22, 2016 at 6:35 PM, Siddharth Jain <[email protected]> > wrote: > >> Hello all, >> I have the first draft of my GSoC proposal on Apache Airavata Monitoring >> Module ready. I will appreciate if you could give any comments or >> suggestions to improve this proposal. >> >> The proposal is available on: >> i) >> https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+Proposal+-+In+Situ+Simulation+Analysis+Using+Airavata >> >> ii) >> https://docs.google.com/document/d/1rm_U-51NzdqfBfTNjOyCzc2eTR2SEg5K1tijhtijcz0/edit?usp=sharing >> >> >> The cwiki.apache.org link points to what it will actually look like, the >> google docs version is just for convenience of commenting. >> >> >> Best regards, >> Siddharth Jain >> > >
