Hi Supun, Yes, I was going to suggest it as a solution to solve the issues described in the first mail. It was a valuable discussion which explains how Airavata architecture is being complicated and hence above issues. But it seems there are different opinions about those issues. I thought to hold on that solution for a while and see other suggestions or maybe someone to prove those are invalid concerns.
Thank you, Shameera. On Fri, Oct 7, 2016 at 9:46 PM Supun Kamburugamuve <[email protected]> wrote: > Hi Shameera, > > The plan we discussed offline is good one and it should solve most of > these issues discussed here. If you can describe that plan to others also, > we may be able to come to a good conclusion. > > Thanks, > Supun.. > > On Fri, Oct 7, 2016 at 5:07 PM, Shameera Rathnayaka < > [email protected]> wrote: > > It seems I might have missed some important parts in Airavata > Architecture. I think it is time for me to step back and re-evaluate again > why I see these big whole in our design. Thanks Amila for following this > thread so far. > > Thanks, > Shameera. > > On Fri, Oct 7, 2016 at 12:50 PM Shameera Rathnayaka < > [email protected]> wrote: > > On Fri, Oct 7, 2016 at 12:18 PM Amila Jayasekara <[email protected]> > wrote: > > You misunderstood. > > Let me re-phrase what mentioned earlier: (You said) "No, it is still > valid, thread goes to thread pool doesn't say *worker* is complete that > request, *it* is waiting until actual hpc job runs on target computer > resoruces". > Try to read above statement and explain to me what you refer as "worker" > and "it" (in bold) ? > > > Worker mean, who consume from the queue (to be specific GFac). "It" means > Worker. > > > Here is how I understood above statement. You submit a job in a thread and > then you put that thread to "wait" state until job finishes. Is that > correct ? > > No, this is bad we are not putting thread to wait. we release it untill > the submitted job finish, next time GFac start to process the same request, > it can be same thread or different thread from thread pool. > > > Also, others let me know whether I read the statement incorrectly. > > -AJ > > On Fri, Oct 7, 2016 at 11:46 AM, Shameera Rathnayaka < > [email protected]> wrote: > > glad you got that, also don't you see that when the message comes(very > first right arrow) and when gfac send the ack for that message(very last > left arrow)? That is the lifetime of a one worker queue message. > > On Fri, Oct 7, 2016 at 9:57 AM Amila Jayasekara <[email protected]> > wrote: > > As per this diagram, it seems the thread that submits the job is not the > same as the thread that handles output. > At least that is what I understand. > > -AJ > > On Thu, Oct 6, 2016 at 4:26 PM, Shameera Rathnayaka < > [email protected]> wrote: > > Previous attachment doesn't work. > > On Thu, Oct 6, 2016 at 4:24 PM, Shameera Rathnayaka < > [email protected]> wrote: > > [image: Work Queue Message Life time.png]Hi Amila, > > Please find work queue message execution sequence diagram below. Hope this > will help to understand how it works in Airavata. > > > > On Thu, Oct 6, 2016 at 4:05 PM Suresh Marru <[email protected]> wrote: > > Just a quick top post. This is informative discussion, please continue :) > > I agree on that Airavata does not do Work Stealing but it implements "Work > Queues”. Conceptually they are similar to the OS Kernel level work queens, > but more in a distributed context - > https://www.kernel.org/doc/Documentation/workqueue.txt > > Suresh > > > On Oct 6, 2016, at 3:52 PM, Amila Jayasekara <[email protected]> > wrote: > > On Thu, Oct 6, 2016 at 3:17 PM, Shameera Rathnayaka < > [email protected]> wrote: > > > > On Thu, Oct 6, 2016 at 2:50 PM Amila Jayasekara <[email protected]> > wrote: > > On Thu, Oct 6, 2016 at 11:07 AM, Shameera Rathnayaka < > [email protected]> wrote: > > Hi Amila, > > -- Please explain how you used "work stealing" in distributed system. That > would be interesting. > > > Airavata depends on work stealing + amqp for followings, > Fault Tolerance - This is one of major distributed system problem which > critical in Airavata, What ever the reason experiment request processing > shouldn't get any effect from internal node failure. Even with the node > failures, Airavata should be capable enough to continue experiment request > processing or hold it until at least one node appear and then continue. How > this is handled in Ariavata is, worker only ack for messages only after it > completely processed it. If the node goes down without sendings ack for > the messages it was processing,then rabbitmq put all these un-ack messages > back to the queue and available to consume again. > > Resource Utilization- Another important goal of distributed system to > effectively use available resources in the system, namely the memory and > processors of components. In Airavata this will decide the throughput and > response time of experiments. Currently, at a given time workers only get > messages up to a preconfigured limit (the limit is prefetch count) But most > of these jobs are async jobs. That means after worker gets fixed amount of > jobs, it won't get any other jobs even worker capable or handling more > jobs, waste of worker resources. > > > > You still did not answer my question. I want to know how you used "work > stealing" in your implementation. In other words how distributed work > stealing works in your implementation. The details you gave above is > unrelated and does not answer my question. > > > I think I have explained, how we use work stealing (work queues). If you > are finding a more analog solution to parallel computing work strealing > then that is hard to explain. > > > No, you have not. :-). > Work stealing != work queues. In a distributed setting I would image > following kind of a work stealing implementation; Every worker > (orchestrator) maintains a request queue locally and it serve requests > coming to the local queue. Whenever one worker runs out of more requests to > serve it will query other distributed workers local queues to see whether > there are requests that it can serve. If there are it can steal requests > from other workers local queues and process. However, this model of > computation is in efficient to do in a distributed environment. I guess > that is the same reason we dont find much distributed work stealing > implementations. > > Anyhow lets stop the discussion about work stealing now. :-) > > > > > > > > > -- I dont see AMQP in the architecture diagram you attached above and I > dont understand why Airavata has to depend on it. One way to figure this > out is think about the architecture without AMQP and figure out what > actually should happend and look for a way to do that using AMQP. > > > Worker Queue is AMQP queue. > > > Does the worker queue needs to be an AMQP queue ? Sorry, I dont know much > about AMQP but it sounds like limitations you are explaining are because of > AMQP. > > > It is not, but good to use well-defined protocol instead of custom one. > Almost all messaging systems have implemented AMQP protocol. > > > Can we figure out whether others have also encountered the same/similar > problem and how they tackled those with AMQP ? Cos the design we have is > pretty straightforward and I believe there are systems analogous to our > design that uses AMQP. > > > > > > > > -- Does this mean that you have a waiting thread or process within > Airavata after submitting the job (for each work) ? > > > No, once the job is submitted to the remote resource, thread goes back to > the thread pool. > > > Then, your previous explanation, (i.e., "The time needs for a worker to > finish the work is depend on the application run time (applications runs on > HPC machine). Theoretically, this can be from few sec to days or even > more."), invalidates. Correct ? > > > No, it is still valid, thread goes to thread pool doesn't say worker is > complete that request, it is waiting until actual hpc job runs on target > computer resoruces. After this hpc jobs completed then outptu data staging > happens. After output stage to storage then it ack to the work queue > message. > > > This is confusing to me. > Does this mean once you return thread to thread pool, it is not reusable > for another request ? Also, how do you wait on a thread after returning it > to the thread pool ? > Also, why do you have to wait for HPC job to complete ? I was under the > impression the communication is asynchronous. i.e. after job completes you > get an email confirmation and then you start output data staging in a > separate thread. > > We should probably meet and verbally discuss this. > > -AJ > > > -- Shameera Rathnayaka
