On Jun 1, 2015, at 10:11 AM, Amila Jayasekara <[email protected]> wrote: > > Hi Shameera, > > I am also having little difficulty relating the problem you are stating and > the solution approach you are proposing. This is maybe because I am > outdated with Airavata developments. > > As per my understanding (from your description) "transparency" is the > problem. One question about the problem is what level of transparency are > you expecting ? Is it transparency depicted in log files or feedback to > user front-ends etc. You may need to clarify this. > > Also if it is the transparency in logs or user front-ends, then the > solution you are proposing doesn't fit exactly to the problem (to my > understanding). > > On the other hand task based approach is a nice way to handle jobs. I guess > you need to design a schedular to execute these tasks also. Some things you > need to be concerned about when designing such a schedular are as follows; > > 1. Some tasks have dependencies and ordering. Probably dependencies and > ordering is defined by user (just like they way handlers are defined in > gfac config files) > 2. Would be nice to execute tasks in parallel whenever they dont have > dependencies > 3. Also be concerned about FT aspects when designing such a schedular. > State of the schedular and also state of the individual tasks may need to > be replicated over all nodes.
Thanks Amila for enumerating these important desired characteristics. As part of the FT you mention in 3, it will be important to checkout and recover capabilities in the tasks. Suresh > > Thanks > -Thejaka Amila > > > > > On Mon, Jun 1, 2015 at 2:09 AM, Sachith Withana <[email protected]> wrote: > >> Hi Shameera, >> >> Wouldn't a job specific distributed logging mechanism solve the problem >> that you mentioned? Allow the user/GWDeveloper to specify which >> logs/messages >> they want to see ( apart from that log everything anyways). >> >> >> >> On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka < >> [email protected] >>> wrote: >> >>> Hi Gary & Borries, >>> >>> It seems architecture mailing list doesn't show(drop) attachments, here >> is >>> the link >>> https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D >>> >>> @Suresh, Could you please have a look ? >>> >>> Thanks, >>> Shameera. >>> >>> On Sun, May 31, 2015 at 5:58 AM, Borries Demeler < >>> [email protected]> wrote: >>> >>>> I think your attachments are stripped somewhere, you would be better >> off >>>> posting a URL. >>>> >>>> Thanks, -b. >>>> >>>>> Hi Gary, >>>>> >>>>> I converted the image to pdf and attached here. in both mails i can >> see >>>>> the >>>>> image, wonder how you all not getting that image. Please let me know >> if >>>>> you >>>>> still can't see it. >>>>> >>>>> Thanks, >>>>> Shameera. >>>>> >>>>> On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>>> On May 30, 2015, at 5:32 PM, Shameera Rathnayaka >>>>>> <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Hi Gary, >>>>>>> >>>>>>> I inserted the diagram as image, let me attache it as attachment. >>>>>> >>>>>> I still see nothing. >>>>>> - Gary >>>>>>> >>>>>>> Thanks, >>>>>>> Shameera. >>>>>>> >>>>>>> On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet < >> [email protected] >>>> >>>>>> wrote: >>>>>>> Shameera, >>>>>>> >>>>>>> On my copy of this email there was no attachment, no following >>>>>> diagram. >>>>>> Would you please send that attachment or a URL that points to it. >>>>>>> >>>>>>> Thanks, >>>>>>> Gary >>>>>>> >>>>>>>> On May 30, 2015, at 3:55 PM, Shameera Rathnayaka >>>>>> <[email protected]> >>>>>> wrote: >>>>>>>> >>>>>>>> Hi Devs, >>>>>>>> >>>>>>>> As we are about to release Airavata 0.15( already cut the >> branch ) >>>>>> we >>>>>> will not add any major changes and it is in testing stage. This will >>>>>> give >>>>>> us time to discuss and finalize requirements for the next release , >> it >>>>>> can >>>>>> be either 0.16 or 1.0. >>>>>>>> >>>>>>>> >>>>>>>> As per the feedback from our user community, they need more >>>>>> transparent view of what Airavata does when they submit an >> experiment >>> to >>>>>> run a job on remote computer resource. Airavata users are science >>>>>> gateway >>>>>> developers, they are not only interested in Experiment level and >>> remote >>>>>> Job >>>>>> level status changes. They would like to know some degree of >>>>>> transparency >>>>>> about pre-processing and post-processing tasks performed by airavata >>>>>> framework, before and after Job submission. For example they would >>> like >>>>>> to >>>>>> see which task is being executed at particular time, does scp file >>>>>> transferring succeed or not. With current Hander architecture, it is >>> not >>>>>> possible to Airavata framework to know which handler does what. User >>> can >>>>>> write and integrate different kind of handlers and integrate it with >>> the >>>>>> execution chain. If Airavata Job submission failed while >> transferring >>>>>> input >>>>>> file to the compute resource. Gateway developer should be able to >> find >>>>>> the >>>>>> reason without any trouble. Current Airavata save the failure reason >>>>>> with >>>>>> stracktrace but that is too low level for a gateway developer. >>>>>>>> >>>>>>>> Here we are thinking of replace this static handler architecture >>>>>> with >>>>>> dynamic task mechanism. Here framework has different type of tasks, >>> lets >>>>>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each >> task >>>>>> clearly know what it need to do and how. When Airavata get an >>> experiment >>>>>> with three inputs, one is simple string and other two are SCP and >> HTTP >>>>>> type >>>>>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP >>> tasks >>>>>> to >>>>>> the dynamic task chain. Then add another Job submission task, let's >>> say >>>>>> job >>>>>> need to submit using ssh keys then Airavata add SSH job submission >>> task. >>>>>> as >>>>>> same add required task for the outputs. Each task has three states >>>>>> Processing, Completed, Failed. In case of failure, framework know >>> which >>>>>> type of works it was doing or which task failed, is it SCP file >>> transfer >>>>>> task or GRIDFTP file transfering task. Then Airavata can >> provide(show) >>>>>> this >>>>>> details to Users by messaging. Please see following diagram to get >> an >>>>>> idea >>>>>> about different level of state transitions. >>>>>>>> >>>>>>>> Yours feedback are highly appreciate. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Shameera. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, >>>>>>> Shameera Rathnayaka. >>>>>>> >>>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com >>>>>>> Blog : http://shameerarathnayaka.blogspot.com/ >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, >>>>> Shameera Rathnayaka. >>>>> >>>>> email: shameera AT apache.org , shameerainfo AT gmail.com >>>>> Blog : http://shameerarathnayaka.blogspot.com/ >>>>> >>>> >>>> >>>> -- >>>> Borries Demeler, Ph.D. >>>> Associate Professor >>>> The University of Texas Health Science Center at San Antonio >>>> Dept. of Biochemistry >>>> Email: [email protected] >>>> >>> >>> >>> >>> -- >>> Best Regards, >>> Shameera Rathnayaka. >>> >>> email: shameera AT apache.org , shameerainfo AT gmail.com >>> Blog : http://shameerarathnayaka.blogspot.com/ >>> >> >> >> >> -- >> Thanks, >> Sachith Withana >>
