I would just like to chime in that I agree with Amila’s general point that having multiple logs capture everything and the Airavata APIs able to return a specified one of them (or set of them) sounds like the most useful model.
Gary > On Jun 1, 2015, at 9:11 AM, Amila Jayasekara <[email protected]> wrote: > > Hi Shameera, > > I am also having little difficulty relating the problem you are stating and > the solution approach you are proposing. This is maybe because I am > outdated with Airavata developments. > > As per my understanding (from your description) "transparency" is the > problem. One question about the problem is what level of transparency are > you expecting ? Is it transparency depicted in log files or feedback to > user front-ends etc. You may need to clarify this. > > Also if it is the transparency in logs or user front-ends, then the > solution you are proposing doesn't fit exactly to the problem (to my > understanding). > > On the other hand task based approach is a nice way to handle jobs. I guess > you need to design a schedular to execute these tasks also. Some things you > need to be concerned about when designing such a schedular are as follows; > > 1. Some tasks have dependencies and ordering. Probably dependencies and > ordering is defined by user (just like they way handlers are defined in > gfac config files) > 2. Would be nice to execute tasks in parallel whenever they dont have > dependencies > 3. Also be concerned about FT aspects when designing such a schedular. > State of the schedular and also state of the individual tasks may need to > be replicated over all nodes. > > Thanks > -Thejaka Amila > > > > > On Mon, Jun 1, 2015 at 2:09 AM, Sachith Withana <[email protected]> wrote: > >> Hi Shameera, >> >> Wouldn't a job specific distributed logging mechanism solve the problem >> that you mentioned? Allow the user/GWDeveloper to specify which >> logs/messages >> they want to see ( apart from that log everything anyways). >> >> >> >> On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka < >> [email protected] >>> wrote: >> >>> Hi Gary & Borries, >>> >>> It seems architecture mailing list doesn't show(drop) attachments, here >> is >>> the link >>> https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D >>> >>> @Suresh, Could you please have a look ? >>> >>> Thanks, >>> Shameera. >>> >>> On Sun, May 31, 2015 at 5:58 AM, Borries Demeler < >>> [email protected]> wrote: >>> >>>> I think your attachments are stripped somewhere, you would be better >> off >>>> posting a URL. >>>> >>>> Thanks, -b. >>>> >>>>> Hi Gary, >>>>> >>>>> I converted the image to pdf and attached here. in both mails i can >> see >>>>> the >>>>> image, wonder how you all not getting that image. Please let me know >> if >>>>> you >>>>> still can't see it. >>>>> >>>>> Thanks, >>>>> Shameera. >>>>> >>>>> On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>>> On May 30, 2015, at 5:32 PM, Shameera Rathnayaka >>>>>> <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Hi Gary, >>>>>>> >>>>>>> I inserted the diagram as image, let me attache it as attachment. >>>>>> >>>>>> I still see nothing. >>>>>> - Gary >>>>>>> >>>>>>> Thanks, >>>>>>> Shameera. >>>>>>> >>>>>>> On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet < >> [email protected] >>>> >>>>>> wrote: >>>>>>> Shameera, >>>>>>> >>>>>>> On my copy of this email there was no attachment, no following >>>>>> diagram. >>>>>> Would you please send that attachment or a URL that points to it. >>>>>>> >>>>>>> Thanks, >>>>>>> Gary >>>>>>> >>>>>>>> On May 30, 2015, at 3:55 PM, Shameera Rathnayaka >>>>>> <[email protected]> >>>>>> wrote: >>>>>>>> >>>>>>>> Hi Devs, >>>>>>>> >>>>>>>> As we are about to release Airavata 0.15( already cut the >> branch ) >>>>>> we >>>>>> will not add any major changes and it is in testing stage. This will >>>>>> give >>>>>> us time to discuss and finalize requirements for the next release , >> it >>>>>> can >>>>>> be either 0.16 or 1.0. >>>>>>>> >>>>>>>> >>>>>>>> As per the feedback from our user community, they need more >>>>>> transparent view of what Airavata does when they submit an >> experiment >>> to >>>>>> run a job on remote computer resource. Airavata users are science >>>>>> gateway >>>>>> developers, they are not only interested in Experiment level and >>> remote >>>>>> Job >>>>>> level status changes. They would like to know some degree of >>>>>> transparency >>>>>> about pre-processing and post-processing tasks performed by airavata >>>>>> framework, before and after Job submission. For example they would >>> like >>>>>> to >>>>>> see which task is being executed at particular time, does scp file >>>>>> transferring succeed or not. With current Hander architecture, it is >>> not >>>>>> possible to Airavata framework to know which handler does what. User >>> can >>>>>> write and integrate different kind of handlers and integrate it with >>> the >>>>>> execution chain. If Airavata Job submission failed while >> transferring >>>>>> input >>>>>> file to the compute resource. Gateway developer should be able to >> find >>>>>> the >>>>>> reason without any trouble. Current Airavata save the failure reason >>>>>> with >>>>>> stracktrace but that is too low level for a gateway developer. >>>>>>>> >>>>>>>> Here we are thinking of replace this static handler architecture >>>>>> with >>>>>> dynamic task mechanism. Here framework has different type of tasks, >>> lets >>>>>> say for input staging we have SCP , GRIDFTP and HTTP tasks. each >> task >>>>>> clearly know what it need to do and how. When Airavata get an >>> experiment >>>>>> with three inputs, one is simple string and other two are SCP and >> HTTP >>>>>> type >>>>>> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP >>> tasks >>>>>> to >>>>>> the dynamic task chain. Then add another Job submission task, let's >>> say >>>>>> job >>>>>> need to submit using ssh keys then Airavata add SSH job submission >>> task. >>>>>> as >>>>>> same add required task for the outputs. Each task has three states >>>>>> Processing, Completed, Failed. In case of failure, framework know >>> which >>>>>> type of works it was doing or which task failed, is it SCP file >>> transfer >>>>>> task or GRIDFTP file transfering task. Then Airavata can >> provide(show) >>>>>> this >>>>>> details to Users by messaging. Please see following diagram to get >> an >>>>>> idea >>>>>> about different level of state transitions. >>>>>>>> >>>>>>>> Yours feedback are highly appreciate. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Shameera. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, >>>>>>> Shameera Rathnayaka. >>>>>>> >>>>>>> email: shameera AT apache.org , shameerainfo AT gmail.com >>>>>>> Blog : http://shameerarathnayaka.blogspot.com/ >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Best Regards, >>>>> Shameera Rathnayaka. >>>>> >>>>> email: shameera AT apache.org , shameerainfo AT gmail.com >>>>> Blog : http://shameerarathnayaka.blogspot.com/ >>>>> >>>> >>>> >>>> -- >>>> Borries Demeler, Ph.D. >>>> Associate Professor >>>> The University of Texas Health Science Center at San Antonio >>>> Dept. of Biochemistry >>>> Email: [email protected] >>>> >>> >>> >>> >>> -- >>> Best Regards, >>> Shameera Rathnayaka. >>> >>> email: shameera AT apache.org , shameerainfo AT gmail.com >>> Blog : http://shameerarathnayaka.blogspot.com/ >>> >> >> >> >> -- >> Thanks, >> Sachith Withana >>
