Hi Amila,
On Mon, Jun 1, 2015 at 10:11 AM, Amila Jayasekara <[email protected]> wrote: > Hi Shameera, > > I am also having little difficulty relating the problem you are stating and > the solution approach you are proposing. This is maybe because I am > outdated with Airavata developments. > > As per my understanding (from your description) "transparency" is the > problem. One question about the problem is what level of transparency are > you expecting ? Is it transparency depicted in log files or feedback to > user front-ends etc. You may need to clarify this. > > Also if it is the transparency in logs or user front-ends, then the > solution you are proposing doesn't fit exactly to the problem (to my > understanding). > > With proposed solution I am going to fix following issues. .Let me explain the requirements clearly and why current airavata implementation failed( or not good enough) to support those. 1. User(Gateway Admin+ Developers) should able to see more details than what we have, currently we have experiment status which say experiment is executing. But sometimes user may need to know more deep, whether input staging in progress or working on job submission etc ... In failure case, experiment failed because input staging issue or job submission issue. Log level data is too low level for a user. In fail case , log answer the question why it failed? but user need to know what failed eg: "scp input stage task failed". We need to handle this in framework level. Currently what framework know is there are set of input handlers then there is a provider and set of output handlers to execute. Framework doesn't know what each handler does, it is hidden from framework. But framework has enough details to know what it need to do for a given experiment and how. eg: for a ssh input type, framework know it need to invoke a handler which ssh given input to the compute resource like wise. 2. Current gfac configuration file allow us to define static execution chains. Let's say we have a static chain with SCP input staging , SSH job submission and then SCP output staging. But in run time user need to change one of his experiment input to use another file transfer mechanism while rest use SCP. And user may change this behavior time to time. Current static approach con't support this dynamic behavior unless it define two static chains and use correct one base on inputs. If we go this way we will have many number of execution chain for a one application. So we need dynamic approach. With proposed solution, framework know what it does because framework generate the task chain. Hence framework can provide more information to user (via amqp message) what happen with each task. Users can get different level of information by listening to different routing keys. Framework will send Experiment , Process and Task level status changes via amqp messages. On the other hand task based approach is a nice way to handle jobs. I guess > you need to design a schedular to execute these tasks also. Some things you > need to be concerned about when designing such a schedular are as follows; > > 1. Some tasks have dependencies and ordering. Probably dependencies and > ordering is defined by user (just like they way handlers are defined in > gfac config files) > 2. Would be nice to execute tasks in parallel whenever they dont have > dependencies > 3. Also be concerned about FT aspects when designing such a schedular. > State of the schedular and also state of the individual tasks may need to > be replicated over all nodes. > For FT we will use the current zookeeper base implementation + registry data. Yes we need to handle task dependency like make environment before copy the input files. and input files can be perform parallel. Once we come up with basic functionalities these are the improvement we need to do. Thanks, Shameera. > > Thanks > -Thejaka Amila > > > > > On Mon, Jun 1, 2015 at 2:09 AM, Sachith Withana <[email protected]> > wrote: > > > Hi Shameera, > > > > Wouldn't a job specific distributed logging mechanism solve the problem > > that you mentioned? Allow the user/GWDeveloper to specify which > > logs/messages > > they want to see ( apart from that log everything anyways). > > > > > > > > On Sun, May 31, 2015 at 8:57 PM, Shameera Rathnayaka < > > [email protected] > > > wrote: > > > > > Hi Gary & Borries, > > > > > > It seems architecture mailing list doesn't show(drop) attachments, here > > is > > > the link > > > https://creately.com/diagram/iaadwt6t1/yWLDNUPgfz60jl4k8WVH0Up3mk%3D > > > > > > @Suresh, Could you please have a look ? > > > > > > Thanks, > > > Shameera. > > > > > > On Sun, May 31, 2015 at 5:58 AM, Borries Demeler < > > > [email protected]> wrote: > > > > > > > I think your attachments are stripped somewhere, you would be better > > off > > > > posting a URL. > > > > > > > > Thanks, -b. > > > > > > > > > Hi Gary, > > > > > > > > > > I converted the image to pdf and attached here. in both mails i can > > see > > > > > the > > > > > image, wonder how you all not getting that image. Please let me > know > > if > > > > > you > > > > > still can't see it. > > > > > > > > > > Thanks, > > > > > Shameera. > > > > > > > > > > On Sat, May 30, 2015 at 9:43 PM, Gary E. Gorbet < > [email protected]> > > > > > wrote: > > > > > > > > > >> > > > > >> > On May 30, 2015, at 5:32 PM, Shameera Rathnayaka > > > > >> <[email protected]> > > > > >> wrote: > > > > >> > > > > > >> > Hi Gary, > > > > >> > > > > > >> > I inserted the diagram as image, let me attache it as > attachment. > > > > >> > > > > >> I still see nothing. > > > > >> - Gary > > > > >> > > > > > >> > Thanks, > > > > >> > Shameera. > > > > >> > > > > > >> > On Sat, May 30, 2015 at 5:35 PM, Gary E. Gorbet < > > [email protected] > > > > > > > > >> wrote: > > > > >> > Shameera, > > > > >> > > > > > >> > On my copy of this email there was no attachment, no following > > > > >> diagram. > > > > >> Would you please send that attachment or a URL that points to it. > > > > >> > > > > > >> > Thanks, > > > > >> > Gary > > > > >> > > > > > >> > > On May 30, 2015, at 3:55 PM, Shameera Rathnayaka > > > > >> <[email protected]> > > > > >> wrote: > > > > >> > > > > > > >> > > Hi Devs, > > > > >> > > > > > > >> > > As we are about to release Airavata 0.15( already cut the > > branch ) > > > > >> we > > > > >> will not add any major changes and it is in testing stage. This > will > > > > >> give > > > > >> us time to discuss and finalize requirements for the next release > , > > it > > > > >> can > > > > >> be either 0.16 or 1.0. > > > > >> > > > > > > >> > > > > > > >> > > As per the feedback from our user community, they need more > > > > >> transparent view of what Airavata does when they submit an > > experiment > > > to > > > > >> run a job on remote computer resource. Airavata users are science > > > > >> gateway > > > > >> developers, they are not only interested in Experiment level and > > > remote > > > > >> Job > > > > >> level status changes. They would like to know some degree of > > > > >> transparency > > > > >> about pre-processing and post-processing tasks performed by > airavata > > > > >> framework, before and after Job submission. For example they would > > > like > > > > >> to > > > > >> see which task is being executed at particular time, does scp file > > > > >> transferring succeed or not. With current Hander architecture, it > is > > > not > > > > >> possible to Airavata framework to know which handler does what. > User > > > can > > > > >> write and integrate different kind of handlers and integrate it > with > > > the > > > > >> execution chain. If Airavata Job submission failed while > > transferring > > > > >> input > > > > >> file to the compute resource. Gateway developer should be able to > > find > > > > >> the > > > > >> reason without any trouble. Current Airavata save the failure > reason > > > > >> with > > > > >> stracktrace but that is too low level for a gateway developer. > > > > >> > > > > > > >> > > Here we are thinking of replace this static handler > architecture > > > > >> with > > > > >> dynamic task mechanism. Here framework has different type of > tasks, > > > lets > > > > >> say for input staging we have SCP , GRIDFTP and HTTP tasks. each > > task > > > > >> clearly know what it need to do and how. When Airavata get an > > > experiment > > > > >> with three inputs, one is simple string and other two are SCP and > > HTTP > > > > >> type > > > > >> file transfer inputs. Then Airavata decide to add SCP and GRIDFTP > > > tasks > > > > >> to > > > > >> the dynamic task chain. Then add another Job submission task, > let's > > > say > > > > >> job > > > > >> need to submit using ssh keys then Airavata add SSH job submission > > > task. > > > > >> as > > > > >> same add required task for the outputs. Each task has three states > > > > >> Processing, Completed, Failed. In case of failure, framework know > > > which > > > > >> type of works it was doing or which task failed, is it SCP file > > > transfer > > > > >> task or GRIDFTP file transfering task. Then Airavata can > > provide(show) > > > > >> this > > > > >> details to Users by messaging. Please see following diagram to get > > an > > > > >> idea > > > > >> about different level of state transitions. > > > > >> > > > > > > >> > > Yours feedback are highly appreciate. > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > Thanks, > > > > >> > > Shameera. > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > -- > > > > >> > Best Regards, > > > > >> > Shameera Rathnayaka. > > > > >> > > > > > >> > email: shameera AT apache.org , shameerainfo AT gmail.com > > > > >> > Blog : http://shameerarathnayaka.blogspot.com/ > > > > >> > > > > >> > > > > > > > > > > > > > > > -- > > > > > Best Regards, > > > > > Shameera Rathnayaka. > > > > > > > > > > email: shameera AT apache.org , shameerainfo AT gmail.com > > > > > Blog : http://shameerarathnayaka.blogspot.com/ > > > > > > > > > > > > > > > > > -- > > > > Borries Demeler, Ph.D. > > > > Associate Professor > > > > The University of Texas Health Science Center at San Antonio > > > > Dept. of Biochemistry > > > > Email: [email protected] > > > > > > > > > > > > > > > > -- > > > Best Regards, > > > Shameera Rathnayaka. > > > > > > email: shameera AT apache.org , shameerainfo AT gmail.com > > > Blog : http://shameerarathnayaka.blogspot.com/ > > > > > > > > > > > -- > > Thanks, > > Sachith Withana > > > -- Best Regards, Shameera Rathnayaka. email: shameera AT apache.org , shameerainfo AT gmail.com Blog : http://shameerarathnayaka.blogspot.com/
