Thanks Shameera for clearly illustrating the refactoring goals. More below:
> On May 30, 2015, at 4:55 PM, Shameera Rathnayaka <[email protected]> wrote: > > Hi Devs, > > As we are about to release Airavata 0.15( already cut the branch ) we will > not add any major changes and it is in testing stage. This will give us time > to discuss and finalize requirements for the next release , it can be either > 0.16 or 1.0. We probably should make a 0.16 and see where we stand. If we can really convince ourselves the API for specific set of advertised functionality can remain stable and the code quality and testing is production ready, we should move ahead to make it 1.0 (after we properly document the design and API). > As per the feedback from our user community, they need more transparent view > of what Airavata does when they submit an experiment to run a job on remote > computer resource. Airavata users are science gateway developers, they are > not only interested in Experiment level and remote Job level status changes. > They would like to know some degree of transparency about pre-processing and > post-processing tasks performed by airavata framework, before and after Job > submission. For example they would like to see which task is being executed > at particular time, does scp file transferring succeed or not. With current > Hander architecture, it is not possible to Airavata framework to know which > handler does what. User can write and integrate different kind of handlers > and integrate it with the execution chain. If Airavata Job submission failed > while transferring input file to the compute resource. Gateway developer > should be able to find the reason without any trouble. Current Airavata save > the failure reason with stracktrace but that is too low level for a gateway > developer. This is good summary. Just to add to it, gateway administrator would care to have a transparent view of “what happened”. The stack trace approach shows “how it happened” which is not so interested and more over require knowledge of Airavata inner workings. If we can clearly illustrate the “whats” through data models then the system can be used more seamlessly. Ofcource there will be lot of “hows” in the airavata logs which can be analyzed. But we should focus on one having one state machine. Currently there are internal vs external state machines which are troubling. > Here we are thinking of replace this static handler architecture with dynamic > task mechanism. Here framework has different type of tasks, lets say for > input staging we have SCP , GRIDFTP and HTTP tasks. each task clearly know > what it need to do and how. When Airavata get an experiment with three > inputs, one is simple string and other two are SCP and HTTP type file > transfer inputs. Then Airavata decide to add SCP and GRIDFTP tasks to the > dynamic task chain. Then add another Job submission task, let's say job need > to submit using ssh keys then Airavata add SSH job submission task. as same > add required task for the outputs. Each task has three states Processing, > Completed, Failed. In case of failure, framework know which type of works it > was doing or which task failed, is it SCP file transfer task or GRIDFTP file > transfering task. Then Airavata can provide(show) this details to Users by > messaging. Please see following diagram to get an idea about different level > of state transitions. > Yours feedback are highly appreciate. Would like to really listen to critical feedback on the proposed changes. I am strongly opinionated on Airavata providing a transparent view to end users and gateway administrators but have no biases on how it can be accomplished. Will appreciate every one’s time on guidance on the design changes Shameera is proposing. Suresh
