Re: User Defined Workflow Execution Framework

[DImuthu Upeksha]

Hi Yasas,

I'm not a expert in XBaya design and use cases but I think Suresh can shed
some light about it. However we no longer use XBaya for workflow
interpretation. So don't get confused with the workflows defined in XBaya
with the description provided in the JIRA ticket. Let's try to make the
concepts clear. We need two levels of Workflows.

1. To run a single experiment of an Application. We call this as a DAG. So
a DAG is statically defined. It can have a set of environment setup tasks,
data staging tasks and a job submission task. For example, a DAG is created
to run a  Gaussian experiment on a compute host
2. To make a chain of Applications. This is what we call an actual
Workflow. In a workflow you can have a Gaussian Experiment running and
followed by a Lammps Experiment. So this is a dynamic workflow. Users can
come up with different combinations of Applications as a workflow

However your claim is true about pausing and restarting workflows. Either
it is a statically defined DAG or a dynamic workflow, we should be able to
do those operations.

I can understand some of the words and terminologies that are in those
resources are confusing and unclear so please feel free to let us know if
you need anything to be clarified.

Thanks
Dimuthu

On Sun, Mar 25, 2018 at 2:45 AM, Yasas Gunarathne  wrote:

> Hi All,
>
> I have few questions to be clarified regarding the user-defined workflow
> execution in Apache Airavata. Here I am talking about the high level
> workflows that are used to chain together multiple applications. This
> related to the issue - Airavata-2717 [1].
>
> In this [2] documentation it says that, the workflow interpreter that
> worked with XBaya provided an interpreted workflow execution framework
> rather than the compiled workflow execution environments, which allowed the
> users to pause the execution of the workflow as necessary and update the
> DAG’s execution states or even the DAG itself and resume execution.
>
> I want to know the actual requirement of having an interpreted workflow
> execution at this level. Is there any domain level advantage in allowing
> users to modify the order of workflow at runtime?
>
> I think we can have, pause, resume, restart, and stop commands available
> even in a compiled workflow execution environment, as long as we don't need
> to change the workflow.
>
> [1] https://issues.apache.org/jira/browse/AIRAVATA-2717
> [2] http://airavata.apache.org/architecture/workflow.html
>
> Regards
> --
> *Yasas Gunarathne*
> Undergraduate at Department of Computer Science and Engineering
> Faculty of Engineering - University of Moratuwa Sri Lanka
> LinkedIn  | GitHub
>  | Mobile : +94 77 4893616
> <+94%2077%20489%203616>
>

Re: [GSoC] Re-architect Output Data Parsing into Airavata core

[DImuthu Upeksha]

Hi Lahiru,

Nice document. And I like how you illustrate the systems through diagrams.
However try to address how you are going to expose parsed data to outside
through thrift APIs and how to design those data APIs in application
specific manner. And in the persisting task, you have to make sure data
integrity is preserved. For example in a Gaussian parsed output, you might
have to validate the parsed output using a schema before persisting them in
the database.

Thanks
Dimuthu

On Sun, Mar 25, 2018 at 5:05 PM, Lahiru Jayathilake <
lahiruj...@cse.mrt.ac.lk> wrote:

> Hi Everyone,
>
> I have shared a draft proposal [1] for the GSoC project, AIRAVATA-2718
> [2]. Any comments would be very helpful to improve it.
>
> [1] https://docs.google.com/document/d/1xhgL1w9Yn_
> c1d5PpabxJJNNLTbkgggasMBM-GsBjVHM/edit?usp=sharing
> [2] https://issues.apache.org/jira/browse/AIRAVATA-2718
>
> Thanks & Regards,
> --
> Lahiru Jayathilake
> Department of Computer Science and Engineering,
> Faculty of Engineering,
> University of Moratuwa
>
> 
>

Re: [GSOC] Apache Airavata Google Summer of Code Projects

[DImuthu Upeksha]

Hi Yasas,

Good writeup. As a feedback, can you please elaborate more on the design of
the new orchestrator and how it fits to the overall system? Some diagrams
will be very helpful.

Thanks
Dimuthu

On Sun, Mar 25, 2018 at 4:04 AM, Yasas Gunarathne  wrote:

> I have uploaded a draft proposal [1] for the project [GSoC] Resurrect
> User-Defined Airavata Workflows [2]. Kindly give your feedbacks.
>
> [1] https://docs.google.com/document/d/1z5N_YOOdau8yiAfzfJ6xjRE75FLplA3BdI
> N6dtOeFRc/edit?usp=sharing
> [2] https://issues.apache.org/jira/browse/AIRAVATA-2717
>
> Regards
>
> On Tue, Feb 27, 2018 at 1:15 PM, Yasas Gunarathne <
> yasasgunarat...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> I am looking forward to contribute in a project under Apache Airavata,
>> mostly related to distributed systems. Any help and guidance to find a
>> project idea would be extremely helpful.
>>
>> Thank You,
>>
>> --
>> *Yasas Gunarathne*
>> Undergraduate at Department of Computer Science and Engineering
>> Faculty of Engineering - University of Moratuwa Sri Lanka
>> LinkedIn  | GitHub
>>  | Mobile : +94 77 4893616
>> <+94%2077%20489%203616>
>>
>
>
>
> --
> *Yasas Gunarathne*
> Undergraduate at Department of Computer Science and Engineering
> Faculty of Engineering - University of Moratuwa Sri Lanka
> LinkedIn  | GitHub
>  | Mobile : +94 77 4893616
> <+94%2077%20489%203616>
>

summer of code

[Connor True]

Hello,

My name is Connor and I am interested in proposing a project to airavata.

[GSoC] Re-architect Output Data Parsing into Airavata core

[Lahiru Jayathilake]

Hi Everyone,

I have shared a draft proposal [1] for the GSoC project, AIRAVATA-2718 [2]. Any
comments would be very helpful to improve it.

[1]
https://docs.google.com/document/d/1xhgL1w9Yn_c1d5PpabxJJNNLTbkgggasMBM-GsBjVHM/edit?usp=sharing

[2] https://issues.apache.org/jira/browse/AIRAVATA-2718

Thanks & Regards,
-- 
Lahiru Jayathilake
Department of Computer Science and Engineering,
Faculty of Engineering,
University of Moratuwa

Re-architect Output Data Parsing into Airavata

[Lahiru Jayathilake]

Hi Everyone,

With the resources mentioned in Airavata-2718 [1] and with the help of
Dimuthu I came up with this [2] architecture for data parsers to act as
Airavata tasks and execute them as Helix DAG's.
Any guidance and comments would be very helpful to improve it


[1] https://issues.apache.org/jira/browse/AIRAVATA-2718
[2]
https://docs.google.com/document/d/1w64ITChLcN-HfKl1twdLy01-6dAue3mpM6t3qZE8mrk/edit?usp=sharing

Thanks
-- 
Lahiru Jayathilake
Department of Computer Science and Engineering,
Faculty of Engineering,
University of Moratuwa

Re: [GSOC] Apache Airavata Google Summer of Code Projects

[Yasas Gunarathne]

I have uploaded a draft proposal [1] for the project [GSoC] Resurrect
User-Defined Airavata Workflows [2]. Kindly give your feedbacks.

[1]
https://docs.google.com/document/d/1z5N_YOOdau8yiAfzfJ6xjRE75FLplA3BdIN6dtOeFRc/edit?usp=sharing
[2] https://issues.apache.org/jira/browse/AIRAVATA-2717

Regards

On Tue, Feb 27, 2018 at 1:15 PM, Yasas Gunarathne  wrote:

> Hi Everyone,
>
> I am looking forward to contribute in a project under Apache Airavata,
> mostly related to distributed systems. Any help and guidance to find a
> project idea would be extremely helpful.
>
> Thank You,
>
> --
> *Yasas Gunarathne*
> Undergraduate at Department of Computer Science and Engineering
> Faculty of Engineering - University of Moratuwa Sri Lanka
> LinkedIn  | GitHub
>  | Mobile : +94 77 4893616
> <+94%2077%20489%203616>
>



-- 
*Yasas Gunarathne*
Undergraduate at Department of Computer Science and Engineering
Faculty of Engineering - University of Moratuwa Sri Lanka
LinkedIn  | GitHub
 | Mobile : +94 77 4893616

User Defined Workflow Execution Framework

[Yasas Gunarathne]

Hi All,

I have few questions to be clarified regarding the user-defined workflow
execution in Apache Airavata. Here I am talking about the high level
workflows that are used to chain together multiple applications. This
related to the issue - Airavata-2717 [1].

In this [2] documentation it says that, the workflow interpreter that
worked with XBaya provided an interpreted workflow execution framework
rather than the compiled workflow execution environments, which allowed the
users to pause the execution of the workflow as necessary and update the
DAG’s execution states or even the DAG itself and resume execution.

I want to know the actual requirement of having an interpreted workflow
execution at this level. Is there any domain level advantage in allowing
users to modify the order of workflow at runtime?

I think we can have, pause, resume, restart, and stop commands available
even in a compiled workflow execution environment, as long as we don't need
to change the workflow.

[1] https://issues.apache.org/jira/browse/AIRAVATA-2717
[2] http://airavata.apache.org/architecture/workflow.html

Regards
-- 
*Yasas Gunarathne*
Undergraduate at Department of Computer Science and Engineering
Faculty of Engineering - University of Moratuwa Sri Lanka
LinkedIn  | GitHub
 | Mobile : +94 77 4893616