Hi all, I was working on standardizing workflow execution. The details of this issue could be found in Jira <https://issues.apache.org/jira/browse/AIRAVATA-2555>. We are thinking about using CWL <https://github.com/common-workflow-language/common-workflow-language> to pass the task DAG information.
We were initially considering using Apache Airflow <https://airflow.incubator.apache.org/> to trigger the constituent tasks in the workflow. But, after a brief research, we found the following reasons to not use it: - Airflow cannot work on a time scale lesser than a minute. Thus, we cannot execute a workflow immediately. - Airflow suggests that tasks should be big enough to run for at least 5 minutes. The tasks in Airavata workflow might be smaller than that - Airflow is mostly used to configure static workflows, whereas we might want to create workflows on the fly - [Unsure] Airflow requires client installation to be done on the executor machines - One of Airflow's biggest design goals is to visualize the workflow. Since we are executing a task inside Airavata, we might not be able to use this feature Currently, we are exploring other tools for the same task. Please correct the above rationale if it is incorrect. Also, please suggest other tools that could solve this problem. Best Wishes, Saurabh Agrawal LinkedIn <https://www.linkedin.com/in/saurabh-agrawal-1a048b118i> Github <https://github.com/saurabhagrawal0412>
