My name is Andrey and I'm developer behind CWL-Airflow.
This message is follow up slack conversation. I copy past some messages from 
there here.


>> Slack chat:

When I've met CWL team there were no pipeline managers to support it. I've 
picked up Airflow to just prove the concept that it is possible.

The same time I was looking for a pipeline manager to use  for bioinformatic 
analysis and asked tons of questions from Airflow team as a result special note 
in documentation: "Beyond the Horizon". Nevertheless, I adopted Airflow for our 
bioinformatic use

There are more than 200 different pipeline managers, and to believe that in 
nearest future there will the only one and perfect one sounds impossible. So, 
to exchange pipeline logic between different pipeline managers and people it is 
good to have a standard (CWL is a a perfect fit) like JavaScript standard and 
different executers, browsers...

Apache taverna (pipeline manager) is working on adopting CWL for a while now, 
we have  code it is already working.

So yes, CWL-Airflow is developed and the use is simple it extends Airflow DAG 
class. However it is still required to put .py file with DAG (CWLDAG in our 
case) to the dag directory. I would like just to put .cwl file into DAG 
directory to simplify the usage

I'm ready to develop what is necessary, but I'm not quite sure (I'm not a big 
expert in airflow code) which way to go, plugin or some native core code, or ...

The project by itself lives https://github.com/Barski-lab/cwl-airflow,  there 
are tons of CWL tests https://ci.commonwl.org/job/airflow-conformance/

Reply via email to