I have not looked at the details of the code yet, it I would like to
understand few things:

1) is the CWL package more of a converter of CWL to Python DAG files (that
can then be scheduled as usual) or whether it is running alongside of the
scheduler and schedules tasks and operators separately using different
scheduling engine?. As a reference there is an
https://github.com/GoogleCloudPlatform/oozie-to-airflow converter from
Oozie XML to airflow DAGs. I think the biggest advantage of Airflow is
being able to modify and iterate quickly using python code so having
aPython Dag generated from CWL might be a good idea - even if it is not
perfect, user can still modify it and extend later manually rather than
relaying that all the features of CWL are implemented.

2 I'd also like to understand what dependencies it introduces on Airflow -
whether it relies on certain internals of Airflow that could make Airflow's
evolution more difficult? Also we have a roadmap for Airflow 2.0 already
and there are certain incompatibilities implemented, more is planned
already (and more to come not planned yet). Is the CWL importer 1.10
compatible or both 1.10 and (current state of)  2.0? Have you been
following some of the discussions with 2.0 and are you aware of some
potential incompatibilities?

3) What are the benefits you see to have Airflow CWL package  managed by
the Airflow community rather than CWL one? It could work both ways - it
could be managed by either of the communities (as usual in case of such
imports), but I think it has to be carefully weighted who maintains it
eventually - it all depends on how much one could rely on other, what is
the release cycle of CWL new versions  vs. Airflow versions etc. Could you
share your thought process and why you think it should be part of Airflow ?


J.

czw., 24 paź 2019, 21:36 użytkownik Michael R. Crusoe <m...@commonwl.org>
napisał:

> Dear Airflow People,
>
> Following up to
>
> http://mail-archives.apache.org/mod_mbox/airflow-dev/201812.mbox/%3c9a85db58-cef7-4b9a-9cfb-20cd5bfce...@gmail.com%3e
>
> The CWL AIrflow paper has been published!
>
> Michael Kotliar, Andrey V Kartashov, Artem Barski, CWL-Airflow: a
> lightweight pipeline manager supporting Common Workflow Language,
> GigaScience, Volume 8, Issue 7, July 2019, giz084,
> https://doi.org/10.1093/gigascience/giz084
>
> The Barski Lab is still interested in their code being unstreamed as part
> of Airflow itself.
> I'm not an Airflow developer myself, but I am the maintainer of the Common
> Workflow Language reference runner that cwl-airflow makes use of.
>
> Please let us know if the community would like to add support for importing
> workflows written in the CWL standard.
>
> Cheers from the Berlin Airflow meetup,
>
> --
> Michael R. Crusoe
> Co-founder & Lead, Common Workflow Language project
> <http://www.commonwl.org/>
> https://orcid.org/0000-0002-2961-9670
> <https://impactstory.org/u/0000-0002-2961-9670>
> m...@commonwl.org
>

Reply via email to