Hello All,
Our team is building a Jupyter lab extension to enable users to construct a ai/data pipeline workflow with notebooks and python scripts using a visual editor. We have been exploring how to add Airflow(awesome) to our list of pipeline/workflow runtimes.(https://elyra.readthedocs.io/en/latest/user_guide/pipelines.html). In a nutshell, we construct a dag using the information provided by the user in the visual editor with a help of a jinja template, however the current experimental REST API nor the one in 2.0 will have the ability to upload one to a dag directory. Have a work in progress branch here: https://github.com/elyra-ai/elyra/pull/490
Our current implementation pushes the constructed dag to an S3 bucket and syncs the bucket to airflow's dag dir via cron. A more official and formal ability to sync to dags to a remote repository is something that would be very useful and has been explored in: https://github.com/apache/airflow/pull/3138 as well as AIP-5 . Are there any plans to revive this effort post 2.0 stability?
Its also been suggested that current best practice(is it?) should be to take advantage of git-syncing the dag bag to a git repo. We are exploring this option as well.
Any comments or input would be greatly appreciated!
