[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tao Feng reassigned AIRFLOW-2221: --------------------------------- Assignee: Chao-Han Tsai (was: Diogo Franco) > Fill up DagBag from remote locations > ------------------------------------ > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core > Affects Versions: 2.0.0 > Reporter: Diogo Franco > Assignee: Chao-Han Tsai > Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)