[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17236084#comment-17236084 ] Ash Berlin-Taylor commented on AIRFLOW-2221: Which isn't to say this won't come, we just haven't planned it yet. Discussion about this is happening in https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+Remote+DAG+Fetcher > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235970#comment-17235970 ] Kamil Bregula commented on AIRFLOW-2221: No, and it is not planned for any future versions. > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17235914#comment-17235914 ] Jeff Zhang commented on AIRFLOW-2221: - What is the status of this ticket ? Will it be included in 2.0 ? > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782187#comment-16782187 ] Chao-Han Tsai commented on AIRFLOW-2221: Got it! > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Chao-Han Tsai >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781297#comment-16781297 ] Tao Feng commented on AIRFLOW-2221: --- [~milton0825], feel free to take it. But since it includes core change, we need to have an AIP(airflow improvement proposal) now. There is already one for this topic([https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-5+DagFetcher)] which you could talk to the author and see if you could take it over for discussion. > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Diogo Franco >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2221) Fill up DagBag from remote locations
[ https://issues.apache.org/jira/browse/AIRFLOW-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781283#comment-16781283 ] Chao-Han Tsai commented on AIRFLOW-2221: [~DiogoFranco] are you still working on this ticket? If not, I am interested in taking this over. > Fill up DagBag from remote locations > > > Key: AIRFLOW-2221 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2221 > Project: Apache Airflow > Issue Type: New Feature > Components: configuration, core >Affects Versions: 2.0.0 >Reporter: Diogo Franco >Assignee: Diogo Franco >Priority: Major > Fix For: 2.0.0 > > > The ability to fill up the DagBag from remote locations (HDFS, S3...) seems > to be deemed useful, e.g. facilitating deployment processes. > This JIRA is to propose an implementation of a *DagFetcher* abstraction on > the DagBag, where the collect_dags method can delegate the walking to a > *FileSystemDagFetcher*, *GitRepoDagFetcher*, *S3DagFetcher*, > *HDFSDagFetcher*, *GCSDagFetcher*, *ArtifactoryDagFetcher* or even > *TarballInS3DagFetcher*. > This was briefly discussed in [this mailing list > thread|https://lists.apache.org/thread.html/03ddcd3a42b7fd6e3dad9711e8adea37fc00391f6053762f73af5b6a@%3Cdev.airflow.apache.org%3E] > I'm happy to start work on this and provide an initial implementation for > review. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
