[jira] [Updated] (AIRFLOW-2499) Dockerised CI pipeline
[ https://issues.apache.org/jira/browse/AIRFLOW-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated AIRFLOW-2499: Fix Version/s: 2.0.0 > Dockerised CI pipeline > -- > > Key: AIRFLOW-2499 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2499 > Project: Apache Airflow > Issue Type: Test > Components: ci, tests >Affects Versions: 1.10.0, 1.10 >Reporter: Gerardo Curiel >Assignee: Gerardo Curiel >Priority: Major > Labels: ci, docker, travis-ci > Fix For: 2.0.0 > > Original Estimate: 72h > Remaining Estimate: 72h > > PR: https://github.com/apache/incubator-airflow/pull/3393 > Currently, running unit tests is a difficult process. Airflow tests depend on > many external services and other custom setup, which makes it hard for > contributors to work on this codebase. CI builds have also been > unreliable[0][1][2][3], and it is hard to reproduce the causes. Having > contributors trying to emulate the build environment every time makes it > easier to get to an "it works on my machine" sort of situation. > This PR implements a dockerised version of the current build pipeline. This > setup has a few advantages: > * TravisCI tests are reproducible locally > * The same build setup can be used to create a local development environment > (there's a request for it [4]) > > *Implementation details* > * I'm using Docker Compose for the container orchestration and configuration. > * MySQL, PostgreSQL, OpenLDAP, krb5 and rabbitmq are now services running > inside their own containers > * I created a separate repo, called incubator-airflow-ci[5] (TravisCI build > here[6]), where a base image with all dependencies is built. In this case, > I'm following the same pattern the CouchDB[7] project follows > * Hadoop, Hive and MiniCluster were moved to this base image > * The current TravisCI pipeline lives here[8]. A few tests are still > failing. It's still WIP. > > *References* > [0] https://issues.apache.org/jira/browse/AIRFLOW-671 > [1] https://issues.apache.org/jira/browse/AIRFLOW-968 > [2] https://issues.apache.org/jira/browse/AIRFLOW-2157 > [3] https://issues.apache.org/jira/browse/AIRFLOW-2272 > [4] https://issues.apache.org/jira/browse/AIRFLOW-1042 > [5] [https://github.com/gerardo/incubator-airflow-ci] > [6] [https://travis-ci.org/gerardo/incubator-airflow-ci] > [7] [https://travis-ci.org/apache/couchdb-ci] > [8] [https://travis-ci.org/gerardo/incubator-airflow] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2499) Dockerised CI pipeline
[ https://issues.apache.org/jira/browse/AIRFLOW-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin updated AIRFLOW-2499: Fix Version/s: (was: 1.10) (was: 1.10.0) > Dockerised CI pipeline > -- > > Key: AIRFLOW-2499 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2499 > Project: Apache Airflow > Issue Type: Test > Components: ci, tests >Affects Versions: 1.10.0, 1.10 >Reporter: Gerardo Curiel >Assignee: Gerardo Curiel >Priority: Major > Labels: ci, docker, travis-ci > Fix For: 2.0.0 > > Original Estimate: 72h > Remaining Estimate: 72h > > PR: https://github.com/apache/incubator-airflow/pull/3393 > Currently, running unit tests is a difficult process. Airflow tests depend on > many external services and other custom setup, which makes it hard for > contributors to work on this codebase. CI builds have also been > unreliable[0][1][2][3], and it is hard to reproduce the causes. Having > contributors trying to emulate the build environment every time makes it > easier to get to an "it works on my machine" sort of situation. > This PR implements a dockerised version of the current build pipeline. This > setup has a few advantages: > * TravisCI tests are reproducible locally > * The same build setup can be used to create a local development environment > (there's a request for it [4]) > > *Implementation details* > * I'm using Docker Compose for the container orchestration and configuration. > * MySQL, PostgreSQL, OpenLDAP, krb5 and rabbitmq are now services running > inside their own containers > * I created a separate repo, called incubator-airflow-ci[5] (TravisCI build > here[6]), where a base image with all dependencies is built. In this case, > I'm following the same pattern the CouchDB[7] project follows > * Hadoop, Hive and MiniCluster were moved to this base image > * The current TravisCI pipeline lives here[8]. A few tests are still > failing. It's still WIP. > > *References* > [0] https://issues.apache.org/jira/browse/AIRFLOW-671 > [1] https://issues.apache.org/jira/browse/AIRFLOW-968 > [2] https://issues.apache.org/jira/browse/AIRFLOW-2157 > [3] https://issues.apache.org/jira/browse/AIRFLOW-2272 > [4] https://issues.apache.org/jira/browse/AIRFLOW-1042 > [5] [https://github.com/gerardo/incubator-airflow-ci] > [6] [https://travis-ci.org/gerardo/incubator-airflow-ci] > [7] [https://travis-ci.org/apache/couchdb-ci] > [8] [https://travis-ci.org/gerardo/incubator-airflow] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2499) Dockerised CI pipeline
[ https://issues.apache.org/jira/browse/AIRFLOW-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gerardo Curiel updated AIRFLOW-2499: External issue URL: https://github.com/apache/incubator-airflow/pull/3393 Description: PR: https://github.com/apache/incubator-airflow/pull/3393 Currently, running unit tests is a difficult process. Airflow tests depend on many external services and other custom setup, which makes it hard for contributors to work on this codebase. CI builds have also been unreliable[0][1][2][3], and it is hard to reproduce the causes. Having contributors trying to emulate the build environment every time makes it easier to get to an "it works on my machine" sort of situation. This PR implements a dockerised version of the current build pipeline. This setup has a few advantages: * TravisCI tests are reproducible locally * The same build setup can be used to create a local development environment (there's a request for it [4]) *Implementation details* * I'm using Docker Compose for the container orchestration and configuration. * MySQL, PostgreSQL, OpenLDAP, krb5 and rabbitmq are now services running inside their own containers * I created a separate repo, called incubator-airflow-ci[5] (TravisCI build here[6]), where a base image with all dependencies is built. In this case, I'm following the same pattern the CouchDB[7] project follows * Hadoop, Hive and MiniCluster were moved to this base image * The current TravisCI pipeline lives here[8]. A few tests are still failing. It's still WIP. *References* [0] https://issues.apache.org/jira/browse/AIRFLOW-671 [1] https://issues.apache.org/jira/browse/AIRFLOW-968 [2] https://issues.apache.org/jira/browse/AIRFLOW-2157 [3] https://issues.apache.org/jira/browse/AIRFLOW-2272 [4] https://issues.apache.org/jira/browse/AIRFLOW-1042 [5] [https://github.com/gerardo/incubator-airflow-ci] [6] [https://travis-ci.org/gerardo/incubator-airflow-ci] [7] [https://travis-ci.org/apache/couchdb-ci] [8] [https://travis-ci.org/gerardo/incubator-airflow] was: Currently, running unit tests is a difficult process. Airflow tests depend on many external services and other custom setup, which makes it hard for contributors to work on this codebase. CI builds have also been unreliable[0][1][2][3], and it is hard to reproduce the causes. Having contributors trying to emulate the build environment every time makes it easier to get to an "it works on my machine" sort of situation. This PR implements a dockerised version of the current build pipeline. This setup has a few advantages: * TravisCI tests are reproducible locally * The same build setup can be used to create a local development environment (there's a request for it [4]) *Implementation details* * I'm using Docker Compose for the container orchestration and configuration. * MySQL, PostgreSQL, OpenLDAP, krb5 and rabbitmq are now services running inside their own containers * I created a separate repo, called incubator-airflow-ci[5] (TravisCI build here[6]), where a base image with all dependencies is built. In this case, I'm following the same pattern the CouchDB[7] project follows * Hadoop, Hive and MiniCluster were moved to this base image * The current TravisCI pipeline lives here[8]. A few tests are still failing. It's still WIP. *References* [0] https://issues.apache.org/jira/browse/AIRFLOW-671 [1] https://issues.apache.org/jira/browse/AIRFLOW-968 [2] https://issues.apache.org/jira/browse/AIRFLOW-2157 [3] https://issues.apache.org/jira/browse/AIRFLOW-2272 [4] https://issues.apache.org/jira/browse/AIRFLOW-1042 [5] https://github.com/gerardo/incubator-airflow-ci [6] https://travis-ci.org/gerardo/incubator-airflow-ci [7] https://travis-ci.org/apache/couchdb-ci [8] https://travis-ci.org/gerardo/incubator-airflow > Dockerised CI pipeline > -- > > Key: AIRFLOW-2499 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2499 > Project: Apache Airflow > Issue Type: Test > Components: ci, tests >Affects Versions: 1.10.0, 1.10 >Reporter: Gerardo Curiel >Assignee: Gerardo Curiel >Priority: Major > Labels: ci, docker, travis-ci > Fix For: 1.10.0, 1.10 > > Original Estimate: 72h > Remaining Estimate: 72h > > PR: https://github.com/apache/incubator-airflow/pull/3393 > Currently, running unit tests is a difficult process. Airflow tests depend on > many external services and other custom setup, which makes it hard for > contributors to work on this codebase. CI builds have also been > unreliable[0][1][2][3], and it is hard to reproduce the causes. Having > contributors trying to emulate the build environment every time makes it > easier to get to an "it works on my machine" sort of situation. > This PR implements a