[
https://issues.apache.org/jira/browse/AIRFLOW-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bolke de Bruin updated AIRFLOW-2499:
------------------------------------
Fix Version/s: 2.0.0
> Dockerised CI pipeline
> ----------------------
>
> Key: AIRFLOW-2499
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2499
> Project: Apache Airflow
> Issue Type: Test
> Components: ci, tests
> Affects Versions: 1.10.0, 1.10
> Reporter: Gerardo Curiel
> Assignee: Gerardo Curiel
> Priority: Major
> Labels: ci, docker, travis-ci
> Fix For: 2.0.0
>
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> PR: https://github.com/apache/incubator-airflow/pull/3393
> Currently, running unit tests is a difficult process. Airflow tests depend on
> many external services and other custom setup, which makes it hard for
> contributors to work on this codebase. CI builds have also been
> unreliable[0][1][2][3], and it is hard to reproduce the causes. Having
> contributors trying to emulate the build environment every time makes it
> easier to get to an "it works on my machine" sort of situation.
> This PR implements a dockerised version of the current build pipeline. This
> setup has a few advantages:
> * TravisCI tests are reproducible locally
> * The same build setup can be used to create a local development environment
> (there's a request for it [4])
>
> *Implementation details*
> * I'm using Docker Compose for the container orchestration and configuration.
> * MySQL, PostgreSQL, OpenLDAP, krb5 and rabbitmq are now services running
> inside their own containers
> * I created a separate repo, called incubator-airflow-ci[5] (TravisCI build
> here[6]), where a base image with all dependencies is built. In this case,
> I'm following the same pattern the CouchDB[7] project follows
> * Hadoop, Hive and MiniCluster were moved to this base image
> * The current TravisCI pipeline lives here[8]. A few tests are still
> failing. It's still WIP.
>
> *References*
> [0] https://issues.apache.org/jira/browse/AIRFLOW-671
> [1] https://issues.apache.org/jira/browse/AIRFLOW-968
> [2] https://issues.apache.org/jira/browse/AIRFLOW-2157
> [3] https://issues.apache.org/jira/browse/AIRFLOW-2272
> [4] https://issues.apache.org/jira/browse/AIRFLOW-1042
> [5] [https://github.com/gerardo/incubator-airflow-ci]
> [6] [https://travis-ci.org/gerardo/incubator-airflow-ci]
> [7] [https://travis-ci.org/apache/couchdb-ci]
> [8] [https://travis-ci.org/gerardo/incubator-airflow]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
