Re: Use Docker for running Airflow tests

[Gerardo Curiel]

Hi Sid,

Definitely, there are a few things about this flow that could be better
documented. I'm planning to address a few things like this as part of
https://issues.apache.org/jira/browse/AIRFLOW-87. Do you have any specifics
about what you would like to be documented about this flow?

The build times have gone down but not by much, given we still need to
download the docker images, although there might be a way to cache those
between builds. I'll have a proper look at some point. The builds seem more
reliable though, given there are fewer failure points now (fewer package
downloads).

Cheers,

On Thu, Sep 6, 2018 at 11:46 AM Sid Anand  wrote:

> Any chance you can document a flow that ties the CI build artifacts to this
> repo and how we would use them? Would our build times go down?
>

-- 
Gerardo Curiel // https://gerar.do

Re: Use Docker for running Airflow tests

[Sid Anand]

Any chance you can document a flow that ties the CI build artifacts to this
repo and how we would use them? Would our build times go down?

-s

On Sun, Aug 19, 2018 at 11:43 AM Driesprong, Fokko 
wrote:

> Hi all,
>
> Gerardo is doing some awesome work on Dockerizing the CI pipeline. The PR
> is still open here: https://github.com/apache/incubator-airflow/pull/3393
>
> To make this work, and not store any Docker images in private repositories,
> I've created a new repository which stores the Docker image which contains
> a lot of the dependencies we use to test against. The repository is here:
> https://github.com/apache/incubator-airflow-ci
> For most of the tests, we use mocks, but sometimes we also use the actual
> service, for example for some Hadoop, Kerberos, Hive functionality. The
> images are build in the Dockerhub service:
> https://hub.docker.com/r/airflowci/incubator-airflow-ci/builds/
> I've created it for now. I'm open for another name, but Airflow was already
> taken, and hyphens are not allowed.
>
> This change will greatly decouple our dependency on Travis and will make it
> much easier to test the code locally. Instead of setting up tox
> environments, we can just use Docker instead. I'm really enthusiastic about
> the change, and really thankful for Gerardo to put all this effort in
> there.
>
> Recently Holden Karau did a lifestream
> , and did her first PR. I
> think the most important learning from it was that testing was not as
> trivial as it should be. Therefore I think we should see if we can merge
> the PR from Gerardo soon.
>
> Cheers, Fokko
>