Hi Gerardo, Very valid points. I'm fully in favor of your proposal. To simplify the stack, I strongly believe we should also strip out tox and fully rely on Docker. Using tox will add another layer that doesn't add a lot of value from my perspective. Also, we should bake all the *.sh bootstrap scripts <https://github.com/apache/incubator-airflow/tree/master/scripts/ci> in the Docker container, instead of having to set this up before running the tests.
In the upcoming months, I might have a bit more time to spend on Airflow, I'm happy to assist you on this one. Cheers, Fokko Op wo 2 jan. 2019 om 06:51 schreef Daniel Imberman < daniel.imber...@gmail.com>: > @gerardo thank you for setting this up. > > I've also been extremely interested in this as well. I've been messing with > GCP VM instances in the past few weeks to try to simplify my local build as > well. Would definitely be interested in helping with the AIP + > implementation. > > One thing I believe we should do is set up the ci base-image with all of > the pip dependencies pre-loaded. A lot of time is wasted pip installing > dependencies. We can auto-generate new images whenever a PR is submitted to > this repository and then specify the tag in the .travis.yml when building. > > On the k8s side, I think we need to move away from minikube for k8s > testing. I discussed in a previous email setting travis to work with GKE. > I'd be careful about coupling k8s stuff too tightly with a docker > infrastructure. That can get pretty dicey. I think as long as we're using a > separate k8s cluster the k8s executor tests only need to gather the IP > addresses + have access to the kubeconfig. > > > On Tue, Jan 1, 2019 at 8:10 PM Gerardo Curiel <gera...@gerar.do> wrote: > > > Hi folks, > > > > I've created an AIP for simplifying Airflow's development workflow: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow > > > > The goal of this proposal is to outline the work needed to make local > > testing significantly easier and standardise the best practices to > > contribute to the Airflow project. > > > > Any input on it would be greatly appreciated. > > > > Cheers, > > > > -- > > Gerardo Curiel // https://gerar.do > > > On Tue, Jan 1, 2019 at 8:10 PM Gerardo Curiel <gera...@gerar.do> wrote: > > > Hi folks, > > > > I've created an AIP for simplifying Airflow's development workflow: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow > > > > The goal of this proposal is to outline the work needed to make local > > testing significantly easier and standardise the best practices to > > contribute to the Airflow project. > > > > Any input on it would be greatly appreciated. > > > > Cheers, > > > > -- > > Gerardo Curiel // https://gerar.do > > >
