Hi guys, I've set up a few sub-projects for this. @gerardo @fokko Lemme know what you guys think
https://cwiki.apache.org/confluence/display/AIRFLOW/Optimizing+Docker+Image+Workflow https://cwiki.apache.org/confluence/display/AIRFLOW/Kubernetes+Testing%3A+Using+GKE+instead+of+Minikube On Tue, Jan 1, 2019 at 11:45 PM Driesprong, Fokko <fo...@driesprong.frl> wrote: > Hi Gerardo, > > Very valid points. I'm fully in favor of your proposal. To simplify the > stack, I strongly believe we should also strip out tox and fully rely on > Docker. Using tox will add another layer that doesn't add a lot of value > from my perspective. Also, we should bake all the *.sh bootstrap scripts > <https://github.com/apache/incubator-airflow/tree/master/scripts/ci> in > the > Docker container, instead of having to set this up before running the > tests. > > In the upcoming months, I might have a bit more time to spend on Airflow, > I'm happy to assist you on this one. > > Cheers, Fokko > > Op wo 2 jan. 2019 om 06:51 schreef Daniel Imberman < > daniel.imber...@gmail.com>: > > > @gerardo thank you for setting this up. > > > > I've also been extremely interested in this as well. I've been messing > with > > GCP VM instances in the past few weeks to try to simplify my local build > as > > well. Would definitely be interested in helping with the AIP + > > implementation. > > > > One thing I believe we should do is set up the ci base-image with all of > > the pip dependencies pre-loaded. A lot of time is wasted pip installing > > dependencies. We can auto-generate new images whenever a PR is submitted > to > > this repository and then specify the tag in the .travis.yml when > building. > > > > On the k8s side, I think we need to move away from minikube for k8s > > testing. I discussed in a previous email setting travis to work with GKE. > > I'd be careful about coupling k8s stuff too tightly with a docker > > infrastructure. That can get pretty dicey. I think as long as we're > using a > > separate k8s cluster the k8s executor tests only need to gather the IP > > addresses + have access to the kubeconfig. > > > > > > On Tue, Jan 1, 2019 at 8:10 PM Gerardo Curiel <gera...@gerar.do> wrote: > > > > > Hi folks, > > > > > > I've created an AIP for simplifying Airflow's development workflow: > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow > > > > > > The goal of this proposal is to outline the work needed to make local > > > testing significantly easier and standardise the best practices to > > > contribute to the Airflow project. > > > > > > Any input on it would be greatly appreciated. > > > > > > Cheers, > > > > > > -- > > > Gerardo Curiel // https://gerar.do > > > > > > On Tue, Jan 1, 2019 at 8:10 PM Gerardo Curiel <gera...@gerar.do> wrote: > > > > > Hi folks, > > > > > > I've created an AIP for simplifying Airflow's development workflow: > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow > > > > > > The goal of this proposal is to outline the work needed to make local > > > testing significantly easier and standardise the best practices to > > > contribute to the Airflow project. > > > > > > Any input on it would be greatly appreciated. > > > > > > Cheers, > > > > > > -- > > > Gerardo Curiel // https://gerar.do > > > > > >
