Re: [DISCUSS] Lineage improvements and standardization of operator signatures

2020-01-22 Thread Tao Feng
Thanks Bolke. For those that are not aware, my team is working with Bolke's team on Amundsen which is a data discovery and metadata project( https://github.com/lyft/amundsen) . I think although it ships with Atlas client(or it used to be), the new API per my understanding is generic enough that

[GitHub] [airflow-on-k8s-operator] kaxil merged pull request #5: Move to kubebuilder v2 generated project

2020-01-22 Thread GitBox
kaxil merged pull request #5: Move to kubebuilder v2 generated project URL: https://github.com/apache/airflow-on-k8s-operator/pull/5 This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [airflow-on-k8s-operator] barney-s commented on issue #3: Move to go modules and update libs

2020-01-22 Thread GitBox
barney-s commented on issue #3: Move to go modules and update libs URL: https://github.com/apache/airflow-on-k8s-operator/pull/3#issuecomment-577058473 Expect a couple more pull requests as i try to get this to a working shape. Will take this oppurtunity to modernize the code base

[GitHub] [airflow-on-k8s-operator] barney-s opened a new pull request #3: Move to go modules and update libs

2020-01-22 Thread GitBox
barney-s opened a new pull request #3: Move to go modules and update libs URL: https://github.com/apache/airflow-on-k8s-operator/pull/3 Moving from gopkg to gomod dependency management. - This removes vendor folder. - Manually vendor in controller-reconciler. Update

Re: Remove Dask Executor in Airflow 2.0 ?

2020-01-22 Thread Darren Weber
Some initial learning and application of dask-distributed unit testing is WIP in - https://github.com/apache/airflow/pull/6984 All the dask-distributed pytest fixtures are already available in Airflow (with the dask option installed). The contribution docs are helpful, i.e. -

Re: [DISCUSS] Lineage improvements and standardization of operator signatures

2020-01-22 Thread Dan Davydov
Just want to preface my reply with the fact that I haven't thought about data lineage very much. This is an awesome idea :)! I like something like 1) personally, e.g. operators could optionally define a .outlet() and .inlet() interface which would return the inlets and outlets of a given task,

[GitHub] [airflow-on-k8s-operator] kaxil commented on issue #4: Add repo name to readme

2020-01-22 Thread GitBox
kaxil commented on issue #4: Add repo name to readme URL: https://github.com/apache/airflow-on-k8s-operator/pull/4#issuecomment-577295517 Good catch @feng-tao This is an automated message from the Apache Git Service. To

[GitHub] [airflow-on-k8s-operator] feng-tao commented on issue #4: Add repo name to readme

2020-01-22 Thread GitBox
feng-tao commented on issue #4: Add repo name to readme URL: https://github.com/apache/airflow-on-k8s-operator/pull/4#issuecomment-577295074 great job to get the repo into Apache! This is an automated message from the Apache

[GitHub] [airflow-on-k8s-operator] kaxil merged pull request #4: Add repo name to readme

2020-01-22 Thread GitBox
kaxil merged pull request #4: Add repo name to readme URL: https://github.com/apache/airflow-on-k8s-operator/pull/4 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

Re: Reverting [AIRFLOW-1467] Dynamic pooling

2020-01-22 Thread Jarek Potiuk
Thanks for contributing it Bin! śr., 22 sty 2020, 07:03 użytkownik Xinbin Huang napisał: > Just an update to this. The feature has been added to the master branch > using the BoringCyborg. Hopefully, we can say goodbye to multiple migration > heads from now on. > > Thanks @kaxil @jarek for the

[GitHub] [airflow-on-k8s-operator] barney-s opened a new pull request #5: Move to kubebuilder v2 generated project

2020-01-22 Thread GitBox
barney-s opened a new pull request #5: Move to kubebuilder v2 generated project URL: https://github.com/apache/airflow-on-k8s-operator/pull/5 Re-generated scaffolding and merged existing controller files into it kubebuilder init --domain apache.org kubebuilder create api --group