Re: Deploy Airflow on Kubernetes using Airflow Operator

[Rob Goretsky]

Barni,
Thank you so much for sharing this!  I'm admittedly far from a Kubernetes
guru, but I'm just trying to wrap my head around the reasons why we'd need
a custom Kubernetes controller to manage Airflow's components, as opposed
to the setup here https://github.com/mumoshu/kube-airflow wherein we simply
deploy Airflow to Kubernetes using standard Kubernetes Services and
Deployments.  I assume you're doing this because, as you mention in the
"Design" section of your repo, "In case of some stateful applications, the
declarative models provided by kubernetes are not sufficient to handle
fault remediation, scaling with data integrity and availability".   Can you
be more specific about what kinds of faults / scaling issues your
Kubernetes Controller might handle for Airflow that otherwise would not be
handled by built-in Kubernetes controllers (Services/Deployments)?

Thanks,
Rob



On Mon, Aug 6, 2018 at 1:55 AM Bolke de Bruin  wrote:

> Really awesome stuff. We are in progress to move over to k8s for Airflow
> (on prem though) and this is really helpful.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
> > Op 3 aug. 2018 om 23:35 heeft Barni Seetharaman 
> het volgende geschreven:
> >
> > Hi
> >
> > We at Google just open-sourced a Kubernetes custom controller (also
> called
> > operator) to make deploying and managing Airflow on kubernetes simple.
> > The operator pattern is a power abstraction in kubernetes.
> > Please watch this repo (in the process of adding docs) for further
> updates.
> >
> > https://github.com/GoogleCloudPlatform/airflow-operator
> >
> > Do reach out if you have any questions.
> >
> > Also created a channel in kubernetes slack  (#airflow-operator)
> >  for any
> > discussions specific to Airflow on Kubernetes (including Daniel's
> > Kubernetes Executor, Kuberenetes operator and this custom controller also
> > called kuberntes airflow operator).
> >
> > regs
> > Barni
>

Re: Deploy Airflow on Kubernetes using Airflow Operator

[Bolke de Bruin]

Really awesome stuff. We are in progress to move over to k8s for Airflow (on 
prem though) and this is really helpful.

B.

Verstuurd vanaf mijn iPad

> Op 3 aug. 2018 om 23:35 heeft Barni Seetharaman  
> het volgende geschreven:
> 
> Hi
> 
> We at Google just open-sourced a Kubernetes custom controller (also called
> operator) to make deploying and managing Airflow on kubernetes simple.
> The operator pattern is a power abstraction in kubernetes.
> Please watch this repo (in the process of adding docs) for further updates.
> 
> https://github.com/GoogleCloudPlatform/airflow-operator
> 
> Do reach out if you have any questions.
> 
> Also created a channel in kubernetes slack  (#airflow-operator)
>  for any
> discussions specific to Airflow on Kubernetes (including Daniel's
> Kubernetes Executor, Kuberenetes operator and this custom controller also
> called kuberntes airflow operator).
> 
> regs
> Barni

Re: Deploy Airflow on Kubernetes using Airflow Operator

[Sid Anand]

Neeto!

-s

On Fri, Aug 3, 2018 at 2:36 PM Barni Seetharaman 
wrote:

> Hi
>
> We at Google just open-sourced a Kubernetes custom controller (also called
> operator) to make deploying and managing Airflow on kubernetes simple.
> The operator pattern is a power abstraction in kubernetes.
> Please watch this repo (in the process of adding docs) for further updates.
>
> https://github.com/GoogleCloudPlatform/airflow-operator
>
> Do reach out if you have any questions.
>
> Also created a channel in kubernetes slack  (#airflow-operator)
>  for any
> discussions specific to Airflow on Kubernetes (including Daniel's
> Kubernetes Executor, Kuberenetes operator and this custom controller also
> called kuberntes airflow operator).
>
> regs
> Barni
>

Deploy Airflow on Kubernetes using Airflow Operator

[Barni Seetharaman]

Hi

We at Google just open-sourced a Kubernetes custom controller (also called
operator) to make deploying and managing Airflow on kubernetes simple.
The operator pattern is a power abstraction in kubernetes.
Please watch this repo (in the process of adding docs) for further updates.

https://github.com/GoogleCloudPlatform/airflow-operator

Do reach out if you have any questions.

Also created a channel in kubernetes slack  (#airflow-operator)
 for any
discussions specific to Airflow on Kubernetes (including Daniel's
Kubernetes Executor, Kuberenetes operator and this custom controller also
called kuberntes airflow operator).

regs
Barni