We use a fork of the KubernetesPodOperator in house (and we don't use the Executor). The steps to use it are roughly:
1) Set up your Kubernetes cluster (can be EKS, GKE, etc.) 2) Make sure your Airflow cluster has network access to the Kubernetes API servers 3) Create a ServiceAccount for the Airflow user, and if using RBAC, grant it access to whatever you'll be creating with it (for instance, we run all Airflow pods within an Airflow-specific namespace) 4) Create a ~/.kube/config file to hold your Kubernetes credentials and define which clusters are available: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#define-clusters-users-and-contexts 5) Create a DAG that imports the operator and provides necessary arguments. It will use the kubectl config file's credentials when submitting to the cluster. You may also need a pipeline for building the images that the operator uses. On Wed, Apr 10, 2019 at 2:41 AM Ashwin Sai Shankar <[email protected]> wrote: > Thanks, James and Kamil! Please let me know if you have any examples of > setting up Kubernetes Executor and Operator. > > On Tue, Apr 9, 2019 at 8:03 AM James Meickle > <[email protected]> wrote: > > > Yes, that summary is correct - the Executor is using Kubernetes to > execute > > all Airflow tasks (each wrapped by a temporary Airflow process), while > the > > PodOperator is using Kubernetes only for that task, to execute one Pod > > (which likely won't run any Airflow code at all). > > > > On Tue, Apr 9, 2019 at 3:17 AM Kamil Gałuszka <[email protected]> > wrote: > > > > > Hi Ashwin, > > > > > > I had exactly same question couple of days ago. Let me try to explain. > If > > > I'm wrong please someone correct me. > > > > > > Kubernetes Executor is used to execute TaskInstance, which means that > Pod > > > is created of that TaskInstance that for ex. could be BashOperator or > > > SlackAPIOperator and after execution of this Task is finished, executor > > pod > > > is cleaned up/removed. > > > > > > If you use KubernetesPodOperator, then there could be used whatever > > > language/library from docker image that will be scheduled by airflow. > For > > > example You parse CSV files with XSV library in Rust, then > > > KubernetesPodOperator is perfect for You as you don't have to spend to > > much > > > time figuring out how to write custom operator, You just have an image > > with > > > executable that is simply used with KubernetesPodOperator. > > > > > > If You use KubernetesPodOperator with KubernetesExecutor then following > > > thing will happen: > > > > > > AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which > > will > > > create KubernetesPodOperator Pod in k8s. > > > > > > KubernetesPodOperator can be used with any other executors to my > > knowledge. > > > > > > Thanks > > > Kamil > > > > > > On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar > > > <[email protected]> wrote: > > > > > > > Hi Airflow users, > > > > What is the difference between Kube executor vs pod operator? > > > > http://airflow.apache.org/kubernetes.html > > > > > > > > Thanks, > > > > Ash > > > > > > > > > >
