Thanks, James and Kamil! Please let me know if you have any examples of setting up Kubernetes Executor and Operator.
On Tue, Apr 9, 2019 at 8:03 AM James Meickle <[email protected]> wrote: > Yes, that summary is correct - the Executor is using Kubernetes to execute > all Airflow tasks (each wrapped by a temporary Airflow process), while the > PodOperator is using Kubernetes only for that task, to execute one Pod > (which likely won't run any Airflow code at all). > > On Tue, Apr 9, 2019 at 3:17 AM Kamil Gałuszka <[email protected]> wrote: > > > Hi Ashwin, > > > > I had exactly same question couple of days ago. Let me try to explain. If > > I'm wrong please someone correct me. > > > > Kubernetes Executor is used to execute TaskInstance, which means that Pod > > is created of that TaskInstance that for ex. could be BashOperator or > > SlackAPIOperator and after execution of this Task is finished, executor > pod > > is cleaned up/removed. > > > > If you use KubernetesPodOperator, then there could be used whatever > > language/library from docker image that will be scheduled by airflow. For > > example You parse CSV files with XSV library in Rust, then > > KubernetesPodOperator is perfect for You as you don't have to spend to > much > > time figuring out how to write custom operator, You just have an image > with > > executable that is simply used with KubernetesPodOperator. > > > > If You use KubernetesPodOperator with KubernetesExecutor then following > > thing will happen: > > > > AirflowScheduler -> will create KubernetesExecutor Pod in k8s -> which > will > > create KubernetesPodOperator Pod in k8s. > > > > KubernetesPodOperator can be used with any other executors to my > knowledge. > > > > Thanks > > Kamil > > > > On Tue, Apr 9, 2019 at 2:56 AM Ashwin Sai Shankar > > <[email protected]> wrote: > > > > > Hi Airflow users, > > > What is the difference between Kube executor vs pod operator? > > > http://airflow.apache.org/kubernetes.html > > > > > > Thanks, > > > Ash > > > > > >
