Executor_config is an attriute you put as an attribute of hte Task/Operators
(or via the default arg of the DAG).
A _very_ incomplete and untested example:
with DAG('my_dag', default_args={'executor_config': { "KubernetesExecutor":
{"namespsace": "client-1"}}}):
BashOperator('sleep', bash_command='sleep 60')
-ash
On Jan 13 2020, at 8:01 pm, Lior Harel <[email protected]> wrote:
> That's interesting. Can you point me in the right direction? Does
> executor_config take a list of namespaces? how to configure which namespace
> should each task/DAG run in?
>
> On Mon, 13 Jan 2020 at 19:29, Daniel Imberman <[email protected]
> (mailto:[email protected])> wrote:
> >
> > You can use the executor_config to run airflow on multiple namespaces in a
> > single Kubernetes clusters. The only caveat is there is a bug that will be
> > solved in 1.10.8 where the airflow scheduler can not delete worker pods
> > created in other namespaces. Other than that it should work just fine :).
> >
> > On 2020/01/13 16:10:26, Jarek Potiuk <[email protected]
> > (mailto:[email protected])> wrote:
> > > Not now as far as I know. But It would be a nice feature to contribute!>
> > >
> > > J.>
> > >
> > > On Mon, Jan 13, 2020 at 2:06 PM Lior Harel <[email protected]
> > > (mailto:[email protected])> wrote:>
> > >
> > > > We run Kubernetes with multiple namespaces. We use them to>
> > > > segregate clients' data, each namespace has the resources for a single>
> > > > client.>
> > > > However, we want to run a single airflow cluster containing DAGs per>
> > > > client. Is there a way to execute each DAG's tasks on a different
> > > > namespace>
> > > > based on some parameter? I see the kubernetes executor has only one>
> > > > namespace parameter.>
> > > >>
> > > > Thanks,>
> > > >>
> > >
> > >
> > > -- >
> > >
> > > Jarek Potiuk>
> > > Polidea <https://www.polidea.com/> | Principal Software Engineer>
> > >
> > > M: +48 660 796 129 <+48660796129>>
> > > [image: Polidea] <https://www.polidea.com/>>
> > >
> > via Newton Mail
> > (https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2)
> >
> >
> >
>
>
>
