This seems super useful to have. We would like to use to autoscale our
infrastructure.
I've been eyeing this JIRA ticket for this feature:
https://issues.apache.org/jira/browse/AIRFLOW-1123
Has anyone else investigated this or know if this is on the roadmap?
Seems like this can be dealt with in
Hi Kevin,
I can answer in more detail in about 1 hour but I'll give a few quick
points:
1. We decided that jobs are pretty risky when it comes to airflow
deployments (imagine a pod that launches a spark job retrying infinitely),
however airflow allows you to define a retry policy
2. We
Hi all,
My team and I have been experimenting with Airflow and Kubernetes, and
there has been a lot of activity recently with the Kubernetes Executor so
hopefully someone can help us out.
Specifically, we are using our own variant of the kubernetes executor to
run some pods on pre-emptible VMs
