Yep +1 for queue parameter.
I had implemented once a POC for a multi-executor that also used 'queue' to
route between any executors (and potentially multiple of them):
* Main commit here:
https://github.com/PolideaInternal/airflow/commit/c87e87a5e7673e67e78d4ce39d5a15a8cc494e56
* fixup here:
ht
+1 for queue param
On Thu, Aug 27, 2020, 04:38 Ping Zhang wrote:
> Hi Daniel,
>
> Thanks for checking that. I like the idea of using the queue field too. I
> will make the change and tag you in the PR.
>
> Best wishes
>
> Ping Zhang
>
>
> On Mon, Aug 24, 2020 at 9:10 AM Daniel Imberman >
> wro
Hi Daniel,
Thanks for checking that. I like the idea of using the queue field too. I
will make the change and tag you in the PR.
Best wishes
Ping Zhang
On Mon, Aug 24, 2020 at 9:10 AM Daniel Imberman
wrote:
> Hi Ping,
>
> This looks great! The only change I might suggest is maybe we check t
Hi Ping,
This looks great! The only change I might suggest is maybe we check the “queue”
field and set the router to KubernetesExecutor is “queue=‘kubernetes’”. What do
you think?
via Newton Mail
[https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2]
On Mon, Aug
ah, did not know that. Thanks for letting me know.
The image is not that important, all the important information is in the
gist:
https://gist.github.com/pingzh/cc44c97336560b658d012c225a2242cc#file-jobdispatcherexecutor-py
Best wishes
Ping Zhang
On Sun, Aug 23, 2020 at 11:59 PM Deng Xiaodong
Hi Ping,
Inserting images in the email body doesn’t work for this mail list. So we
cannot actually see the pictures you shared.
If you find them important, you may share the pictures somewhere then
provide the links in the mail body.
Thanks.
XD
On Mon, Aug 24, 2020 at 08:56 Ping Zhang wrote:
Great ideas.
Actually, when I evaluated using KubernetesExecutor in Airbnb, I created
*JobDispatcherExecutor*, which is similar to the idea 2.
We have tons of sensor tasks launched at around UTC 00:00 and we enabled
the SmartSensor, which means each sensor task is very short running. If we
choose
+1 for pod mutation in config. It's not a "yamly" thing - it's a python
code to run, so it's quite appropriate to have it in config.
On Wed, Aug 12, 2020 at 9:43 PM Daniel Imberman
wrote:
> I think pod_mutation is important as an “admin override”. E.g. there might
> be a label that is required b
I think pod_mutation is important as an “admin override”. E.g. there might be a
label that is required by security policy for all tasks launched by airflow. I
do agree that the executor_config should take in a V1Pod object
One option is that the executor_config could take in a pod_mutation_hook
I'll just leave some passing thoughts here. How the pod template file is
modified by the executor config might not be very well defined. Currently
when a user passes in a k8s.V1Pod object through executor_config we kind of
arbitrarily merge the two pod definitions to what we imagine the client
woul
+1 for the idea, thanks Daniel! I agree that multi-executor and pod
templates should be 2.1.
T.
On Wed, Aug 12, 2020 at 5:13 PM Daniel Imberman
wrote:
> (Also funny enough we could use a lot of the existing infrastructure int
> 1.10 to create that migration script. Would just need to take the p
(Also funny enough we could use a lot of the existing infrastructure int 1.10
to create that migration script. Would just need to take the python dictionary
and use the python yaml library)
via Newton Mail
[https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2]
On
100% agreed on timing. I think 2.0 should be for the breaking aspect (losing
the configurations) and then 2.1/2.2 we can start adding on the new features. I
also like the idea of a migration tool. We can make a script that takes your
airflow.cfg and converts that into a yaml file.
via Newton Ma
Big +1. All the arguments are very appealing to me and simplifying the
Kubernetes Executor down to YAML-configurable one seems like a no-brainer
especially if we provide some migration tools. I've lost countless hours on
debugging some configuration problems, simply because the relevant
Kubernetes-
14 matches
Mail list logo
