Re: Repurposing the `queue` field for non-celery executors

2017-08-06 Thread Daniel Imberman
I think its a really good idea from the kubernetes executor perspective. There are a lot of elements that should be automated when launching a pod (i.e. a bash command wouldn't need the same amount of compute as building a model with scikitlearn) @Alexander > don't you think that you give too

Re: Repurposing the `queue` field for non-celery executors

2017-08-06 Thread Alexander Shorin
I think it's a good intention, but don't you think that you give too much power to operators to configure how executeros should acts like? Why executtor itself can't decide which config to apply per queue basics? It's a question of ops domain, not devs. -- ,,,^..^,,, On Sun, Aug 6, 2017 at

Repurposing the `queue` field for non-celery executors

2017-08-06 Thread grantnicholas2...@u.northwestern.edu
Basic proposal: Change the queue column in the TaskInstance table from a varchar(n) => pickle. Additionally rename the column from queue => executor_config. Why? Currently the `queue` column is only used for the CeleryExecutor since the concept of a `queue` doesn't make sense for other