Hey Alex, with resource management I meant the Pod CPU and Memory limits <http://kubernetes.io/docs/admin/limitrange/> which can be set in k8s deployments. By default the pods will use all available resources on the node it's on. In theory this could mean a pod ends up by itself on 1 node when it consumes a lot of resources.
I agree with your remark about delegation in GCP, I tried keeping workers small so I could run multiple on a single node, and scale the number of workers up/down accordingly. The reason this failed for me was because I also had these resource limits apply on scheduler and rabbitmq pods. This resulted in SIGTERM errors, most likely due to lack of resources. Where your ConfigMap is now populated from a settings file, I think an override from env vars would also be very nice. I'll definitely give this a try when I have some time! On Mon, Dec 5, 2016 at 1:20 PM, Alex Van Boxel <[email protected]> wrote: > Hey Koen, I tried running it with 2 but one is for my use-case good enough. > I don't exactly know what you "in-this-context" mean by managing resources, > but on GCloud you want to delegate as much as possible to the services > (DataProc/DataFlow/BigQuery) and soon also other docker containers in the > same cluster, so I like to keep the workers as light as possible. But good > point about the max (I should make it configurable). > > My cluster for now is 2x4CPU for my Kubernetes cluster, but it has other > stuff running aside from Airflow. > > On Mon, Dec 5, 2016 at 12:42 PM Koen Mevissen <[email protected]> > wrote: > > > Nice one, thanks! I tried this a while ago, and got stuck on resource > > issues - probably because I capped the pods max resources to hard. > > > > I see in the worker yaml it creates 1 worker replica, are you running it > > with multiple workers as well? Are you managing any resources on the > pods, > > or you let the workers consume whatever's available on the nodes? > > > > > > On Fri, Dec 2, 2016 at 10:14 PM, Chris Riccomini <[email protected]> > > wrote: > > > > > Nice, thanks! :) > > > > > > On Fri, Dec 2, 2016 at 4:55 AM, Alex Van Boxel <[email protected]> > wrote: > > > > > > > Hi all, > > > > > > > > I think I pulled it off to have kind of "one-execute" install of > > Airflow > > > on > > > > Google Container Engine. Personally I find this my preferred setup > > > (because > > > > I can have a production environment and staging environment). It's > > what I > > > > use for our production/staging setup. I think someone else could pull > > it > > > > off with everything in this repo: > > > > > > > > https://github.com/alexvanboxel/airflow-gcp-k8s > > > > > > > > It has a README that should be enough to get it setup. > > > > > > > > Now I'll work on my sync'er so my dags will refresh as soon as I push > > to > > > > git :-) > > > > > > > > > > > > > > > -- > > Kind regards, > > Met vriendelijke groet, > > > > *Koen Mevissen* > > Principal BI Developer > > > > > > *Travix Nederland B.V.* > > Piet Heinkade 55 > > 1019 GM Amsterdam > > The Netherlands > > > > T. +31 (0)20 203 3241 <+31%2020%20203%203241> > > E: [email protected] > > www.travix.com > > > > *Brands: * CheapTickets | Vliegwinkel | Vayama | BudgetAir | > > Flugladen > > > -- Kind regards, Met vriendelijke groet, *Koen Mevissen* Principal BI Developer *Travix Nederland B.V.* Piet Heinkade 55 1019 GM Amsterdam The Netherlands T. +31 (0)20 203 3241 E: [email protected] www.travix.com *Brands: * CheapTickets | Vliegwinkel | Vayama | BudgetAir | Flugladen
