While I think it's a good idea in theory, +1 with Daniel on this as the complexity is actually quite high.
1. It would require a complete code audit/refactoring for all config access to make sure no one is caching values. To me that seems very hard to do in an effective way. 2. Some config values can be dynamically reloaded safely while others cannot. We would get race conditions for those reloaded configs and it would be hard to hunt down and document all the edge cases. Focusing on making scheduler restarts safe and fast would (mostly) get rid of the need for dynamic config reloads. On Fri, Dec 8, 2017 at 9:07 AM, Daniel Imberman <[email protected]> wrote: > At least for the k8s executor it would be easier to just restart the > scheduler pod. I can see some potential issues if some configs are stored > in local variables. > On Wed, Dec 6, 2017 at 11:45 AM Feng Lu <[email protected]> wrote: > > > Hi, > > > > It's probably well-known that Airflow only loads config file (i.e., > > airflow.cfg) at instance creation time, if one needs to change the config > > file, all Airflow instances have to be restarted (understand that Airflow > > worker does restart itself for each task execution and therefore picks up > > the latest config updates). > > > > Are people from this mailing group interested in adding dynamic config > > loading support inside AirflowConfigParser ( > > > > https://github.com/apache/incubator-airflow/blob/master/ > airflow/configuration.py#L110 > > )? > > > > Initial implementation ideas: > > - introduce threadling.Rlock to AirflowConfigParser and guard all method > > access > > - add a periodical timer task that reads in the config file (of course > > needs to acquire the lock beforehand). > > > > Since config data is always accessed via the AirflowConfigParser object, > > this essentially gives us dynamic config update without restarting > Airflow > > scheduler/webserver. > > > > Thoughts? > > Thank you. > > > > Feng > > >
