potiuk edited a comment on issue #18582: URL: https://github.com/apache/airflow/issues/18582#issuecomment-933026630
Yeah. That is an interesting one - I am not 100% sure whether this is a good idea to do it this way, to be honest. Should it be done by scheduler ? Webservers ? Which one, if we have multiple of those? I think there is no good answer to that - similarly as in case of adding new users, I think it should be kept outside of existing airflow components. I feel we are trying to put some band-aid where we need a better-thought solution. Similarly as we do not feel comfortable about running migrations automatically when we start airflow components, modifying the pools at startup of the "regular" components sounds like a bit too magical. There are some issues with that which might be very counter-intuitive when we try to implement all the cases - including deletion/incremental updates. However maybe there is a better solution to that. Maybe that is the right moment that we think about a separate "airflow config-sync" (maybe better name could be invented) kind of component, wich could take care about configuring all DB-related configuration from a code: * runing migrations * creating users * creating pools * possibly updating Airlfow Variables/Connections from exported files * maybe we have some other DB-related config that could come from outside too I think that could nicely solve a number of problems we have (and which we tried to solve different for a number of different use cases - differently in the "user-community" chart, differently in the "official" chart, diferently in 'quick-start docker compose" and differently in the image. Instead of running "commands" that we have now (airflow db upgrade, airflow users create etc.) it could use files to store desired configuration state and it could do everythig that is needed to bring the configuration to the state that is described in the files. It would serve similar purpose as the `terraform "state"` description - this would make the approach of "infrastrucure as a code" much easier and more straightforward. We could run such "config-sync" and make airflow depend on it being "ready" or "complete" in order to run and replace the "post-install" step of ours in the Helm chart if we make proper dependency management between components - same in docker-compose. I saw a number of questions from users who wanted to do it (and some of the features of the user-community chart implemented were responding to that - not in a best way as there was no good support from Airflow). And if we make such "config-sync" a first-class citizen in Airflow, we might address the need of our users in elegant and manageable way. Just a thought - I'd love what you think about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
