This is an interesting idea, but I’d like to pick up Saša Jurić’s [call]( http://www.theerlangelist.com/article/rethinking_app_env) to move as much as possible from static application configurations and toward more dynamic runtime configuration. For the main Elixir app that I deploy, we use distillery to generate the release, and then generate an AMI image with Ansible scripts that customize the installed sys.config (this does periodically result in issues where the Ansible sys.config needs updating). We are looking at shifting toward Consul and Vault, but that is still some time away for us, so the continuity of sys.config is more important to us than anything else until that point, and when we *do* switch, I’m not sure that this will provide anything meaningful for us.
(This may be more appropriate for Heroku and Heroku-like deploy environments, but I am increasingly inclined to avoid those as much as possible.) -a On Mon, May 21, 2018 at 8:12 AM José Valim <[email protected]> wrote: > One of the major differences between running your application as a release > and as a Mix project is the differences in configuration. Mix evaluates the > configuration right before the application starts, releases evaluates the > configuration when your application is compiled. > > This implies in a large mismatch of how those two environments are used. > For releases, environment variables (read by `System.get_env/1`) need to be > set when the application is compiled and such information may not be > available at this point. > > Ideally, we would want a release to evaluate the configurations files in > `config` when the release starts. One approach would be to copy the > configuration files as is to the release but that's hard to achieve in > practice for two reasons: > > 1. A config file may import other config files and often importing those > files happen dynamically. For example: `import_config "#{Mix.env()}.exs"`. > The dynamic import makes it hard for release tools to know which > configuration files must be copied to a release, especially in cases like > umbrella projects, where a developer may load configuration across projects > > 2. Even we copy today's configuration files to a release, those > configuration files rely on `Mix`, which is a build tool and therefore it > is not available during releases > > To solve those issues, we need to make sure we can discover all imports of > a configuration file without evaluating its contents. We also need to > introduce a new module for configuration that does not depend on Mix. > > This is the goal of this proposal. > > *## Application.Config* > > This proposal is about introducing a module named `Application.Config`. It > will work similarly to the existing `Mix.Config`, except it belongs to the > `:elixir` application instead of `:mix`. This allows releases to leverage > configuration without depending on Mix. > > The user API of `Application.Config` is quite similar to `Mix.Config`. > There is `config/2` and `config/3` to define configurations. There still is > `import_config/1` to import new configuration files with one important > difference: the argument to `import_config/1` must be a literal string. So > interpolation, variables or any other dynamic pattern is no longer allowed. > > In order to help with configuration management, we will introduce a > project option in your `mix.exs`, named `:config_paths` to help manage > multiple required and optional configuration files. > > In the next section we will provide an example of how configuration files > used by projects like Nerves and Phoenix will have to be rewritten and then > we will discuss how integration with release tools such as distillery will > work. > > *### A common example* > > Projects like Nerves and Phoenix generate files with built-in > multi-environment configuration. Today, this configuration has an entry > point `config/config.exs` file that imports an environment specific > configuration at the bottom: > > # config/config.exs > use Mix.Config > > config :my_app, :some_shared_configuration, ... > > import_config "#{Mix.env()}.exs" > > > > And then each `config/{dev,test,prod}.exs` provides environment specific > configuration. For instance: > > # config/dev.exs > use Mix.Config > > config :my_app, :some_dev_configuration, ... > > > The issue in the example above is the use of dynamic imports, such as > `import_config "#{Mix.env()}.exs"`. We will address this by defining both > `config/config.exs` and `config/#{Mix.env()}.exs` as configuration entry > points in your `mix.exs`: > > # mix.exs > def project do > [ > ..., > config_paths: ~w(config/config.exs config/#{Mix.env()}.exs), > ... > ] > end > > > And now we can define those configuration files without dynamic imports: > > # config/config.exs > use Application.Config > > config :my_app, :some_shared_configuration, ... > > > > # config/dev.exs > use Application.Config > > config :my_app, :some_dev_configuration, ... > > > In Phoenix, the `config/prod.exs` case may link to a separate > `prod.secret.exs` file. While we could also refer to this file in the > `:config_paths` configuration in the `mix.exs` file, because it is only > specific to production, it is more straight-forward to continue importing > it at the bottom. So a `config/prod.exs` would look like this: > > # config/prod.exs > use Application.Config > > config :my_app, :some_prod_configuration, ... > > import_config "prod.secret.exs" > > > By adding `:config_paths`, we are able to move the dynamic configuration > to the `mix.exs` file and make the order that configuration files are > loaded clearer. > > *### A FarmBot example* > > Nerves projects tend to rely extensively on configuration files. So let's > look into existing open source Nerves projects and see how this proposal > will fare. Let's take a look at [FarmBot v6.4.1]( > https://github.com/FarmBot/farmbot_os/tree/v6.4.1). > > The questions we want to answer are: if we move the FarmBot project to the > proposed `Application.Config`, will they be able to express of all the > existing idioms they do today? And, even further, will their configuration > files become simpler or more complex? > > From looking at its [config/config.exs]( > https://github.com/FarmBot/farmbot_os/blob/v6.4.1/config/config.exs), we > can already see a pattern that won't work in releases: [the use of > `Mix.env` and `Mix.Project.config`]( > https://github.com/FarmBot/farmbot_os/blob/v6.4.1/config/config.exs#L3-L5 > ). > > We can see [those variables are used to dynamically import configuration]( > https://github.com/FarmBot/farmbot_os/blob/v6.4.1/config/config.exs#L69-L77), > which `Application.Config` won't allow. > > Those idioms are perfectly fine with how configurations work in Mix today. > But they will no longer with a release built on top of `Application.Config`. > > The solution is to move all of those imports to the `:config_paths` option > in `mix.exs`. However, note that some of those dynamic imports are > optional, so we will also need the ability to explicitly tag them as such: > > > # farmbot/mix.exs > def project do > [ > ..., > config_path: ~w(config/config.exs config/#{Mix.env()}.exs) ++ > optional_config_paths(@target, Mix.env()) > ... > ] > end > > defp optional_config_paths("host", env), > do: [{:optional, "config/host/#{env}.exs"}] > > defp optional_config_paths(target, env), > do: [{:optional, "config/target/#{env}.exs"}, {:optional, > "config/target/#{target}.exs"}] > > > > We believe this approach is an improvement to the previous one because it > allows all environment and target specific handling to remain in the > `mix.exs` file and not scattered around multiple configuration files. > > *## Using it in releases* > > In the previous sections, we have outlined `Application.Config` which no > longer depends on Mix and has a restricted `import_config`. > > Now that we are able to see all of the configuration files that affect our > system, a release tool, such as discovery, should be able to traverse all > of those configuration files and merge them into a final > `config/release.exs` that will be part of your release. In fact, Elixir > will provide a convenient API that performs such operation, streamlining > the release assembling process. > > *## Unresolved topics* > > There are two important topics that we have not included in this proposal > and they will be discussed in a further step. > > 1. What about umbrella projects? Umbrella projects also rely on > configuration and we need to make sure the listed mechanisms also work well > with umbrellas. > > 2. How to avoid common pitfalls? Even though we will migrate to > `Application.Config`, there is nothing stopping a developer from accessing > Mix (and the module defined in the `mix.exs` file) from their new config > files. As we have seen, this may lead to errors when running releases, as > releases do not have Mix available. To address this, we may introduce > checks when assembling releases that make sure `Mix` is not invoked in > configuration files, raising appropriate error messages in case they do. > > *## Summing up* > > We propose a new `Application.Config` module and a new `:config_paths` > project option that allows release tools to discover all of the relevant > configurations in a system. Release tools can then merge and copy those > configuration into releases and execute them as part of the release > process, allowing dynamic calls such as `System.get_env/1` to work in > development and in production transparently, with or without releases. > > > > *José Valimwww.plataformatec.com.br > <http://www.plataformatec.com.br/>Founder and Director of R&D* > > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L%2BAsNewcj%2BPF0fogpgDQEAc6WDMDy7X-W5evx19WPkrA%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L%2BAsNewcj%2BPF0fogpgDQEAc6WDMDy7X-W5evx19WPkrA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Austin Ziegler • [email protected] • [email protected] http://www.halostatue.ca/ • http://twitter.com/halostatue -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQuQ5F0zCQyyd6ijaCj9hPryqJ7tZk3FzZN55btXKXvW%3DA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
