FYI, most of the discussion is happening on the forum: https://elixirforum.com/t/proposal-moving-towards-discoverable-config-files/14302/73
Allen Madsen http://www.allenmadsen.com On Thu, May 24, 2018 at 12:33 AM, Austin Ziegler <[email protected]> wrote: > This is an interesting idea, but I’d like to pick up Saša Jurić’s [call]( > http://www.theerlangelist.com/article/rethinking_app_env) to move as much > as possible from static application configurations and toward more dynamic > runtime configuration. For the main Elixir app that I deploy, we use > distillery to generate the release, and then generate an AMI image with > Ansible scripts that customize the installed sys.config (this does > periodically result in issues where the Ansible sys.config needs updating). > We are looking at shifting toward Consul and Vault, but that is still some > time away for us, so the continuity of sys.config is more important to us > than anything else until that point, and when we *do* switch, I’m not > sure that this will provide anything meaningful for us. > > (This may be more appropriate for Heroku and Heroku-like deploy > environments, but I am increasingly inclined to avoid those as much as > possible.) > > -a > > On Mon, May 21, 2018 at 8:12 AM José Valim <[email protected]> wrote: > >> One of the major differences between running your application as a >> release and as a Mix project is the differences in configuration. Mix >> evaluates the configuration right before the application starts, releases >> evaluates the configuration when your application is compiled. >> >> This implies in a large mismatch of how those two environments are used. >> For releases, environment variables (read by `System.get_env/1`) need to be >> set when the application is compiled and such information may not be >> available at this point. >> >> Ideally, we would want a release to evaluate the configurations files in >> `config` when the release starts. One approach would be to copy the >> configuration files as is to the release but that's hard to achieve in >> practice for two reasons: >> >> 1. A config file may import other config files and often importing >> those files happen dynamically. For example: `import_config >> "#{Mix.env()}.exs"`. The dynamic import makes it hard for release tools to >> know which configuration files must be copied to a release, especially in >> cases like umbrella projects, where a developer may load configuration >> across projects >> >> 2. Even we copy today's configuration files to a release, those >> configuration files rely on `Mix`, which is a build tool and therefore it >> is not available during releases >> >> To solve those issues, we need to make sure we can discover all imports >> of a configuration file without evaluating its contents. We also need to >> introduce a new module for configuration that does not depend on Mix. >> >> This is the goal of this proposal. >> >> *## Application.Config* >> >> This proposal is about introducing a module named `Application.Config`. >> It will work similarly to the existing `Mix.Config`, except it belongs to >> the `:elixir` application instead of `:mix`. This allows releases to >> leverage configuration without depending on Mix. >> >> The user API of `Application.Config` is quite similar to `Mix.Config`. >> There is `config/2` and `config/3` to define configurations. There still is >> `import_config/1` to import new configuration files with one important >> difference: the argument to `import_config/1` must be a literal string. So >> interpolation, variables or any other dynamic pattern is no longer allowed. >> >> In order to help with configuration management, we will introduce a >> project option in your `mix.exs`, named `:config_paths` to help manage >> multiple required and optional configuration files. >> >> In the next section we will provide an example of how configuration files >> used by projects like Nerves and Phoenix will have to be rewritten and then >> we will discuss how integration with release tools such as distillery will >> work. >> >> *### A common example* >> >> Projects like Nerves and Phoenix generate files with built-in >> multi-environment configuration. Today, this configuration has an entry >> point `config/config.exs` file that imports an environment specific >> configuration at the bottom: >> >> # config/config.exs >> use Mix.Config >> >> config :my_app, :some_shared_configuration, ... >> >> import_config "#{Mix.env()}.exs" >> >> >> >> And then each `config/{dev,test,prod}.exs` provides environment specific >> configuration. For instance: >> >> # config/dev.exs >> use Mix.Config >> >> config :my_app, :some_dev_configuration, ... >> >> >> The issue in the example above is the use of dynamic imports, such as >> `import_config "#{Mix.env()}.exs"`. We will address this by defining both >> `config/config.exs` and `config/#{Mix.env()}.exs` as configuration entry >> points in your `mix.exs`: >> >> # mix.exs >> def project do >> [ >> ..., >> config_paths: ~w(config/config.exs config/#{Mix.env()}.exs), >> ... >> ] >> end >> >> >> And now we can define those configuration files without dynamic imports: >> >> # config/config.exs >> use Application.Config >> >> config :my_app, :some_shared_configuration, ... >> >> >> >> # config/dev.exs >> use Application.Config >> >> config :my_app, :some_dev_configuration, ... >> >> >> In Phoenix, the `config/prod.exs` case may link to a separate >> `prod.secret.exs` file. While we could also refer to this file in the >> `:config_paths` configuration in the `mix.exs` file, because it is only >> specific to production, it is more straight-forward to continue importing >> it at the bottom. So a `config/prod.exs` would look like this: >> >> # config/prod.exs >> use Application.Config >> >> config :my_app, :some_prod_configuration, ... >> >> import_config "prod.secret.exs" >> >> >> By adding `:config_paths`, we are able to move the dynamic configuration >> to the `mix.exs` file and make the order that configuration files are >> loaded clearer. >> >> *### A FarmBot example* >> >> Nerves projects tend to rely extensively on configuration files. So let's >> look into existing open source Nerves projects and see how this proposal >> will fare. Let's take a look at [FarmBot v6.4.1](https://github.com/ >> FarmBot/farmbot_os/tree/v6.4.1). >> >> The questions we want to answer are: if we move the FarmBot project to >> the proposed `Application.Config`, will they be able to express of all the >> existing idioms they do today? And, even further, will their configuration >> files become simpler or more complex? >> >> From looking at its [config/config.exs](https:// >> github.com/FarmBot/farmbot_os/blob/v6.4.1/config/config.exs), we can >> already see a pattern that won't work in releases: [the use of `Mix.env` >> and `Mix.Project.config`](https://github.com/FarmBot/farmbot_os/ >> blob/v6.4.1/config/config.exs#L3-L5). >> >> We can see [those variables are used to dynamically import configuration]( >> https://github.com/FarmBot/farmbot_os/blob/v6.4.1/config/config.exs#L69- >> L77), which `Application.Config` won't allow. >> >> Those idioms are perfectly fine with how configurations work in Mix >> today. But they will no longer with a release built on top of >> `Application.Config`. >> >> The solution is to move all of those imports to the `:config_paths` >> option in `mix.exs`. However, note that some of those dynamic imports are >> optional, so we will also need the ability to explicitly tag them as such: >> >> >> # farmbot/mix.exs >> def project do >> [ >> ..., >> config_path: ~w(config/config.exs config/#{Mix.env()}.exs) ++ >> optional_config_paths(@target, Mix.env()) >> ... >> ] >> end >> >> defp optional_config_paths("host", env), >> do: [{:optional, "config/host/#{env}.exs"}] >> >> defp optional_config_paths(target, env), >> do: [{:optional, "config/target/#{env}.exs"}, {:optional, >> "config/target/#{target}.exs"}] >> >> >> >> We believe this approach is an improvement to the previous one because it >> allows all environment and target specific handling to remain in the >> `mix.exs` file and not scattered around multiple configuration files. >> >> *## Using it in releases* >> >> In the previous sections, we have outlined `Application.Config` which no >> longer depends on Mix and has a restricted `import_config`. >> >> Now that we are able to see all of the configuration files that affect >> our system, a release tool, such as discovery, should be able to traverse >> all of those configuration files and merge them into a final >> `config/release.exs` that will be part of your release. In fact, Elixir >> will provide a convenient API that performs such operation, streamlining >> the release assembling process. >> >> *## Unresolved topics* >> >> There are two important topics that we have not included in this proposal >> and they will be discussed in a further step. >> >> 1. What about umbrella projects? Umbrella projects also rely on >> configuration and we need to make sure the listed mechanisms also work well >> with umbrellas. >> >> 2. How to avoid common pitfalls? Even though we will migrate to >> `Application.Config`, there is nothing stopping a developer from accessing >> Mix (and the module defined in the `mix.exs` file) from their new config >> files. As we have seen, this may lead to errors when running releases, as >> releases do not have Mix available. To address this, we may introduce >> checks when assembling releases that make sure `Mix` is not invoked in >> configuration files, raising appropriate error messages in case they do. >> >> *## Summing up* >> >> We propose a new `Application.Config` module and a new `:config_paths` >> project option that allows release tools to discover all of the relevant >> configurations in a system. Release tools can then merge and copy those >> configuration into releases and execute them as part of the release >> process, allowing dynamic calls such as `System.get_env/1` to work in >> development and in production transparently, with or without releases. >> >> >> >> *José Valimwww.plataformatec.com.br >> <http://www.plataformatec.com.br/>Founder and Director of R&D* >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-core" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit https://groups.google.com/d/ >> msgid/elixir-lang-core/CAGnRm4L%2BAsNewcj%2BPF0fogpgDQEAc6WDMDy7X- >> W5evx19WPkrA%40mail.gmail.com >> <https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L%2BAsNewcj%2BPF0fogpgDQEAc6WDMDy7X-W5evx19WPkrA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> For more options, visit https://groups.google.com/d/optout. >> > > > -- > Austin Ziegler • [email protected] • [email protected] > http://www.halostatue.ca/ • http://twitter.com/halostatue > > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elixir-lang-core/CAJ4ekQuQ5F0zCQyyd6ijaCj9hPryq > J7tZk3FzZN55btXKXvW%3DA%40mail.gmail.com > <https://groups.google.com/d/msgid/elixir-lang-core/CAJ4ekQuQ5F0zCQyyd6ijaCj9hPryqJ7tZk3FzZN55btXKXvW%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAK-y3CuyVSmmGJJxBpgpeMwB%2BykkQkzTs9uAHmgD86Am%3DNDbsw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
