Note: this proposal is also available in a gist - https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6
Streamlining child specs Hello everyone, This is a proposal for improving how supervisors are defined and used in Elixir. Before we go to the improvements, let's discuss the pain points of the current API. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#the-current-state-of-the-art>The current state of the art <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#module-based-supervisors>Module-based supervisors Supervisors in Elixir can be defined using modules: defmodule MySupervisor do use Supervisor def start_link do Supervisor.start_link(__MODULE__, []) end def init([]) do children = [ worker(Child, []) ] supervise(children, strategy: :one_for_one) endend Notice we introduced the functions worker and supervise, which are part of the Supervisor.Spec module, to take care of defining the tuples used behind the scenes. However, the above is still verbose for the cases we want to simply start a supervisor. That's when we introduced callback-less supervisors. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#callback-less-supervisors>Callback-less supervisors The idea behind callback-less supervisors is that you can start them directly without needing callbacks: import Supervisor.Spec children = [ worker(Child, []) ] Supervisor.start_link(children, strategy: :one_for_one) While the example above is an improvement compared to previous versions, especially the early ones that required tuples to be passed around instead of using the worker/supervisor functions, it has the following issues: - Importing Supervisor.Spec which exists in a separate module from Supervisor is confusing in terms of learning and exploring the API and documentation - Children inside callback-less supervisors cannot be hot code upgraded, since the definition is provided before the supervisor even starts * The worker(Child, []) API is confusing in regards to which function it will invoke and with which arguments. The fact worker(Child, [1, 2, 3]) invokes Child.start_link(1, 2, 3) which is then packed into an argument given to GenServer.start_link/3 start is not helpful. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#application-integration>Application integration Perhaps the most troubling aspect of supervisors is the amount of code necessary to integrate any supervised process into a new application. For example, imagine you have created a new application with mix new my_app and after a while you decide it needs an agent. In order to correct supervise your agent implenentation, you will need to: 1. Define a new module that use Application and implements the start/2 callback which starts a supervisor with the agent as a worker defmodule MyApp do use Application def start(_type, _args) do import Supervisor.Spec children = [ worker(MyAgent, [], restart: :transient) ] Supervisor.start_link(children, strategy: :one_for_one) endend 2. Make sure to define the `:mod key in the mix.exs: elixir def application do [mod: {MyApp, []}] end The amount of code to go from unsupervised to supervised often lead people down the wrong road of not adding supervision at all. Another sign the code above is a common boilerplate is that it has been automatized under the mix new my_app --sup flag. The --sup flag has been helpful but it is useless in projects you have already created. The other problem with the code above is that the agent configuration ends-up spread through multiple files. Properties such as :shutdown and its type (worker or supervisor) is most times better to be co-located with its implementation rather than specified separately in the supervision tree. Specifying it at the supervision tree is useful only in cases the same implementation is started multiple times under the same or different trees, which is not the most common scenario. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#a-streamlined-solution>A streamlined solution In the previous section we have explored the current state of application and supervisors in Elixir. In this section, we propose a streamlined solution that is going to touch all of the problems outlined above, starting with the application one. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#default-application-callbacks>Default application callbacks Ideally, we want to make it as straight-forward as possible to go from non-supervised to supervised code. If most applications callbacks end-up having the exact same structure, such boilerplate can likely be addressed by Elixir. Therefore, instead of forcing users to define an application callback with a boilerplate supervision tree, Elixir could ship with the application definition where we only need to tell Elixir which processes to supervise: def application do [extra_applications: [:logger], supervise: [MyAgent, {Registry, name: MyRegistry}]]end The :supervise option allows us to specify a list of module names or tuples with two elements, where the first element is the module name and second element is any argument. When the application starts, Elixir will traverse the list invoking the child_spec/1 function on each module, passing the arguments. In other words, we will push developers to co-locate their supervision specification with the module definition. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#the-child_spec1-function>The child_spec/1 function The heart of the proposal is in allowing modules to specify the supervision specification for the processes they implement. For example, our agent definition may look like this now: defmodule MyAgent do def child_spec(_) do %{id: MyAgent, start: {__MODULE__, :start_link, []}, restart: :transient} end # ...end With the definition above, we are now able to use MyAgent as a supervised process in our application, as shown in the previous section. If your application requires multiple supervisors, Supervisors can also be added to the application's :superviseoption, as long as they also implement the child_spec/1 function. Furthermore, notice the Supervisor API, such as start_link/2 will also be updated to allow a list of modules, in the same format as the application's :supervise: defmodule MySup do def child_spec(_) do %{id: MySup, start: {__MODULE__, :start_link, []}, type: :supervisor} end def start_link do Supervisor.start_link([OtherAgent], strategy: :simple_one_for_one, name: MySup) endend Besides keeping the agent configuration in the same module it is defined, the module based child_spec/1 has a big advantage of also being hot code swap friendly, because by changing the child_spec/1 function in the MyAgentmodule, we can patch how the agent runs in production. Not only that, by colocating child_spec/1 with the code, we make it simpler to start abstractions such as Phoenix.Endpoint or Ecto.Repo under a supervision tree, as those abstractions can automatically define the child_spec/1 function, no longer forcing developers to know if those entries have the type worker or supervisor. Today developers likely have the following code in their Phoenix applications: children = [ supervisor(MyApp.Endpoint, []), supervisor(MyApp.Repo, []) ] which constraints both Ecto and Phoenix because now they are forced to always run a supervisor at the top level in order to maintain backwards compatibility. Even worse, if a developer add those entries by hand with the wrong type, their applications will be misconfigured. By storing the child_spec directly in the repository and endpoint, developers only need to write: supervise: [MyApp.Endpoint, MyApp.Repo] which is cleaner and less error prone. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#catching-up-with-otp>Catching up with OTP In the previous section we have mentioned that Supervisor.start_link/2 will also accept a list of modules or a list of tuples with two elements on Supervisor.start_link/2. In such cases, the supervisor will invoke child_spec/1appropriately on the list elements. You may have also noticed we have used maps to build the child_spec/1 functions: def child_spec(_) do %{id: MySup, start: {__MODULE__, :start_link, []}, type: :supervisor}end The reason we chose a map with the keys above is to mirror the API introduced in Erlang/OTP 18. Overall, Supervisor.start_link/2 expect a list of children where each child must be: - an atom - such as MyApp. In such cases, MyApp.child_spec([]) will be invoked when the supervisor starts - a tuple with two elements - such as {MyApp, arg}. In such cases, MyApp.child_spec(arg) will be invoked when the supervisor starts - a map - containing the keys :id (required), :start (required), :shutdown, :restart, :type or :modules Introducing maps will allow a more natural and data-centric approach for configuring supervisors, no longer needing the helper functions defined in Supervised.Spec today. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#a-practical-example>A practical example In order to show how the changes above will affect existing applications, we checked how applications generated by Phoenix would leverage them. The answer is quite positive. The whole lib/demo.ex file will no longer exist: defmodule Demo do use Application # See http://elixir-lang.org/docs/stable/elixir/Application.html # for more information on OTP Applications def start(_type, _args) do import Supervisor.Spec # Define workers and child supervisors to be supervised children = [ # Start the Ecto repository supervisor(Demo.Repo, []), # Start the endpoint when the application starts supervisor(Demo.Endpoint, []), # Start your own worker by calling: Demo.Worker.start_link(arg1, arg2, arg3) # worker(Demo.Worker, [arg1, arg2, arg3]), ] # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html # for other strategies and supported options opts = [strategy: :one_for_one, name: Demo.Supervisor] Supervisor.start_link(children, opts) endend instead it will be replaced by: def application do [extra_applications: [:logger], supervise: [Demo.Repo, Demo.Endpoint]]end which Mix will translate behind the scenes to: def application do [extra_applications: [:logger], mod: {Application.Default, %{supervise: [Demo.Repo, Demo.Endpoint]}}]end Overall we propose the following additions to def application: - :supervise - expects a list of module names to supervise when the application starts (maps as childspecs are not supported as that would make them compile time) - :max_restarts - the amount of restarts supported by application supervision tree in :max_seconds (defaults to 1) - :max_seconds - the amount of time for :max_restarts (defaults to 5) Those options will be handled by the default application module implemented in Application.Default. Mix will error building the application file if any of the options above are given alongside :mod. The application tree supervisor will have strategy of :one_for_one and such is not configurable. If you need other supervision strategies, then all you need to do is define your own supervisor with a custom strategy and list it under :supervise in def application. At the end of the day, the changes above will render both mix new --sup and Supervisor.Spec obsolete, as we now co-locate the child_spec/1 with the module definition, leading to better code and hot code upgrades, also making it easier to move from non-supervised to supervised applications. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#automating-child_spec1> Automating child_spec/1 We have explicitly defined the child_spec/1 function in all of the examples above. This poses a problem: what if someone defines a Supervisor module and forget to mark its type as :supervisor in the specification? Given the Supervisor module already knows sensible defaults, we should rely on them for building specs. Therefore, we also propose use Agent, use Task, use Supervisor and use GenServer automatically define a child_spec/1function that is overridable: defmodule MyAgent do use Agent, restart: :transient # Generated on: use Agent # def child_spec(opts) do # %{id: MyAgent, start: {__MODULE__, :start_link, [opts]}, restart: :transient} # end def start_link(opts) do Agent.start_link(fn -> %{} end, opts) end ...end The default child_spec/1 will pass the argument given to child_spec/1 directly to start_link/1. This means MyAgentcould be started with no name in a supervision tree as follows: supervise: [MyAgent] and with a given name as: supervise: [{MyAgent, name: FooBar}] In other words, the changes in this proposal will push developers towards a single start_link/1 function that will receive options as a single argument. This mirrors the API exposed in GenServer, Supervisor and friends, removing the common confusion with passing more than one argument to start_link. Of course, any of the defaults above, as well as the rare cases developers wants to dynamically customize child_spec/1 can be achieved by overriding the default implementation of child_spec/1. <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#summing-up>Summing up This proposal aims to streamline the move from non-supervised applications to supervised applications. To do so, we introduced the ability to colocate the child_spec/1 definition with the module implementation, which leads to more self-contained abstractions, such as Ecto.Repo and Phoenix.Endpoint. Those changes also allowed us to overhaul the supervision system, getting rid of Supervisor.Spec and making it more data-centric. While those changes remove some of the importance given to application callbacks today, we believe this is a positive change. There is no need to talk about application callbacks if the majority of developers are using application callbacks simply to start supervision trees. The application callback will still remain import for scenarios where custom start, config_change or stop are necessary, such as those relying on included applications. Since Elixir is a stable language, all of the code available today will continue to work, although this proposal opens up the possibility to deprecate Supervisor.Spec and mix new --sup in a far ahead future, preparing for an eventual removal in Elixir 2.0. *Thanks to Dave Thomas for initiating the proposal and the Elixir team for further discussions and reviews. Also thanks to Chris McCord and Saša Jurić for feedback.* *José Valim* www.plataformatec.com.br Skype: jv.ptec Founder and Director of R&D -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4J1mmJQ_QUML5VeSnT81kprkTWBtjekD3hpHMUhVDJBZg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
