I definitely like this proposal overall. There are some nuances I don't understand yet, but it sounds like y'all are asking the right questions.
In our code base we've already been defined a function like `child_spec` a few times for this exact purpose, so it neatly aligns with how I'd like to do things. We've been calling the function `supervisor_spec` (e.g. it's the function to get the spec for a supervisor to supervise a process defined by this module) instead of `child_spec` and I think that's a better name -- but it sounds like there's some inertia around `child_spec` from other projects. Myron On Tuesday, February 21, 2017 at 10:17:11 AM UTC-8, José Valim wrote: > > Note: this proposal is also available in a gist - > https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6 > > Streamlining child specs > > Hello everyone, > > This is a proposal for improving how supervisors are defined and used in > Elixir. Before we go to the improvements, let's discuss the pain points of > the current API. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#the-current-state-of-the-art>The > > current state of the art > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#module-based-supervisors>Module-based > > supervisors > > Supervisors in Elixir can be defined using modules: > > defmodule MySupervisor do > use Supervisor > > def start_link do > Supervisor.start_link(__MODULE__, []) > end > > def init([]) do > children = [ > worker(Child, []) > ] > > supervise(children, strategy: :one_for_one) > endend > > Notice we introduced the functions worker and supervise, which are part > of the Supervisor.Spec module, to take care of defining the tuples used > behind the scenes. > > However, the above is still verbose for the cases we want to simply start > a supervisor. That's when we introduced callback-less supervisors. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#callback-less-supervisors>Callback-less > > supervisors > > The idea behind callback-less supervisors is that you can start them > directly without needing callbacks: > > import Supervisor.Spec > > children = [ > worker(Child, []) > ] > Supervisor.start_link(children, strategy: :one_for_one) > > While the example above is an improvement compared to previous versions, > especially the early ones that required tuples to be passed around instead > of using the worker/supervisor functions, it has the following issues: > > - > > Importing Supervisor.Spec which exists in a separate module from > Supervisor is confusing in terms of learning and exploring the API and > documentation > - > > Children inside callback-less supervisors cannot be hot code upgraded, > since the definition is provided before the supervisor even starts > > * The worker(Child, []) API is confusing in regards to which function > it will invoke and with which arguments. The fact worker(Child, [1, 2, > 3]) invokes Child.start_link(1, 2, 3) which is then packed into an > argument given to GenServer.start_link/3 start is not helpful. > > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#application-integration>Application > > integration > > Perhaps the most troubling aspect of supervisors is the amount of code > necessary to integrate any supervised process into a new application. For > example, imagine you have created a new application with mix new my_app and > after a while you decide it needs an agent. In order to correct supervise > your agent implenentation, you will need to: > > 1. > > Define a new module that use Application and implements the start/2 > callback > which starts a supervisor with the agent as a worker > > defmodule MyApp do > use Application > > def start(_type, _args) do > import Supervisor.Spec > > children = [ > worker(MyAgent, [], restart: :transient) > ] > > Supervisor.start_link(children, strategy: :one_for_one) > endend > > 2. Make sure to define the `:mod key in the mix.exs: > > elixir def application do [mod: {MyApp, []}] end > > The amount of code to go from unsupervised to supervised often lead people > down the wrong road of not adding supervision at all. Another sign the code > above is a common boilerplate is that it has been automatized under the mix > new my_app --sup flag. The --sup flag has been helpful but it is useless > in projects you have already created. > > The other problem with the code above is that the agent configuration > ends-up spread through multiple files. Properties such as :shutdown and > its type (worker or supervisor) is most times better to be co-located > with its implementation rather than specified separately in the supervision > tree. Specifying it at the supervision tree is useful only in cases the > same implementation is started multiple times under the same or different > trees, which is not the most common scenario. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#a-streamlined-solution>A > > streamlined solution > > In the previous section we have explored the current state of application > and supervisors in Elixir. In this section, we propose a streamlined > solution that is going to touch all of the problems outlined above, > starting with the application one. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#default-application-callbacks>Default > > application callbacks > > Ideally, we want to make it as straight-forward as possible to go from > non-supervised to supervised code. If most applications callbacks end-up > having the exact same structure, such boilerplate can likely be addressed > by Elixir. > > Therefore, instead of forcing users to define an application callback with > a boilerplate supervision tree, Elixir could ship with the application > definition where we only need to tell Elixir which processes to supervise: > > def application do > [extra_applications: [:logger], > supervise: [MyAgent, {Registry, name: MyRegistry}]]end > > The :supervise option allows us to specify a list of module names or > tuples with two elements, where the first element is the module name and > second element is any argument. When the application starts, Elixir will > traverse the list invoking the child_spec/1 function on each module, > passing the arguments. > > In other words, we will push developers to co-locate their supervision > specification with the module definition. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#the-child_spec1-function>The > > child_spec/1 function > > The heart of the proposal is in allowing modules to specify the > supervision specification for the processes they implement. For example, > our agent definition may look like this now: > > defmodule MyAgent do > def child_spec(_) do > %{id: MyAgent, start: {__MODULE__, :start_link, []}, restart: :transient} > end > # ...end > > With the definition above, we are now able to use MyAgent as a supervised > process in our application, as shown in the previous section. > > If your application requires multiple supervisors, Supervisors can also be > added to the application's :superviseoption, as long as they also > implement the child_spec/1 function. Furthermore, notice the Supervisor API, > such as start_link/2 will also be updated to allow a list of modules, in > the same format as the application's :supervise: > > defmodule MySup do > def child_spec(_) do > %{id: MySup, start: {__MODULE__, :start_link, []}, type: :supervisor} > end > > def start_link do > Supervisor.start_link([OtherAgent], strategy: :simple_one_for_one, name: > MySup) > endend > > Besides keeping the agent configuration in the same module it is defined, > the module based child_spec/1 has a big advantage of also being hot code > swap friendly, because by changing the child_spec/1 function in the > MyAgentmodule, we can patch how the agent runs in production. > > Not only that, by colocating child_spec/1 with the code, we make it > simpler to start abstractions such as Phoenix.Endpoint or Ecto.Repo under > a supervision tree, as those abstractions can automatically define the > child_spec/1 function, no longer forcing developers to know if those > entries have the type worker or supervisor. > > Today developers likely have the following code in their Phoenix > applications: > > children = [ > supervisor(MyApp.Endpoint, []), > supervisor(MyApp.Repo, []) > ] > > which constraints both Ecto and Phoenix because now they are forced to > always run a supervisor at the top level in order to maintain backwards > compatibility. Even worse, if a developer add those entries by hand with > the wrong type, their applications will be misconfigured. > > By storing the child_spec directly in the repository and endpoint, > developers only need to write: > > supervise: [MyApp.Endpoint, MyApp.Repo] > > which is cleaner and less error prone. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#catching-up-with-otp>Catching > > up with OTP > > In the previous section we have mentioned that Supervisor.start_link/2 will > also accept a list of modules or a list of tuples with two elements on > Supervisor.start_link/2. In such cases, the supervisor will invoke > child_spec/1appropriately on the list elements. > > You may have also noticed we have used maps to build the child_spec/1 > functions: > > def child_spec(_) do > %{id: MySup, start: {__MODULE__, :start_link, []}, type: :supervisor}end > > The reason we chose a map with the keys above is to mirror the API > introduced in Erlang/OTP 18. Overall, Supervisor.start_link/2 expect a > list of children where each child must be: > > - an atom - such as MyApp. In such cases, MyApp.child_spec([]) will be > invoked when the supervisor starts > - a tuple with two elements - such as {MyApp, arg}. In such cases, > MyApp.child_spec(arg) will be invoked when the supervisor starts > - a map - containing the keys :id (required), :start (required), > :shutdown, :restart, :type or :modules > > Introducing maps will allow a more natural and data-centric approach for > configuring supervisors, no longer needing the helper functions defined in > Supervised.Spec today. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#a-practical-example>A > > practical example > > In order to show how the changes above will affect existing applications, > we checked how applications generated by Phoenix would leverage them. The > answer is quite positive. The whole lib/demo.ex file will no longer exist: > > defmodule Demo do > use Application > # See http://elixir-lang.org/docs/stable/elixir/Application.html # for > more information on OTP Applications > def start(_type, _args) do > import Supervisor.Spec > # Define workers and child supervisors to be supervised > children = [ # Start the Ecto repository > supervisor(Demo.Repo, []), # Start the endpoint when the > application starts > supervisor(Demo.Endpoint, []), # Start your own worker by calling: > Demo.Worker.start_link(arg1, arg2, arg3) # worker(Demo.Worker, [arg1, > arg2, arg3]), > ] > # See http://elixir-lang.org/docs/stable/elixir/Supervisor.html # for > other strategies and supported options > opts = [strategy: :one_for_one, name: Demo.Supervisor] > Supervisor.start_link(children, opts) > endend > > instead it will be replaced by: > > def application do > [extra_applications: [:logger], > supervise: [Demo.Repo, Demo.Endpoint]]end > > which Mix will translate behind the scenes to: > > def application do > [extra_applications: [:logger], > mod: {Application.Default, %{supervise: [Demo.Repo, Demo.Endpoint]}}]end > > Overall we propose the following additions to def application: > > - :supervise - expects a list of module names to supervise when the > application starts (maps as childspecs are not supported as that would > make > them compile time) > - :max_restarts - the amount of restarts supported by application > supervision tree in :max_seconds (defaults to 1) > - :max_seconds - the amount of time for :max_restarts (defaults to 5) > > Those options will be handled by the default application module > implemented in Application.Default. Mix will error building the > application file if any of the options above are given alongside :mod. > > The application tree supervisor will have strategy of :one_for_one and > such is not configurable. If you need other supervision strategies, then > all you need to do is define your own supervisor with a custom strategy and > list it under :supervise in def application. > > At the end of the day, the changes above will render both mix new --sup > and Supervisor.Spec obsolete, as we now co-locate the child_spec/1 with > the module definition, leading to better code and hot code upgrades, also > making it easier to move from non-supervised to supervised applications. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#automating-child_spec1> > Automating child_spec/1 > > We have explicitly defined the child_spec/1 function in all of the > examples above. This poses a problem: what if someone defines a Supervisor > module > and forget to mark its type as :supervisor in the specification? > > Given the Supervisor module already knows sensible defaults, we should > rely on them for building specs. Therefore, we also propose use Agent, use > Task, use Supervisor and use GenServer automatically define a > child_spec/1function > that is overridable: > > defmodule MyAgent do > use Agent, restart: :transient > # Generated on: use Agent # def child_spec(opts) do # %{id: MyAgent, > start: {__MODULE__, :start_link, [opts]}, restart: :transient} # end > > def start_link(opts) do > Agent.start_link(fn -> %{} end, opts) > end > > ...end > > The default child_spec/1 will pass the argument given to child_spec/1 > directly > to start_link/1. This means MyAgentcould be started with no name in a > supervision tree as follows: > > supervise: [MyAgent] > > and with a given name as: > > supervise: [{MyAgent, name: FooBar}] > > In other words, the changes in this proposal will push developers towards > a single start_link/1 function that will receive options as a single > argument. This mirrors the API exposed in GenServer, Supervisor and > friends, removing the common confusion with passing more than one argument > to start_link. > > Of course, any of the defaults above, as well as the rare cases developers > wants to dynamically customize child_spec/1 can be achieved by overriding > the default implementation of child_spec/1. > > <https://gist.github.com/josevalim/82fb1d20c9738bb3cc96449e67407df6#summing-up>Summing > > up > > This proposal aims to streamline the move from non-supervised applications > to supervised applications. To do so, we introduced the ability to colocate > the child_spec/1 definition with the module implementation, which leads > to more self-contained abstractions, such as Ecto.Repo and > Phoenix.Endpoint. Those changes also allowed us to overhaul the > supervision system, getting rid of Supervisor.Spec and making it more > data-centric. > > While those changes remove some of the importance given to application > callbacks today, we believe this is a positive change. There is no need to > talk about application callbacks if the majority of developers are using > application callbacks simply to start supervision trees. The application > callback will still remain import for scenarios where custom start, > config_change or stop are necessary, such as those relying on included > applications. > > Since Elixir is a stable language, all of the code available today will > continue to work, although this proposal opens up the possibility to > deprecate Supervisor.Spec and mix new --sup in a far ahead future, > preparing for an eventual removal in Elixir 2.0. > > *Thanks to Dave Thomas for initiating the proposal and the Elixir team for > further discussions and reviews. Also thanks to Chris McCord and Saša Jurić > for feedback.* > > > *José Valim* > www.plataformatec.com.br > Skype: jv.ptec > Founder and Director of R&D > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/0073924d-9f9c-4984-87ff-7229efb5c647%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
