The big question is which properties we want to exhibit. For example, do we want async tests and async modules to run at the same time? Or even async tests from different modules together? If the answer is no, then we can potentially leave parallelism on the table, because we may be waiting for one test to finish in one module while N other tests from another module could run.
If we say yes, the downside is that we could potentially use too many resources (not CPU wise but stuff like database connections), as in the worst case scenario we will run M * T processes at once (M = Modules, T = Tests). We could try to introduce coordination between M * T to adhere to a limit of max_cases, but that will likely be too complex. On Mon, Sep 4, 2023 at 2:53 PM Yiming Chen <dsdsh...@gmail.com> wrote: > Thanks for confirming and referencing the past discussion! > > 100% agree that `async: :per_test | :per_module | true | false` is the way > to go 👍 > Most people can then switch to it with a global search & replace. > > I looked into the current ExUnit implementation and came up with several > design directions. > > ## Current implementation: > > 1. register test modules > 1. ExUnit.Case calls ExUnit.Server.add_sync_module in after_compile > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/case.ex#L528-L532> > 2. ExUnit.Server.add_async_module > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/server.ex#L12> > 2. mix test > 1. Mix.Tasks.Test > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/mix/lib/mix/tasks/test.ex#L1> > match test files > 2. Mix.Compilers.Test.require_and_run/4 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/mix/lib/mix/compilers/test.ex#L25> > require and run matched test files > 3. run tests > 1. ExUnit.run/0 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit.ex#L380> > 2. ExUnit.Runner.run/2 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L8> > 3. ExUnit.Runner.async_loop/3 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L95> > 4. ExUnit.Server.take_async_modules/1 > 5. ExUnit.Runner.spawn_modules/2 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L154-L165> > 6. spawn_monitor -> ExUnit.Runner.run_module/2 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L214> > 7. ExUnit.EventManager > 1. module_started > 2. test_started > 3. test_finished > 4. module_finished > 8. ExUnit.Runner.run_module/3 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L278> > 1. run_setup_all > 2. spawn_monitor -> module.__ex_unit__(:setup_all, > test_module.tags) > 9. ExUnit.Runner.run_tests/3 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L345> > 10. ExUnit.Runner.run_test/3 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L356> > 11. ExUnit.Runner.spawn_test/3 > > <https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L374> > 1. spawn_test_monitor/4 > 2. receive_test_reply/4 > 3. exec_on_exit/3 > > ## possible design directions: > > 1. registration > 1. save async test cases separately from async modules > 2. save async_per_test modules & async_per_module modules > 2. taking > 1. return async test cases from > ExUnit.Server.take_async_per_test_cases/1 > 2. return async_per_test modules from > ExUnit.Server.take_async_modules(:per_test, count) > return async_per_module modules from > ExUnit.Server.take_async_modules(:per_module, count) > 3. return both modules from ExUnit.Server.take_async_modules/1 > 3. running > 1. spawn_tests/3 before spawn_modules/2 > 2. spawn_tests/3 inside spawn_modules/2 if it's a per_test async > module > > I'd prefer 1.2 + 2.2 + 3.1: > > - save async_per_test modules & async_per_module modules > - return async_per_test modules from > ExUnit.Server.take_async_modules(:per_test, count) > - spawn_tests/3 before spawn_modules/2 > > What do you think? > > Best, > Yiming > > On Sunday, September 3, 2023 at 10:48:11 PM UTC+8 José Valim wrote: > >> Yes, it is desirable and it has come up in the past: >> https://github.com/elixir-lang/elixir/pull/11949#issuecomment-1177262901 >> >> Although I think async: :per_module is what most people want, since the >> tests in the same module tend to access the same resource, opting-in for it >> to be per test will be welcome tho. >> >> On Thu, Aug 31, 2023 at 4:57 PM Yiming Chen <dsds...@gmail.com> wrote: >> >>> Currently, ExUnit's `async: true` option would run test cases in this >>> module synchronously, >>> but only run this module asynchronously along with other `async: true` >>> modules. >>> >>> This is to propose we add an option for ExUnit to run asynchronously by >>> test cases. >>> >>> # Background >>> 1. Async by module was a surprise >>> >>> My initial understanding of `async: true` is async by cases, instead of >>> modules. It's a bit surprising the behavior is later. >>> >>> 2. Async by module would behave more like synchronous tests as a module >>> gets more test cases >>> >>> As we grow our libraries/apps, a test module will have more and more >>> test cases. >>> It's tedious to break them into separate modules to speed up the test >>> suite run. >>> And breaking them into modules has the cost of making related tests >>> further from each other. >>> >>> # Benefits >>> 1. speed up test suite runs for libraries, apps almost effortlessly >>> 2. more accurate `async: xxx seconds, sync: yyy seconds` metrics >>> >>> # Caveats >>> >>> 1. Async by test cases may not run faster than async by modules: >>> - managing these test cases has a cost on its own >>> - communicating these test cases between ExUnit Server and Runner >>> has costs as well >>> 2. backward compatibility with current `async: true` behavior >>> >>> some libs or apps may rely on the async by module behavior. >>> we should still allow user to use `async: ture` by default, >>> and make async by test cases an easily opt-in feature. >>> >>> 3. Async by test cases may complex the ExUnit implementation even further >>> >>> # Potential solution >>> >>> I looked into current ExUnit implementation a little bit >>> I think `async by test cases` is doable, but I don't have a concrete >>> solution yet >>> >>> A initial idea is to: >>> 1. instead of saving modules in ExUnit Server, we save test cases (mfa) >>> in ExUnit Server >>> 2. when Runner asks for more async tests, ExUnit Server returns test >>> cases (and also modules) for Runner >>> >>> This seems to be a huge change, >>> so I'd like to know if this feature is desirable/feasible from the core >>> team's PoV before I dig more into it. >>> >>> Best, >>> Yiming >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elixir-lang-core" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elixir-lang-co...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elixir-lang-core/51ad9575-71b9-4afe-8996-1dd9e2aea7b8n%40googlegroups.com >>> <https://groups.google.com/d/msgid/elixir-lang-core/51ad9575-71b9-4afe-8996-1dd9e2aea7b8n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-core" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elixir-lang-core+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-core/3ca6d5aa-a073-4144-98c2-0a2d5bcc86f5n%40googlegroups.com > <https://groups.google.com/d/msgid/elixir-lang-core/3ca6d5aa-a073-4144-98c2-0a2d5bcc86f5n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "elixir-lang-core" group. To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-core+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4JKfYkJst%3D%3Dwr2awBdX5Ax-HBD-L9%2BCvkuHvi7VY%3DS%2BQg%40mail.gmail.com.
