Thanks for confirming and referencing the past discussion!
100% agree that `async: :per_test | :per_module | true | false` is the way
to go 👍
Most people can then switch to it with a global search & replace.
I looked into the current ExUnit implementation and came up with several
design directions.
## Current implementation:
1. register test modules
1. ExUnit.Case calls ExUnit.Server.add_sync_module in after_compile
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/case.ex#L528-L532>
2. ExUnit.Server.add_async_module
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/server.ex#L12>
2. mix test
1. Mix.Tasks.Test
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/mix/lib/mix/tasks/test.ex#L1>
match test files
2. Mix.Compilers.Test.require_and_run/4
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/mix/lib/mix/compilers/test.ex#L25>
require and run matched test files
3. run tests
1. ExUnit.run/0
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit.ex#L380>
2. ExUnit.Runner.run/2
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L8>
3. ExUnit.Runner.async_loop/3
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L95>
4. ExUnit.Server.take_async_modules/1
5. ExUnit.Runner.spawn_modules/2
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L154-L165>
6. spawn_monitor -> ExUnit.Runner.run_module/2
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L214>
7. ExUnit.EventManager
1. module_started
2. test_started
3. test_finished
4. module_finished
8. ExUnit.Runner.run_module/3
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L278>
1. run_setup_all
2. spawn_monitor -> module.__ex_unit__(:setup_all,
test_module.tags)
9. ExUnit.Runner.run_tests/3
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L345>
10. ExUnit.Runner.run_test/3
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L356>
11. ExUnit.Runner.spawn_test/3
<https://github.com/elixir-lang/elixir/blob/d34b708d53a808be8ed569b263797bdf6ca3d6e0/lib/ex_unit/lib/ex_unit/runner.ex#L374>
1. spawn_test_monitor/4
2. receive_test_reply/4
3. exec_on_exit/3
## possible design directions:
1. registration
1. save async test cases separately from async modules
2. save async_per_test modules & async_per_module modules
2. taking
1. return async test cases from ExUnit.Server.take_async_per_test_cases/1
2. return async_per_test modules from
ExUnit.Server.take_async_modules(:per_test, count)
return async_per_module modules from
ExUnit.Server.take_async_modules(:per_module, count)
3. return both modules from ExUnit.Server.take_async_modules/1
3. running
1. spawn_tests/3 before spawn_modules/2
2. spawn_tests/3 inside spawn_modules/2 if it's a per_test async
module
I'd prefer 1.2 + 2.2 + 3.1:
- save async_per_test modules & async_per_module modules
- return async_per_test modules from
ExUnit.Server.take_async_modules(:per_test, count)
- spawn_tests/3 before spawn_modules/2
What do you think?
Best,
Yiming
On Sunday, September 3, 2023 at 10:48:11 PM UTC+8 José Valim wrote:
> Yes, it is desirable and it has come up in the past:
> https://github.com/elixir-lang/elixir/pull/11949#issuecomment-1177262901
>
> Although I think async: :per_module is what most people want, since the
> tests in the same module tend to access the same resource, opting-in for it
> to be per test will be welcome tho.
>
> On Thu, Aug 31, 2023 at 4:57 PM Yiming Chen <dsds...@gmail.com> wrote:
>
>> Currently, ExUnit's `async: true` option would run test cases in this
>> module synchronously,
>> but only run this module asynchronously along with other `async: true`
>> modules.
>>
>> This is to propose we add an option for ExUnit to run asynchronously by
>> test cases.
>>
>> # Background
>> 1. Async by module was a surprise
>>
>> My initial understanding of `async: true` is async by cases, instead of
>> modules. It's a bit surprising the behavior is later.
>>
>> 2. Async by module would behave more like synchronous tests as a module
>> gets more test cases
>>
>> As we grow our libraries/apps, a test module will have more and more test
>> cases.
>> It's tedious to break them into separate modules to speed up the test
>> suite run.
>> And breaking them into modules has the cost of making related tests
>> further from each other.
>>
>> # Benefits
>> 1. speed up test suite runs for libraries, apps almost effortlessly
>> 2. more accurate `async: xxx seconds, sync: yyy seconds` metrics
>>
>> # Caveats
>>
>> 1. Async by test cases may not run faster than async by modules:
>> - managing these test cases has a cost on its own
>> - communicating these test cases between ExUnit Server and Runner has
>> costs as well
>> 2. backward compatibility with current `async: true` behavior
>>
>> some libs or apps may rely on the async by module behavior.
>> we should still allow user to use `async: ture` by default,
>> and make async by test cases an easily opt-in feature.
>>
>> 3. Async by test cases may complex the ExUnit implementation even further
>>
>> # Potential solution
>>
>> I looked into current ExUnit implementation a little bit
>> I think `async by test cases` is doable, but I don't have a concrete
>> solution yet
>>
>> A initial idea is to:
>> 1. instead of saving modules in ExUnit Server, we save test cases (mfa)
>> in ExUnit Server
>> 2. when Runner asks for more async tests, ExUnit Server returns test
>> cases (and also modules) for Runner
>>
>> This seems to be a huge change,
>> so I'd like to know if this feature is desirable/feasible from the core
>> team's PoV before I dig more into it.
>>
>> Best,
>> Yiming
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elixir-lang-core" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elixir-lang-co...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elixir-lang-core/51ad9575-71b9-4afe-8996-1dd9e2aea7b8n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/elixir-lang-core/51ad9575-71b9-4afe-8996-1dd9e2aea7b8n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elixir-lang-core+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elixir-lang-core/3ca6d5aa-a073-4144-98c2-0a2d5bcc86f5n%40googlegroups.com.
