Re: Use ExUnit to write unit tests.
On Thu, May 23, 2019 at 11:04 AM Joan Touzet wrote: > > On 2019-05-23 11:15, Paul Davis wrote: > > I'm pretty happy with the ExUnit we've got going for the HTTP > > interface and would be an enthusiastic +1 on starting to use it for > > internals as well. > > Wait, where's the full ExUnit implementation for the HTTP interface? Is > that Ilya's PR, or something that Cloudant runs internally? > > If you mean the slow conversion of the JS tests over to Elixir, I wasn't > aware that these were implemented in ExUnit already. Learn something new > every day! > Just the slow conversion is all I meant. There's no magical HTTP test suite hiding anywhere. :P > > The only thing I'd say is that the adapter concept while interesting > > doesn't feel like it would be that interesting for our particular > > situation. I could see it being useful for the 5984/5986 distinction > > since its the same code underneath and we'd only be munging a few > > differences for testing. However, as Garren points out 5986 is going > > to disappear one way or another so long term not a huge deal. > > +1, the intent was to deprecate 5986 for CouchDB 3.0, and obviously it's > gone for 4.0. > > -Joan >
Re: Use ExUnit to write unit tests.
On 2019-05-23 11:15, Paul Davis wrote: > I'm pretty happy with the ExUnit we've got going for the HTTP > interface and would be an enthusiastic +1 on starting to use it for > internals as well. Wait, where's the full ExUnit implementation for the HTTP interface? Is that Ilya's PR, or something that Cloudant runs internally? If you mean the slow conversion of the JS tests over to Elixir, I wasn't aware that these were implemented in ExUnit already. Learn something new every day! > The only thing I'd say is that the adapter concept while interesting > doesn't feel like it would be that interesting for our particular > situation. I could see it being useful for the 5984/5986 distinction > since its the same code underneath and we'd only be munging a few > differences for testing. However, as Garren points out 5986 is going > to disappear one way or another so long term not a huge deal. +1, the intent was to deprecate 5986 for CouchDB 3.0, and obviously it's gone for 4.0. -Joan
Re: Use ExUnit to write unit tests.
I'm pretty happy with the ExUnit we've got going for the HTTP interface and would be an enthusiastic +1 on starting to use it for internals as well. The only thing I'd say is that the adapter concept while interesting doesn't feel like it would be that interesting for our particular situation. I could see it being useful for the 5984/5986 distinction since its the same code underneath and we'd only be munging a few differences for testing. However, as Garren points out 5986 is going to disappear one way or another so long term not a huge deal. For testing HTTP vs Fabric layer I'd wager that it'll turn out to be not very useful. There's a *lot* of code in chttpd that mutates results returned from fabric functions. Attempting to do meaningful tests using logic based on adapaters seems to me like it means that you'd have to either reimplement chttpd bug-complete in the fabric adapter, or do some sort of weird inverted-chttpd in the HTTP adapter such that your tests can make the same assertions. And neither of those ideas sounds like a good idea to me. Currently my ideal approach would be to start by implementing a test suite that covers fabric thoroughly (based on at least code coverage and possibly unnamed other tooling), and then move to chttpd and do the same. That way each test suite is focused on a particular layer of concern rather than trying to formulate a single suite that tests two different layers. On Thu, May 23, 2019 at 6:21 AM Ilya Khlopotov wrote: > > Hi Joan, > > My answers inline > > On 2019/05/22 20:16:18, Joan Touzet wrote: > > Hi Ilya, thanks for starting this thread. Comments inline. > > > > On 2019-05-22 14:42, Ilya Khlopotov wrote: > > > The eunit testing framework is very hard to maintain. In particular, it > > > has the following problems: > > > - the process structure is designed in such a way that failure in setup > > > or teardown of one test affects the execution environment of subsequent > > > tests. Which makes it really hard to locate the place where the problem > > > is coming from. > > > > I've personally experienced this a lot when reviewing failed logfiles, > > trying to find the *first* failure where things go wrong. It's a huge > > problem. > > > > > - inline test in the same module as the functions it tests might be > > > skipped > > > - incorrect usage of ?assert vs ?_assert is not detectable since it makes > > > tests pass > > > - there is a weird (and hard to debug) interaction when used in > > > combination with meck > > >- https://github.com/eproxus/meck/issues/133#issuecomment-113189678 > > >- https://github.com/eproxus/meck/issues/61 > > >- meck:unload() must be used instead of meck:unload(Module) > > > > Eep! I wasn't aware of this one. That's ugly. > > > > > - teardown is not always run, which affects all subsequent tests > > > > Have first-hand experienced this one too. > > > > > - grouping of tests is tricky > > > - it is hard to group tests so individual tests have meaningful > > > descriptions > > > > > > We believe that with ExUnit we wouldn't have these problems: > > > > Who's "we"? > Wrong pronoun read it as I. > > > > > > - on_exit function is reliable in ExUnit > > > - it is easy to group tests using `describe` directive > > > - code-generation is trivial, which makes it is possible to generate > > > tests from formal spec (if/when we have one) > > > > Can you address the timeout question w.r.t. EUnit that I raised > > elsewhere for cross-platform compatibility testing? I know that > > Peng ran into the same issues I did here and was looking into extending > > timeouts. > > > > Many of our tests suffer from failures where CI resources are slow and > > simply fail due to taking longer than expected. Does ExUnit have any > > additional support here? > > > > A suggestion was made (by Jay Doane, I believe, on IRC) that perhaps we > > simply remove all timeout==failure logic (somehow?) and consider a > > timeout a hung test run, which would eventually fail the entire suite. > > This would ultimately lead to better deterministic testing, but we'd > > probably uncover quite a few bugs in the process (esp. against CouchDB > > <= 4.0). > > There is one easy workaround. We could set trace: true in the config > because one of the side effects of it is timeout = infinity (see here > https://github.com/elixir-lang/elixir/blob/master/lib/ex_unit/lib/ex_unit/runner.ex#L410). > However this approach has an important caveat: > - all tests would be run sequentially which means that we wouldn't be able to > parallelize them latter. > > > > > > > Here are a few examples: > > > > > > # Test adapters to test different interfaces using same test suite > > > > This is neat. I'd like someone else to comment whether this the approach > > you define will handle the polymorphic interfaces gracefully, or if the > > effort to parametrise/DRY out the tests will be more difficulty than > > simply maintaining 4 sets of tests. > > > > > > > # Using same
Re: Use ExUnit to write unit tests.
Hi Joan, My answers inline On 2019/05/22 20:16:18, Joan Touzet wrote: > Hi Ilya, thanks for starting this thread. Comments inline. > > On 2019-05-22 14:42, Ilya Khlopotov wrote: > > The eunit testing framework is very hard to maintain. In particular, it has > > the following problems: > > - the process structure is designed in such a way that failure in setup or > > teardown of one test affects the execution environment of subsequent tests. > > Which makes it really hard to locate the place where the problem is coming > > from. > > I've personally experienced this a lot when reviewing failed logfiles, > trying to find the *first* failure where things go wrong. It's a huge > problem. > > > - inline test in the same module as the functions it tests might be skipped > > - incorrect usage of ?assert vs ?_assert is not detectable since it makes > > tests pass > > - there is a weird (and hard to debug) interaction when used in combination > > with meck > >- https://github.com/eproxus/meck/issues/133#issuecomment-113189678 > >- https://github.com/eproxus/meck/issues/61 > >- meck:unload() must be used instead of meck:unload(Module) > > Eep! I wasn't aware of this one. That's ugly. > > > - teardown is not always run, which affects all subsequent tests > > Have first-hand experienced this one too. > > > - grouping of tests is tricky > > - it is hard to group tests so individual tests have meaningful descriptions > > > > We believe that with ExUnit we wouldn't have these problems: > > Who's "we"? Wrong pronoun read it as I. > > > - on_exit function is reliable in ExUnit > > - it is easy to group tests using `describe` directive > > - code-generation is trivial, which makes it is possible to generate tests > > from formal spec (if/when we have one) > > Can you address the timeout question w.r.t. EUnit that I raised > elsewhere for cross-platform compatibility testing? I know that > Peng ran into the same issues I did here and was looking into extending > timeouts. > > Many of our tests suffer from failures where CI resources are slow and > simply fail due to taking longer than expected. Does ExUnit have any > additional support here? > > A suggestion was made (by Jay Doane, I believe, on IRC) that perhaps we > simply remove all timeout==failure logic (somehow?) and consider a > timeout a hung test run, which would eventually fail the entire suite. > This would ultimately lead to better deterministic testing, but we'd > probably uncover quite a few bugs in the process (esp. against CouchDB > <= 4.0). There is one easy workaround. We could set trace: true in the config because one of the side effects of it is timeout = infinity (see here https://github.com/elixir-lang/elixir/blob/master/lib/ex_unit/lib/ex_unit/runner.ex#L410). However this approach has an important caveat: - all tests would be run sequentially which means that we wouldn't be able to parallelize them latter. > > > > Here are a few examples: > > > > # Test adapters to test different interfaces using same test suite > > This is neat. I'd like someone else to comment whether this the approach > you define will handle the polymorphic interfaces gracefully, or if the > effort to parametrise/DRY out the tests will be more difficulty than > simply maintaining 4 sets of tests. > > > > # Using same test suite to compare new implementation of the same interface > > with the old one > > > > Imagine that we are doing a major rewrite of a module which would implement > > the same interface. > > *tries to imagine such a 'hypothetical' rewrite* :) > > How do we compare both implementations return the same results for the same > > input? > > It is easy in Elixir, here is a sketch: > > Sounds interesting. I'd again like an analysis (from someone else) as to > how straightforward this would be to implement. > > -Joan > >
Re: Use ExUnit to write unit tests.
Hi iilya, This is really great, I've found writing tests in Elixir to be quite pleasant whereas when I use eunit I spend a lot of time shouting at my screen. How would the current elixir test be integrated in? Would we rewrite them to use your new adapter setup so we can test again the various interfaces? I took a quick look at the code and it looks really good, my own concern is maintaining the three different adapters for testing. That could be a lot of work. From what I remember we are looking at removing support for the backdoor ports (*5986), so maybe we can remove the need to maintain that one. Otherwise I'm +1 on this. Cheers Garren On Wed, May 22, 2019 at 10:16 PM Joan Touzet wrote: > Hi Ilya, thanks for starting this thread. Comments inline. > > On 2019-05-22 14:42, Ilya Khlopotov wrote: > > The eunit testing framework is very hard to maintain. In particular, it > has the following problems: > > - the process structure is designed in such a way that failure in setup > or teardown of one test affects the execution environment of subsequent > tests. Which makes it really hard to locate the place where the problem is > coming from. > > I've personally experienced this a lot when reviewing failed logfiles, > trying to find the *first* failure where things go wrong. It's a huge > problem. > > > - inline test in the same module as the functions it tests might be > skipped > > - incorrect usage of ?assert vs ?_assert is not detectable since it > makes tests pass > > - there is a weird (and hard to debug) interaction when used in > combination with meck > >- https://github.com/eproxus/meck/issues/133#issuecomment-113189678 > >- https://github.com/eproxus/meck/issues/61 > >- meck:unload() must be used instead of meck:unload(Module) > > Eep! I wasn't aware of this one. That's ugly. > > > - teardown is not always run, which affects all subsequent tests > > Have first-hand experienced this one too. > > > - grouping of tests is tricky > > - it is hard to group tests so individual tests have meaningful > descriptions > > > > We believe that with ExUnit we wouldn't have these problems: > > Who's "we"? > I think I am one of the "we", I've chatted to iilya a few times around finding a better way to write and run tests in CouchDB. > > > - on_exit function is reliable in ExUnit > > - it is easy to group tests using `describe` directive > > - code-generation is trivial, which makes it is possible to generate > tests from formal spec (if/when we have one) > > Can you address the timeout question w.r.t. EUnit that I raised > elsewhere for cross-platform compatibility testing? I know that > Peng ran into the same issues I did here and was looking into extending > timeouts. > > Many of our tests suffer from failures where CI resources are slow and > simply fail due to taking longer than expected. Does ExUnit have any > additional support here? > > A suggestion was made (by Jay Doane, I believe, on IRC) that perhaps we > simply remove all timeout==failure logic (somehow?) and consider a > timeout a hung test run, which would eventually fail the entire suite. > This would ultimately lead to better deterministic testing, but we'd > probably uncover quite a few bugs in the process (esp. against CouchDB > <= 4.0). > > > > > Here are a few examples: > > > > # Test adapters to test different interfaces using same test suite > > This is neat. I'd like someone else to comment whether this the approach > you define will handle the polymorphic interfaces gracefully, or if the > effort to parametrise/DRY out the tests will be more difficulty than > simply maintaining 4 sets of tests. > > > > # Using same test suite to compare new implementation of the same > interface with the old one > > > > Imagine that we are doing a major rewrite of a module which would > implement the same interface. > > *tries to imagine such a 'hypothetical' rewrite* :) > > > How do we compare both implementations return the same results for the > same input? > > It is easy in Elixir, here is a sketch: > > Sounds interesting. I'd again like an analysis (from someone else) as to > how straightforward this would be to implement. > > -Joan > >
