I think doctests are extremely helpful and powerful since they encourage 
developers to write testable, accurate documentation for their code. But 
the lack of some of the niceties that exist in ExUnit might make it less 
likely for developers to choose doctests instead of ExUnit tests, so I've 
been thinking of ways to narrow this gap.

Currently if we are writing doctests and want to assert that the output of 
the function we're testing matches a pattern instead of testing the output 
for strict equality, we need to do one of the following two things:

```
iex> match?(%{c: _}, Map.put(%{a: :b}, :c, :d))
true

iex> %{c: _} = Map.put(%{a: :b}, :c, :d)
%{a: :b, c: :d}
```

There are a few problems I see with this at the moment. First is the 
failure messages - they're not nearly as good as the failure messages in 
ExUnit at the moment for pattern matching.

```
iex> %{f: _} = Map.put(%{a: :b}, :c, :d)
%{a: :b, c: :d}
```

  1) doctest Benchee.Configuration.init/1 (1) (Benchee.ConfigurationTest)
     test/benchee/configuration_test.exs:3
     ** (MatchError) no match of right hand side value: %{a: :b, c: :d}
     stacktrace:
       (for doctest at) lib/benchee/configuration.ex:167: (test)


And then for the other one we get an even less helpful error message:

```
iex> match?(%{f: _}, Map.put(%{a: :b}, :c, :d))
true
```

  1) doctest Benchee.Configuration.init/1 (1) (Benchee.ConfigurationTest)
     test/benchee/configuration_test.exs:3
     Doctest failed
     doctest:
       iex> match?(%{f: _}, Map.put(%{a: :b}, :c, :d))
       true
     code:  match?(%{f: _}, Map.put(%{a: :b}, :c, :d)) === true
     left:  false
     right: true
     stacktrace:
       lib/benchee/configuration.ex:167: Benchee.Configuration (module)

To get the very nice new colored diff for matches available in 1.10 in our 
failure message we can do the following:

```
iex> assert %{f: _} = Map.put(%{a: :b}, :c, :d)
%{a: :b, c: :d}
```

  1) doctest Benchee.Configuration.init/1 (1) (Benchee.ConfigurationTest)
     test/benchee/configuration_test.exs:3
     match (=) failed
     code:  assert %{f: _} = Map.put(%{a: :b}, :c, :d)
     left:  %{f: _}
     right: %{a: :b, c: :d}
     stacktrace:
       (for doctest at) lib/benchee/configuration.ex:167: (test)

But putting this assertion in a doctest takes away from value of the 
example as documentation since the example is no longer really able to be 
cut and pasted - it's now more test than documentation and doesn't really 
belong here.

Also, If you want to pattern match in the final line of a doctest, you 
still need to put the return value of the match (which is always the value 
being matched against) at the end of the test or else it's not parsed as a 
valid test, which makes the pattern match useless since you're already 
making the more strict comparison with `===`. Basically, we can't do this 
today:

```
iex> %{f: _} = Map.put(%{a: :b}, :c, :d)

iex> %{e: _} = Map.put(%{a: :b}, :e, :f)

iex> %{f: _} = Map.put(%{a: :b}, :f, :e)
```

My proposal is that we allow this to be a valid doctest, and that we use it 
to give really good failure messages. Anytime the last line of a doctest is 
a pattern match, we convert that into `assert pattern = expr` when parsing 
and running the test.

The other thing I came up with (that I don't particularly like but figured 
I'd put it out there just in case someone builds something good off of it) 
is to use the anonymous function shorthand syntax to represent the output 
of the previous line and do something like this:

```
iex> Map.put(%{a: :b}, :c, :d)
%{f: _} = &1

iex> Map.put(%{a: :b}, :e, :f)
%{e: _} = &1
```

This gets a little further away from the intended purpose of documentation 
and towards more of a test, but it would still give a better failure 
message for developers and keeps the requirement that the last line of a 
doctest includes something to test against.

If folks have any better ideas for this I'd love to hear them!

-- 
You received this message because you are subscribed to the Google Groups 
"elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elixir-lang-core+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elixir-lang-core/b93ea3e4-9ebe-4090-b14c-bae138c60f4f%40googlegroups.com.

Reply via email to