>> but I might end up needing users with e.g. genuine Norwegian addresses.
I think that's an interesting point, and it's a problem I've encountered several times myself. However I think it can be solved several ways. 1) If your validation function "norwegian-addr?" is a simple predicate, then the answer is actually pretty trivial, you can create a generator for that predicate. There are some test.check generators for arbitrary regexes, and those can be composed. 2) If your validation function hits a database of know addresses, then you have another possibility: generate your test data from the accepted data set. I've taken this approach on several projects. I have a database of known Product IDs, and I query that when creating my test.check generators. 3) If your validation function hits a database you can also generate that database from test.check. A "is-valid-addr?" function that takes a database should be generic enough to query for any address that exists in a DB. So generate your DB with data from test.check, then generate the inputs to your functions, then let it run. #3 here is especially powerful. When we talk about creating generators for "domain objects", it should be recognized that the domain itself could be generated by test.check. On Sun, Oct 23, 2016 at 3:38 AM, Erik Assum <[email protected]> wrote: > Colin, > > I think the points you bring up here are very interesting, and reflect > ideas that I have not yet been able to formulate. > > I certainly would think there is enough "stuff" in here for a talk, and > also some ideas into how to grnerative test "bread and butter" code, as can > I, and if so, how, generative-test my REST-api. Can I do it in some more > meaningful way than just checking for a non 5xx status code? How do I > property base test my functions which take my domain "objects". Do I have > to write generators for them, and am I then ending up in the same pain as I > do writing mocks in Java? I know that I can compose generators, but I might > end up needing users with e.g. genuine Norwegian addresses. > > These are my questions wrt generative testing in my world. > > Erik. > -- > i farta > > Den 22. okt. 2016 kl. 23.48 skrev Colin Fleming < > [email protected]>: > > Yes, that is a major selling point. Generative testing is really cool, and > you should definitely not be disheartened - it's a tool like any other, > with its strong points and weak points (like static typing, too). It's > definitely not universally applicable, and even more than standard testing, > it requires you to write your code keeping in mind how you will test it. It > happens that in my case I believe that static typing is a better fit, but > that doesn't mean that it won't work well for your use case. You should > definitely be excited to try it and see - if it works well in your case, > it's pretty magical. > > The limitation in my case is that generative testing works best when > testing pure functions. Because so much of my system involves calls out to > a huge mutable blob of Java, it doesn't work well for me. > > i actually don't have a lot of experience with generative testing, but > here are two examples from the little experience I have, one which worked > well and one which didn't. > > The one that worked well was testing a binary object serialiser. This was > in Java several years ago, so a lot of the complexity in my code was > writing generators - spec (via test.check) really helps with this and it > would be relatively trivial to achieve nowadays what took me a long time > back then. But once I had a generator which generated random objects, I > then just ran a bunch of tests which generated an object, serialised it, > deserialised the resulting bytes, and checked that the deserialised object > was equal to the original one. It was really useful, and caught a whole lot > of edge case bugs I would never have caught otherwise. Things to note are > that the serialisation and deserialisation are pure functions, and the > success condition is very easy to define. > > One that I have not yet found a good way to test generatively is the > paredit code in Cursive, which occasionally suffers from edge case bugs. > I'd also like to test code refactorings. However here, the generation is > much much harder (I'd have to generate valid code in some way) and then I'd > have to randomly refactor it and somehow check that the refactored version > is equivalent. Here both the generation and the success condition are > extremely hard to define, and the risk is that in your success condition > you really just end up reimplementing your original algorithm and then what > you're testing is actually that those two algorithms are the same, not > necessarily that they do what you expect. I was lucky enough to talk at > length to Thomas Arts at Curry On last year and I asked him about this. He > had done something similar for Erlang code, and had a very complicated code > generator. They generated code that produced some output, then randomly > refactored it, compiled both versions, ran them both and checked that the > output was the same. His conclusion was that in that case, the tests > weren't worth it - they took forever to run and were very brittle. > > On 22 October 2016 at 02:54, Daniel <[email protected]> wrote: > >> > In this sort of situation, a static type system which provides >> universal guarantees (this value can never be null) is more useful than a >> contract system (no null values have been seen yet for the test inputs >> you've tried). There's simply no way I can test all combinations, or >> reproduce all combinations that users might have running. >> >> Isn't a major selling point of generative testing was that it creates >> loads of unique cases you can't invent on your own? >> >> You don't trust it to do that? Is that from personal experience? >> Genuinely curious because I am a little excited about using it in a project >> at work but this is disheartening. >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to [email protected] >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to [email protected] > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to [email protected] > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- “One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.” (Robert Firth) -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
