Re: Thoughts on clojure.spec

[Dean Thompson]

For others, like me, who are finding this old thread interesting reading, 
there's a newer library that nicely addresses a concern expressed here, "Do 
you find it frustrating that there's no way to turn on instrumentation of 
function outputs for manual testing?"

I am really enjoying and getting value from orchestra 
.

Dean

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Ben Brinckerhoff]

I would certainly welcome and use an `overinstrument` function if added to 
schpec (or another library).

One reason spec is great because it provides "a la carte" validation: a can 
add an appropriate amount of validation to the parts of my code that would 
most benefit from it. In my experience using spec so far, optionally 
running `fn` and `ret` specs during instrumentation would give me more 
options to pick the correct amount of validation.

For instance, since side-effecting functions are harder to unit test (and 
difficult to test generatively), I would like to spec the functions and 
then run integration tests. Running `ret`/`fn` specs would help me detect 
errors closer to their source and also confirm the specs are accurate, 
which improves the value specs as documentation. In my current project, I 
have been avoiding writing `ret` specs for side-effecting functions because 
I worry they'll get out-of-sync and mislead other programmers.

Also, having `ret`/`fn` specs checked during instrumentation would add 
value during development when I'm not yet ready to build generative tests. 
For example, during early development of a function, I might just spec a 
parameter with `map?` even though it actually requires certain keys. That 
spec far too vague for generative testing, but immediately adds value when 
I turn on instrumentation, since it catches bugs where I've passed, say, 
`nil` as the parameter. When the feature is more stable, I can invest the 
time in creating a generator and running generative tests. With the current 
implementation of `instrument`, I find that I've been tempted to over-spec 
early in development in order to get working generative tests that will 
confirm my `ret` and `fn` specs are correct.

I understand Rich and Alex have thought all this through and have provided 
a detailed rationale for the current behavior. But in my own workflow, 
being able to turn on `ret`/`fn` specs during instrumentation would be a 
big help, and I'd very much like to see this feature added to schpec. I'm 
happy to look at this when my schedule allows (not this week or next due to 
travel). Perhaps the best approach is to add an `over-instrument` function 
that has options for turning on/off `args`, `fn`, and `ret` specs (enabling 
all by default) and see how people use it in practice?

Thanks to everyone who have worked so hard on all aspects of spec!
Ben










On Tuesday, December 6, 2016 at 4:42:44 PM UTC-7, Brandon Bloom wrote:
>
> I was just very surprised to discover my :ret specs were not being checked 
> via instrument. I've read the rationale above, but I'm not yet convinced. 
> While of course I can (and will) use spec/assert from a post condition, I 
> lose the nice selective instrumentation features.
>
> I'd also like to make a counter-point to this reasoning:
>  
>
>> Running return-value instrument-style checking on whatever few 
>> hand-written tests you might have isn’t going to give you better coverage 
>> than a simple (even hardwired) generator that captures similar ranges.
>
>
> In my case, I was already testing the root-level return value, but this 
> function is deeply recursive. A small non-conforming value deep in the 
> evaluation stack was yielding non-conformance at a path deep in the root 
> return value. I was confused by why the :ret check wasn't finding it 
> earlier. By relying on the top-level check rather than inline checks 
> per-call (as per instrument or :pre/:post) I lost access to the call stack. 
> Adding the post condition pin-pointed the problem instantly.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Brandon Bloom]

I was just very surprised to discover my :ret specs were not being checked 
via instrument. I've read the rationale above, but I'm not yet convinced. 
While of course I can (and will) use spec/assert from a post condition, I 
lose the nice selective instrumentation features.

I'd also like to make a counter-point to this reasoning:
 

> Running return-value instrument-style checking on whatever few 
> hand-written tests you might have isn’t going to give you better coverage 
> than a simple (even hardwired) generator that captures similar ranges.


In my case, I was already testing the root-level return value, but this 
function is deeply recursive. A small non-conforming value deep in the 
evaluation stack was yielding non-conformance at a path deep in the root 
return value. I was confused by why the :ret check wasn't finding it 
earlier. By relying on the top-level check rather than inline checks 
per-call (as per instrument or :pre/:post) I lost access to the call stack. 
Adding the post condition pin-pointed the problem instantly.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Gary Fredericks]

Here's a library you could add that functionality to: 
https://github.com/gfredericks/schpec

Gary

On Tuesday, July 19, 2016 at 1:30:27 AM UTC-5, Beau Fabry wrote:
>
> Right, and I don't think the "this is closed we shouldn't discuss it 
> anymore" line is great when people are advocating for a piece of 
> functionality. I understand Alex doesn't want endless threads bikeshedding 
> basically arbitrary naming choices, but that's not the same as people 
> making simple points of "I think X would be a good addition because of Y" 
> with no back and forth.
>
> Maybe enough people saying "yes that sounds like a good idea because Y" in 
> this thread will convince someone else that they should create a lib that 
> mirrors the old functionality, this is the general Clojure group and not 
> clojure-dev after all.
>
> Sorry about the meta.
>
> On Tuesday, July 19, 2016 at 3:51:09 PM UTC+10, Sean Corfield wrote:
>>
>> Well, both Alex and Rich have said the change is deliberate and there are 
>> no plans to change that decision – and Rich talked about ways you can add 
>> return value testing manually based on specs (if you want, but he won’t 
>> help you) – so it seems like a “closed” topic to me? (and Alex has shut 
>> down a couple of other threads that have continued on past a clear line of 
>> decision)
>>
>>  
>>
>> I was sad to see :ret checking go away but I accept Rich’s line of 
>> thinking on this and I’ll adjust my workflow accordingly. I find Rich’s 
>> point that instrumentation is now about ensuring functions are _*called*_ 
>> correctly rather than trying to establish that they _*behave*_ correctly 
>> oddly compelling, now that I’ve had some time to think about it and play 
>> with it 
>>
>>  
>>
>> Sean Corfield -- (904) 302-SEAN
>> An Architect's View -- http://corfield.org
>>
>>  
>>
>> *From: *Beau Fabry
>> *Sent: *Monday, July 18, 2016 8:50 PM
>> *To: *Clojure
>> *Subject: *Re: Thoughts on clojure.spec
>>
>>  
>>
>> I think that was an explanation of why it's not particularly valuable in 
>> unit tests, but not really an explanation of why it wouldn't be useful in 
>> lower environments or canary boxes in distributed deployments. This thread 
>> has also touched on how not everything is gen-testable because of 
>> complexity, and I'd add that side-effects are another reason for that. We 
>> also have "you can just use assert on the return value" which is true, but 
>> seeing as I already have a database of expected return values that I've 
>> defined then it seems natural to be able to use that database to gain some 
>> extra testing value rather than define it again.
>>
>>  
>>
>> I'm not trying to argue for inclusion, if clojure core doesn't want to 
>> implement the feature then those who see value in it can trivially 
>> implement it themselves, but I haven't read anything that's made me think 
>> it wouldn't be useful.
>>
>>
>> On Tuesday, July 19, 2016 at 12:53:49 PM UTC+10, Sean Corfield wrote:
>>
>> Rich has given a pretty good explanation of why this was removed 
>> elsewhere. And in this thread, a week ago, he explained again why 
>> gen-testing :ret and :fn specs was the better approach.
>>
>>  
>>
>> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
>> An Architect's View -- http://corfield.org/
>>
>> "If you're not annoying somebody, you're not really alive."
>> -- Margaret Atwood
>>
>>  
>>
>> On 7/18/16, 7:46 PM, "Oliver George" <clo...@googlegroups.com on behalf 
>> of oli...@condense.com.au> wrote:
>>
>>  
>>
>> Here's the commit removing that aspect of instrument-all.  Not a big 
>> change.
>>
>>  
>>
>>
>> https://github.com/clojure/clojure/commit/30dd3d8554ff96f1acda7cbe31470d92df2f565a
>>
>>  
>>
>> As an aside, I also love the idea of the Clojure community fostering a 
>> culture of gen testing each chunk of well defined functionality.  If it's 
>> truly achievable the Clojure community could become known as an unstoppable 
>> force of robust code.
>>
>>  
>>
>> It would be something of a challenge for many of us... especially those 
>> wanting this particular feature!
>>
>>  
>>
>>  
>>
>> On 19 July 2016 at 10:36, Beau Fabry <imf...@gmail.com> wrote:
>>
>> > Do you find it frustrating that there's no way to turn on 
>> instrumentation of function outputs for manual

Re: Thoughts on clojure.spec

[Beau Fabry]

Right, and I don't think the "this is closed we shouldn't discuss it 
anymore" line is great when people are advocating for a piece of 
functionality. I understand Alex doesn't want endless threads bikeshedding 
basically arbitrary naming choices, but that's not the same as people 
making simple points of "I think X would be a good addition because of Y" 
with no back and forth.

Maybe enough people saying "yes that sounds like a good idea because Y" in 
this thread will convince someone else that they should create a lib that 
mirrors the old functionality, this is the general Clojure group and not 
clojure-dev after all.

Sorry about the meta.

On Tuesday, July 19, 2016 at 3:51:09 PM UTC+10, Sean Corfield wrote:
>
> Well, both Alex and Rich have said the change is deliberate and there are 
> no plans to change that decision – and Rich talked about ways you can add 
> return value testing manually based on specs (if you want, but he won’t 
> help you) – so it seems like a “closed” topic to me? (and Alex has shut 
> down a couple of other threads that have continued on past a clear line of 
> decision)
>
>  
>
> I was sad to see :ret checking go away but I accept Rich’s line of 
> thinking on this and I’ll adjust my workflow accordingly. I find Rich’s 
> point that instrumentation is now about ensuring functions are _*called*_ 
> correctly rather than trying to establish that they _*behave*_ correctly 
> oddly compelling, now that I’ve had some time to think about it and play 
> with it 
>
>  
>
> Sean Corfield -- (904) 302-SEAN
> An Architect's View -- http://corfield.org
>
>  
>
> *From: *Beau Fabry 
> *Sent: *Monday, July 18, 2016 8:50 PM
> *To: *Clojure 
> *Subject: *Re: Thoughts on clojure.spec
>
>  
>
> I think that was an explanation of why it's not particularly valuable in 
> unit tests, but not really an explanation of why it wouldn't be useful in 
> lower environments or canary boxes in distributed deployments. This thread 
> has also touched on how not everything is gen-testable because of 
> complexity, and I'd add that side-effects are another reason for that. We 
> also have "you can just use assert on the return value" which is true, but 
> seeing as I already have a database of expected return values that I've 
> defined then it seems natural to be able to use that database to gain some 
> extra testing value rather than define it again.
>
>  
>
> I'm not trying to argue for inclusion, if clojure core doesn't want to 
> implement the feature then those who see value in it can trivially 
> implement it themselves, but I haven't read anything that's made me think 
> it wouldn't be useful.
>
>
> On Tuesday, July 19, 2016 at 12:53:49 PM UTC+10, Sean Corfield wrote:
>
> Rich has given a pretty good explanation of why this was removed 
> elsewhere. And in this thread, a week ago, he explained again why 
> gen-testing :ret and :fn specs was the better approach.
>
>  
>
> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
> An Architect's View -- http://corfield.org/
>
> "If you're not annoying somebody, you're not really alive."
> -- Margaret Atwood
>
>  
>
> On 7/18/16, 7:46 PM, "Oliver George" <clo...@googlegroups.com on behalf 
> of oli...@condense.com.au> wrote:
>
>  
>
> Here's the commit removing that aspect of instrument-all.  Not a big 
> change.
>
>  
>
>
> https://github.com/clojure/clojure/commit/30dd3d8554ff96f1acda7cbe31470d92df2f565a
>
>  
>
> As an aside, I also love the idea of the Clojure community fostering a 
> culture of gen testing each chunk of well defined functionality.  If it's 
> truly achievable the Clojure community could become known as an unstoppable 
> force of robust code.
>
>  
>
> It would be something of a challenge for many of us... especially those 
> wanting this particular feature!
>
>  
>
>  
>
> On 19 July 2016 at 10:36, Beau Fabry <imf...@gmail.com> wrote:
>
> > Do you find it frustrating that there's no way to turn on 
> instrumentation of function outputs for manual testing?
>
>  
>
> Yes. I've mentioned this elsewhere but I think being able to turn on 
> output checking in lower environments (dev, test, master, staging) is 
> getting extra values from specs basically for free. Being able to do it 
> seems pragmatic. I'm hoping it won't be too difficult to write an 
> `overinstrument-all` that gives me that when I want it.
>
>
> On Tuesday, July 12, 2016 at 5:36:39 PM UTC+10, Maarten Truyens wrote:
>
> I would also truly appreciate instrumentation of function outputs for 
> manual outputs. I understand the rationale for not having it as t

RE: Thoughts on clojure.spec

[sean]

Well, both Alex and Rich have said the change is deliberate and there are no 
plans to change that decision – and Rich talked about ways you can add return 
value testing manually based on specs (if you want, but he won’t help you) – so 
it seems like a “closed” topic to me? (and Alex has shut down a couple of other 
threads that have continued on past a clear line of decision)

I was sad to see :ret checking go away but I accept Rich’s line of thinking on 
this and I’ll adjust my workflow accordingly. I find Rich’s point that 
instrumentation is now about ensuring functions are _called_ correctly rather 
than trying to establish that they _behave_ correctly oddly compelling, now 
that I’ve had some time to think about it and play with it 

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org

From: Beau Fabry

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Beau Fabry]

I think that was an explanation of why it's not particularly valuable in 
unit tests, but not really an explanation of why it wouldn't be useful in 
lower environments or canary boxes in distributed deployments. This thread 
has also touched on how not everything is gen-testable because of 
complexity, and I'd add that side-effects are another reason for that. We 
also have "you can just use assert on the return value" which is true, but 
seeing as I already have a database of expected return values that I've 
defined then it seems natural to be able to use that database to gain some 
extra testing value rather than define it again.

I'm not trying to argue for inclusion, if clojure core doesn't want to 
implement the feature then those who see value in it can trivially 
implement it themselves, but I haven't read anything that's made me think 
it wouldn't be useful.

On Tuesday, July 19, 2016 at 12:53:49 PM UTC+10, Sean Corfield wrote:
>
> Rich has given a pretty good explanation of why this was removed 
> elsewhere. And in this thread, a week ago, he explained again why 
> gen-testing :ret and :fn specs was the better approach.
>
>  
>
> Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
> An Architect's View -- http://corfield.org/
>
> "If you're not annoying somebody, you're not really alive."
> -- Margaret Atwood
>
>  
>
> On 7/18/16, 7:46 PM, "Oliver George"   on behalf of oli...@condense.com.au > wrote:
>
>  
>
> Here's the commit removing that aspect of instrument-all.  Not a big 
> change.
>
>  
>
>
> https://github.com/clojure/clojure/commit/30dd3d8554ff96f1acda7cbe31470d92df2f565a
>
>  
>
> As an aside, I also love the idea of the Clojure community fostering a 
> culture of gen testing each chunk of well defined functionality.  If it's 
> truly achievable the Clojure community could become known as an unstoppable 
> force of robust code.
>
>  
>
> It would be something of a challenge for many of us... especially those 
> wanting this particular feature!
>
>  
>
>  
>
> On 19 July 2016 at 10:36, Beau Fabry  
> wrote:
>
> > Do you find it frustrating that there's no way to turn on 
> instrumentation of function outputs for manual testing?
>
>  
>
> Yes. I've mentioned this elsewhere but I think being able to turn on 
> output checking in lower environments (dev, test, master, staging) is 
> getting extra values from specs basically for free. Being able to do it 
> seems pragmatic. I'm hoping it won't be too difficult to write an 
> `overinstrument-all` that gives me that when I want it.
>
>
> On Tuesday, July 12, 2016 at 5:36:39 PM UTC+10, Maarten Truyens wrote:
>
> I would also truly appreciate instrumentation of function outputs for 
> manual outputs. I understand the rationale for not having it as the 
> default, but could it perhaps be specified as an option s/instrument? 
> (Considering that it was present in the first alphas, I would assume that 
> such option should not be far-fetched.)
>
> -- 
>
>  
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Sean Corfield]

Rich has given a pretty good explanation of why this was removed elsewhere. And 
in this thread, a week ago, he explained again why gen-testing :ret and :fn 
specs was the better approach.

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

On 7/18/16, 7:46 PM, "Oliver George"  wrote:

 

Here's the commit removing that aspect of instrument-all.  Not a big change.

 

https://github.com/clojure/clojure/commit/30dd3d8554ff96f1acda7cbe31470d92df2f565a

 

As an aside, I also love the idea of the Clojure community fostering a culture 
of gen testing each chunk of well defined functionality.  If it's truly 
achievable the Clojure community could become known as an unstoppable force of 
robust code.

 

It would be something of a challenge for many of us... especially those wanting 
this particular feature!

 

 

On 19 July 2016 at 10:36, Beau Fabry  wrote:

> Do you find it frustrating that there's no way to turn on instrumentation of 
> function outputs for manual testing?

 

Yes. I've mentioned this elsewhere but I think being able to turn on output 
checking in lower environments (dev, test, master, staging) is getting extra 
values from specs basically for free. Being able to do it seems pragmatic. I'm 
hoping it won't be too difficult to write an `overinstrument-all` that gives me 
that when I want it.


On Tuesday, July 12, 2016 at 5:36:39 PM UTC+10, Maarten Truyens wrote:

I would also truly appreciate instrumentation of function outputs for manual 
outputs. I understand the rationale for not having it as the default, but could 
it perhaps be specified as an option s/instrument? (Considering that it was 
present in the first alphas, I would assume that such option should not be 
far-fetched.)

-- 


 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Oliver George]

Here's the commit removing that aspect of instrument-all.  Not a big change.

https://github.com/clojure/clojure/commit/30dd3d8554ff96f1acda7cbe31470d92df2f565a

As an aside, I also love the idea of the Clojure community fostering a
culture of gen testing each chunk of well defined functionality.  If it's
truly achievable the Clojure community could become known as an unstoppable
force of robust code.

It would be something of a challenge for many of us... especially those
wanting this particular feature!


On 19 July 2016 at 10:36, Beau Fabry  wrote:

> > Do you find it frustrating that there's no way to turn on
> instrumentation of function outputs for manual testing?
>
> Yes. I've mentioned this elsewhere but I think being able to turn on
> output checking in lower environments (dev, test, master, staging) is
> getting extra values from specs basically for free. Being able to do it
> seems pragmatic. I'm hoping it won't be too difficult to write an
> `overinstrument-all` that gives me that when I want it.
>
> On Tuesday, July 12, 2016 at 5:36:39 PM UTC+10, Maarten Truyens wrote:
>>
>> I would also truly appreciate instrumentation of function outputs for
>> manual outputs. I understand the rationale for not having it as the
>> default, but could it perhaps be specified as an option s/instrument?
>> (Considering that it was present in the first alphas, I would assume that
>> such option should not be far-fetched.)
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Clojure" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/clojure/jcVnjk1MOWY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Oliver George
Director, Condense
0428 740 978

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Beau Fabry]

> Do you find it frustrating that there's no way to turn on instrumentation 
of function outputs for manual testing?

Yes. I've mentioned this elsewhere but I think being able to turn on output 
checking in lower environments (dev, test, master, staging) is getting 
extra values from specs basically for free. Being able to do it seems 
pragmatic. I'm hoping it won't be too difficult to write an 
`overinstrument-all` that gives me that when I want it.

On Tuesday, July 12, 2016 at 5:36:39 PM UTC+10, Maarten Truyens wrote:
>
> I would also truly appreciate instrumentation of function outputs for 
> manual outputs. I understand the rationale for not having it as the 
> default, but could it perhaps be specified as an option s/instrument? 
> (Considering that it was present in the first alphas, I would assume that 
> such option should not be far-fetched.)
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Maarten Truyens]

I would also truly appreciate instrumentation of function outputs for 
manual outputs. I understand the rationale for not having it as the 
default, but could it perhaps be specified as an option s/instrument? 
(Considering that it was present in the first alphas, I would assume that 
such option should not be far-fetched.)

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Alan Thompson]

Do you find it frustrating that there's no way to turn on instrumentation
of function outputs for manual testing?

Yes.


On Mon, Jul 11, 2016 at 4:16 PM, Oliver George 
wrote:

>
> Do you find it frustrating that there's no way to turn on instrumentation
>> of function outputs for manual testing?
>
>
> Yes.
>
> In particular when I'm refactoring code and want to know when I'm
> returning something surprising.
>
> I haven't yet had much success with spec generators for CLJS / UI code.
> Too many non-clojure types to deal with - my gen-fu is limited.
>
> I suspect the argument for reading an instrument flag to turn on return
> value testing is just that:
>
>1. to help those who aren't able to leverage generators in their
>problem domain.
>2. informal exploration testing
>
> I see that spec/assert would give me the same results.  That's a fantastic
> addition but is also a code intrusion.
>
> Final though: I suspect you can have it for the cost of a small helper
> function.  The instrument-all code doesn't seem complex but there may be a
> few private vars between you and happiness.
>
>
> On Sunday, 10 July 2016 20:04:39 UTC+10, puzzler wrote:
>>
>> I've played around now with implementing specs for a couple of my
>> projects.
>>
>> What I'm finding is that writing specs to check the inputs of a function
>> is easy-ish, but building useful random generators is very hard -- in my
>> projects, this seems too hard to do with any reasonable amount of effort.
>>
>> This isn't due to anything inherent in clojure.spec, it's just that for
>> non-trivial functions, coming up with relevant random input is a very hard
>> problem.  For example, let's say I have a function that takes two
>> integers.  Odds are that not any two randomly chosen integers will work.
>> Some combinations of integers are non-sensical and could trigger an error,
>> other combinations may cause the function to run way too long.  As a
>> concrete example, I just tried to spec out a SAT solver (which tries to
>> solve NP-complete problems).  The input should be a vector of vectors of
>> ints, but many combinations of inputs will just run forever.  How to
>> generate "good" SAT problems?  I have no idea.
>>
>> So for the most part, I've ignored the generation aspect of specs because
>> it just feels like too much work.  But without the generators,
>> clojure.spec's utility is significantly diminished.
>>
>> 1. No way to test function output specs.  For documentation purposes, I
>> want to have an output spec on my function.  However, as far as I know,
>> after instrumentation-triggered checking of output specs was removed a
>> couple of alphas ago, the only way remaining to check against output specs
>> is to use randomly generated tests.  So if I can't make good generators, I
>> have no way to confirm that my output spec works the way I think it does.
>> My documentation could be totally out of touch with reality, and that
>> displeases me.
>>
>> 2. Limited ability for testing that functions take and receive what you
>> expect.  Whereas a static type system can prove certain properties about
>> whether functions are called with valid inputs, with spec, you only get
>> those guarantees if you pump a function with enough valid random data to
>> trigger the function calling all its callees with interesting combinations
>> of data.  But if I use the naive generators, the function will never even
>> complete with most of the randomly generated input, let alone call other
>> functions in a useful way.  And in many cases I don't see how to generate
>> something of better quality.
>>
>> So looking back at my experiments, my preliminary impression is that by
>> adding specs to my public APIs, I've gained some useful documentation, and
>> I've given users the ability to instrument functions in order to get
>> high-quality assertion-checking of the inputs.  In some cases, the error
>> messages for bad input when instrumented are also more useful than I would
>> have otherwise gotten, but in some cases they aren't.  Overall, I've
>> struggled to write generators, and without them, the value proposition
>> isn't as great.
>>
>> One other issue I've had, unrelated to generators, is that I'm struggling
>> to express higher-order type constraints, for example, this function takes
>> a vector v1 of anything and a vector v2 of anything, but the type of the
>> things in vector v1 better match the type of the things in vector v2.
>>
>> What are other people finding?  Do you find it easy/hard to write
>> generators?  (If you think it's easy, I'd love to know your tricks).  Do
>> you find it easy/hard to read specs as a form of documentation about the
>> contract of a function?  Do you find it frustrating that there's no way to
>> turn on instrumentation of function outputs for manual testing?  Do you
>> feel your generators are providing sufficient code coverage when exercising
>> callee functions?
>>
> --
> You received this 

Re: Thoughts on clojure.spec

[Oliver George]

> Do you find it frustrating that there's no way to turn on instrumentation 
> of function outputs for manual testing?


Yes.  

In particular when I'm refactoring code and want to know when I'm returning 
something surprising.  

I haven't yet had much success with spec generators for CLJS / UI code. 
 Too many non-clojure types to deal with - my gen-fu is limited.

I suspect the argument for reading an instrument flag to turn on return 
value testing is just that: 

   1. to help those who aren't able to leverage generators in their problem 
   domain.
   2. informal exploration testing

I see that spec/assert would give me the same results.  That's a fantastic 
addition but is also a code intrusion.

Final though: I suspect you can have it for the cost of a small helper 
function.  The instrument-all code doesn't seem complex but there may be a 
few private vars between you and happiness.


On Sunday, 10 July 2016 20:04:39 UTC+10, puzzler wrote:
>
> I've played around now with implementing specs for a couple of my projects.
>
> What I'm finding is that writing specs to check the inputs of a function 
> is easy-ish, but building useful random generators is very hard -- in my 
> projects, this seems too hard to do with any reasonable amount of effort.
>
> This isn't due to anything inherent in clojure.spec, it's just that for 
> non-trivial functions, coming up with relevant random input is a very hard 
> problem.  For example, let's say I have a function that takes two 
> integers.  Odds are that not any two randomly chosen integers will work.  
> Some combinations of integers are non-sensical and could trigger an error, 
> other combinations may cause the function to run way too long.  As a 
> concrete example, I just tried to spec out a SAT solver (which tries to 
> solve NP-complete problems).  The input should be a vector of vectors of 
> ints, but many combinations of inputs will just run forever.  How to 
> generate "good" SAT problems?  I have no idea.
>
> So for the most part, I've ignored the generation aspect of specs because 
> it just feels like too much work.  But without the generators, 
> clojure.spec's utility is significantly diminished.
>
> 1. No way to test function output specs.  For documentation purposes, I 
> want to have an output spec on my function.  However, as far as I know, 
> after instrumentation-triggered checking of output specs was removed a 
> couple of alphas ago, the only way remaining to check against output specs 
> is to use randomly generated tests.  So if I can't make good generators, I 
> have no way to confirm that my output spec works the way I think it does.  
> My documentation could be totally out of touch with reality, and that 
> displeases me.
>
> 2. Limited ability for testing that functions take and receive what you 
> expect.  Whereas a static type system can prove certain properties about 
> whether functions are called with valid inputs, with spec, you only get 
> those guarantees if you pump a function with enough valid random data to 
> trigger the function calling all its callees with interesting combinations 
> of data.  But if I use the naive generators, the function will never even 
> complete with most of the randomly generated input, let alone call other 
> functions in a useful way.  And in many cases I don't see how to generate 
> something of better quality.
>
> So looking back at my experiments, my preliminary impression is that by 
> adding specs to my public APIs, I've gained some useful documentation, and 
> I've given users the ability to instrument functions in order to get 
> high-quality assertion-checking of the inputs.  In some cases, the error 
> messages for bad input when instrumented are also more useful than I would 
> have otherwise gotten, but in some cases they aren't.  Overall, I've 
> struggled to write generators, and without them, the value proposition 
> isn't as great.
>
> One other issue I've had, unrelated to generators, is that I'm struggling 
> to express higher-order type constraints, for example, this function takes 
> a vector v1 of anything and a vector v2 of anything, but the type of the 
> things in vector v1 better match the type of the things in vector v2.
>
> What are other people finding?  Do you find it easy/hard to write 
> generators?  (If you think it's easy, I'd love to know your tricks).  Do 
> you find it easy/hard to read specs as a form of documentation about the 
> contract of a function?  Do you find it frustrating that there's no way to 
> turn on instrumentation of function outputs for manual testing?  Do you 
> feel your generators are providing sufficient code coverage when exercising 
> callee functions?
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, 

Re: Thoughts on clojure.spec

[Rich Hickey]

Kovas is right on the money here. Inherently when you make something 
‘programmable’ generating correct programs for it is hard.

I would say though, I frequently see people struggling to spec more complex 
logic are going directly to independent input generation. Thus, how will inputs 
be reasonably coordinated etc. The better approach is to first generate a 
consistent model, then generate inputs from that model. gen/bind is your 
friend. Typically the approach is:

(gen/bind model-generator (fn [model] data-generator))

Each model then need not be exhaustively large because many models will be 
generated, and the model lets you share things between what would otherwise be 
independent generators.

On the specific points:

> 1. No way to test function output specs.  For documentation purposes, I want 
> to have an output spec on my function.  However, as far as I know, after 
> instrumentation-triggered checking of output specs was removed a couple of 
> alphas ago, the only way remaining to check against output specs is to use 
> randomly generated tests.  So if I can't make good generators, I have no way 
> to confirm that my output spec works the way I think it does.  My 
> documentation could be totally out of touch with reality, and that displeases 
> me.

Running return-value instrument-style checking on whatever few hand-written 
tests you might have isn’t going to give you better coverage than a simple 
(even hardwired) generator that captures similar ranges. And you can directly 
exercise your :ret spec - it’s just another spec. You can also spec/assert on 
your return - the tests will disappear in production, although this is 
similarly as weak as instrument-triggered return checking, so I’m not going to 
help you do it, but you can do it yourself :)

> 2. Limited ability for testing that functions take and receive what you 
> expect.  Whereas a static type system can prove certain properties about 
> whether functions are called with valid inputs, with spec, you only get those 
> guarantees if you pump a function with enough valid random data to trigger 
> the function calling all its callees with interesting combinations of data.  
> But if I use the naive generators, the function will never even complete with 
> most of the randomly generated input, let alone call other functions in a 
> useful way.  And in many cases I don't see how to generate something of 
> better quality.

You can certainly use core.typed if you want to check the kinds of things it 
can check in the way it can check them. But saying specs aren’t types says 
little about either, and using the word ‘valid’ in both contexts without 
qualification equates things that are not the same. It’s not as if most type 
systems can check the predicates spec can (Idris et al aside). There’s a big 
difference between type-correct inputs/outputs and stakeholder-correct 
programs. spec is oriented towards the latter, but, the bar, as you have seen, 
is generation. It is, however, a local challenge - ‘this particular fn is hard 
to gen for’.  The bar for static typing is flowing types everywhere, a bar I 
consider to be more global, harder to meet, and less expressive.

> One other issue I've had, unrelated to generators, is that I'm struggling to 
> express higher-order type constraints, for example, this function takes a 
> vector v1 of anything and a vector v2 of anything, but the type of the things 
> in vector v1 better match the type of the things in vector v2.

Parameterized types definitely have advantages for the subset of things they 
can check in this regard. But again, it is a logical fallacy to equate spec == 
type then switch to static type checking ‘advantages' as if they had the same 
expressive power - in general they don’t, since specs are predicative and of 
runtime values. Most type systems can’t tell you that the things in v1 satisfy 
the same predicates as do the things in v2 for any predicate other than ‘is 
statically a T’ (Idris et al aside). That said, I have been thinking about the 
‘parameterized spec’ problem, nothing concrete to show yet.

Rich

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[kovas boguta]

On Sun, Jul 10, 2016 at 6:04 AM, Mark Engelberg 
wrote:

>
> This isn't due to anything inherent in clojure.spec, it's just that for
> non-trivial functions, coming up with relevant random input is a very hard
> problem.
>

This is a very interesting observation.

I've written generators (we'd call it enumeration) for a fair number of
computational systems when I was at Wolfram, so I have some observations
and suggestions.

This problem comes up anytime the *behavior* of the function is nontrivial.
For most human-coded functions, behavior is trivial, its only the
implementation that is nontrivial. Clojure.spec optimizes for this case.

All NP-complete problems, by definition, exhibit a kind of universality,
meaning they can be programmed. In general there is no way to know the
behavior of the inputs without an irreducible amount of computation.  In
specific cases, there are ornate mathematical theories, which can also help
with generation, but which have no unifying framework universal across all
systems.

So I don't think there is any general solution clojure.spec can offer.

On generating formulas:

I would just enumerate them systematically with increasing number of
variables (starting at 1), and see how far I can get within an allotted
runtime. Being able to time-out surely must be important also.

With most nontrivial computational systems you don't have go far to hit
nontrivial behavior. So only being able to enumerate to a modest size isn't
a deal breaker.

If you want to generate big, satisfiable formulas, i think you'll have to
make them satisfiable by construction. You can do with this with a simple
term-rewriting system. Anytime you have a satisfiable expr, you can "or-in"
failing branches. It would actually probably be pretty illuminating so see
the runtime performance under various schemas for doing this.

If the goal is to test with a diverse set of formulas, your best bet is to
generate them using using a secondary computational system (for the
elementary cellular automata) and then map its behavior into formulas.

You might find the following links interesting, both for potential methods
and for building intuition of what might one expect when enumerating
through mathematical systems:

http://www.wolframscience.com/nksonline/section-12.9
http://www.wolframscience.com/nksonline/notes-section-12.9

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Thoughts on clojure.spec

[Michael Gardner]

It might be possible to leverage something like American Fuzzy Lop[1] for 
better random input generation. I've never used AFL myself, but I know SQLite 
(one of the best-tested libraries I know of) has had good success with it[2], 
and it does work on Java.

[1] http://lcamtuf.coredump.cx/afl/
[2] https://www.sqlite.org/testing.html#aflfuzz

> On Jul 10, 2016, at 03:04, Mark Engelberg  wrote:
> 
> I've played around now with implementing specs for a couple of my projects.
> 
> What I'm finding is that writing specs to check the inputs of a function is 
> easy-ish, but building useful random generators is very hard -- in my 
> projects, this seems too hard to do with any reasonable amount of effort.
> 
> This isn't due to anything inherent in clojure.spec, it's just that for 
> non-trivial functions, coming up with relevant random input is a very hard 
> problem.  For example, let's say I have a function that takes two integers.  
> Odds are that not any two randomly chosen integers will work.  Some 
> combinations of integers are non-sensical and could trigger an error, other 
> combinations may cause the function to run way too long.  As a concrete 
> example, I just tried to spec out a SAT solver (which tries to solve 
> NP-complete problems).  The input should be a vector of vectors of ints, but 
> many combinations of inputs will just run forever.  How to generate "good" 
> SAT problems?  I have no idea.
> 
> So for the most part, I've ignored the generation aspect of specs because it 
> just feels like too much work.  But without the generators, clojure.spec's 
> utility is significantly diminished.
> 
> 1. No way to test function output specs.  For documentation purposes, I want 
> to have an output spec on my function.  However, as far as I know, after 
> instrumentation-triggered checking of output specs was removed a couple of 
> alphas ago, the only way remaining to check against output specs is to use 
> randomly generated tests.  So if I can't make good generators, I have no way 
> to confirm that my output spec works the way I think it does.  My 
> documentation could be totally out of touch with reality, and that displeases 
> me.
> 
> 2. Limited ability for testing that functions take and receive what you 
> expect.  Whereas a static type system can prove certain properties about 
> whether functions are called with valid inputs, with spec, you only get those 
> guarantees if you pump a function with enough valid random data to trigger 
> the function calling all its callees with interesting combinations of data.  
> But if I use the naive generators, the function will never even complete with 
> most of the randomly generated input, let alone call other functions in a 
> useful way.  And in many cases I don't see how to generate something of 
> better quality.
> 
> So looking back at my experiments, my preliminary impression is that by 
> adding specs to my public APIs, I've gained some useful documentation, and 
> I've given users the ability to instrument functions in order to get 
> high-quality assertion-checking of the inputs.  In some cases, the error 
> messages for bad input when instrumented are also more useful than I would 
> have otherwise gotten, but in some cases they aren't.  Overall, I've 
> struggled to write generators, and without them, the value proposition isn't 
> as great.
> 
> One other issue I've had, unrelated to generators, is that I'm struggling to 
> express higher-order type constraints, for example, this function takes a 
> vector v1 of anything and a vector v2 of anything, but the type of the things 
> in vector v1 better match the type of the things in vector v2.
> 
> What are other people finding?  Do you find it easy/hard to write generators? 
>  (If you think it's easy, I'd love to know your tricks).  Do you find it 
> easy/hard to read specs as a form of documentation about the contract of a 
> function?  Do you find it frustrating that there's no way to turn on 
> instrumentation of function outputs for manual testing?  Do you feel your 
> generators are providing sufficient code coverage when exercising callee 
> functions?
> 
> -- 
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your 
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" 

Thoughts on clojure.spec

[Mark Engelberg]

I've played around now with implementing specs for a couple of my projects.

What I'm finding is that writing specs to check the inputs of a function is
easy-ish, but building useful random generators is very hard -- in my
projects, this seems too hard to do with any reasonable amount of effort.

This isn't due to anything inherent in clojure.spec, it's just that for
non-trivial functions, coming up with relevant random input is a very hard
problem.  For example, let's say I have a function that takes two
integers.  Odds are that not any two randomly chosen integers will work.
Some combinations of integers are non-sensical and could trigger an error,
other combinations may cause the function to run way too long.  As a
concrete example, I just tried to spec out a SAT solver (which tries to
solve NP-complete problems).  The input should be a vector of vectors of
ints, but many combinations of inputs will just run forever.  How to
generate "good" SAT problems?  I have no idea.

So for the most part, I've ignored the generation aspect of specs because
it just feels like too much work.  But without the generators,
clojure.spec's utility is significantly diminished.

1. No way to test function output specs.  For documentation purposes, I
want to have an output spec on my function.  However, as far as I know,
after instrumentation-triggered checking of output specs was removed a
couple of alphas ago, the only way remaining to check against output specs
is to use randomly generated tests.  So if I can't make good generators, I
have no way to confirm that my output spec works the way I think it does.
My documentation could be totally out of touch with reality, and that
displeases me.

2. Limited ability for testing that functions take and receive what you
expect.  Whereas a static type system can prove certain properties about
whether functions are called with valid inputs, with spec, you only get
those guarantees if you pump a function with enough valid random data to
trigger the function calling all its callees with interesting combinations
of data.  But if I use the naive generators, the function will never even
complete with most of the randomly generated input, let alone call other
functions in a useful way.  And in many cases I don't see how to generate
something of better quality.

So looking back at my experiments, my preliminary impression is that by
adding specs to my public APIs, I've gained some useful documentation, and
I've given users the ability to instrument functions in order to get
high-quality assertion-checking of the inputs.  In some cases, the error
messages for bad input when instrumented are also more useful than I would
have otherwise gotten, but in some cases they aren't.  Overall, I've
struggled to write generators, and without them, the value proposition
isn't as great.

One other issue I've had, unrelated to generators, is that I'm struggling
to express higher-order type constraints, for example, this function takes
a vector v1 of anything and a vector v2 of anything, but the type of the
things in vector v1 better match the type of the things in vector v2.

What are other people finding?  Do you find it easy/hard to write
generators?  (If you think it's easy, I'd love to know your tricks).  Do
you find it easy/hard to read specs as a form of documentation about the
contract of a function?  Do you find it frustrating that there's no way to
turn on instrumentation of function outputs for manual testing?  Do you
feel your generators are providing sufficient code coverage when exercising
callee functions?

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.