Re: Transitive spec generation
In case anyone was wondering, it looks like this is probably a known issue:
https://dev.clojure.org/jira/browse/CLJ-2079
On Wednesday, February 28, 2018 at 12:05:31 AM UTC-6, James Gatannah wrote:
>
>
>
> On Tuesday, February 27, 2018 at 12:45:26 AM UTC-6, Didier wrote:
>>
>> Don't fully understand what you are doing,
>>
>
> (Slightly) bigger picture, I have a pair of functions that
> serialize/deserialize maps where the values are mostly fixed-length byte
> arrays.
>
> I want to verify that they round-trip correctly.
>
> This specific example is about two of those fields which really have the
> same spec. I'm trying to define this in a way that's similar to a C typedef
> (or possibly just something like using OOP inheritance for the sake of
> documentation):
>
> (s/def ::extension ...)
> (s/def ::server-extension ::extension)
> (s/def ::client-extension ::extension)
>
> I'm doing this in an attempt to trying to reduce my nesting levels.
>
> This is what the weird subject is about: I'm trying to take advantage of
> the transitive property of specs.
>
>
>
>> but when you run test.check, it seems to not always be generating random
>> sample, it'll grow the samples. When you run sample, it'll always start
>> back from the beginning.
>>
>
>> You can see that by running:
>>
>> (sgen/sample (s/gen ::test) 100)
>>
>> See how the generated samples are getting longer and longer?
>>
>
> Now instead try running this 10 times:
> (sgen/sample (s/gen ::test) 10)
>
> And notice how you always get the same short samples?
>
>
>
> (I'm sorry about the quote mangling).
>
> Since I have to have a specific length, I'm specifying that as a parameter
> to sgen/vector.
>
> This part of it works. I can run s/sample on my generator and validate its
> output against the spec without any trouble.
>
> Actually, if I define the ::extension spec using with-gen, it also looks
> as though it works (I just ran across this, so haven't tested it
> thoroughly). But then my library has a runtime dependency on test.check.
> That's not the end of the world, but it's a definite downside.
>
> I haven't been able to find real documentation covering the arguments to
> s/gen. But it looks as though I can pass in a map of 0-arity functions to
> override the different generators. The keys to that map override specs
> being generated. I can't remember now where I found this, but it can't be
> totally undocumented. I haven't studied the source code thoroughly enough
> to have picked out that sort of detail.
>
> So:
>
> (s/gen (s/keys :req [::server-extension
> ::client-extension])
>{::server-extension #(gen/fmap byte-array (gen/vector (gen/choose
> -128 127) extension-length))
> ::client-extension #(gen/fmap byte-array (gen/vector (gen/choose
> -128 127) extension-length))})
>
>
> calls the function in the value associated with each key to override the
> generator for that spec.
>
> This is the part that seems to only work sometimes.
>
> When I extend this to include all the keys in the full spec, these are the
> only two that have problems. They're also the only ones that are directly
> defined as another spec definition. (Which is why I'm pointing my finger at
> my attempt to use the "transitive" property, though that could totally be a
> red herring).
>
> When I run this inside deftest, it seems fine. Really, that's all I need.
> But I'd really like to understand the seemingly non-deterministic failures.
>
> I wrote a function called manual-check that takes that generator and calls
> sgen/sample on it.
>
> When I call that function directly from the REPL, it very nearly always
> fails.
>
> When I call it 40 times in a reduce inside deftest, I haven't seen it fail
> yet.
>
> * While I was writing this, I stumbled across another piece that seems
> like an interesting clue:
>
> If I override the generator for ::extension rather than the individual
> specs that I've defined transitively, it seems to start working.
>
> i.e.
>
> (s/gen (s/keys :req [::server-extension
> ::client-extension])
>{::extension #(gen/fmap byte-array (gen/vector (gen/choose -128
> 127) extension-length))})
>
> seems to work.
>
> It's still less than ideal, in case I ever want to change the underlying
> spec that's the "base" definition, since I'll have to remember to change
> the base generator. But I really shouldn't have this sort of thing
> scattered around everywhere. Especially since it's most interesting inside
> deftest anyway, where both approaches always seem to work (I haven't tested
> this out thoroughly yet either).
>
> My current hypothesis is that there's something about the deftest macro
> that interacts with test.check in a way that makes gen more capable in
> terms of resolving which spec I actually mean/want to override. I know
> (based on digging through internals) that it calls something (I think it's
> named specize) that resolves the spec's
Re: Transitive spec generation
On Tuesday, February 27, 2018 at 12:45:26 AM UTC-6, Didier wrote:
>
> Don't fully understand what you are doing,
>
(Slightly) bigger picture, I have a pair of functions that
serialize/deserialize maps where the values are mostly fixed-length byte
arrays.
I want to verify that they round-trip correctly.
This specific example is about two of those fields which really have the
same spec. I'm trying to define this in a way that's similar to a C typedef
(or possibly just something like using OOP inheritance for the sake of
documentation):
(s/def ::extension ...)
(s/def ::server-extension ::extension)
(s/def ::client-extension ::extension)
I'm doing this in an attempt to trying to reduce my nesting levels.
This is what the weird subject is about: I'm trying to take advantage of
the transitive property of specs.
> but when you run test.check, it seems to not always be generating random
> sample, it'll grow the samples. When you run sample, it'll always start
> back from the beginning.
>
> You can see that by running:
>
> (sgen/sample (s/gen ::test) 100)
>
> See how the generated samples are getting longer and longer?
>
Now instead try running this 10 times:
(sgen/sample (s/gen ::test) 10)
And notice how you always get the same short samples?
(I'm sorry about the quote mangling).
Since I have to have a specific length, I'm specifying that as a parameter
to sgen/vector.
This part of it works. I can run s/sample on my generator and validate its
output against the spec without any trouble.
Actually, if I define the ::extension spec using with-gen, it also looks as
though it works (I just ran across this, so haven't tested it thoroughly).
But then my library has a runtime dependency on test.check. That's not the
end of the world, but it's a definite downside.
I haven't been able to find real documentation covering the arguments to
s/gen. But it looks as though I can pass in a map of 0-arity functions to
override the different generators. The keys to that map override specs
being generated. I can't remember now where I found this, but it can't be
totally undocumented. I haven't studied the source code thoroughly enough
to have picked out that sort of detail.
So:
(s/gen (s/keys :req [::server-extension
::client-extension])
{::server-extension #(gen/fmap byte-array (gen/vector (gen/choose
-128 127) extension-length))
::client-extension #(gen/fmap byte-array (gen/vector (gen/choose
-128 127) extension-length))})
calls the function in the value associated with each key to override the
generator for that spec.
This is the part that seems to only work sometimes.
When I extend this to include all the keys in the full spec, these are the
only two that have problems. They're also the only ones that are directly
defined as another spec definition. (Which is why I'm pointing my finger at
my attempt to use the "transitive" property, though that could totally be a
red herring).
When I run this inside deftest, it seems fine. Really, that's all I need.
But I'd really like to understand the seemingly non-deterministic failures.
I wrote a function called manual-check that takes that generator and calls
sgen/sample on it.
When I call that function directly from the REPL, it very nearly always
fails.
When I call it 40 times in a reduce inside deftest, I haven't seen it fail
yet.
* While I was writing this, I stumbled across another piece that seems like
an interesting clue:
If I override the generator for ::extension rather than the individual
specs that I've defined transitively, it seems to start working.
i.e.
(s/gen (s/keys :req [::server-extension
::client-extension])
{::extension #(gen/fmap byte-array (gen/vector (gen/choose -128 127)
extension-length))})
seems to work.
It's still less than ideal, in case I ever want to change the underlying
spec that's the "base" definition, since I'll have to remember to change
the base generator. But I really shouldn't have this sort of thing
scattered around everywhere. Especially since it's most interesting inside
deftest anyway, where both approaches always seem to work (I haven't tested
this out thoroughly yet either).
My current hypothesis is that there's something about the deftest macro
that interacts with test.check in a way that makes gen more capable in
terms of resolving which spec I actually mean/want to override. I know
(based on digging through internals) that it calls something (I think it's
named specize) that resolves the spec's name. Then it has to use that
name/those names in the overriding map parameter to pick out the
appropriate function to call to get the generator.
I'm very skeptical, but could that possibly hold water?
Thank you to anyone who actually took the time to read this!
>
>
> On Monday, 26 February 2018 21:39:09 UTC-8, James Gatannah wrote:
>>
>> Fairly minimalist example available at
>>
Re: Transitive spec generation
Don't fully understand what you are doing, but when you run test.check, it
seems to not always be generating random sample, it'll grow the samples.
When you run sample, it'll always start back from the beginning.
You can see that by running:
(sgen/sample (s/gen ::test) 100)
See how the generated samples are getting longer and longer?
Now instead try running this 10 times:
(sgen/sample (s/gen ::test) 10)
And notice how you always get the same short samples?
On Monday, 26 February 2018 21:39:09 UTC-8, James Gatannah wrote:
>
> Fairly minimalist example available at
> https://gist.github.com/jimrthy/21851c52a8cd6b04a31ed08b1d0a7f04
>
> When I call gen/sample from inside a unit test, it seems to pass with
> flying colors.
>
> When I directly eval the gen/sample form or call (manual-check) from the
> REPL (I checked both CIDER and the boot CLI, just in case), I usually get
> the "Couldn't satisfy such-that predicate after 100 tries."
>
> To be a little more specific about this:
>
> Calling (manual-check) failed 49/50 times.
>
> Calling (transitive-indirect) passed 50 times in a row. If you haven't
> bothered looking at the gist, each of those calls (manual-check) 40 times.
>
> I was quite surprised by this. Does anyone have any suggestions about why
> wrapping the call in a unit test might help it succeed more often?
>
> If I just call gen/sample on the generator I'm trying to use, followed by
> s/valid? for one of the specs I'm trying to generate, it seems fine. I
> haven't dug into the source here (and I'm not positive what all's going on
> inside the spec.gen namespace), but I thought that's what gen/generate does
> when you define a custom generator for your spec.
>
> Except that isn't really what I'm doing.
>
> I'm trying to avoid adding extra runtime dependencies on a library like
> tools.check, so I'm trying to do this with overrides in the test namespaces
> to try to limit the extra dependencies to test time.
>
> Could that be where I'm breaking core assumptions that don't seem to cause
> trouble for anyone else?
>
> Thanks,
> James
>
>
>
> On Sunday, February 25, 2018 at 6:45:39 PM UTC-6, James Gatannah wrote:
>>
>> I have a spec for an array of 16 bytes:
>>
>> (s/def ::extension (s/and bytes?
>>#(= (count %) 16))
>>
>> Then I have a couple of other specs that are really just renaming it:
>>
>> (s/def ::client-extension ::extension)
>> (s/def ::server-extension ::extension)
>>
>> I started doing some refactoring today, and the definitions wound up
>> needing to move to a different namespace.
>>
>> So now the original definitions have changed to
>>
>> (s/def ::client-extension ::refactored/client-extension)
>>
>> I also started dabbling with generators, and came up with this:
>>
>> (gen/generate (s/gen ::client-extension
>> {::client-extension #(gen/fmap
>> byte-array (gen/vector (gen/choose -128 127) 16)}))
>>
>> When I define things this way, I get a "Couldn't satisfy such-that
>> predicate after 100 tries." exception a little more than half the time.
>>
>> If I rearrange things so that either
>> a) The refactored namespace defines the spec directly
>> or
>> b) I change my generator override to specify the top-level spec that the
>> others are copying
>>
>> i.e.
>> a) would mean changing the refactored ns such that I have
>> (s/def ::client-extension (s/and bytes?
>> #(= (count %) 16))
>>
>> b) changing the generator to
>> (gen/generate (s/gen ::client-extension
>> {::refactored/extension #(gen/fmap
>> byte-array (gen/vector (gen/choose -128 127) 16)}))
>>
>> it seems to fail (with the same problem) about 1 time in 5.
>>
>> I haven't seen it fail yet if I undo my refactoring and move the spec
>> back to the original location.
>>
>> I haven't collected any sorts of real numbers on this, much less tried to
>> make enough test runs to collect a statistically significant sample. I know
>> the next real steps are to put together a minimalist example.
>>
>> But before I do that, I figured it might be asking whether anyone sees
>> anything obviously wrong in what I'm trying to do, or whether there's a
>> better way to do it.
>>
>> Thanks in advance,
>> James
>>
>>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit
Re: Transitive spec generation
Fairly minimalist example available
at https://gist.github.com/jimrthy/21851c52a8cd6b04a31ed08b1d0a7f04
When I call gen/sample from inside a unit test, it seems to pass with
flying colors.
When I directly eval the gen/sample form or call (manual-check) from the
REPL (I checked both CIDER and the boot CLI, just in case), I usually get
the "Couldn't satisfy such-that predicate after 100 tries."
To be a little more specific about this:
Calling (manual-check) failed 49/50 times.
Calling (transitive-indirect) passed 50 times in a row. If you haven't
bothered looking at the gist, each of those calls (manual-check) 40 times.
I was quite surprised by this. Does anyone have any suggestions about why
wrapping the call in a unit test might help it succeed more often?
If I just call gen/sample on the generator I'm trying to use, followed by
s/valid? for one of the specs I'm trying to generate, it seems fine. I
haven't dug into the source here (and I'm not positive what all's going on
inside the spec.gen namespace), but I thought that's what gen/generate does
when you define a custom generator for your spec.
Except that isn't really what I'm doing.
I'm trying to avoid adding extra runtime dependencies on a library like
tools.check, so I'm trying to do this with overrides in the test namespaces
to try to limit the extra dependencies to test time.
Could that be where I'm breaking core assumptions that don't seem to cause
trouble for anyone else?
Thanks,
James
On Sunday, February 25, 2018 at 6:45:39 PM UTC-6, James Gatannah wrote:
>
> I have a spec for an array of 16 bytes:
>
> (s/def ::extension (s/and bytes?
>#(= (count %) 16))
>
> Then I have a couple of other specs that are really just renaming it:
>
> (s/def ::client-extension ::extension)
> (s/def ::server-extension ::extension)
>
> I started doing some refactoring today, and the definitions wound up
> needing to move to a different namespace.
>
> So now the original definitions have changed to
>
> (s/def ::client-extension ::refactored/client-extension)
>
> I also started dabbling with generators, and came up with this:
>
> (gen/generate (s/gen ::client-extension
> {::client-extension #(gen/fmap
> byte-array (gen/vector (gen/choose -128 127) 16)}))
>
> When I define things this way, I get a "Couldn't satisfy such-that
> predicate after 100 tries." exception a little more than half the time.
>
> If I rearrange things so that either
> a) The refactored namespace defines the spec directly
> or
> b) I change my generator override to specify the top-level spec that the
> others are copying
>
> i.e.
> a) would mean changing the refactored ns such that I have
> (s/def ::client-extension (s/and bytes?
> #(= (count %) 16))
>
> b) changing the generator to
> (gen/generate (s/gen ::client-extension
> {::refactored/extension #(gen/fmap
> byte-array (gen/vector (gen/choose -128 127) 16)}))
>
> it seems to fail (with the same problem) about 1 time in 5.
>
> I haven't seen it fail yet if I undo my refactoring and move the spec back
> to the original location.
>
> I haven't collected any sorts of real numbers on this, much less tried to
> make enough test runs to collect a statistically significant sample. I know
> the next real steps are to put together a minimalist example.
>
> But before I do that, I figured it might be asking whether anyone sees
> anything obviously wrong in what I'm trying to do, or whether there's a
> better way to do it.
>
> Thanks in advance,
> James
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Transitive spec generation
I have a spec for an array of 16 bytes:
(s/def ::extension (s/and bytes?
#(= (count %) 16))
Then I have a couple of other specs that are really just renaming it:
(s/def ::client-extension ::extension)
(s/def ::server-extension ::extension)
I started doing some refactoring today, and the definitions wound up
needing to move to a different namespace.
So now the original definitions have changed to
(s/def ::client-extension ::refactored/client-extension)
I also started dabbling with generators, and came up with this:
(gen/generate (s/gen ::client-extension
{::client-extension #(gen/fmap byte-array
(gen/vector (gen/choose -128 127) 16)}))
When I define things this way, I get a "Couldn't satisfy such-that
predicate after 100 tries." exception a little more than half the time.
If I rearrange things so that either
a) The refactored namespace defines the spec directly
or
b) I change my generator override to specify the top-level spec that the
others are copying
i.e.
a) would mean changing the refactored ns such that I have
(s/def ::client-extension (s/and bytes?
#(= (count %) 16))
b) changing the generator to
(gen/generate (s/gen ::client-extension
{::refactored/extension #(gen/fmap
byte-array (gen/vector (gen/choose -128 127) 16)}))
it seems to fail (with the same problem) about 1 time in 5.
I haven't seen it fail yet if I undo my refactoring and move the spec back
to the original location.
I haven't collected any sorts of real numbers on this, much less tried to
make enough test runs to collect a statistically significant sample. I know
the next real steps are to put together a minimalist example.
But before I do that, I figured it might be asking whether anyone sees
anything obviously wrong in what I'm trying to do, or whether there's a
better way to do it.
Thanks in advance,
James
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
