Re: std.experimental.testing formal review
On Tuesday, 29 September 2015 at 19:15:13 UTC, Meta wrote: On Tuesday, 29 September 2015 at 14:34:42 UTC, Atila Neves wrote: runTests with no optional arguments will run the tests in threads. There's nothing about purity enforcement there. In fact, I tried using pure unit tests yesterday with std.experimental.testing and couldn't. The compiler inferred the functions to not be pure. I tried adding pure to them and descended into a madness of adding pure all over phobos until I got fed up. It seems to be something to do with `format` not being pure, I have no idea why. It really should be. There are various unit tests in Phobos that are annotated with @pure, @nogc, nothrow, etc. to test that those attributes will be inferred by the compiler. In light of that, this could pose a problem. Which is why I tested with a pure unittest. The problem is that `text` and `to` from `std.conv` aren't pure. That can and should be fixed, but unless I write my own pure versions, it's got nothing to do with this proposal. Atila
Re: std.experimental.testing formal review
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. Will `runTests` automatically assert that all pure unittests by default are parallellized and all non-pure are serialized? If so why is the @serial UDA needed? Will it make use of D's builtin threadpool or will every unittest run in its own thread? IMHO, we should strive for threadpool usage here.
Re: std.experimental.testing formal review
On Tuesday, 29 September 2015 at 10:45:23 UTC, Per Nordlöw wrote: On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. Will `runTests` automatically assert that all pure unittests by default are parallellized and all non-pure are serialized? If so why is the @serial UDA needed? runTests with no optional arguments will run the tests in threads. There's nothing about purity enforcement there. In fact, I tried using pure unit tests yesterday with std.experimental.testing and couldn't. The compiler inferred the functions to not be pure. I tried adding pure to them and descended into a madness of adding pure all over phobos until I got fed up. It seems to be something to do with `format` not being pure, I have no idea why. It really should be. @serial is needed to _not_ run tests in threads, which you'd need if those tests call functions that mutate global state. Even "thread-global" is bad here since those tests could run in the same or different threads depending on the run. Will it make use of D's builtin threadpool or will every unittest run in its own thread? IMHO, we should strive for threadpool usage here. It uses the thread pool. Atila
Re: std.experimental.testing formal review
On Tuesday, 29 September 2015 at 14:34:42 UTC, Atila Neves wrote: runTests with no optional arguments will run the tests in threads. There's nothing about purity enforcement there. In fact, I tried using pure unit tests yesterday with std.experimental.testing and couldn't. The compiler inferred the functions to not be pure. I tried adding pure to them and descended into a madness of adding pure all over phobos until I got fed up. It seems to be something to do with `format` not being pure, I have no idea why. It really should be. There are various unit tests in Phobos that are annotated with @pure, @nogc, nothrow, etc. to test that those attributes will be inferred by the compiler. In light of that, this could pose a problem.
Re: std.experimental.testing formal review
On Monday, 28 September 2015 at 10:03:14 UTC, Robert burner Schadek wrote: Review of std.experimental.testing formal review the two weeks of the formal review phase are over. The review thread was very shallow. Dicebot again expressed this disaffection with the assert function names "should(BeTrue|BeFalse|...)" No agreement could be found. Personally, I'm not sure if an agreement can be found, as this is more a personal style question rather then technical question. Some minor comments still need to be addressed: std.experimental.testing.reflection.TestData fields have no DDoc I will start the voting thread a week from now. Added the DDoc. Atila
Re: std.experimental.testing formal review
Review of std.experimental.testing formal review the two weeks of the formal review phase are over. The review thread was very shallow. Dicebot again expressed this disaffection with the assert function names "should(BeTrue|BeFalse|...)" No agreement could be found. Personally, I'm not sure if an agreement can be found, as this is more a personal style question rather then technical question. Some minor comments still need to be addressed: std.experimental.testing.reflection.TestData fields have no DDoc I will start the voting thread a week from now.
Re: std.experimental.testing formal review
On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote: 1) being able to mark test case as fatal (i.e. if internal handshake or sanity check fails there is no point in trying to run other tests) I'm leaning towards not including this now and concentrating on getting it approved - a PR to change std.experimental.testing later will be much easier to deal with than doing it now. 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs I did the check for random runs and renamed @singleThreaded to @serial in the PR. Atila
Re: std.experimental.testing formal review
Sure, it isn't really important and does not impact my opinion anyway. Was simply sharing experience of writing similar purpose library.
Re: std.experimental.testing formal review
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. PR: https://github.com/D-Programming-Language/phobos/pull/3207 Dub: http://code.dlang.org/packages/unit-threaded Doc: See CyberShadow/DAutoTest for up-to-date documentation build Previous Thread: http://forum.dlang.org/post/uzocokshugchescba...@forum.dlang.org I've updated the PR with a proposal on the functionality to auto-generate the file containing the main function to run the unit tests. Atila
Re: std.experimental.testing formal review
On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote: On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg wrote: On 2015-09-12 15:34, Dicebot wrote: I also don't like mixing unittest and higher level functional tests (with setup and cleanup phases) into the same buckets - this doesn't fit nice with D module system. Latter should be placed in a separate modules/package to avoid being picked up by rdmd & Co when compiled as dependency. Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory? On related topic - there are 2 things that are currently missing in `TestCase` when applied for functional test purpose: 1) being able to mark test case as fatal (i.e. if internal handshake or sanity check fails there is no point in trying to run other tests) I'll see if I can add this before the end of the review phase. 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs How about I change the name from @singleThreaded to @serial and make sure it works with randomised runs as well (it probably already does, I just have to check)? Atila
Re: std.experimental.testing formal review
On Sunday, 13 September 2015 at 10:44:30 UTC, Atila Neves wrote: 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs There's `@singleThreaded` for that: all tests in a module with that UDA run in series (other modules are still run in parallel). I didn't think one was needed for random ordering. Atila I had inverse thing in mind - all tests within a module / block run in parallel, but blocks run sequentially. It is not a good feature for unit tests but quite important one to higher level ones which deal with nasty environment issues.
Re: std.experimental.testing formal review
On Tuesday, 15 September 2015 at 08:27:29 UTC, Dicebot wrote:
On Sunday, 13 September 2015 at 10:44:30 UTC, Atila Neves wrote:
2) being able to do weak ordering of tests (by defining
strict sequence of groups so that
parallelization/randomization only happens within such group)
- I have used something as simple as numerical priority value
so far for my needs
There's `@singleThreaded` for that: all tests in a module with
that UDA run in series (other modules are still run in
parallel). I didn't think one was needed for random ordering.
Atila
I had inverse thing in mind - all tests within a module / block
run in parallel, but blocks run sequentially. It is not a good
feature for unit tests but quite important one to higher level
ones which deal with nasty environment issues.
I'm not sure we're understanding each other. With the current
implementation and a module like this:
@singleThreaded
@name("serial1") unittest { ... }
@singleThreaded
@name("serial2") unittest { ... }
@name("parallel1") unittest {... }
@name("parallel2") unittest { ...}
3 tasks would get scheduled in parallel: parallel1, parallel2,
and a composite task that does serial1 then serial2. If there are
any other modules, all of the other tests run in parallel with
these 3 tasks.
I'm proposing to extend the same behaviour to randomised running
of tests, but if that's the case the name would change.
Atila
Re: std.experimental.testing formal review
On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg wrote: On 2015-09-12 15:34, Dicebot wrote: I also don't like mixing unittest and higher level functional tests (with setup and cleanup phases) into the same buckets - this doesn't fit nice with D module system. Latter should be placed in a separate modules/package to avoid being picked up by rdmd & Co when compiled as dependency. Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory? On related topic - there are 2 things that are currently missing in `TestCase` when applied for functional test purpose: 1) being able to mark test case as fatal (i.e. if internal handshake or sanity check fails there is no point in trying to run other tests) 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs
Re: std.experimental.testing formal review
On Sunday, 13 September 2015 at 09:59:18 UTC, Dicebot wrote: On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg wrote: On 2015-09-12 15:34, Dicebot wrote: I also don't like mixing unittest and higher level functional tests (with setup and cleanup phases) into the same buckets - this doesn't fit nice with D module system. Latter should be placed in a separate modules/package to avoid being picked up by rdmd & Co when compiled as dependency. Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory? On related topic - there are 2 things that are currently missing in `TestCase` when applied for functional test purpose: 1) being able to mark test case as fatal (i.e. if internal handshake or sanity check fails there is no point in trying to run other tests) I've never heard of functionality like that, but should be easy to implement. 2) being able to do weak ordering of tests (by defining strict sequence of groups so that parallelization/randomization only happens within such group) - I have used something as simple as numerical priority value so far for my needs There's `@singleThreaded` for that: all tests in a module with that UDA run in series (other modules are still run in parallel). I didn't think one was needed for random ordering. Atila Atila
Re: std.experimental.testing formal review
On 09/11/2015 01:27 PM, Atila Neves wrote: >> How about Fuzz-tests, randomize input for test on each run? > > Like QuickCheck? Robert has something for that. There is also https://github.com/MartinNowak/qcheck for that.
Re: std.experimental.testing formal review
On 2015-09-13 12:44, Atila Neves wrote: I've never heard of functionality like that, but should be easy to implement. We're using that at work, but on a different level. We have two separate jobs in Jenkins, one depends on the other one. If the first one fails, the second one is not run. -- /Jacob Carlborg
Re: std.experimental.testing formal review
On Friday, 11 September 2015 at 15:54:57 UTC, Jacob Carlborg wrote: On 2015-09-11 13:27, Atila Neves wrote: Unit tests should run in a fraction of a second... no, there's no such functionality. Why would I not use this for other kinds of tests? I guess you're right. In any case, getting what's in there to get accepted is work enough as it is. Atila
Re: std.experimental.testing formal review
On 2015-09-12 11:52, Atila Neves wrote: I guess you're right. In any case, getting what's in there to get accepted is work enough as it is. Yes, absolutely. I wasn't arguing for the feature, just the reason not to add it :) -- /Jacob Carlborg
Re: std.experimental.testing formal review
On 2015-09-12 15:34, Dicebot wrote: I also don't like mixing unittest and higher level functional tests (with setup and cleanup phases) into the same buckets - this doesn't fit nice with D module system. Latter should be placed in a separate modules/package to avoid being picked up by rdmd & Co when compiled as dependency. Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory? -- /Jacob Carlborg
Re: std.experimental.testing formal review
On Saturday, 12 September 2015 at 14:50:32 UTC, Jacob Carlborg wrote: Not sure I understand the problem. Does this prevent one from writing functional tests in a completely separate directory? Nope but the fact that they are treated the same (no separate category, no output separation, no documentation warning) encourages to do so - while in practice it is almost always a bad idea.
Re: std.experimental.testing formal review
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. PR: https://github.com/D-Programming-Language/phobos/pull/3207 Dub: http://code.dlang.org/packages/unit-threaded Doc: See CyberShadow/DAutoTest for up-to-date documentation build Previous Thread: http://forum.dlang.org/post/uzocokshugchescba...@forum.dlang.org My opinion hasn't changed since last review - it is a very solid and useful library but I don't want provided API to become standard (== part of Phobos). I also don't like mixing unittest and higher level functional tests (with setup and cleanup phases) into the same buckets - this doesn't fit nice with D module system. Latter should be placed in a separate modules/package to avoid being picked up by rdmd & Co when compiled as dependency. At the same time actual test running facilities (parallel, randomized, etc) look generally applicable and uncontroversial.
Re: std.experimental.testing formal review
On Wednesday, 9 September 2015 at 18:54:30 UTC, Brian Schott wrote: On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. PR: https://github.com/D-Programming-Language/phobos/pull/3207 Dub: http://code.dlang.org/packages/unit-threaded Doc: See CyberShadow/DAutoTest for up-to-date documentation build Previous Thread: http://forum.dlang.org/post/uzocokshugchescba...@forum.dlang.org Package-level documentation seems to be missing from the auto-generated documentation. The gen_ut_main link on the side bar is also a 404. std.experimental.testing.options.Options and std.experimental.testing.reflection.TestData fields have no DDoc, so they don't show up in the generated documentation. The autotester docs are... different from `make html`. I fixed it, but just in case here's my version of dlang.org with the docs: http://atilaneves.github.io/phobos/ Is there going to be a shouldEqual that's specialized for floating point, or should shouldBeTrue(approxEqual(...)) be used instead? (If so, this should be documented) Implemented in the latest version and unit tested. Atila
Re: std.experimental.testing formal review
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. Some questions from a Java programmer: How would I go about making test fixtures. Ex. in JUnit you have @Before and @BeforeClass Is it possible to categorize tests? How about Fuzz-tests, randomize input for test on each run? Time limited test? If this tests runs more than 5min then fail
Re: std.experimental.testing formal review
On Friday, 11 September 2015 at 10:02:22 UTC, NVolcz wrote: On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. Some questions from a Java programmer: How would I go about making test fixtures. Ex. in JUnit you have @Before and @BeforeClass The py.test way: write a function that creates the fixture, call it from the unit test. Is it possible to categorize tests? D's module system does that already. How about Fuzz-tests, randomize input for test on each run? Like QuickCheck? Robert has something for that. Time limited test? If this tests runs more than 5min then fail Unit tests should run in a fraction of a second... no, there's no such functionality. Atila
Re: std.experimental.testing formal review
On Friday, 11 September 2015 at 11:27:59 UTC, Atila Neves wrote: On Friday, 11 September 2015 at 10:02:22 UTC, NVolcz wrote: Is it possible to categorize tests? D's module system does that already. Along with things like version(unittestFeatureA) or version(unittestPrecision)
Re: std.experimental.testing formal review
On 2015-09-11 13:27, Atila Neves wrote: Unit tests should run in a fraction of a second... no, there's no such functionality. Why would I not use this for other kinds of tests? -- /Jacob Carlborg
Re: std.experimental.testing formal review
On Wednesday, 9 September 2015 at 18:54:30 UTC, Brian Schott wrote: On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. PR: https://github.com/D-Programming-Language/phobos/pull/3207 Dub: http://code.dlang.org/packages/unit-threaded Doc: See CyberShadow/DAutoTest for up-to-date documentation build Previous Thread: http://forum.dlang.org/post/uzocokshugchescba...@forum.dlang.org Package-level documentation seems to be missing from the auto-generated documentation. I think I generated the docs before there were any, let me go see. The gen_ut_main link on the side bar is also a 404. Good catch, I'll take a look. I'm not looking forward to trying to recreate the site... it really should be easier. std.experimental.testing.options.Options and std.experimental.testing.reflection.TestData fields have no DDoc, so they don't show up in the generated documentation. I'll add DDoc. Is there going to be a shouldEqual that's specialized for floating point, or should shouldBeTrue(approxEqual(...)) be used instead? (If so, this should be documented) Good question. std.experimental.testing.testcase.TestCase.numTestsRun should be @property? Sure. Atila
Re: std.experimental.testing formal review
On 09-Sep-2015 18:20, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. PR: https://github.com/D-Programming-Language/phobos/pull/3207 Dub: http://code.dlang.org/packages/unit-threaded Doc: See CyberShadow/DAutoTest for up-to-date documentation build Previous Thread: http://forum.dlang.org/post/uzocokshugchescba...@forum.dlang.org Where is the announce thread? -- Dmitry Olshansky
Re: std.experimental.testing formal review
On Thursday, 10 September 2015 at 14:03:31 UTC, Dmitry Olshansky wrote: On 09-Sep-2015 18:20, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. PR: https://github.com/D-Programming-Language/phobos/pull/3207 Dub: http://code.dlang.org/packages/unit-threaded Doc: See CyberShadow/DAutoTest for up-to-date documentation build Previous Thread: http://forum.dlang.org/post/uzocokshugchescba...@forum.dlang.org Where is the announce thread? http://forum.dlang.org/post/tekjnfyvvmrozmqix...@forum.dlang.org
Re: std.experimental.testing formal review
On Wednesday, 9 September 2015 at 15:20:41 UTC, Robert burner Schadek wrote: This post marks the start of the two week review process of std.experimental.testing. PR: https://github.com/D-Programming-Language/phobos/pull/3207 Dub: http://code.dlang.org/packages/unit-threaded Doc: See CyberShadow/DAutoTest for up-to-date documentation build Previous Thread: http://forum.dlang.org/post/uzocokshugchescba...@forum.dlang.org Package-level documentation seems to be missing from the auto-generated documentation. The gen_ut_main link on the side bar is also a 404. std.experimental.testing.options.Options and std.experimental.testing.reflection.TestData fields have no DDoc, so they don't show up in the generated documentation. Is there going to be a shouldEqual that's specialized for floating point, or should shouldBeTrue(approxEqual(...)) be used instead? (If so, this should be documented) std.experimental.testing.testcase.TestCase.numTestsRun should be @property?
