Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Tue, Mar 27, 2018 at 09:27:16AM -0400, Steven Schveighoffer via Digitalmars-d-announce wrote: > On 3/26/18 9:26 AM, Atila Neves wrote: > > On Friday, 23 March 2018 at 14:54:57 UTC, Steven Schveighoffer wrote: [...] > > > It's simple. Unittests in imported modules should not be visible. > > > They should be compiled as if -unittest was not passed. > > > > > > Even Walter and Andrei are supportive: > > > https://github.com/dlang/dmd/pull/6375#issuecomment-373487247 > > > > > > > That would completely break __traits(getUnitTests). > > I'm sure we could find a way to keep the features here, 99.99% of the > time, you don't care about, nor want to parse or semantic, an imported > module's unit tests. Only specialized unit test frameworks care about > this feature. > > It could be as simple as, if you use __traits(getUnitTests, > modulename) anywhere in a module, then that module is imported the > traditional way. Or we could create a specialized "import unittest" > syntax for this purpose. [...] Yeah, since __traits(getUnitTests) is inside the compiler, it could in theory be as simple as: - `import abc;` - compiler lexes and parses abc, but leaves unittests alone (but still present as AST nodes with no semantic run). Basically, module abc is parsed as if `-unittest` is not present. - `__traits(getUnitTests)` triggers semantic on unittest AST nodes of the target module. Does __traits(getUnitTests) work if you don't specify `-unittest`? If so, then the solution is probably already there. T -- Written on the window of a clothing store: No shirt, no shoes, no service.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On 3/26/18 9:26 AM, Atila Neves wrote: On Friday, 23 March 2018 at 14:54:57 UTC, Steven Schveighoffer wrote: On 3/22/18 6:59 AM, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila It's simple. Unittests in imported modules should not be visible. They should be compiled as if -unittest was not passed. Even Walter and Andrei are supportive: https://github.com/dlang/dmd/pull/6375#issuecomment-373487247 That would completely break __traits(getUnitTests). I'm sure we could find a way to keep the features here, 99.99% of the time, you don't care about, nor want to parse or semantic, an imported module's unit tests. Only specialized unit test frameworks care about this feature. It could be as simple as, if you use __traits(getUnitTests, modulename) anywhere in a module, then that module is imported the traditional way. Or we could create a specialized "import unittest" syntax for this purpose. I think we can have the best of both worlds, with the common case being preferred. -Steve
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Friday, 23 March 2018 at 14:54:57 UTC, Steven Schveighoffer wrote: On 3/22/18 6:59 AM, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila It's simple. Unittests in imported modules should not be visible. They should be compiled as if -unittest was not passed. Even Walter and Andrei are supportive: https://github.com/dlang/dmd/pull/6375#issuecomment-373487247 -Steve That would completely break __traits(getUnitTests). Atila
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Sun, Mar 25, 2018 at 04:22:32PM +, pineapple via Digitalmars-d-announce wrote: > I really like `unittest`. It's my personal conviction that a developer > should not be able to view the documentation, tests, or implementation > for some part of a code base in isolation. `unittest` makes it easier > for me to work this way. > > Automated tests are vital for stability and documentation is vital for > maintainability and making it possible for others to understand your > code. But when they aren't right there in the same place as the > implementation, I've noticed that it is the inevitable habit of > developers to think of tests and documentation as second-class, if > they are thought of at all. The developer updates the implementation > and doesn't think to update the tests or the docs. This happens a few > times. Then instead of updating the tests to work with the updated > implementation, the tests are discarded as an inconvenience. And then > end users become confused by docs that have since become inapplicable. Exactly. This is why inline unittests are one of the best design decisions ever made in D. Along with built-in documentation (in spite of all ddoc's warts). Before D, almost none of my code was ever thoroughly tested -- I just couldn't be bothered to install, setup, write, and then keep in-sync a separate unittesting system. Ditto for a documentation system. I tried DOxygen for one project, but it did not even get past writing a Doxyfile. :-( Fortunately, now I'm in the process of migrating that project from C++ to D, and now I have 90%+ test coverage, integrated into the build system so that it will fail loudly should any test fail. A large part of the tests are inline unittests -- otherwise they would just be too cumbersome to maintain and I would've given up long ago. (The external testsuite is there only out of necessity, due to the original C++ design; if I had the started from a blank slate, I would've hands-down preferred inline unittests.) D's unittests and inline docs (in spite of said warts) have significantly (and measurably) improved the quality of my code. And the fundamental reason is exactly as you said: proximity to the actual code make a huge difference. External docs does have the tendency of quickly falling out-of-date, and external tests tend to get ignored and eventually thrown out -- I've seen this with my own eyes in large projects with large numbers of developers, where the code simply changes way too fast for any external docs/tests to be kept up-to-date without lots of external motivation (like an angry manager holding threats over your head, :-P or an irate QA department complaining about regressions). T -- Philosophy: how to make a career out of daydreaming.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
I really like `unittest`. It's my personal conviction that a developer should not be able to view the documentation, tests, or implementation for some part of a code base in isolation. `unittest` makes it easier for me to work this way. Automated tests are vital for stability and documentation is vital for maintainability and making it possible for others to understand your code. But when they aren't right there in the same place as the implementation, I've noticed that it is the inevitable habit of developers to think of tests and documentation as second-class, if they are thought of at all. The developer updates the implementation and doesn't think to update the tests or the docs. This happens a few times. Then instead of updating the tests to work with the updated implementation, the tests are discarded as an inconvenience. And then end users become confused by docs that have since become inapplicable.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On 2018-03-23 21:43, H. S. Teoh wrote: Yep. As I mentioned elsewhere, recently I've had to resort to external testing for one of my projects, and I'm still working on that right now. And already, I'm seeing a liability: rather than quickly locating a unittest immediately following a particular function, now I have to remember "oh which subdirectory was it that the tests were put in? and which file was it that a particular test of this function was done?". Put the tests in a directory called "tests" and follow the same file and directory structure as the regular code. It's just a matter of configuring a command in an editor that will find the corresponding test file and navigate to it. Or dump your editor if it doesn't support custom commands 'cos it sux and you deserve better. :-D I mean, this is 2018, not 1995, we shouldn't have to be stuck with the handicap of clicking through files and directories to find the right test file anymore. :D -- /Jacob Carlborg
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On 3/23/18 9:54 PM, Tony wrote: I said my "original reply", meaning the one where I first mentioned Test-Driven Development. That was to something that Steven Schveighoffer said (although I did not reply directly to his message, but replied to his comment that was still in H.S. Teoh's message): "I've worked on a project where the testing was separated from the code, and it was a liability IMO. Things would get missed and not tested properly." He doesn't explicitly specify development or maintenance, but I assume it was development. It was somewhat development and somewhat maintenance, but I was not using TDD. The code was already written, I was adding to it. I was also adding tests afterwards to make sure what I did worked properly. In my case, I had no choice but to use separate files, as I was writing firmware for an embedded device, and the tests were running on the host PC (i.e. testing that when I communicated with the device, it was doing what I expected). But even in my normal work where tests and the code run on the same system, my style of development just doesn't fit TDD, I often times don't know what the code or functionality is going to look like at the end when I'm done (I rewrote what eventually became iopipe 5 times because I wasn't sure of the best way to do it). I can see where if you use TDD, it would be OK to separate the tests from the program, but I still feel that the cost of having them separated from the code it's testing outweighs the benefits of less recompilations. I'm just not as picky as Atila when it comes to build time :) -Steve
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Saturday, 24 March 2018 at 01:15:46 UTC, Jonathan M Davis wrote A number of us have nothing good to say about TDD. That's fine. That's why they have menus in restaurants. But saying it is an inferior method is different than saying it won't work or can't be used in a maintenance situation. On Saturday, 24 March 2018 at 01:15:46 UTC, Jonathan M Davis wrote: But as far as whether or not it can be done with maintenance code, my original reply that mentioned it was to someone who appeared to be talking about a new project not getting everything tested, not a maintenance project. So saying "can't do it for maintenance" doesn't even apply to my reply. You were replying to H. S. Teoh talking about adding tests to an existing project, in which case, it's very much about maintenance. I said my "original reply", meaning the one where I first mentioned Test-Driven Development. That was to something that Steven Schveighoffer said (although I did not reply directly to his message, but replied to his comment that was still in H.S. Teoh's message): "I've worked on a project where the testing was separated from the code, and it was a liability IMO. Things would get missed and not tested properly." He doesn't explicitly specify development or maintenance, but I assume it was development.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Saturday, March 24, 2018 00:51:07 Tony via Digitalmars-d-announce wrote: > On Saturday, 24 March 2018 at 00:12:23 UTC, Jonathan M Davis > > wrote: > > On Friday, March 23, 2018 22:42:34 Tony via > > > > Digitalmars-d-announce wrote: > >> On Friday, 23 March 2018 at 22:32:50 UTC, H. S. Teoh wrote: > >> > On Fri, Mar 23, 2018 at 09:45:33PM +, Tony via > >> > > >> > Digitalmars-d-announce wrote: > >> >> On Friday, 23 March 2018 at 20:43:15 UTC, H. S. Teoh wrote: > >> >> > On Friday, 23 March 2018 at 19:56:03 UTC, Steven > >> >> > > >> >> > Schveighoffer wrote: > >> >> > > I've worked on a project where the testing was separated > >> >> > > from the code, and it was a liability IMO. Things would > >> >> > > get missed and not tested properly. > >> >> > >> >> That's where Test Driven Development comes in. > >> > > >> > That's not an option when you have an existing codebase that > >> > you have to work with. You basically have to start out with > >> > tons of code and no tests, and incrementally add them. > >> > Having to also maintain a separate test tree mirroring the > >> > source tree is simply far too much overhead to be worth the > >> > effort. > >> > >> I think that you could "Test Driven Develop" the code you are > >> adding or changing. > > > > Insisting on writing the tests before writing the code doesn't > > help with the kind of situation that H. S. Teoh is describing. > > And arguably it exacerbates the problem. Regardless, it doesn't > > help when the code has already been written. > > I don't see how it exacerbates it and I don't see how it doesn't > help. The point of Test-Driven Development it to make sure you > have written a test for all your code. You can also do > test-driven development in unittest blocks. TDD makes it worse, because when you do TDD, you're constantly hopping between your code and the tests instead of just writing the code and then writing the tests. That still involves some back and forth, but it avoids having to hop back and forth while you're still designing the function, whereas with TDD, you have to go change the tests any time you change the function before the function is even done. But whatever. If you prefer TDD, then use it. A number of us have nothing good to say about TDD. You'd pretty much have to hold a gun to my head before I'd ever do anything along those lines. I'm all for writing tests, but I hate TDD. Regardless, having the tests right next to the functions reduces how much hopping around you have to do to edit the code and tests, because you have everything in one place instead of in separate files. > But as far as whether or not it can be done with maintenance > code, my original reply that mentioned it was to someone who > appeared to be talking about a new project not getting everything > tested, not a maintenance project. So saying "can't do it for > maintenance" doesn't even apply to my reply. You were replying to H. S. Teoh talking about adding tests to an existing project, in which case, it's very much about maintenance. - Jonathan M Davis
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Saturday, 24 March 2018 at 00:12:23 UTC, Jonathan M Davis wrote: On Friday, March 23, 2018 22:42:34 Tony via Digitalmars-d-announce wrote: On Friday, 23 March 2018 at 22:32:50 UTC, H. S. Teoh wrote: > On Fri, Mar 23, 2018 at 09:45:33PM +, Tony via > > Digitalmars-d-announce wrote: >> On Friday, 23 March 2018 at 20:43:15 UTC, H. S. Teoh wrote: >> > On Friday, 23 March 2018 at 19:56:03 UTC, Steven >> > >> > Schveighoffer wrote: >> > > I've worked on a project where the testing was separated >> > > from the code, and it was a liability IMO. Things would >> > > get missed and not tested properly. >> >> That's where Test Driven Development comes in. > > That's not an option when you have an existing codebase that > you have to work with. You basically have to start out with > tons of code and no tests, and incrementally add them. > Having to also maintain a separate test tree mirroring the > source tree is simply far too much overhead to be worth the > effort. I think that you could "Test Driven Develop" the code you are adding or changing. Insisting on writing the tests before writing the code doesn't help with the kind of situation that H. S. Teoh is describing. And arguably it exacerbates the problem. Regardless, it doesn't help when the code has already been written. I don't see how it exacerbates it and I don't see how it doesn't help. The point of Test-Driven Development it to make sure you have written a test for all your code. You can also do test-driven development in unittest blocks. But as far as whether or not it can be done with maintenance code, my original reply that mentioned it was to someone who appeared to be talking about a new project not getting everything tested, not a maintenance project. So saying "can't do it for maintenance" doesn't even apply to my reply.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Friday, March 23, 2018 22:42:34 Tony via Digitalmars-d-announce wrote: > On Friday, 23 March 2018 at 22:32:50 UTC, H. S. Teoh wrote: > > On Fri, Mar 23, 2018 at 09:45:33PM +, Tony via > > > > Digitalmars-d-announce wrote: > >> On Friday, 23 March 2018 at 20:43:15 UTC, H. S. Teoh wrote: > >> > On Friday, 23 March 2018 at 19:56:03 UTC, Steven > >> > > >> > Schveighoffer wrote: > >> > > I've worked on a project where the testing was separated > >> > > from the code, and it was a liability IMO. Things would > >> > > get missed and not tested properly. > >> > >> That's where Test Driven Development comes in. > > > > That's not an option when you have an existing codebase that > > you have to work with. You basically have to start out with > > tons of code and no tests, and incrementally add them. Having > > to also maintain a separate test tree mirroring the source tree > > is simply far too much overhead to be worth the effort. > > I think that you could "Test Driven Develop" the code you are > adding or changing. Insisting on writing the tests before writing the code doesn't help with the kind of situation that H. S. Teoh is describing. And arguably it exacerbates the problem. Regardless, it doesn't help when the code has already been written. - Jonathan M Davis
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Friday, 23 March 2018 at 22:32:50 UTC, H. S. Teoh wrote: On Fri, Mar 23, 2018 at 09:45:33PM +, Tony via Digitalmars-d-announce wrote: On Friday, 23 March 2018 at 20:43:15 UTC, H. S. Teoh wrote: > On Friday, 23 March 2018 at 19:56:03 UTC, Steven > Schveighoffer wrote: > > I've worked on a project where the testing was separated > > from the code, and it was a liability IMO. Things would > > get missed and not tested properly. That's where Test Driven Development comes in. That's not an option when you have an existing codebase that you have to work with. You basically have to start out with tons of code and no tests, and incrementally add them. Having to also maintain a separate test tree mirroring the source tree is simply far too much overhead to be worth the effort. I think that you could "Test Driven Develop" the code you are adding or changing.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Fri, Mar 23, 2018 at 09:45:33PM +, Tony via Digitalmars-d-announce wrote: > On Friday, 23 March 2018 at 20:43:15 UTC, H. S. Teoh wrote: > > > > > I've worked on a project where the testing was separated from the > > > code, and it was a liability IMO. Things would get missed and not > > > tested properly. > > That's where Test Driven Development comes in. That's not an option when you have an existing codebase that you have to work with. You basically have to start out with tons of code and no tests, and incrementally add them. Having to also maintain a separate test tree mirroring the source tree is simply far too much overhead to be worth the effort. [...] > > Yep. As I mentioned elsewhere, recently I've had to resort to > > external testing for one of my projects, and I'm still working on > > that right now. And already, I'm seeing a liability: rather than > > quickly locating a unittest immediately following a particular > > function, now I have to remember "oh which subdirectory was it that > > the tests were put in? and which file was it that a particular test > > of this function was done?". It's an additional mental burden to > > have to keep doing the mapping between current source location <-> > > test code location (even if it's a 1-to-1 mapping), and a physical > > burden to have to continually open external files (and typing a > > potentially long path for them) rather than just "bookmark, jump to > > end of function, navigate unittest blocks" in the same file. > > There are pluses and minuses to both approaches, but I don't think > that a separate file approach is as difficult as you are suggesting. > The naming is typically identical to the project entities being > tested, with a prefix like "Test_" tacked onto the front of the > project, modules, classes and functions, making finding things > straightforward. Well, in my case it wasn't so simple, because it's an external testsuite. Meaning the mapping of tests to function names is non-trivial, because you have to go through the interface language in order to get at the function at all. > And most modern editors/IDEs will allow multiple files and projects to > be open at the same time, allowing test code to be opened only once > per coding session. [...] Being able to open multiple files or not, isn't the problem. (An editor that couldn't do that would be worthless as a coding editor!) The problem lies in having to open multiple files in the first place, for what's essentially the same piece of functionality. When I'm working with a large project, I already have enough files open to cross-reference stuff that depends on each other. Having to open yet another file just to work with the tests is an additional, and IMO unnecessary, burden. Even if your IDE can open 50 files at the same time, you still end up having to scroll through the list of 50 open files in order to find the one you're looking for -- which is not much different from having to open a new window and opening a currently-unopened file. Whereas if the unittest is already right there *in the current file*, and better yet, *right next* to the function you're writing, you can just go to it immediately. No additional fuss of typing filenames or navigating big lists of open files (and locating, within said additional file, the right place where the test is found). Now of course, a sufficiently-powerful editor like Vim could let you define macros for automatically finding and opening the test file corresponding with your current location, and finding the unittest defined therein for you. But still, it begs the question of why bother with such encumbrances when it's really not necessary in the first place? T -- When solving a problem, take care that you do not become part of the problem.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Friday, 23 March 2018 at 20:43:15 UTC, H. S. Teoh wrote: I've worked on a project where the testing was separated from the code, and it was a liability IMO. Things would get missed and not tested properly. That's where Test Driven Development comes in. Yep. As I mentioned elsewhere, recently I've had to resort to external testing for one of my projects, and I'm still working on that right now. And already, I'm seeing a liability: rather than quickly locating a unittest immediately following a particular function, now I have to remember "oh which subdirectory was it that the tests were put in? and which file was it that a particular test of this function was done?". It's an additional mental burden to have to keep doing the mapping between current source location <-> test code location (even if it's a 1-to-1 mapping), and a physical burden to have to continually open external files (and typing a potentially long path for them) rather than just "bookmark, jump to end of function, navigate unittest blocks" in the same file. There are pluses and minuses to both approaches, but I don't think that a separate file approach is as difficult as you are suggesting. The naming is typically identical to the project entities being tested, with a prefix like "Test_" tacked onto the front of the project, modules, classes and functions, making finding things straightforward. And most modern editors/IDEs will allow multiple files and projects to be open at the same time, allowing test code to be opened only once per coding session.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Friday, March 23, 2018 13:43:15 H. S. Teoh via Digitalmars-d-announce wrote: > Yep. As I mentioned elsewhere, recently I've had to resort to external > testing for one of my projects, and I'm still working on that right now. > And already, I'm seeing a liability: rather than quickly locating a > unittest immediately following a particular function, now I have to > remember "oh which subdirectory was it that the tests were put in? and > which file was it that a particular test of this function was done?". > It's an additional mental burden to have to keep doing the mapping > between current source location <-> test code location (even if it's a > 1-to-1 mapping), and a physical burden to have to continually open > external files (and typing a potentially long path for them) rather than > just "bookmark, jump to end of function, navigate unittest blocks" in > the same file. When I've done unit testing in C++, I've had the tests in separate files, and when I do that, I usually put all of the test files in the same place (e.g. include/ for the .h files, source/ for the .cpp files and tests/ for the .h and .cpp files with the tests) and have a 1-to-1 relationship between the .h/.cpp pair containing the code and the .h/.cpp pair containing the tests. Also, the test functions are usually named after the function that they're testing. So, it's all fairly organized, but it's still way more of a pain than just having the unittest block right after the function being tested, and it makes it easy for folks to just ignore the tests and easy for you to miss that something wasn't tested. Having dealt with both that and putting all of the unit tests next to stuff in D, I find having the tests right after the functions to be _far_ more maintainable. In fact, in std.datetime.interval, most of the tests are outside the templated types that they're testing in order to avoid having the tests included in every template instantiation, and that's turned out to be _way_ more of a pain to maintain than having the tests right after the functions. And that doesn't even involve a separate file. Obviously, YMMV, but in my experience, having the tests _immediately_ after what they're testing is vastly more maintainable than having the tests elsewhere. - Jonathan M Davis
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Fri, Mar 23, 2018 at 03:56:03PM -0400, Steven Schveighoffer via Digitalmars-d-announce wrote: [..] > Note that a frequent complaint of std.datetime (at least when it was > one module) is that the file was too big. While it does hold a lot of > functionality, the majority of the file size is unittests. This means > that it can be hard to surf the file for functionality. Actually, I didn't have a problem with std.datetime being a huge file. What I *did* have a problem with was that it took too much memory to compile and the unittests took too long to run. And also that it consisted of a bunch of somewhat loosely-coupled chunks that could be more meaningfully encapsulated as submodules. > But on the flip side, there aren't a lot of datetime bugs! Exactly. > I personally believe that there should be unit tests for every > function, and be right next to the function. I don't want to go on a > search for such things, or have to rely on manual documentation to > know what is testing what. It would be nice to have your editor hide > the unit tests unless you want to work on them. It's just a matter of configuring folding in an editor that supports it. Or dump your editor if it doesn't support folding, 'cos it sux and you deserve better. :-D I mean, this is 2018, not 1995, we shouldn't have to be stuck with the handicap of navigating by scrollbar and paging up/down anymore. > I've worked on a project where the testing was separated from the > code, and it was a liability IMO. Things would get missed and not > tested properly. [...] Yep. As I mentioned elsewhere, recently I've had to resort to external testing for one of my projects, and I'm still working on that right now. And already, I'm seeing a liability: rather than quickly locating a unittest immediately following a particular function, now I have to remember "oh which subdirectory was it that the tests were put in? and which file was it that a particular test of this function was done?". It's an additional mental burden to have to keep doing the mapping between current source location <-> test code location (even if it's a 1-to-1 mapping), and a physical burden to have to continually open external files (and typing a potentially long path for them) rather than just "bookmark, jump to end of function, navigate unittest blocks" in the same file. Fortunately, this problem is somewhat alleviated by having cumulative code coverage (see other thread), which can quickly identify what hasn't been tested yet. Currently, with the help of coverage, I've managed to get 100% coverage for 4 of my modules, and 90%+ for another 5 modules, and working on the remainder right now. It's an uphill battle at times, but already I've been reaping the benefits: buggy corner cases that I missed were discovered, fixed, and now have a corresponding test case / unittest where possible. T -- I've been around long enough to have seen an endless parade of magic new techniques du jour, most of which purport to remove the necessity of thought about your programming problem. In the end they wind up contributing one or two pieces to the collective wisdom, and fade away in the rearview mirror. -- Walter Bright
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Fri, Mar 23, 2018 at 07:46:34PM +, Johan Engelen via Digitalmars-d-announce wrote: > On Thursday, 22 March 2018 at 15:18:40 UTC, Jacob Carlborg wrote: [...] > > I completely agree. Although my reason is mostly because there will > > be too much code in a single file if the regular code and unit tests > > are mixed in the same file. > > Fully agree with this "too much code in a single file" point. I am > confident that part of the reason of Phobos unittesting being very > incomplete, is that adding unittests further clutters the codebase. > Moving all unittests to the bottom of the file (pulling them out of > classes too) would resolve this issue in part. [...] I beg to differ. From my experience with working on Phobos, file size or perceived "clutter" simply isn't an issue. Good chunks of Phobos have long lists of unittests following an overload set, and often even interspersed between overload sets, and we have never shied away from adding more unittests to the list as bug fixes get added, nor have we ever felt the pressure to move the unittests elsewhere. Using a code editor with modern capabilities helps to a great degree, of course. If your primary way of navigating a source file is by paging up/down or by pulling a scrollbar, IMO you're doing it wrong. You should rather be using actually useful navigation, like Vim's search / fold / jump to matching brace/parenthesis / jump to bookmark / ctags + jump to tag, etc., or whatever a GUI IDE's equivalents of these are. Dragging scrollbars around is so 90's, and simply isn't up to task for navigating and working with large files. In my own projects, I regularly deal with 3000-line source files, and the size of the file has never been an issue as far as navigation is concerned. Of course, there are other considerations as to why stuffing everything in one file might not be such a good idea, such as encapsulation, but certainly I have never been deterred because of the fear that adding unittests might make a file "too big". When I do split a file into multiple modules, it's purely for encapsulation reasons rather than file size considerations. In fact, I prefer to keep related code, e.g. code + unittest for that code, together in the same file, because otherwise things tend to go out-of-sync. It's always easier to put related things together in one place, rather than scatter them across multiple places and then have to worry about keeping each piece in sync with the others. If a well-encapsulated module contains 1000 lines of code and 5000 lines of unittests, then so much the better. But of course, YMMV. T -- What did the alien say to Schubert? "Take me to your lieder."
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On 3/23/18 3:46 PM, Johan Engelen wrote: On Thursday, 22 March 2018 at 15:18:40 UTC, Jacob Carlborg wrote: On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ I completely agree. Although my reason is mostly because there will be too much code in a single file if the regular code and unit tests are mixed in the same file. Fully agree with this "too much code in a single file" point. I am confident that part of the reason of Phobos unittesting being very incomplete, is that adding unittests further clutters the codebase. Moving all unittests to the bottom of the file (pulling them out of classes too) would resolve this issue in part. Note that a frequent complaint of std.datetime (at least when it was one module) is that the file was too big. While it does hold a lot of functionality, the majority of the file size is unittests. This means that it can be hard to surf the file for functionality. But on the flip side, there aren't a lot of datetime bugs! I personally believe that there should be unit tests for every function, and be right next to the function. I don't want to go on a search for such things, or have to rely on manual documentation to know what is testing what. It would be nice to have your editor hide the unit tests unless you want to work on them. I've worked on a project where the testing was separated from the code, and it was a liability IMO. Things would get missed and not tested properly. -Steve
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 15:18:40 UTC, Jacob Carlborg wrote: On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ I completely agree. Although my reason is mostly because there will be too much code in a single file if the regular code and unit tests are mixed in the same file. Fully agree with this "too much code in a single file" point. I am confident that part of the reason of Phobos unittesting being very incomplete, is that adding unittests further clutters the codebase. Moving all unittests to the bottom of the file (pulling them out of classes too) would resolve this issue in part. - Johan
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On 3/22/18 6:59 AM, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila It's simple. Unittests in imported modules should not be visible. They should be compiled as if -unittest was not passed. Even Walter and Andrei are supportive: https://github.com/dlang/dmd/pull/6375#issuecomment-373487247 -Steve
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Friday, 23 March 2018 at 00:47:20 UTC, Tony wrote: I think unittest blocks are good for write-once and quick-and-dirty projects, or as a first-cut of testing that ultimately gets moved to a full-grown test suite in a separate project. I'd prefer not to read source code that has unittest blocks inter-mixed with the actual code. I partially agree with this. It's really annoying if you have to look for an implementation and then half the module is unittests and searching isn't always straightforward.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
I think unittest blocks are good for write-once and quick-and-dirty projects, or as a first-cut of testing that ultimately gets moved to a full-grown test suite in a separate project. I'd prefer not to read source code that has unittest blocks inter-mixed with the actual code.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thu, Mar 22, 2018 at 05:29:52PM +, Anton Fediushin via Digitalmars-d-announce wrote: [...] > Oh heck yeah! I think that dub is only one of the D tools which never > disappoints me. I'm glad you found dub to your liking. I'm afraid I can't say the same, though for reasons not really relevant to this thread. T -- If the comments and the code disagree, it's likely that *both* are wrong. -- Christopher
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thu, Mar 22, 2018 at 05:21:41PM +, Atila Neves via
Digitalmars-d-announce wrote:
> On Thursday, 22 March 2018 at 16:30:37 UTC, H. S. Teoh wrote:
[...]
> > being tested, was a big slowdown for me. I have to stop to think
> > about which subdirectory under test/ I should put the relevant
> > file(s), or if there's already a test there I have to stop to think
> > about which filename I saved it under, etc.. It's just yet another
> > mental hurdle to jump over while my already-busy brain is thinking
> > about the code itself.
>
> Then I'd recommend:
>
> 1) To write the tests inline when the mental burden is high
> 2) Move them afterwards when one's brain can think of where to place
> them.
I don't like having to move code around after it's written. Besides,
since all sufficiently complex (i.e., sufficiently interesting) code
inevitably has bugs, the set of unittests following each function is
likely to keep growing, so I'd like to keep them all in one place where
possible.
But at the end of the day, it really comes down to just personal
preferences. The case could be equally argued both ways.
[...]
> > 2) Compilation times: perhaps if your unittests are too heavy,
> > compilation times could become a problem,
>
> Maybe I wasn't clear in what I wrote: I'm not saying that I notice the
> increase in compilation times of the tests themselves. I probably
> haven't but I'd have to measure to know for sure. I'm saying that if
> you change anything in a D module, it must be assumed that any other
> module that imports the module you just edited, even if transitively,
> must be recompiled. So if I edit a test in D, under normal
> circumstances, I _have_ to recompile several files. It's not the
> unittest itself that is heavy, it's recompiling everyone who depends
> on the module that said unittest happens to find itself in.
Ah, I see your point. So the idea is to split off the unittests into a
separate file that isn't among the dependencies, so that changing
unittests won't force everything else to recompile. Makes sense to me.
Though I have yet to work with a project large enough where this would
cause a significant degradation in compilation times.
(Having said that, though, I find the building with dub in general is
extremely slow due to all the other stuff that it does besides building
the code. So if the addition of a few more modules to recompile causes
a noticeable slowdown in dub, I wouldn't be surprised.)
> > but IME, they haven't been.
>
> IME most other people find it bizarre I get angry at 2s incremental
> rebuild times. Anything over 100ms is an eternity.
Haha... well, then maybe you could help us reduce the compilation cost
of certain common Phobos modules, like std.format. ;-) I *love*
std.format for its functionality, and use it almost everywhere in my
code, but recently I've been rather disappointed at the huge slowdowns
it adds to my compilation times. :-( I wish the cost of using
std.format is more proportional to how much of it I'm actually using.
Currently, even something as trivial as format("%s",str) adds a
significant hit to compilation times, even though conceptually it should
really just be a single call to output a string.
[...]
> > As for the dub-specific problems introduced by version(unittest):
> > IMO that's a flaw in dub.
>
> It's not dub-specific at all. It's same problem that you reference in:
>
> > I remember that in Phobos we used to merge PRs containing code
> > that compiles fine with -unittest, but in real-world code doesn't
> > compile because it has stuff outside unittests that depend on
> > imports/declarations inside version(unittest).
>
> I used dub as an example. Anyone else would have the same problems if
> they hand-wrote Makefiles using git submodules as dependencies.
[...]
True. That's why I'm inclined to say version(unittest) is an
antipattern.
T
--
The trouble with TCP jokes is that it's like hearing the same joke over and
over.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Agreed on most counts though I’d say D simply produced a language without regard for build tools and large projects. Many small annoying things like version(unittest) stem from that fact. In addition I think many things could easily take a ton of time to compile and run tests. ctRegex is one such monster. Inadvertently pluging that into somebody else unittest build would be murder ;) It’s a given in some other languages that you need some library support to write tests (sometimes any tests) so they usually have cleancut prod dependecies and test dependencies. Our unitest being built-in practically forces the style. Atila
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 13:50:39 UTC, Atila Neves wrote: On Thursday, 22 March 2018 at 11:19:46 UTC, Basile B. wrote: On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ Sorry for the forum spam. Atila I don't agree at all. Everything is so much faster when unittest blocks are in the same module as the stuff that are tested. Generally i think that it makes things easier. Faster as measured by what? Projects definitely take longer to build and iterate on. If you think it makes things easier, great! The blog post is my opinion. Atila I was able to reach almost 100% coverage (99.2263) in my project Yatol just because tests are next to the code. This project has really convinced me of the fact that unittest are better in the code. This is also very good for tooling, e.g workflow in my IDE relies on tests being with the code (there's an action called Run file unittests that allows to test + cover a single module that's part of bigger project.) I honestly think that the project i mention would be less tested if the test suite was put apart. I don't say you are wrong, but the two or three occasional issues you mention in your blog post are really marginal things IMO, although i don't deny they are possible.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 17:09:55 UTC, Marco Leise wrote: I understand your opinion and I think it is all reasonable. You talk about longer compile times since every D module is like a C++ header. That touches one of my pet peeves with the language or eco system as it stands and I wonder if you would agree with me on the following: Libraries should be tied into applications using interface files (*.di) that are auto-generated by the compiler for the _library author_ with inferred function attributes. If after a code change, a declaration in the *.di file changes, the library's interface changed and a new minor version must be released. The language must allow to explicitly declare a function or method as @gc, impure, etc. so the auto-inferred attributes don't later become an issue when the implementation changes from e.g. a pure to an impure one. Opaque struct pointers as seen in most C APIs should also be considered for *.di files to reduce the number of imports for member fields. That means: * No more fuzzyness about whether a library function will remain @nogc, @safe, etc. in the next update. * Explicit library boundaries that don't recursively import the world. I like it but I'd have to think about it a bit more. My current idea to save me from staring at the screen for 2s at a time several times a day is to write a program using dmd as a library and integrate it into reggae that gets all of the dependencies from a given module. Then the build system reggae would generate would _not_ declare those dependencies, but actually .di files generated from them. That way a rebuild would only happen if the .di file changed, which the program would be smart enough to not overwrite if the hash/diff hasn't changed. Atila
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 17:15:55 UTC, Anton Fediushin wrote: So? Am I wrong about dub? Let me investigate I'm not wrong! It works as expected: only package you are working with compiles with `-unittest` option. Test repo: https://github.com/ohdatboi/dub-please-be-as-cool-as-i-think-you-are Cd to `app` directory and run: --- $ dub Performing "debug" build using /usr/bin/dmd for x86_64. lib1 ~master: building configuration "library"... lib1: Compiled without -unittest option app ~master: building configuration "application"... app: Compiled without -unittest option Linking... Running ./app --- --- $ dub -b unittest Performing "unittest" build using /usr/bin/dmd for x86_64. lib1 ~master: building configuration "library"... lib1: Compiled without -unittest option app ~master: building configuration "application"... app: Compiled with -unittest option Linking... Running ./app --- Oh heck yeah! I think that dub is only one of the D tools which never disappoints me.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 16:30:37 UTC, H. S. Teoh wrote: On Thu, Mar 22, 2018 at 10:59:56AM +, Atila Neves via Digitalmars-d-announce wrote: Blog post: https://atilanevesoncode.wordpress.com/ [...] I realize this is your opinion, but I disagree with them because: Disagreeing is more than fine. :) 1) I've found that having unittests built into the language is a big win, *especially* because I can write tests next to the code being tested. I'm confused. I don't know what I wrote that made you think I believe otherwise. I wrote "fantastically successful for the language" and that "Let me start by saying that some tests should go along with the production code". being tested, was a big slowdown for me. I have to stop to think about which subdirectory under test/ I should put the relevant file(s), or if there's already a test there I have to stop to think about which filename I saved it under, etc.. It's just yet another mental hurdle to jump over while my already-busy brain is thinking about the code itself. Then I'd recommend: 1) To write the tests inline when the mental burden is high 2) Move them afterwards when one's brain can think of where to place them. I think that mirroring the production source tree is probably the way to go most of the time, i.e. test/foo/bar/baz.d for src/foo/bar/baz.d. 2) Compilation times: perhaps if your unittests are too heavy, compilation times could become a problem, Maybe I wasn't clear in what I wrote: I'm not saying that I notice the increase in compilation times of the tests themselves. I probably haven't but I'd have to measure to know for sure. I'm saying that if you change anything in a D module, it must be assumed that any other module that imports the module you just edited, even if transitively, must be recompiled. So if I edit a test in D, under normal circumstances, I _have_ to recompile several files. It's not the unittest itself that is heavy, it's recompiling everyone who depends on the module that said unittest happens to find itself in. but IME, they haven't been. IME most other people find it bizarre I get angry at 2s incremental rebuild times. Anything over 100ms is an eternity. Plus, my opinion is that when you're compiling the code, you *should* be running unittests anyway (otherwise regressions inevitably slip in), Yes. so you're going to have to pay for the time taken to compile them regardless. Yes. In that sense, it's actually better to have them in the same file so that the compiler doesn't have to open up another file and allocate resources for handling another module. Noo. As for the dub-specific problems introduced by version(unittest): IMO that's a flaw in dub. It's not dub-specific at all. It's same problem that you reference in: I remember that in Phobos we used to merge PRs containing code that compiles fine with -unittest, but in real-world code doesn't compile because it has stuff outside unittests that depend on imports/declarations inside version(unittest). I used dub as an example. Anyone else would have the same problems if they hand-wrote Makefiles using git submodules as dependencies. Atila
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 16:54:18 UTC, Anton Fediushin wrote: On Thursday, 22 March 2018 at 16:30:37 UTC, H. S. Teoh wrote: As for the dub-specific problems introduced by version(unittest): IMO that's a flaw in dub. I should not need to contort my code just to accomodate some flaw in dub. Having said that, though, I do agree that version(unittest) itself is a bad idea, so code written the way I recommend would not have this problem even given dub's flaws. There's no "dub-specific problems". Article is wrong about that: when you run `dub test` only package you are working with is compiled with '-unittest' option. This way it is _impossible_ to have any kind of problems with `version(unittest)` if you are writing libraries IMHO dub does it exactly right - I most definitely do _not_ want to build my dependencies's unittests, nor do I want to run them. Imagine if I had to parse and run the tests from libclang or Qt (worse: both!) every time I wrote a C++ program... Atila
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 17:08:18 UTC, Jonathan M Davis
wrote:
I could be wrong, but I am _quite_ sure that dub builds all
dependencies with their test targets when you build your
project with its test target.
I thought so too, but I just checked and it doesn't do that. I'd
better create a test repository for that, maybe I'm doing
something wrong.
I had to add this to dub.json
"buildTypes":
{
"doTests":
{
"buildOptions": ["unittests", "debugMode",
"debugInfo"],
"versions": ["dxmlTests"]
},
"doCov":
{
"buildOptions": ["unittests", "debugMode",
"debugInfo",
"coverage"],
"versions": ["dxmlTests"]
}
}
Well, that's just ugly. And that `versions` thing looks exactly
like the one from Atila's article.
So? Am I wrong about dub? Let me investigate
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 13:58:50 UTC, Atila Neves wrote: On Thursday, 22 March 2018 at 12:26:14 UTC, Anton Fediushin Tests in their own file is something from 90-s. It's 2018 and I want to be able to write tests anywhere I want. You _can_ write them wherever you want. I'm not arguing for taking any flexibility away, I'm arguing that for most projects it's a bad idea, and I stated the reasons why I think that's the case. Feel free to disagree. Since your article is wrong about dub's way of handling `unittest` configuration (take a look at my last post), the only valid point is "they increase build times". The only choice you have to make is: you take the red pill and choose convenience over compilation speed or you take the blue pill for speed over the convenience. None of the choices is "good" or "bad". They're just slightly different. "They increase build times" - fix compiler, not my code. If by "fix the compiler" you mean "make it so that the compiler knows exactly what files need to be recompiled if I only edited the body of a unittest block" (which is what would be needed), then I don't think that'll happen any time soon. But sure, if the compiler worked that way I'd happily advocate for writing unittests in the same file. But for now, I'd rather spend a lot less time staring at the screen for 2s waiting for my code to be built. That's my trade-off, and reasonable people may (and apparently quite obviously do) disagree. I'm not a compiler developer, so I don't mean anything in particular. Just saying. I didn't knew that change in unittest block causes rebuilding of many modules, so that's good to know. "version(unittest) will cause you problems if you write libraries" - fix dub, not my code. This is _not_ a dub issue. version(unittest) _will_ bite you as soon as you compile foo.d with -unittest because its imports will be parsed differently. This is one to stick in the "fix the compiler" column. See the link to the since-reverted Phobos PR in the blog. Yeah, I was wrong about dub. I'm so sorry since it turned out that dub handles `dub test` in the way that never causes problems with `version(unittest)`. What about `version(unittest)` itself - as any of the conditional compilation features it should be handled with care. Especially when we are talking about cases where dub isn't used.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
I understand your opinion and I think it is all reasonable. You talk about longer compile times since every D module is like a C++ header. That touches one of my pet peeves with the language or eco system as it stands and I wonder if you would agree with me on the following: Libraries should be tied into applications using interface files (*.di) that are auto-generated by the compiler for the _library author_ with inferred function attributes. If after a code change, a declaration in the *.di file changes, the library's interface changed and a new minor version must be released. The language must allow to explicitly declare a function or method as @gc, impure, etc. so the auto-inferred attributes don't later become an issue when the implementation changes from e.g. a pure to an impure one. Opaque struct pointers as seen in most C APIs should also be considered for *.di files to reduce the number of imports for member fields. That means: * No more fuzzyness about whether a library function will remain @nogc, @safe, etc. in the next update. * Explicit library boundaries that don't recursively import the world. -- Marco
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, March 22, 2018 16:54:18 Anton Fediushin via Digitalmars-d-
announce wrote:
> On Thursday, 22 March 2018 at 16:30:37 UTC, H. S. Teoh wrote:
> > As for the dub-specific problems introduced by
> > version(unittest): IMO that's a flaw in dub. I should not need
> > to contort my code just to accomodate some flaw in dub. Having
> > said that, though, I do agree that version(unittest) itself is
> > a bad idea, so code written the way I recommend would not have
> > this problem even given dub's flaws.
>
> There's no "dub-specific problems". Article is wrong about that:
> when you run `dub test` only package you are working with is
> compiled with '-unittest' option. This way it is _impossible_ to
> have any kind of problems with `version(unittest)` if you are
> writing libraries
I could be wrong, but I am _quite_ sure that dub builds all dependencies
with their test targets when you build your project with its test target. I
was forced to work around that with dxml by adding a version identifier
specifically for dxml's tests _and_ create a test target specifically for
dxml. Simply adding a dxml-specific version identifier to its test target
was not enough, because any project which depended on dxml ended up with the
dxml-specific version identifier defined when its tests were built, because
dub was building dxml's test target and not its debug or release target. The
only way I found around the problem was to create a target specific to dxml
for building its unit tests and define the version identifier for that
target only.
I had to add this to dub.json
"buildTypes":
{
"doTests":
{
"buildOptions": ["unittests", "debugMode", "debugInfo"],
"versions": ["dxmlTests"]
},
"doCov":
{
"buildOptions": ["unittests", "debugMode", "debugInfo",
"coverage"],
"versions": ["dxmlTests"]
}
}
And then I have scripts such as
test.sh
===
#!/bin/sh
dub test --build=doTests
===
to run the tests. I had to actively work around dub and what it does with
unit tests in order to not have all of dxml's tests compiled into any
project which had dxml as a dependency.
- Jonathan M Davis
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, March 22, 2018 09:30:37 H. S. Teoh via Digitalmars-d-announce wrote: > 2) Compilation times: perhaps if your unittests are too heavy, > compilation times could become a problem, but IME, they haven't been. Personally, I agree with you, but Atila is one of those folks who gets annoyed when tests take even a fraction of a second longer, so stuff that many of us would not consider pain points at all tends to annoy him. I prefer it when tests run fast, but if they take 1 second rather than 500 milliseconds, I don't consider it a big deal. Atila would. So, given Atila's preferences, it makes sense to remove the tests from the module if that speeds up compilation even a little, whereas personally, I'd much rather have them next to the code they're testing so that I can see what's being tested, and I don't have to go track down another file and edit the tests there when I'm editing the code being tested. I'd prefer that that not harm compilation time, but if it does, I'm generally going to put up with it rather than move the tests elsewhere. > 3) version(unittest): it's true that this can be a problem. I remember > that in Phobos we used to merge PRs containing code that compiles fine > with -unittest, but in real-world code doesn't compile because it has > stuff outside unittests that depend on imports/declarations inside > version(unittest). This is actually one of the reasons I was (and still > am) a big supporter of local/scoped imports. It may be more convenient > to just put global imports at the top of the module, but it just creates > too many implicit dependencies from mostly-unrelated chunks of code, > that I'm inclined actually to call global imports an anti-pattern. In > fact, I'd even go so far to say that version(unittest) in general is a > bad idea. It is better to factor out stuff inside version(unittest) to > a different module dedicated to unittest-specific stuff, and have each > unittest that needs it import that module. This way you avoid polluting > the non-unittest namespace with unittest-specific symbols. I don't think that global imports are really an anti-pattern, though there are advantages to local imports. The big problem with global imports is when they're versioned, because then it's way too easy to screw them up. If they're not versioned, then worst case, you're importing something which wouldn't necessarily have to be imported if local imports were used. > As for the dub-specific problems introduced by version(unittest): IMO > that's a flaw in dub. I should not need to contort my code just to > accomodate some flaw in dub. Having said that, though, I do agree that > version(unittest) itself is a bad idea, so code written the way I > recommend would not have this problem even given dub's flaws. dub makes the problem worse, but it's inherent in how D currently works. When I compile my project with -unittest, and I'm importing your library, your version(unittest) code gets compiled in. I'm not sure exactly what happens with the unittest blocks, since the code for them isn't generated unless the module is explicitly compiled, but I think that it's the case that all of the semantic analysis still gets done for them, in which case, anything they import would need to be available, and that's a huge negative. We really need a DIP to sort this out. Now, unfortunately, dub makes the whole thing worse by compiling dependencies with their unittest target when you build your project with its unittest target, so you get all of the unit tests of all of your dependencies regardless of what dmd does, but even if dub were not doing that, we'd still have a problem with the language itself. - Jonathan M Davis
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thu, Mar 22, 2018 at 10:59:56AM +, Atila Neves via Digitalmars-d-announce wrote: > Blog post: > > https://atilanevesoncode.wordpress.com/ [...] I realize this is your opinion, but I disagree with them because: 1) I've found that having unittests built into the language is a big win, *especially* because I can write tests next to the code being tested. This is a big win because needing to open up a new file and finding the right place to write the test is so much slower, and distracts my focus from the code being written, and also when browsing the code it's so much easier to find related unittests nearby to see if a particular case has been tested or not, as opposed to having to open up another file and searching for tests that *might* be relevant. Case in point: just this week, for a particular project of mine I needed to write external tests because refactoring would be too hard. So I cooked up a simple test driver that scans a set of directories to pick up test inputs and expected outputs. It was great that in D, this could be done in just a couple of days' worth of work. However, I did find that needing to open up another file to create a new test, a second file to put the expected output, and a third file to look at the source code being tested, was a big slowdown for me. I have to stop to think about which subdirectory under test/ I should put the relevant file(s), or if there's already a test there I have to stop to think about which filename I saved it under, etc.. It's just yet another mental hurdle to jump over while my already-busy brain is thinking about the code itself. In my case it's a necessary evil, but would I do it for a project that could just use inline unittests? Definitely not. But of course, YMMV. 2) Compilation times: perhaps if your unittests are too heavy, compilation times could become a problem, but IME, they haven't been. Plus, my opinion is that when you're compiling the code, you *should* be running unittests anyway (otherwise regressions inevitably slip in), so you're going to have to pay for the time taken to compile them regardless. In that sense, it's actually better to have them in the same file so that the compiler doesn't have to open up another file and allocate resources for handling another module. 3) version(unittest): it's true that this can be a problem. I remember that in Phobos we used to merge PRs containing code that compiles fine with -unittest, but in real-world code doesn't compile because it has stuff outside unittests that depend on imports/declarations inside version(unittest). This is actually one of the reasons I was (and still am) a big supporter of local/scoped imports. It may be more convenient to just put global imports at the top of the module, but it just creates too many implicit dependencies from mostly-unrelated chunks of code, that I'm inclined actually to call global imports an anti-pattern. In fact, I'd even go so far to say that version(unittest) in general is a bad idea. It is better to factor out stuff inside version(unittest) to a different module dedicated to unittest-specific stuff, and have each unittest that needs it import that module. This way you avoid polluting the non-unittest namespace with unittest-specific symbols. As for the dub-specific problems introduced by version(unittest): IMO that's a flaw in dub. I should not need to contort my code just to accomodate some flaw in dub. Having said that, though, I do agree that version(unittest) itself is a bad idea, so code written the way I recommend would not have this problem even given dub's flaws. T -- Programming is not just an act of telling a computer what to do: it is also an act of telling other programmers what you wished the computer to do. Both are important, and the latter deserves care. -- Andrew Morton
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ I completely agree. Although my reason is mostly because there will be too much code in a single file if the regular code and unit tests are mixed in the same file. Also, having utilities/helpers only used by the tests is easier, or easier to find a location for. -- /Jacob Carlborg
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 12:26:14 UTC, Anton Fediushin wrote: On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila I *love* built-in unittests. Putting them right after each function makes things so much easier. If it works for you, great. Tests in their own file is something from 90-s. It's 2018 and I want to be able to write tests anywhere I want. You _can_ write them wherever you want. I'm not arguing for taking any flexibility away, I'm arguing that for most projects it's a bad idea, and I stated the reasons why I think that's the case. Feel free to disagree. "They increase build times" - fix compiler, not my code. If by "fix the compiler" you mean "make it so that the compiler knows exactly what files need to be recompiled if I only edited the body of a unittest block" (which is what would be needed), then I don't think that'll happen any time soon. But sure, if the compiler worked that way I'd happily advocate for writing unittests in the same file. But for now, I'd rather spend a lot less time staring at the screen for 2s waiting for my code to be built. That's my trade-off, and reasonable people may (and apparently quite obviously do) disagree. "version(unittest) will cause you problems if you write libraries" - fix dub, not my code. This is _not_ a dub issue. version(unittest) _will_ bite you as soon as you compile foo.d with -unittest because its imports will be parsed differently. This is one to stick in the "fix the compiler" column. See the link to the since-reverted Phobos PR in the blog. Built-in unittests are part of D's specification. As long as it allowes to do so, I'll carry on writing unittests in my code and nobody can take it from me. Who's trying to take it from you? Atila
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 11:19:46 UTC, Basile B. wrote: On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ Sorry for the forum spam. Atila I don't agree at all. Everything is so much faster when unittest blocks are in the same module as the stuff that are tested. Generally i think that it makes things easier. Faster as measured by what? Projects definitely take longer to build and iterate on. If you think it makes things easier, great! The blog post is my opinion. Atila
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 12:25:59 UTC, bauss wrote: On Thursday, 22 March 2018 at 11:47:34 UTC, Alexandru Ermicioi wrote: On Thursday, 22 March 2018 at 11:19:46 UTC, Basile B. wrote: On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: [...] Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ Sorry for the forum spam. Atila I don't agree at all. Everything is so much faster when unittest blocks are in the same module as the stuff that are tested. Generally i think that it makes things easier. It's easier, true. But not all code can be nicely unittested using small chunks of unittests near the actual code. Consider unittested objects/structs, in each unittest you'll have to instantiate and stuff with mocks if needed and test a small piece of functionality from them. Also there is a problem with mocked objects where they will go? If every bit of functionality will be tested in same module with source code, then unittest / line of actual code will be much more to unittests part. It will make your module hard to navigate and discover what your source code is actually doing. So having a separate module with unittests indeed will help in making more claner and more understandable since they won't interfere with actual code. Also if you use templates with unittests sometimes private symbols may succeed in the unittest, but fail when the template is used in practice. Such bugs have existed in Phobos plenty of times. Yes, but for the record they got extinct since ~ one year once CircleCi with its setup test extraction pipeline started to enforce this. There's no reason DMD couldn't do this by default internally when it finds a public unittest. Or your favorite build tool could do such an extraction though granted then you do loose the "built-in", simplistic aspect of unittests.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila I *love* built-in unittests. Putting them right after each function makes things so much easier. Tests in their own file is something from 90-s. It's 2018 and I want to be able to write tests anywhere I want. "They increase build times" - fix compiler, not my code. "version(unittest) will cause you problems if you write libraries" - fix dub, not my code. Built-in unittests are part of D's specification. As long as it allowes to do so, I'll carry on writing unittests in my code and nobody can take it from me.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 11:47:34 UTC, Alexandru Ermicioi wrote: On Thursday, 22 March 2018 at 11:19:46 UTC, Basile B. wrote: On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ Sorry for the forum spam. Atila I don't agree at all. Everything is so much faster when unittest blocks are in the same module as the stuff that are tested. Generally i think that it makes things easier. It's easier, true. But not all code can be nicely unittested using small chunks of unittests near the actual code. Consider unittested objects/structs, in each unittest you'll have to instantiate and stuff with mocks if needed and test a small piece of functionality from them. Also there is a problem with mocked objects where they will go? If every bit of functionality will be tested in same module with source code, then unittest / line of actual code will be much more to unittests part. It will make your module hard to navigate and discover what your source code is actually doing. So having a separate module with unittests indeed will help in making more claner and more understandable since they won't interfere with actual code. Also if you use templates with unittests sometimes private symbols may succeed in the unittest, but fail when the template is used in practice. Such bugs have existed in Phobos plenty of times.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 11:19:46 UTC, Basile B. wrote: On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ Sorry for the forum spam. Atila I don't agree at all. Everything is so much faster when unittest blocks are in the same module as the stuff that are tested. Generally i think that it makes things easier. It's easier, true. But not all code can be nicely unittested using small chunks of unittests near the actual code. Consider unittested objects/structs, in each unittest you'll have to instantiate and stuff with mocks if needed and test a small piece of functionality from them. Also there is a problem with mocked objects where they will go? If every bit of functionality will be tested in same module with source code, then unittest / line of actual code will be much more to unittests part. It will make your module hard to navigate and discover what your source code is actually doing. So having a separate module with unittests indeed will help in making more claner and more understandable since they won't interfere with actual code.
Re: Why think unit tests should be in their own source code hierarchy instead of side-by-side
On Thursday, 22 March 2018 at 11:00:31 UTC, Atila Neves wrote: On Thursday, 22 March 2018 at 10:59:56 UTC, Atila Neves wrote: Blog post: https://atilanevesoncode.wordpress.com/ Atila Direct link: https://atilanevesoncode.wordpress.com/2018/03/22/keep-d-unittests-separated-from-production-code/ Sorry for the forum spam. Atila I don't agree at all. Everything is so much faster when unittest blocks are in the same module as the stuff that are tested. Generally i think that it makes things easier.
