On Thu, Jan 24, 2013 at 9:24 PM, Tommaso Cucinotta <[email protected]> wrote: > On 07/01/13 21:48, Scott Kostyshak wrote: >> I'm curious about is why there are >> so few tests for LyX. >> >> In particular, I'm interested in the following two questions: > > let's try the funny ride... :-) > >> 1) Why don't you write tests? > > I can tell u why I wrote a few: after a tiny change in the logic for Advanced > Find and Replace, I needed a way to ensure that all the basic usages of the > feature were keeping working as expected, with various options > enabled/disabled, in various cases, and it's impossible to run such tests > manually (I was doing that until I discovered the amazing monkey testing > framework by McCabe-Dansted to automate sending key presses to LyX -- only > that I didn't need random presses, but specific ones creating my needed test > scenarios). > >> 2) Why do you think others don't write tests? > > a real testing framework was not really designed/advertised, AFAIK. Just, > when needed for critical regressions, someone wrote a few [semi-]automated > tests. > >> a. I do. They're called assertions. > > hmmm... there's quite some difference, but assertions help of course (see > below). > >> b. The time it would take me to write a test would be better spent on >> some other aspect of LyX development. > > Actually, with such a short time to dedicate to the project, and just for > fun, why having fun doing the non-funny stuff :-)? > But, it's also non-funny when the d..n thing crashes while you're writing the > d..n paper! > >> c. No one runs the tests. > > ehm..., perhaps whoever takes care of the release might take such > responsibility? > (assuming all tests would work on the platform/OS used for the purpose, but I > suspect it's not really the case) > >> f. The initial setup (dependencies, and cmake vs. autotools) of the >> getting the tests to work is too annoying. > > guess not more annoying than setting up the whole development environment for > LyX, with all the needed libraries, headers and tools. > >> Some tests seem difficult to write. For example, I find Tommmaso's >> advanced find tests to be very creative but I imagine they were >> time-consuming to write. Thus, my next question is: >> >> 3) What are the types of tests that are the most useful in the context >> of LyX and is there anything I can do to make writing those tests >> easier? > > very subjective: useful for what? Is there any kind of statistics about > the most annoying bugs/problems, that would allow for prioritizing > bugs? Guess we have only the criticality in the trac. One possible > action might be: when the fixed bug was leading to a SEGFAULT, write > a test ensuring it doesn't repeat in the future. > >> My own attempt at an answer: >> The tests that are the most useful are those that are the easiest to >> write (because then we will actually write them), which in LyX are >> tests that >> (a) can be expressed by a command sequence, and >> (b) trigger an assertion or crash (so there's no need to redirect with >> a debug flag to a log and use pcregrep; and because a crash is often a >> bad type of bug from the user perspective). > > But, most of the tests I currently have in the findadv scenarios, don't > trigger assertion, but they simply would fail to find the right stuff > if the find logic has some problems (or, they find what should not be > found etc.). > Nor it would make sense to add "triggerable" assertions to LyX, I mean, > I trigger an LFUN that's supposed to find smth., but it doesn't find it, > LyX is supposed to give a notice to the user and that's it, not to assert! > That's why the "white box" type of test, that exploits that ugly debug > log. Though, most tests need only to know exactly the position of the > found match and compare with a known position. An equivalent way to get > such information out of lyx would suffice. > > Also, debug messages might easily be considered completely optional > and suppressible once there's confidence that the functionality works, > but if one removes some key messages, or even alters only slightly them, > then all findadv tests will fail! I don't like such fragility of course. > A testing framework might include a special TestLog() service (or as a > debugging level) useful to tag those messages that must not be changed, > and whose output goes to a special testing log (rather than the debug > log). Just a proposal. > > Of course, one can write > independent test programs that link only the relevant findadv.cpp > files, and invoke those methods in a very specific way, so to get rid > altogether of the need for launching the GUI for testing the find logic, > and speeding up the whole tests execution etc. > >> Currently, writing this type of tests is pretty easy, but perhaps it >> could be made even easier by just asking the developer to add one line >> to a file, say LFUNcrashes.txt: >> >> command-sequence lfun1; lfun2; lfun3 # see [trac bug number] >
Thanks for all of the great comments and explanations, Tommaso. I like your idea of TestLog(). Independent test programs also sound like a good alternative in some cases. I guess my questions were too broad. As you pointed out, there are many subjective decisions that a programmer must make (this is what makes programming fun sometimes though). And each test scenario is different so the "best" approach taken to implement that test might be unique. Regardless, I learned some good general lessons that have given me a better understanding of testing in a LyX context. I appreciate your comments. Scott
