Re: [notmuch] A functional (but rudimentary) test suite for notmuch
On Thu, 11 Feb 2010 13:42:08 -0800, Carl Worth cwo...@cworth.org wrote: Interesting. My only real concern at this point is that some of the new files have a copyright header identifying Junio as the copyright holder, but no license information. Meanwhile, the implicit license of git (GPLv2 only) is incompatible with that of notmuch (GPLv3+). I'm not lawyer, but I'd say it should be no problem to use GPLv2 test suite to test your GPLv3 application. You are not linking them together. On the other hand, if you want the tests cases to be GPLv3 then you are probably right. Is this the case? Michal ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [notmuch] A functional (but rudimentary) test suite for notmuch
On Fri, 12 Feb 2010 15:47:56 +0100, Michal Sojka sojk...@fel.cvut.cz wrote: I'm not lawyer, but I'd say it should be no problem to use GPLv2 test suite to test your GPLv3 application. You are not linking them together. Right. We could do that. But we'd have to be careful to document things carefully to keep track of things, (which the current patches at least, don't do). On the other hand, if you want the tests cases to be GPLv3 then you are probably right. Is this the case? I think things are a lot simpler if we keep everything in the repository under a single license. So I'd prefer that. -Carl pgpbi3d4r5W7x.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [notmuch] A functional (but rudimentary) test suite for notmuch
Hi Michal, I found myself today *really* needing to add a test that exercises some code in our emacs client. (Eric found that a message that includes a : in the From: address breaks things due to a buggy regexp.) And I wanted to see if I could merge your conversion of the test suite before I added a feature as big as that. See below for my feedback. -Carl On Mon, 8 Feb 2010 16:14:24 +0100, Michal Sojka sojk...@fel.cvut.cz wrote: I converted the actual version of notmuch-test to git test framework. The result is in the followup patches. Interesting. My only real concern at this point is that some of the new files have a copyright header identifying Junio as the copyright holder, but no license information. Meanwhile, the implicit license of git (GPLv2 only) is incompatible with that of notmuch (GPLv3+). So if you'd like to ask Junio for permission to re-use the files under the GPLv3+ then we could see what he says. In the meantime, you've done some nice modularization work here which we should be able to take right away, (and I can write tiny implementations of the functions we need so that we can just drop in git's test-lib.sh if we get permission). So we won't yet have any of the fancy features of the framework, (expected failures, reports of fixed tests, pre-requisites for tests), but we're not *using* any of those features yet so it shouldn't matter. The conversion was not as straightforward as I expected mainly because of problems with quoting. There are several sources of quotation problems one being Carl's hashed array parameters. I thing it would be sufficient to use plain variables for passing additional parameters. Instead of: add_message [from]=\Sender sen...@example.com\ \ [to]=test_su...@notmuchmail.org \ ... I'd do: msg_from=Sender sen...@example.com msg_to=test_su...@notmuchmail.org add_message Yeah. I almost wrote it that way, but named parameters (even if faked) seemed so much nicer. I'll agree that the quoting was quite nasty though. A possible additional improvement is elimination of execute_expecting(). Combination of action (running notmuch) and testing of a result in a single function makes it hard to distinguish whether the problem is in the action or in the output. A fair point. It also caused problems when I wanted to do tests that didn't directly test the output of notmuch, (such as a test that uses diff to compare a result against the contents in a file). -Carl pgpeQnT3z8ZkP.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [notmuch] A functional (but rudimentary) test suite for notmuch
On Thursday 04 of February 2010 21:50:18 Carl Worth wrote: The test suite is still extremely rudimentary. Here are some things I'd like to improve about it: I converted the actual version of notmuch-test to git test framework. The result is in the followup patches. I'd like to know opinion of others. If Carl agrees that it could be merged I can do additional work as I describe below. The conversion was not as straightforward as I expected mainly because of problems with quoting. There are several sources of quotation problems one being Carl's hashed array parameters. I thing it would be sufficient to use plain variables for passing additional parameters. Instead of: add_message [from]=\Sender sen...@example.com\ \ [to]=test_su...@notmuchmail.org \ [subject]=notmuch-reply-test \ [date]=\Tue, 05 Jan 2010 15:43:56 -0800\ \ [body]=\basic reply test\ I'd do: ( msg_from=Sender sen...@example.com msg_to=test_su...@notmuchmail.org msg_subject=notmuch-reply-test msg_date=Tue, 05 Jan 2010 15:43:56 -0800 msg_body=basic reply test add_message ) A possible additional improvement is elimination of execute_expecting(). Combination of action (running notmuch) and testing of a result in a single function makes it hard to distinguish whether the problem is in the action or in the output. For example, if notmuch is killed because of segfault, it would look like that no output was produced. So instead of: execute_expecting new No new mail. Removed 3 messages. I'd write something like: echo No new mail. Removed 3 messages. expected test_expect_success 'Run notmuch' 'notmuch_filter_output new actual' test_expect_success 'Compare actual and correct output' 'test_cmp expected actual' where test_cmp is basically diff -u. This has also the advantage that longer output (e.g. as in t0003-reply.sh) is more readable if there is a difference. On Thursday 04 of February 2010 22:27:52 Oliver Charles wrote: Carl, have you considered outputting the test suite in the same format as the test anything protocol? [1] I only mention this because it might be a nice way to easily do some reporting (or perhaps even continuous integration) notmuch, with trivial effort. According to http://testanything.org/wiki/index.php/TAP_Producers#Git_Project Git test output does not conform to TAP, but from my quick look it could be easily changed to conform. On Friday 05 of February 2010 00:29:27 Carl Worth wrote: Looking at TAP, one thing I don't like is that it prints the success/failure of the test first, before the description of the test. That's not so nice in the case of a long-running (perhaps infinitely running) test where you might need to interrupt it, but you'd still want to know *what* was running for so long. Git test framework also outputs first the result and then the test. It can be easily changed by -v switch. To conclude, the output of running make in test/ directory now looks like this: *** t-basic.sh *** * ok 1: test that mail store was created * ok 2: mail store should be empty * ok 3: NOTMUCH_CONFIG is set and points to an existing file * ok 4: PATH is set to this repository * ok 5: success is reported like this * still broken 6: pretend we have a known breakage * FIXED 7: pretend we have fixed a known breakage * ok 8: test runs if prerequisite is satisfied * skip 9: unmet prerequisite causes test to be skipped * fixed 1 known breakage(s) * still have 1 known breakage(s) * passed all remaining 8 test(s) *** t0001-new.sh *** * ok 1: No new messages * ok 2: Single new message * ok 3: Multiple new messages * ok 4: No new messages (non-empty DB) * ok 5: New directories * ok 6: Alternate inode order * ok 7: Message moved in * ok 8: Renamed message * ok 9: Deleted message * ok 10: Renamed directory * ok 11: Deleted directory * ok 12: New directory (at end of list) * ok 13: Deleted directory (end of list) * ok 14: New symlink to directory * ok 15: New symlink to a file * ok 16: New two-level directory * ok 17: Deleted two-level directory * passed all 17 test(s) *** t0002-search.sh *** * ok 1: Search body * ok 2: Search by from: * ok 3: Search by to: * ok 4: Search by subject: * ok 5: Search by id: * ok 6: Search by tag: * ok 7: Search by thread: * ok 8: Search body (phrase) * ok 9: Search by from: (address) * ok 10: Search by from: (name) * ok 11: Search by to: (address) * ok 12: Search by to: (name) * ok 13: Search by subject: (phrase) * passed all 13 test(s) *** t0003-reply.sh *** * ok 1: Basic reply * ok 2: Multiple recipients * ok 3: Reply with CC * ok 4: Reply from alternate address * ok 5: Support for Reply-To * ok 6: Un-munging Reply-To * passed all 6 test(s) *** t0004-uuencoded-data.sh *** * ok 1: Generate message * ok 2: Ensure content before uu data is indexed * ok 3: Ensure uu
