[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-12 Thread Michal Sojka
On Thu, 11 Feb 2010 13:42:08 -0800, Carl Worth  wrote:
> Interesting. My only real concern at this point is that some of the new files
> have a copyright header identifying Junio as the copyright holder, but
> no license information. Meanwhile, the implicit license of git (GPLv2
> only) is incompatible with that of notmuch (GPLv3+).

I'm not lawyer, but I'd say it should be no problem to use GPLv2 test
suite to test your GPLv3 application. You are not linking them
together.

On the other hand, if you want the tests cases to be GPLv3 then you are
probably right. Is this the case?


Michal


[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-12 Thread Carl Worth
On Fri, 12 Feb 2010 15:47:56 +0100, Michal Sojka  wrote:
> I'm not lawyer, but I'd say it should be no problem to use GPLv2 test
> suite to test your GPLv3 application. You are not linking them
> together.

Right. We could do that. But we'd have to be careful to document things
carefully to keep track of things, (which the current patches at least,
don't do).

> On the other hand, if you want the tests cases to be GPLv3 then you are
> probably right. Is this the case?

I think things are a lot simpler if we keep everything in the repository
under a single license. So I'd prefer that.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 



Re: [notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-12 Thread Michal Sojka
On Thu, 11 Feb 2010 13:42:08 -0800, Carl Worth cwo...@cworth.org wrote:
 Interesting. My only real concern at this point is that some of the new files
 have a copyright header identifying Junio as the copyright holder, but
 no license information. Meanwhile, the implicit license of git (GPLv2
 only) is incompatible with that of notmuch (GPLv3+).

I'm not lawyer, but I'd say it should be no problem to use GPLv2 test
suite to test your GPLv3 application. You are not linking them
together.

On the other hand, if you want the tests cases to be GPLv3 then you are
probably right. Is this the case?


Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-12 Thread Carl Worth
On Fri, 12 Feb 2010 15:47:56 +0100, Michal Sojka sojk...@fel.cvut.cz wrote:
 I'm not lawyer, but I'd say it should be no problem to use GPLv2 test
 suite to test your GPLv3 application. You are not linking them
 together.

Right. We could do that. But we'd have to be careful to document things
carefully to keep track of things, (which the current patches at least,
don't do).

 On the other hand, if you want the tests cases to be GPLv3 then you are
 probably right. Is this the case?

I think things are a lot simpler if we keep everything in the repository
under a single license. So I'd prefer that.

-Carl


pgpbi3d4r5W7x.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-11 Thread Carl Worth
Hi Michal,

I found myself today *really* needing to add a test that exercises some
code in our emacs client. (Eric found that a message that includes a ":"
in the From: address breaks things due to a buggy regexp.) And I
wanted to see if I could merge your conversion of the test suite before
I added a feature as big as that. See below for my feedback.

-Carl

On Mon, 8 Feb 2010 16:14:24 +0100, Michal Sojka  wrote:
> I converted the actual version of notmuch-test to git test framework.
> The result is in the followup patches.

Interesting. My only real concern at this point is that some of the new files
have a copyright header identifying Junio as the copyright holder, but
no license information. Meanwhile, the implicit license of git (GPLv2
only) is incompatible with that of notmuch (GPLv3+).

So if you'd like to ask Junio for permission to re-use the files under
the GPLv3+ then we could see what he says.

In the meantime, you've done some nice modularization work here which we
should be able to take right away, (and I can write tiny implementations
of the functions we need so that we can just drop in git's test-lib.sh
if we get permission).

So we won't yet have any of the fancy features of the framework,
(expected failures, reports of fixed tests, pre-requisites for tests),
but we're not *using* any of those features yet so it shouldn't matter.

> The conversion was not as straightforward as I expected mainly because
> of problems with quoting. There are several sources of quotation problems
> one being Carl's hashed array parameters. I thing it would be
> sufficient to use plain variables for passing additional parameters.
> Instead of:
> add_message [from]="\"Sender \"" \
> [to]=test_suite at notmuchmail.org \
...
> I'd do:
> msg_from="Sender "
> msg_to=test_suite at notmuchmail.org 
> add_message

Yeah. I almost wrote it that way, but named parameters (even if faked)
seemed so much nicer. I'll agree that the quoting was quite nasty
though.

> A possible additional improvement is elimination of
> execute_expecting(). Combination of action (running notmuch) and
> testing of a result in a single function makes it hard to distinguish
> whether the problem is in the action or in the output.

A fair point. It also caused problems when I wanted to do tests that
didn't directly test the output of notmuch, (such as a test that uses
diff to compare a result against the contents in a file).

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 



Re: [notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-11 Thread Carl Worth
Hi Michal,

I found myself today *really* needing to add a test that exercises some
code in our emacs client. (Eric found that a message that includes a :
in the From: address breaks things due to a buggy regexp.) And I
wanted to see if I could merge your conversion of the test suite before
I added a feature as big as that. See below for my feedback.

-Carl

On Mon, 8 Feb 2010 16:14:24 +0100, Michal Sojka sojk...@fel.cvut.cz wrote:
 I converted the actual version of notmuch-test to git test framework.
 The result is in the followup patches.

Interesting. My only real concern at this point is that some of the new files
have a copyright header identifying Junio as the copyright holder, but
no license information. Meanwhile, the implicit license of git (GPLv2
only) is incompatible with that of notmuch (GPLv3+).

So if you'd like to ask Junio for permission to re-use the files under
the GPLv3+ then we could see what he says.

In the meantime, you've done some nice modularization work here which we
should be able to take right away, (and I can write tiny implementations
of the functions we need so that we can just drop in git's test-lib.sh
if we get permission).

So we won't yet have any of the fancy features of the framework,
(expected failures, reports of fixed tests, pre-requisites for tests),
but we're not *using* any of those features yet so it shouldn't matter.

 The conversion was not as straightforward as I expected mainly because
 of problems with quoting. There are several sources of quotation problems
 one being Carl's hashed array parameters. I thing it would be
 sufficient to use plain variables for passing additional parameters.
 Instead of:
 add_message [from]=\Sender sen...@example.com\ \
 [to]=test_su...@notmuchmail.org \
...
 I'd do:
 msg_from=Sender sen...@example.com
 msg_to=test_su...@notmuchmail.org 
 add_message

Yeah. I almost wrote it that way, but named parameters (even if faked)
seemed so much nicer. I'll agree that the quoting was quite nasty
though.

 A possible additional improvement is elimination of
 execute_expecting(). Combination of action (running notmuch) and
 testing of a result in a single function makes it hard to distinguish
 whether the problem is in the action or in the output.

A fair point. It also caused problems when I wanted to do tests that
didn't directly test the output of notmuch, (such as a test that uses
diff to compare a result against the contents in a file).

-Carl


pgpeQnT3z8ZkP.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-08 Thread Michal Sojka
On Thursday 04 of February 2010 21:50:18 Carl Worth wrote:
> The test suite is still extremely rudimentary. Here are some things I'd
> like to improve about it:

I converted the actual version of notmuch-test to git test framework.
The result is in the followup patches.

I'd like to know opinion of others. If Carl agrees that it could be
merged I can do additional work as I describe below.

The conversion was not as straightforward as I expected mainly because
of problems with quoting. There are several sources of quotation problems
one being Carl's hashed array parameters. I thing it would be
sufficient to use plain variables for passing additional parameters.
Instead of:
add_message [from]="\"Sender \"" \
[to]=test_suite at notmuchmail.org \
[subject]=notmuch-reply-test \
[date]="\"Tue, 05 Jan 2010 15:43:56 -0800\"" \
[body]="\"basic reply test\"" &&
I'd do:
(
msg_from="Sender "
msg_to=test_suite at notmuchmail.org 
msg_subject=notmuch-reply-test
msg_date="Tue, 05 Jan 2010 15:43:56 -0800"
msg_body="basic reply test"
add_message
)

A possible additional improvement is elimination of
execute_expecting(). Combination of action (running notmuch) and
testing of a result in a single function makes it hard to distinguish
whether the problem is in the action or in the output. For example, if
notmuch is killed because of segfault, it would look like that no
output was produced. So instead of:

execute_expecting new "No new mail. Removed 3 messages."

I'd write something like:

echo "No new mail. Removed 3 messages." > expected

  test_expect_success 'Run notmuch'
'notmuch_filter_output new > actual'
  test_expect_success 'Compare actual and correct output'
'test_cmp expected actual'

where test_cmp is basically diff -u. This has also the advantage that
longer output (e.g. as in t0003-reply.sh) is more readable if there is
a difference.

On Thursday 04 of February 2010 22:27:52 Oliver Charles wrote:
> Carl, have you considered outputting the test suite in the same format
> as the test anything protocol? [1] I only mention this because it
> might be a nice way to easily do some reporting (or perhaps even
> continuous integration) notmuch, with trivial effort.

According to http://testanything.org/wiki/index.php/TAP_Producers#Git_Project 
Git test output does not conform to TAP, but from my quick look it could be 
easily changed to conform.


On Friday 05 of February 2010 00:29:27 Carl Worth wrote:
> Looking at TAP, one thing I don't like is that it prints the
> success/failure of the test first, before the description of the
> test. That's not so nice in the case of a long-running (perhaps
> infinitely running) test where you might need to interrupt it, but you'd
> still want to know *what* was running for so long.

Git test framework also outputs first the result and then the test. It can be 
easily changed by -v switch.

To conclude, the output of running make in test/ directory now looks like 
this:
*** t-basic.sh ***
*   ok 1: test that mail store was created
*   ok 2: mail store should be empty
*   ok 3: NOTMUCH_CONFIG is set and points to an existing file
*   ok 4: PATH is set to this repository
*   ok 5: success is reported like this
*   still broken 6: pretend we have a known breakage
*   FIXED 7: pretend we have fixed a known breakage
*   ok 8: test runs if prerequisite is satisfied
* skip 9: unmet prerequisite causes test to be skipped
* fixed 1 known breakage(s)
* still have 1 known breakage(s)
* passed all remaining 8 test(s)
*** t0001-new.sh ***
*   ok 1: No new messages
*   ok 2: Single new message
*   ok 3: Multiple new messages
*   ok 4: No new messages (non-empty DB)
*   ok 5: New directories
*   ok 6: Alternate inode order
*   ok 7: Message moved in
*   ok 8: Renamed message
*   ok 9: Deleted message
*   ok 10: Renamed directory
*   ok 11: Deleted directory
*   ok 12: New directory (at end of list)
*   ok 13: Deleted directory (end of list)
*   ok 14: New symlink to directory
*   ok 15: New symlink to a file
*   ok 16: New two-level directory
*   ok 17: Deleted two-level directory
* passed all 17 test(s)
*** t0002-search.sh ***
*   ok 1: Search body
*   ok 2: Search by from:
*   ok 3: Search by to:
*   ok 4: Search by subject:
*   ok 5: Search by id:
*   ok 6: Search by tag:
*   ok 7: Search by thread:
*   ok 8: Search body (phrase)
*   ok 9: Search by from: (address)
*   ok 10: Search by from: (name)
*   ok 11: Search by to: (address)
*   ok 12: Search by to: (name)
*   ok 13: Search by subject: (phrase)
* passed all 13 test(s)
*** t0003-reply.sh ***
*   ok 1: Basic reply
*   ok 2: Multiple recipients
*   ok 3: Reply with CC
*   ok 4: Reply from alternate address
*   ok 5: Support for Reply-To
*   ok 6: Un-munging Reply-To
* passed all 6 test(s)
*** t0004-uuencoded-data.sh ***
*   ok 1: Generate message
*   ok 2: Ensure content before uu data is indexed
*   ok 3: Ensure 

Re: [notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-08 Thread Michal Sojka
On Thursday 04 of February 2010 21:50:18 Carl Worth wrote:
 The test suite is still extremely rudimentary. Here are some things I'd
 like to improve about it:

I converted the actual version of notmuch-test to git test framework.
The result is in the followup patches.

I'd like to know opinion of others. If Carl agrees that it could be
merged I can do additional work as I describe below.

The conversion was not as straightforward as I expected mainly because
of problems with quoting. There are several sources of quotation problems
one being Carl's hashed array parameters. I thing it would be
sufficient to use plain variables for passing additional parameters.
Instead of:
add_message [from]=\Sender sen...@example.com\ \
[to]=test_su...@notmuchmail.org \
[subject]=notmuch-reply-test \
[date]=\Tue, 05 Jan 2010 15:43:56 -0800\ \
[body]=\basic reply test\ 
I'd do:
(
msg_from=Sender sen...@example.com
msg_to=test_su...@notmuchmail.org 
msg_subject=notmuch-reply-test
msg_date=Tue, 05 Jan 2010 15:43:56 -0800
msg_body=basic reply test
add_message
)

A possible additional improvement is elimination of
execute_expecting(). Combination of action (running notmuch) and
testing of a result in a single function makes it hard to distinguish
whether the problem is in the action or in the output. For example, if
notmuch is killed because of segfault, it would look like that no
output was produced. So instead of:

execute_expecting new No new mail. Removed 3 messages.

I'd write something like:

echo No new mail. Removed 3 messages.  expected

  test_expect_success 'Run notmuch'
'notmuch_filter_output new  actual'
  test_expect_success 'Compare actual and correct output'
'test_cmp expected actual'

where test_cmp is basically diff -u. This has also the advantage that
longer output (e.g. as in t0003-reply.sh) is more readable if there is
a difference.

On Thursday 04 of February 2010 22:27:52 Oliver Charles wrote:
 Carl, have you considered outputting the test suite in the same format
 as the test anything protocol? [1] I only mention this because it
 might be a nice way to easily do some reporting (or perhaps even
 continuous integration) notmuch, with trivial effort.

According to http://testanything.org/wiki/index.php/TAP_Producers#Git_Project 
Git test output does not conform to TAP, but from my quick look it could be 
easily changed to conform.


On Friday 05 of February 2010 00:29:27 Carl Worth wrote:
 Looking at TAP, one thing I don't like is that it prints the
 success/failure of the test first, before the description of the
 test. That's not so nice in the case of a long-running (perhaps
 infinitely running) test where you might need to interrupt it, but you'd
 still want to know *what* was running for so long.

Git test framework also outputs first the result and then the test. It can be 
easily changed by -v switch.

To conclude, the output of running make in test/ directory now looks like 
this:
*** t-basic.sh ***
*   ok 1: test that mail store was created
*   ok 2: mail store should be empty
*   ok 3: NOTMUCH_CONFIG is set and points to an existing file
*   ok 4: PATH is set to this repository
*   ok 5: success is reported like this
*   still broken 6: pretend we have a known breakage
*   FIXED 7: pretend we have fixed a known breakage
*   ok 8: test runs if prerequisite is satisfied
* skip 9: unmet prerequisite causes test to be skipped
* fixed 1 known breakage(s)
* still have 1 known breakage(s)
* passed all remaining 8 test(s)
*** t0001-new.sh ***
*   ok 1: No new messages
*   ok 2: Single new message
*   ok 3: Multiple new messages
*   ok 4: No new messages (non-empty DB)
*   ok 5: New directories
*   ok 6: Alternate inode order
*   ok 7: Message moved in
*   ok 8: Renamed message
*   ok 9: Deleted message
*   ok 10: Renamed directory
*   ok 11: Deleted directory
*   ok 12: New directory (at end of list)
*   ok 13: Deleted directory (end of list)
*   ok 14: New symlink to directory
*   ok 15: New symlink to a file
*   ok 16: New two-level directory
*   ok 17: Deleted two-level directory
* passed all 17 test(s)
*** t0002-search.sh ***
*   ok 1: Search body
*   ok 2: Search by from:
*   ok 3: Search by to:
*   ok 4: Search by subject:
*   ok 5: Search by id:
*   ok 6: Search by tag:
*   ok 7: Search by thread:
*   ok 8: Search body (phrase)
*   ok 9: Search by from: (address)
*   ok 10: Search by from: (name)
*   ok 11: Search by to: (address)
*   ok 12: Search by to: (name)
*   ok 13: Search by subject: (phrase)
* passed all 13 test(s)
*** t0003-reply.sh ***
*   ok 1: Basic reply
*   ok 2: Multiple recipients
*   ok 3: Reply with CC
*   ok 4: Reply from alternate address
*   ok 5: Support for Reply-To
*   ok 6: Un-munging Reply-To
* passed all 6 test(s)
*** t0004-uuencoded-data.sh ***
*   ok 1: Generate message
*   ok 2: Ensure content before uu data is indexed
*   ok 3: Ensure uu 

[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-06 Thread Oliver Charles
On Thu, Feb 4, 2010 at 11:29 PM, Carl Worth  wrote:
> Looking at TAP, one thing I don't like is that it prints the
> success/failure of the test first, before the description of the
> test. That's not so nice in the case of a long-running (perhaps
> infinitely running) test where you might need to interrupt it, but you'd
> still want to know *what* was running for so long.

Ah, the way this works for Test::Harness in Perl is that while a test
suite is running the output is of the form:

t/edit_artist_add_alias.t .. 1/?

And then when it completes it will change to:

t/edit_artist_add_alias.t .. OK

So you can see stuff that's failing to terminate (as I just found
while running my test suites ;))

-- 
Oliver Charles / aCiD2


[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-04 Thread Oliver Charles
Carl, have you considered outputting the test suite in the same format
as the test anything protocol? [1] I only mention this because it
might be a nice way to easily do some reporting (or perhaps even
continuous integration) notmuch, with trivial effort.

-- 
Oliver Charles / aCiD2

[1]: http://en.wikipedia.org/wiki/Test_Anything_Protocol


[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-04 Thread Carl Worth
On Thu, 4 Feb 2010 21:27:52 +, Oliver Charles  wrote:
> Carl, have you considered outputting the test suite in the same format
> as the test anything protocol? [1] I only mention this because it
> might be a nice way to easily do some reporting (or perhaps even
> continuous integration) notmuch, with trivial effort.
...
> [1]: http://en.wikipedia.org/wiki/Test_Anything_Protocol

No, I hadn't considered that. But perhaps because I wasn't at all aware
of TAP.

It looks like it would be quite trivial to switch to an output format
like that if somebody had something they'd like to hook the notmuch test
suite up to, (but I wouldn't likely switch before).

Looking at TAP, one thing I don't like is that it prints the
success/failure of the test first, before the description of the
test. That's not so nice in the case of a long-running (perhaps
infinitely running) test where you might need to interrupt it, but you'd
still want to know *what* was running for so long.

-Carl
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: 



[notmuch] A functional (but rudimentary) test suite for notmuch

2010-02-04 Thread Carl Worth
I've just pushed out some improvements to make the little notmuch-test
script I had posted earlier into an actual test suite. I've included the
output of a run of the test suite below so that you can get an idea of
what it does so far.

What I'd like to see going forward is that new features get submitted
along with new test cases for the test suite. And it would be great if
everyone submitting patches could get into the habit of running:

./test/notmuch-test

before submitting.

The test suite is still extremely rudimentary. Here are some things I'd
like to improve about it:

  * Make the test suite more modular. Right now it's just one long shell
script---so it's not even easy to see where one test ends and the
next begins.

  * Allow for executing a subset of the test suite, (eg. notmuch-test reply)

I just wrote a bunch of "notmuch reply" tests and it was annoying to
have to wait for all of the "notmuch new" tests to run each time.

  * Actually document what it takes to write a test, (what the test can
assume about the environment in which its running, what functions
are available to call, etc.)

I hope others find this useful.

-Carl

Testing "notmuch new" in several variations:
 No new messages... PASS
 Single new message...  PASS
 Multiple new messages...   PASS
 No new messages (non-empty DB)...  PASS
 New directories... PASS
 Alternate inode order...   PASS
 Message moved in...PASS
 Renamed message... PASS
 Deleted message... PASS
 Renamed directory...   PASS
 Deleted directory...   PASS
 New directory (at end of list)...  PASS
 Deleted directory (end of list)... PASS
 New symlink to directory...PASS
 New symlink to a file...   PASS
 New two-level directory... PASS
 Deleted two-level directory... PASS

Testing "notmuch reply" in several variations:
 Basic reply... PASS
 Multiple recipients... PASS
 Reply with CC...   PASS
 Reply from alternate address...PASS
 Support for Reply-To...PASS
 Un-munging Reply-To... PASS
Notmuch test suite complete.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: