notmuch.readthedocs.org

2014-11-18 Thread Ben Gamari
Justus Winter <4winter at informatik.uni-hamburg.de> writes:

> Hey Sebastian,
>
> Quoting Sebastian Spaeth (2014-11-18 09:44:15)
>> Let me know if I should be deleting the site. Or hand it over, or
>> something :-).
>
> If I remember correctly, you handed notmuch.rtfd.org over to me.  Or
> you deleted the project, and I created a new one in its place, I don't
> recall.
>
> Ben, I added you as maintainer.  I cannot, however, seem to be able to
> remove myself as maintainer.
>
No worries.

I've updated the description and added notmuch-python as a
subproject. For what it's worth the notmuch-python documentation can now
be accessed at
http://notmuch.rtfd.org/projects/notmuch-python/en/latest/ in addition
to its usual home at http://notmuch-python.rtfd.org/

- Ben
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 472 bytes
Desc: not available
URL: 



Re: notmuch.readthedocs.org

2014-11-18 Thread Ben Gamari
Justus Winter 4win...@informatik.uni-hamburg.de writes:

 Hey Sebastian,

 Quoting Sebastian Spaeth (2014-11-18 09:44:15)
 Let me know if I should be deleting the site. Or hand it over, or
 something :-).

 If I remember correctly, you handed notmuch.rtfd.org over to me.  Or
 you deleted the project, and I created a new one in its place, I don't
 recall.

 Ben, I added you as maintainer.  I cannot, however, seem to be able to
 remove myself as maintainer.

No worries.

I've updated the description and added notmuch-python as a
subproject. For what it's worth the notmuch-python documentation can now
be accessed at
http://notmuch.rtfd.org/projects/notmuch-python/en/latest/ in addition
to its usual home at http://notmuch-python.rtfd.org/

- Ben


pgpeKpgZZlHEh.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


notmuch.readthedocs.org

2014-11-17 Thread Ben Gamari
Justus Winter <4winter at informatik.uni-hamburg.de> writes:

> Quoting David Bremner (2014-11-16 21:23:21)
>> I'm not an admin of the readthedocs project. I guess Justus is?
>> The seperate project approach seems good enough to me too. Justus
>> (assuming you are an admin), how about adding Ben so he fix things up a
>> bit?
>
> Sure thing! Ben, what's your username at rtfd?
>
bgamari

Thanks,

- Ben

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 472 bytes
Desc: not available
URL: 



Re: notmuch.readthedocs.org

2014-11-17 Thread Ben Gamari
Justus Winter 4win...@informatik.uni-hamburg.de writes:

 Quoting David Bremner (2014-11-16 21:23:21)
 I'm not an admin of the readthedocs project. I guess Justus is?
 The seperate project approach seems good enough to me too. Justus
 (assuming you are an admin), how about adding Ben so he fix things up a
 bit?

 Sure thing! Ben, what's your username at rtfd?

bgamari

Thanks,

- Ben



pgp_YAFRYgwxP.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


notmuch.readthedocs.org

2014-11-16 Thread Ben Gamari
David Bremner  writes:

> It seems that this site has bitrotted.  I hit the build button and it
> now builds the CLI docs instead of the python bindings.  I assume since
> Sebastian has moved on from maintaining the bindings he's also not
> intersted in maintaining this site. If someone would like to take over
> and fix what needs to be fixed, then great, otherwise I'll ask the
> readthedocs admins to purge the site. It seems like it should be
> possible to have CLI and python docs in one sphinx tree, so maybe that's
> something we can do upstream to help.
>
Also, I just noticed that readthedocs supports a notion of
subprojects. Perhaps try adding notmuch-python to the subprojects list
of notmuch. I'm frankly not sure what the subproject relationship
entails beyond exposing notmuch-python in the notmuch.readthedocs.org
subdomain.

Cheers,

- Ben
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 472 bytes
Desc: not available
URL: 



notmuch.readthedocs.org

2014-11-16 Thread Ben Gamari
David Bremner  writes:

> It seems that this site has bitrotted.  I hit the build button and it
> now builds the CLI docs instead of the python bindings.  I assume since
> Sebastian has moved on from maintaining the bindings he's also not
> intersted in maintaining this site. If someone would like to take over
> and fix what needs to be fixed, then great, otherwise I'll ask the
> readthedocs admins to purge the site. It seems like it should be
> possible to have CLI and python docs in one sphinx tree, so maybe that's
> something we can do upstream to help.
>
I have set up a separate project for the Python bindings here [1]. I
looked into how one might convince Sphinx to build both sets of
documentation in one run but it seems this is not trivial without unduly
coupling the two. Having two separate projects seems like a fair
solution for now.

David, perhaps you could update the description of the notmuch RtD
project to reflect that fact that it is no longer describing the Python
bindings? Perhaps it might also make sense to add a link to the
notmuch-python project in the project description. I'd be happy to keep
an eye on the notmuch project if you want to add me to the maintainers
list.

Cheers,

- Ben


[1] http://notmuch-python.readthedocs.org/en/latest/
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 472 bytes
Desc: not available
URL: 



Re: notmuch.readthedocs.org

2014-11-16 Thread Ben Gamari
David Bremner da...@tethera.net writes:

 It seems that this site has bitrotted.  I hit the build button and it
 now builds the CLI docs instead of the python bindings.  I assume since
 Sebastian has moved on from maintaining the bindings he's also not
 intersted in maintaining this site. If someone would like to take over
 and fix what needs to be fixed, then great, otherwise I'll ask the
 readthedocs admins to purge the site. It seems like it should be
 possible to have CLI and python docs in one sphinx tree, so maybe that's
 something we can do upstream to help.

I have set up a separate project for the Python bindings here [1]. I
looked into how one might convince Sphinx to build both sets of
documentation in one run but it seems this is not trivial without unduly
coupling the two. Having two separate projects seems like a fair
solution for now.

David, perhaps you could update the description of the notmuch RtD
project to reflect that fact that it is no longer describing the Python
bindings? Perhaps it might also make sense to add a link to the
notmuch-python project in the project description. I'd be happy to keep
an eye on the notmuch project if you want to add me to the maintainers
list.

Cheers,

- Ben


[1] http://notmuch-python.readthedocs.org/en/latest/


pgp5CxvL2mmCj.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: notmuch.readthedocs.org

2014-11-16 Thread Ben Gamari
David Bremner da...@tethera.net writes:

 It seems that this site has bitrotted.  I hit the build button and it
 now builds the CLI docs instead of the python bindings.  I assume since
 Sebastian has moved on from maintaining the bindings he's also not
 intersted in maintaining this site. If someone would like to take over
 and fix what needs to be fixed, then great, otherwise I'll ask the
 readthedocs admins to purge the site. It seems like it should be
 possible to have CLI and python docs in one sphinx tree, so maybe that's
 something we can do upstream to help.

Also, I just noticed that readthedocs supports a notion of
subprojects. Perhaps try adding notmuch-python to the subprojects list
of notmuch. I'm frankly not sure what the subproject relationship
entails beyond exposing notmuch-python in the notmuch.readthedocs.org
subdomain.

Cheers,

- Ben


pgpiFNfy_E7kn.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] Add NEWS entry for notmuch-compact

2013-11-20 Thread Ben Gamari
Signed-off-by: Ben Gamari 
---
 NEWS | 8 
 1 file changed, 8 insertions(+)

diff --git a/NEWS b/NEWS
index dbbbe9f..314d518 100644
--- a/NEWS
+++ b/NEWS
@@ -30,6 +30,14 @@ Improved `notmuch new` performance for unchanged folders
   RFC 2047-encodes the headers, making the output a valid RFC 2822
   message.  The JSON/sexp format is unchanged.

+`notmuch compact` command
+
+  The new `compact` command exposes Xapian's compaction
+  functionality through a more convenient interface than
+  `xapian-compact`. `notmuch compact` will compact the database to a
+  temporary location, optionally backup the original database, and
+  move the compacted database into place.
+
 Emacs Interface
 ---

-- 
1.8.3.2



[PATCH] notmuch compact support (v4)

2013-11-20 Thread Ben Gamari
Does this look sufficient?

Thanks,

- Ben




Re: [PATCH] notmuch compact support (v4)

2013-11-20 Thread Ben Gamari
Does this look sufficient?

Thanks,

- Ben


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] Add NEWS entry for notmuch-compact

2013-11-20 Thread Ben Gamari
Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 NEWS | 8 
 1 file changed, 8 insertions(+)

diff --git a/NEWS b/NEWS
index dbbbe9f..314d518 100644
--- a/NEWS
+++ b/NEWS
@@ -30,6 +30,14 @@ Improved `notmuch new` performance for unchanged folders
   RFC 2047-encodes the headers, making the output a valid RFC 2822
   message.  The JSON/sexp format is unchanged.
 
+`notmuch compact` command
+
+  The new `compact` command exposes Xapian's compaction
+  functionality through a more convenient interface than
+  `xapian-compact`. `notmuch compact` will compact the database to a
+  temporary location, optionally backup the original database, and
+  move the compacted database into place.
+
 Emacs Interface
 ---
 
-- 
1.8.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH v2 00/11] compactor changes v2

2013-11-05 Thread Ben Gamari
Tomi Ollila  writes:

> On Tue, Nov 05 2013, David Bremner  wrote:
>
>> Jani Nikula  writes:
>>
>>> Hi all, this is v2 of [1], incorporating compact related patches from
>>> David, some changes from David's review, some new patches, man page
>>> update, test update.
>>>
>>> Cheers,
>>> Jani.
>>
>> Aside from the one quibble that I already sent seperately, the first 8
>> patches look OK to me. I have no technical objections to the last 3, but
>> since the change the UI, I'd like to wait a bit for more input from
>> other people.
>
>
> The whole series looks OK to me. The only thing I'm, a bit worried about
> what happens is I Ctrl-C compaction at any state. Is it possible that I
> end up with corrupted database ? Someone may accidentally execute
> notmuch compact -- for example by picking the command from history.
>
You should in principle never end up in a situation where the original
database is corrupted. Xapian's Compactor interface reads from the
existing database and writes out the compacted database to a new
directory. The notmuch compact command keeps the original database in
place until the last possible moment.

When the compacted database is completely constructed it, deletes or
moves the old database out of the way and then moves the new database
into place. The worst thing that could happen is that you manage to
interrupt the process between deleting the old database and moving the
new one into place (a very small window). In this case you'd simply need
to move the new database into place manually (although you'd need to
figure out where the compacted database is located).

Cheers,

- Ben

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



[PATCH v2 00/11] compactor changes v2

2013-11-05 Thread Ben Gamari
Jani Nikula  writes:

> Hi all, this is v2 of [1], incorporating compact related patches from
> David, some changes from David's review, some new patches, man page
> update, test update.
>
Alas, more silly oversights on my part. Thanks for catching these!

No objections from my side. Given that I've never seen a compact fail, I
think switching off backups by default seems fine. Personally I'll
likely always run with --verbose as even with an SSD compact can take
quite a while and there's nothing like watching a teapot to pass the
day. Regardless, I have no objection to the flag's existence.

Cheers,

- Ben

-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re: [PATCH v2 00/11] compactor changes v2

2013-11-05 Thread Ben Gamari
Jani Nikula j...@nikula.org writes:

 Hi all, this is v2 of [1], incorporating compact related patches from
 David, some changes from David's review, some new patches, man page
 update, test update.

Alas, more silly oversights on my part. Thanks for catching these!

No objections from my side. Given that I've never seen a compact fail, I
think switching off backups by default seems fine. Personally I'll
likely always run with --verbose as even with an SSD compact can take
quite a while and there's nothing like watching a teapot to pass the
day. Regardless, I have no objection to the flag's existence.

Cheers,

- Ben



pgpznIRgGDxug.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH v2 00/11] compactor changes v2

2013-11-05 Thread Ben Gamari
Tomi Ollila tomi.oll...@iki.fi writes:

 On Tue, Nov 05 2013, David Bremner da...@tethera.net wrote:

 Jani Nikula j...@nikula.org writes:

 Hi all, this is v2 of [1], incorporating compact related patches from
 David, some changes from David's review, some new patches, man page
 update, test update.

 Cheers,
 Jani.

 Aside from the one quibble that I already sent seperately, the first 8
 patches look OK to me. I have no technical objections to the last 3, but
 since the change the UI, I'd like to wait a bit for more input from
 other people.


 The whole series looks OK to me. The only thing I'm, a bit worried about
 what happens is I Ctrl-C compaction at any state. Is it possible that I
 end up with corrupted database ? Someone may accidentally execute
 notmuch compact -- for example by picking the command from history.

You should in principle never end up in a situation where the original
database is corrupted. Xapian's Compactor interface reads from the
existing database and writes out the compacted database to a new
directory. The notmuch compact command keeps the original database in
place until the last possible moment.

When the compacted database is completely constructed it, deletes or
moves the old database out of the way and then moves the new database
into place. The worst thing that could happen is that you manage to
interrupt the process between deleting the old database and moving the
new one into place (a very small window). In this case you'd simply need
to move the new database into place manually (although you'd need to
figure out where the compacted database is located).
  
Cheers,

- Ben



pgpgVxTz1otii.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 4/4] database: Handle error while deleting uncompacted database

2013-11-04 Thread Ben Gamari
David Bremner  writes:

> Ben Gamari  writes:
>
>> We never checked to ensure that the rmtree() of the old database
>> succeeded.
>>
>
> Is there some reason these patches (and the existing compact code in the
> lib) is using fprintf and not the status callback function? Most places
> we don't (yet) have logging callbacks, but here we do.
>
Nope. I simply wasn't aware of this infrastructure. I can put together a
patch to fix this.

Cheers,

- Ben
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20131104/eca30aa1/attachment.pgp>


Re: [PATCH 4/4] database: Handle error while deleting uncompacted database

2013-11-04 Thread Ben Gamari
David Bremner da...@tethera.net writes:

 Ben Gamari bgamari.f...@gmail.com writes:

 We never checked to ensure that the rmtree() of the old database
 succeeded.


 Is there some reason these patches (and the existing compact code in the
 lib) is using fprintf and not the status callback function? Most places
 we don't (yet) have logging callbacks, but here we do.

Nope. I simply wasn't aware of this infrastructure. I can put together a
patch to fix this.

Cheers,

- Ben


pgpvJAZVnBQDT.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/4] test: Add compact test

2013-10-31 Thread Ben Gamari
Signed-off-by: Ben Gamari 
---
 test/compact  | 32 
 test/notmuch-test |  1 +
 2 files changed, 33 insertions(+)
 create mode 100755 test/compact

diff --git a/test/compact b/test/compact
new file mode 100755
index 000..5bb5cea
--- /dev/null
+++ b/test/compact
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+test_description='"notmuch compact"'
+. ./test-lib.sh
+
+add_message '[subject]=One'
+add_message '[subject]=Two'
+add_message '[subject]=Three'
+
+notmuch tag +tag1 \*
+notmuch tag +tag2 subject:Two
+notmuch tag -tag1 +tag3 subject:Three
+
+test_expect_success "Running compact" "notmuch compact"
+
+test_begin_subtest "Compact preserves database"
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal "$output" "\
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
+
+test_begin_subtest "Restoring backup"
+rm -Rf ${TEST_TMPDIR}/mail/xapian
+mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
+
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal "$output" "\
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index aa28bb0..ec94baf 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -19,6 +19,7 @@ cd $(dirname "$0")
 TESTS="
   basic
   help-test
+  compact
   config
   setup
   new
-- 
1.8.3.2



[PATCH 1/4] test: Add compact test

2013-10-31 Thread Ben Gamari
Tomi Ollila  writes:

> On Tue, Oct 29 2013, Ben Gamari  wrote:
>
>> Signed-off-by: Ben Gamari 
>> ---
>
> Patches 2, 3 & 4 Looks OK To Me. Thanks. A few comments on this patch 
> inline:
>
Thanks again for the review!

>>  test/compact  | 35 +++
>>  test/notmuch-test |  1 +
>>  2 files changed, 36 insertions(+)
>>  create mode 100755 test/compact
>>
>> diff --git a/test/compact b/test/compact
>> new file mode 100755
>> index 000..54e85ab
>> --- /dev/null
>> +++ b/test/compact
>> @@ -0,0 +1,35 @@
>> +#!/usr/bin/env bash
>> +test_description='"notmuch compact"'
>> +. ./test-lib.sh
>> +
>> +add_message '[subject]=One'
>> +add_message '[subject]=Two'
>> +add_message '[subject]=Three'
>> +
>> +notmuch tag +tag1 \*
>> +notmuch tag +tag2 subject:Two
>> +notmuch tag -tag1 +tag3 subject:Three
>> +
>> +test_begin_subtest "Compacting"
>> +notmuch compact
>> +test_expect_success "compact" "notmuch compact"
>
> test_expect_success executes "$2" ("notmuch compact" in this case)
> do you mean to run notmuch 'compact twice' ?
>
It's been a while but I suspect I just never cleaned up the patch after
coming to this realization. Anyways, it's fixed in the revised patch
coming shortly.

>> +notmuch search \*
>> +output=$(notmuch search \* | notmuch_search_sanitize)
>
> Now do you mean to run 'notmuch search \*' twice ?
>
>> +test_expect_equal "$output" "\
>> +thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
>> +thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 
>> unread)
>> +thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
>> +
>> +test_begin_subtest "Restoring backup"
>> +rm -Rf ${TEST_TMPDIR}/mail/xapian
>> +mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
>> +
>> +notmuch search \*
>> +output=$(notmuch search \* | notmuch_search_sanitize)
>
> again... ?
>
>
> ... actually David Bremner had even more insightful comments in 
> id:874n8cw2yq.fsf at zancas.localnet to these same issues :D
>
Hmm, it seems I overlooked these. Thanks for the reference.

Cheers,

- Ben
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20131031/f45ed614/attachment.pgp>


[PATCH 1/4] test: Add compact test

2013-10-30 Thread Ben Gamari
Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 test/compact  | 32 
 test/notmuch-test |  1 +
 2 files changed, 33 insertions(+)
 create mode 100755 test/compact

diff --git a/test/compact b/test/compact
new file mode 100755
index 000..5bb5cea
--- /dev/null
+++ b/test/compact
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+test_description='notmuch compact'
+. ./test-lib.sh
+
+add_message '[subject]=One'
+add_message '[subject]=Two'
+add_message '[subject]=Three'
+
+notmuch tag +tag1 \*
+notmuch tag +tag2 subject:Two
+notmuch tag -tag1 +tag3 subject:Three
+
+test_expect_success Running compact notmuch compact
+
+test_begin_subtest Compact preserves database
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal $output \
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
+
+test_begin_subtest Restoring backup
+rm -Rf ${TEST_TMPDIR}/mail/xapian
+mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
+
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal $output \
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index aa28bb0..ec94baf 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -19,6 +19,7 @@ cd $(dirname $0)
 TESTS=
   basic
   help-test
+  compact
   config
   setup
   new
-- 
1.8.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 1/4] test: Add compact test

2013-10-30 Thread Ben Gamari
Tomi Ollila tomi.oll...@iki.fi writes:

 On Tue, Oct 29 2013, Ben Gamari bgamari.f...@gmail.com wrote:

 Signed-off-by: Ben Gamari bgamari.f...@gmail.com
 ---

 Patches 2, 3  4 Looks OK To Me. Thanks. A few comments on this patch 
 inline:

Thanks again for the review!

  test/compact  | 35 +++
  test/notmuch-test |  1 +
  2 files changed, 36 insertions(+)
  create mode 100755 test/compact

 diff --git a/test/compact b/test/compact
 new file mode 100755
 index 000..54e85ab
 --- /dev/null
 +++ b/test/compact
 @@ -0,0 +1,35 @@
 +#!/usr/bin/env bash
 +test_description='notmuch compact'
 +. ./test-lib.sh
 +
 +add_message '[subject]=One'
 +add_message '[subject]=Two'
 +add_message '[subject]=Three'
 +
 +notmuch tag +tag1 \*
 +notmuch tag +tag2 subject:Two
 +notmuch tag -tag1 +tag3 subject:Three
 +
 +test_begin_subtest Compacting
 +notmuch compact
 +test_expect_success compact notmuch compact

 test_expect_success executes $2 (notmuch compact in this case)
 do you mean to run notmuch 'compact twice' ?

It's been a while but I suspect I just never cleaned up the patch after
coming to this realization. Anyways, it's fixed in the revised patch
coming shortly.

 +notmuch search \*
 +output=$(notmuch search \* | notmuch_search_sanitize)

 Now do you mean to run 'notmuch search \*' twice ?

 +test_expect_equal $output \
 +thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
 +thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 
 unread)
 +thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
 +
 +test_begin_subtest Restoring backup
 +rm -Rf ${TEST_TMPDIR}/mail/xapian
 +mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
 +
 +notmuch search \*
 +output=$(notmuch search \* | notmuch_search_sanitize)

 again... ?


 ... actually David Bremner had even more insightful comments in 
 id:874n8cw2yq.fsf@zancas.localnet to these same issues :D

Hmm, it seems I overlooked these. Thanks for the reference.

Cheers,

- Ben


pgp93a9BTV1N7.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 4/4] database: Handle error while deleting uncompacted database

2013-10-28 Thread Ben Gamari
We never checked to ensure that the rmtree() of the old database
succeeded.

Signed-off-by: Ben Gamari 
---
 lib/database.cc | 14 +-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 34753ab..bfc5dac 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -934,7 +934,19 @@ notmuch_database_compact (const char* path,
goto DONE;
}
 } else {
-   rmtree(xapian_path);
+   if (rmtree(xapian_path)) {
+   fprintf (stderr, "Error removing old database: %s\n",
+strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Old database: %s\n", xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Please remove the old database and move the 
compacted one in to place manually with\n");
+   fprintf (stderr, "\n");
+   fprintf (stderr, "mv %s %s\n", compact_xapian_path, 
xapian_path);
+   fprintf (stderr, "\n");
+   ret = NOTMUCH_STATUS_FILE_ERROR;
+   goto DONE;
+   }
 }

 if (rename(compact_xapian_path, xapian_path)) {
-- 
1.8.3.2



[PATCH 3/4] compact: Provide user with more error feedback

2013-10-28 Thread Ben Gamari
Provide instructions on what to do when we couldn't move the compacted
database into place.

Signed-off-by: Ben Gamari 
---
 lib/database.cc | 20 +++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 57c2292..34753ab 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -938,7 +938,25 @@ notmuch_database_compact (const char* path,
 }

 if (rename(compact_xapian_path, xapian_path)) {
-   fprintf (stderr, "Error moving compacted database\n");
+   fprintf (stderr, "Error moving compacted database into place: %s\n", 
strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Encountered error while moving the compacted 
database\n");
+   fprintf (stderr, "\n");
+   fprintf (stderr, "%s\n", compact_xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "to\n");
+   fprintf (stderr, "\n");
+   fprintf (stderr, "%s\n", xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Please identify the reason for this and move the 
compacted database\n");
+   fprintf (stderr, "into place manually.\n");
+   if (old_xapian_path != NULL) {
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Alternatively you can revert to the uncompacted 
database with\n");
+   fprintf (stderr, "\n");
+   fprintf (stderr, "mv %s %s\n", old_xapian_path, xapian_path);
+   fprintf (stderr, "\n");
+   }
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
 }
-- 
1.8.3.2



[PATCH 2/4] compact: Give user more feedback on failure renaming

2013-10-28 Thread Ben Gamari
Provide the user with instructions after we fail to move the old
un-compacted database out of the way.

Signed-off-by: Ben Gamari 
---
 lib/database.cc | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 06f1c0a..57c2292 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -922,7 +922,14 @@ notmuch_database_compact (const char* path,

 if (old_xapian_path != NULL) {
if (rename(xapian_path, old_xapian_path)) {
-   fprintf (stderr, "Error moving old database out of the way\n");
+   fprintf (stderr, "Error moving old database out of the way: %s\n",
+strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Old database: %s\n", xapian_path);
+   fprintf (stderr, "Compacted database: %s\n", compact_xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "At this point it's probably best to remove the 
compacted database,\n");
+   fprintf (stderr, "find the cause of this error, and try compacting 
again.\n");
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
-- 
1.8.3.2



[PATCH 1/4] test: Add compact test

2013-10-28 Thread Ben Gamari
Signed-off-by: Ben Gamari 
---
 test/compact  | 35 +++
 test/notmuch-test |  1 +
 2 files changed, 36 insertions(+)
 create mode 100755 test/compact

diff --git a/test/compact b/test/compact
new file mode 100755
index 000..54e85ab
--- /dev/null
+++ b/test/compact
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+test_description='"notmuch compact"'
+. ./test-lib.sh
+
+add_message '[subject]=One'
+add_message '[subject]=Two'
+add_message '[subject]=Three'
+
+notmuch tag +tag1 \*
+notmuch tag +tag2 subject:Two
+notmuch tag -tag1 +tag3 subject:Three
+
+test_begin_subtest "Compacting"
+notmuch compact
+test_expect_success "compact" "notmuch compact"
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal "$output" "\
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
+
+test_begin_subtest "Restoring backup"
+rm -Rf ${TEST_TMPDIR}/mail/xapian
+mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal "$output" "\
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index aa28bb0..ec94baf 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -19,6 +19,7 @@ cd $(dirname "$0")
 TESTS="
   basic
   help-test
+  compact
   config
   setup
   new
-- 
1.8.3.2



[PATCH 3/3] compact: Provide user with more error feedback

2013-10-28 Thread Ben Gamari
Tomi Ollila  writes:

> On Sun, Oct 27 2013, Ben Gamari  wrote:
>
>> Provide instructions on what to do when we couldn't move the compacted
>> database into place.
>>
>> Signed-off-by: Ben Gamari 
>> ---
>>  lib/database.cc | 18 +-
>>  1 file changed, 17 insertions(+), 1 deletion(-)
>>
>> diff --git a/lib/database.cc b/lib/database.cc
>> index 57c2292..6f9fed1 100644
>> --- a/lib/database.cc
>> +++ b/lib/database.cc
>> @@ -938,7 +938,23 @@ notmuch_database_compact (const char* path,
>>  }
>>  
>>  if (rename(compact_xapian_path, xapian_path)) {
>> -fprintf (stderr, "Error moving compacted database\n");
>> +fprintf (stderr, "Error moving compacted database into place: %s\n", 
>> strerror(errno));
>> +fprintf (stderr, "\n");
>> +fprintf (stderr, "Encountered error %s while moving the compacted 
>> database,\n",
>> + strerror(errno));
>
> You probably missed my comments in id:m2d2mz7qv7.fsf at guru.guru-group.fi ?
>
Indeed I did. My apologies. New set coming.

Cheers,

- Ben
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20131028/6c698f89/attachment.pgp>


Re: [PATCH 3/3] compact: Provide user with more error feedback

2013-10-28 Thread Ben Gamari
Tomi Ollila tomi.oll...@iki.fi writes:

 On Sun, Oct 27 2013, Ben Gamari bgamari.f...@gmail.com wrote:

 Provide instructions on what to do when we couldn't move the compacted
 database into place.

 Signed-off-by: Ben Gamari bgamari.f...@gmail.com
 ---
  lib/database.cc | 18 +-
  1 file changed, 17 insertions(+), 1 deletion(-)

 diff --git a/lib/database.cc b/lib/database.cc
 index 57c2292..6f9fed1 100644
 --- a/lib/database.cc
 +++ b/lib/database.cc
 @@ -938,7 +938,23 @@ notmuch_database_compact (const char* path,
  }
  
  if (rename(compact_xapian_path, xapian_path)) {
 -fprintf (stderr, Error moving compacted database\n);
 +fprintf (stderr, Error moving compacted database into place: %s\n, 
 strerror(errno));
 +fprintf (stderr, \n);
 +fprintf (stderr, Encountered error %s while moving the compacted 
 database,\n,
 + strerror(errno));

 You probably missed my comments in id:m2d2mz7qv7@guru.guru-group.fi ?

Indeed I did. My apologies. New set coming.

Cheers,

- Ben


pgptHpiHJ_QFy.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/4] compact: Give user more feedback on failure renaming

2013-10-28 Thread Ben Gamari
Provide the user with instructions after we fail to move the old
un-compacted database out of the way.

Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 lib/database.cc | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 06f1c0a..57c2292 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -922,7 +922,14 @@ notmuch_database_compact (const char* path,
 
 if (old_xapian_path != NULL) {
if (rename(xapian_path, old_xapian_path)) {
-   fprintf (stderr, Error moving old database out of the way\n);
+   fprintf (stderr, Error moving old database out of the way: %s\n,
+strerror(errno));
+   fprintf (stderr, \n);
+   fprintf (stderr, Old database: %s\n, xapian_path);
+   fprintf (stderr, Compacted database: %s\n, compact_xapian_path);
+   fprintf (stderr, \n);
+   fprintf (stderr, At this point it's probably best to remove the 
compacted database,\n);
+   fprintf (stderr, find the cause of this error, and try compacting 
again.\n);
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
-- 
1.8.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 4/4] database: Handle error while deleting uncompacted database

2013-10-28 Thread Ben Gamari
We never checked to ensure that the rmtree() of the old database
succeeded.

Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 lib/database.cc | 14 +-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 34753ab..bfc5dac 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -934,7 +934,19 @@ notmuch_database_compact (const char* path,
goto DONE;
}
 } else {
-   rmtree(xapian_path);
+   if (rmtree(xapian_path)) {
+   fprintf (stderr, Error removing old database: %s\n,
+strerror(errno));
+   fprintf (stderr, \n);
+   fprintf (stderr, Old database: %s\n, xapian_path);
+   fprintf (stderr, \n);
+   fprintf (stderr, Please remove the old database and move the 
compacted one in to place manually with\n);
+   fprintf (stderr, \n);
+   fprintf (stderr, mv %s %s\n, compact_xapian_path, 
xapian_path);
+   fprintf (stderr, \n);
+   ret = NOTMUCH_STATUS_FILE_ERROR;
+   goto DONE;
+   }
 }
 
 if (rename(compact_xapian_path, xapian_path)) {
-- 
1.8.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/4] compact: Provide user with more error feedback

2013-10-28 Thread Ben Gamari
Provide instructions on what to do when we couldn't move the compacted
database into place.

Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 lib/database.cc | 20 +++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 57c2292..34753ab 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -938,7 +938,25 @@ notmuch_database_compact (const char* path,
 }
 
 if (rename(compact_xapian_path, xapian_path)) {
-   fprintf (stderr, Error moving compacted database\n);
+   fprintf (stderr, Error moving compacted database into place: %s\n, 
strerror(errno));
+   fprintf (stderr, \n);
+   fprintf (stderr, Encountered error while moving the compacted 
database\n);
+   fprintf (stderr, \n);
+   fprintf (stderr, %s\n, compact_xapian_path);
+   fprintf (stderr, \n);
+   fprintf (stderr, to\n);
+   fprintf (stderr, \n);
+   fprintf (stderr, %s\n, xapian_path);
+   fprintf (stderr, \n);
+   fprintf (stderr, Please identify the reason for this and move the 
compacted database\n);
+   fprintf (stderr, into place manually.\n);
+   if (old_xapian_path != NULL) {
+   fprintf (stderr, \n);
+   fprintf (stderr, Alternatively you can revert to the uncompacted 
database with\n);
+   fprintf (stderr, \n);
+   fprintf (stderr, mv %s %s\n, old_xapian_path, xapian_path);
+   fprintf (stderr, \n);
+   }
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
 }
-- 
1.8.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/4] test: Add compact test

2013-10-28 Thread Ben Gamari
Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 test/compact  | 35 +++
 test/notmuch-test |  1 +
 2 files changed, 36 insertions(+)
 create mode 100755 test/compact

diff --git a/test/compact b/test/compact
new file mode 100755
index 000..54e85ab
--- /dev/null
+++ b/test/compact
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+test_description='notmuch compact'
+. ./test-lib.sh
+
+add_message '[subject]=One'
+add_message '[subject]=Two'
+add_message '[subject]=Three'
+
+notmuch tag +tag1 \*
+notmuch tag +tag2 subject:Two
+notmuch tag -tag1 +tag3 subject:Three
+
+test_begin_subtest Compacting
+notmuch compact
+test_expect_success compact notmuch compact
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal $output \
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
+
+test_begin_subtest Restoring backup
+rm -Rf ${TEST_TMPDIR}/mail/xapian
+mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal $output \
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index aa28bb0..ec94baf 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -19,6 +19,7 @@ cd $(dirname $0)
 TESTS=
   basic
   help-test
+  compact
   config
   setup
   new
-- 
1.8.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] compact: Provide user with more error feedback

2013-10-27 Thread Ben Gamari
Provide instructions on what to do when we couldn't move the compacted
database into place.

Signed-off-by: Ben Gamari 
---
 lib/database.cc | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 57c2292..6f9fed1 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -938,7 +938,23 @@ notmuch_database_compact (const char* path,
 }

 if (rename(compact_xapian_path, xapian_path)) {
-   fprintf (stderr, "Error moving compacted database\n");
+   fprintf (stderr, "Error moving compacted database into place: %s\n", 
strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Encountered error %s while moving the compacted 
database,\n",
+strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "%s\n", compact_xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "to\n");
+   fprintf (stderr, "\n");
+   fprintf (stderr, "%s\n", xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Please identify the reason for this and move the 
compacted database into place manually.\n");
+   if (backup_path != NULL) {
+   fprintf (stderr, "Otherwise, you can revert to the backup database 
located at,\n");
+   fprintf (stderr, "\n");
+   fprintf (stderr, "%s\n", backup_path);
+   }
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
 }
-- 
1.8.3.2



[PATCH 2/3] compact: Give user more feedback on failure renaming

2013-10-27 Thread Ben Gamari
Provide the user with instructions after we fail to move the old
un-compacted database out of the way.

Signed-off-by: Ben Gamari 
---
 lib/database.cc | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 06f1c0a..57c2292 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -922,7 +922,14 @@ notmuch_database_compact (const char* path,

 if (old_xapian_path != NULL) {
if (rename(xapian_path, old_xapian_path)) {
-   fprintf (stderr, "Error moving old database out of the way\n");
+   fprintf (stderr, "Error moving old database out of the way: %s\n",
+strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Old database: %s\n", xapian_path);
+   fprintf (stderr, "Compacted database: %s\n", compact_xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "At this point it's probably best to remove the 
compacted database,\n");
+   fprintf (stderr, "find the cause of this error, and try compacting 
again.\n");
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
-- 
1.8.3.2



[PATCH 1/3] test: Add compact test

2013-10-27 Thread Ben Gamari
Signed-off-by: Ben Gamari 
---
 test/compact  | 35 +++
 test/notmuch-test |  1 +
 2 files changed, 36 insertions(+)
 create mode 100755 test/compact

diff --git a/test/compact b/test/compact
new file mode 100755
index 000..54e85ab
--- /dev/null
+++ b/test/compact
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+test_description='"notmuch compact"'
+. ./test-lib.sh
+
+add_message '[subject]=One'
+add_message '[subject]=Two'
+add_message '[subject]=Three'
+
+notmuch tag +tag1 \*
+notmuch tag +tag2 subject:Two
+notmuch tag -tag1 +tag3 subject:Three
+
+test_begin_subtest "Compacting"
+notmuch compact
+test_expect_success "compact" "notmuch compact"
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal "$output" "\
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
+
+test_begin_subtest "Restoring backup"
+rm -Rf ${TEST_TMPDIR}/mail/xapian
+mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal "$output" "\
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index aa28bb0..ec94baf 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -19,6 +19,7 @@ cd $(dirname "$0")
 TESTS="
   basic
   help-test
+  compact
   config
   setup
   new
-- 
1.8.3.2



[PATCH] Compact tests and error handling

2013-10-27 Thread Ben Gamari
David Bremner  writes:

> Ben Gamari  writes:
>
>> Here are a few patches adding a test case and some more user feedback on
>> rename failure during compaction.
>>
>
> Hi Ben;
>
> I'd like to include this series in the next release, freezing on
> Nov. 8. Will you have a chance to revise it before then?
>
Yes, I actually have the set sitting around waiting to be sent out. I'll
take care of this right now.

Cheers,

- Ben
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20131027/255c731a/attachment.pgp>


[PATCH 2/3] compact: Give user more feedback on failure renaming

2013-10-27 Thread Ben Gamari
Provide the user with instructions after we fail to move the old
un-compacted database out of the way.

Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 lib/database.cc | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 06f1c0a..57c2292 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -922,7 +922,14 @@ notmuch_database_compact (const char* path,
 
 if (old_xapian_path != NULL) {
if (rename(xapian_path, old_xapian_path)) {
-   fprintf (stderr, Error moving old database out of the way\n);
+   fprintf (stderr, Error moving old database out of the way: %s\n,
+strerror(errno));
+   fprintf (stderr, \n);
+   fprintf (stderr, Old database: %s\n, xapian_path);
+   fprintf (stderr, Compacted database: %s\n, compact_xapian_path);
+   fprintf (stderr, \n);
+   fprintf (stderr, At this point it's probably best to remove the 
compacted database,\n);
+   fprintf (stderr, find the cause of this error, and try compacting 
again.\n);
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
-- 
1.8.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/3] test: Add compact test

2013-10-27 Thread Ben Gamari
Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 test/compact  | 35 +++
 test/notmuch-test |  1 +
 2 files changed, 36 insertions(+)
 create mode 100755 test/compact

diff --git a/test/compact b/test/compact
new file mode 100755
index 000..54e85ab
--- /dev/null
+++ b/test/compact
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+test_description='notmuch compact'
+. ./test-lib.sh
+
+add_message '[subject]=One'
+add_message '[subject]=Two'
+add_message '[subject]=Three'
+
+notmuch tag +tag1 \*
+notmuch tag +tag2 subject:Two
+notmuch tag -tag1 +tag3 subject:Three
+
+test_begin_subtest Compacting
+notmuch compact
+test_expect_success compact notmuch compact
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal $output \
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
+
+test_begin_subtest Restoring backup
+rm -Rf ${TEST_TMPDIR}/mail/xapian
+mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal $output \
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index aa28bb0..ec94baf 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -19,6 +19,7 @@ cd $(dirname $0)
 TESTS=
   basic
   help-test
+  compact
   config
   setup
   new
-- 
1.8.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] Compact tests and error handling

2013-10-27 Thread Ben Gamari
David Bremner da...@tethera.net writes:

 Ben Gamari bgamari.f...@gmail.com writes:

 Here are a few patches adding a test case and some more user feedback on
 rename failure during compaction.


 Hi Ben;

 I'd like to include this series in the next release, freezing on
 Nov. 8. Will you have a chance to revise it before then?

Yes, I actually have the set sitting around waiting to be sent out. I'll
take care of this right now.

Cheers,

- Ben


pgpcKBnCAAfyl.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] compact: Provide user with more error feedback

2013-10-14 Thread Ben Gamari
Provide instructions on what to do when we couldn't move the compacted
database into place.

Signed-off-by: Ben Gamari 
---
 lib/database.cc | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 57c2292..6f9fed1 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -938,7 +938,23 @@ notmuch_database_compact (const char* path,
 }

 if (rename(compact_xapian_path, xapian_path)) {
-   fprintf (stderr, "Error moving compacted database\n");
+   fprintf (stderr, "Error moving compacted database into place: %s\n", 
strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Encountered error %s while moving the compacted 
database,\n",
+strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "%s\n", compact_xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "to\n");
+   fprintf (stderr, "\n");
+   fprintf (stderr, "%s\n", xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Please identify the reason for this and move the 
compacted database into place manually.\n");
+   if (backup_path != NULL) {
+   fprintf (stderr, "Otherwise, you can revert to the backup database 
located at,\n");
+   fprintf (stderr, "\n");
+   fprintf (stderr, "%s\n", backup_path);
+   }
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
 }
-- 
1.8.1.2



[PATCH 2/3] compact: Give user more feedback on failure renaming

2013-10-14 Thread Ben Gamari
Provide the user with instructions after we fail to move the old
un-compacted database out of the way.

Signed-off-by: Ben Gamari 
---
 lib/database.cc | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 06f1c0a..57c2292 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -922,7 +922,14 @@ notmuch_database_compact (const char* path,

 if (old_xapian_path != NULL) {
if (rename(xapian_path, old_xapian_path)) {
-   fprintf (stderr, "Error moving old database out of the way\n");
+   fprintf (stderr, "Error moving old database out of the way: %s\n",
+strerror(errno));
+   fprintf (stderr, "\n");
+   fprintf (stderr, "Old database: %s\n", xapian_path);
+   fprintf (stderr, "Compacted database: %s\n", compact_xapian_path);
+   fprintf (stderr, "\n");
+   fprintf (stderr, "At this point it's probably best to remove the 
compacted database,\n");
+   fprintf (stderr, "find the cause of this error, and try compacting 
again.\n");
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
-- 
1.8.1.2



[PATCH 1/3] test: Add compact test

2013-10-14 Thread Ben Gamari
Signed-off-by: Ben Gamari 
---
 test/compact  | 35 +++
 test/notmuch-test |  1 +
 2 files changed, 36 insertions(+)
 create mode 100755 test/compact

diff --git a/test/compact b/test/compact
new file mode 100755
index 000..54e85ab
--- /dev/null
+++ b/test/compact
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+test_description='"notmuch compact"'
+. ./test-lib.sh
+
+add_message '[subject]=One'
+add_message '[subject]=Two'
+add_message '[subject]=Three'
+
+notmuch tag +tag1 \*
+notmuch tag +tag2 subject:Two
+notmuch tag -tag1 +tag3 subject:Three
+
+test_begin_subtest "Compacting"
+notmuch compact
+test_expect_success "compact" "notmuch compact"
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal "$output" "\
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
+
+test_begin_subtest "Restoring backup"
+rm -Rf ${TEST_TMPDIR}/mail/xapian
+mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal "$output" "\
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)"
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index aa28bb0..ec94baf 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -19,6 +19,7 @@ cd $(dirname "$0")
 TESTS="
   basic
   help-test
+  compact
   config
   setup
   new
-- 
1.8.1.2



[PATCH] Compact tests and error handling

2013-10-14 Thread Ben Gamari
Here are a few patches adding a test case and some more user feedback on
rename failure during compaction.

Cheers,

- Ben




[PATCH 1/3] test: Add compact test

2013-10-14 Thread Ben Gamari
Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 test/compact  | 35 +++
 test/notmuch-test |  1 +
 2 files changed, 36 insertions(+)
 create mode 100755 test/compact

diff --git a/test/compact b/test/compact
new file mode 100755
index 000..54e85ab
--- /dev/null
+++ b/test/compact
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+test_description='notmuch compact'
+. ./test-lib.sh
+
+add_message '[subject]=One'
+add_message '[subject]=Two'
+add_message '[subject]=Three'
+
+notmuch tag +tag1 \*
+notmuch tag +tag2 subject:Two
+notmuch tag -tag1 +tag3 subject:Three
+
+test_begin_subtest Compacting
+notmuch compact
+test_expect_success compact notmuch compact
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal $output \
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
+
+test_begin_subtest Restoring backup
+rm -Rf ${TEST_TMPDIR}/mail/xapian
+mv ${TEST_TMPDIR}/mail/xapian.old ${TEST_TMPDIR}/mail/xapian
+
+notmuch search \*
+output=$(notmuch search \* | notmuch_search_sanitize)
+test_expect_equal $output \
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; One (inbox tag1 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Two (inbox tag1 tag2 unread)
+thread:XXX   2001-01-05 [1/1] Notmuch Test Suite; Three (inbox tag3 unread)
+
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index aa28bb0..ec94baf 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -19,6 +19,7 @@ cd $(dirname $0)
 TESTS=
   basic
   help-test
+  compact
   config
   setup
   new
-- 
1.8.1.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/3] compact: Give user more feedback on failure renaming

2013-10-14 Thread Ben Gamari
Provide the user with instructions after we fail to move the old
un-compacted database out of the way.

Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 lib/database.cc | 9 -
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 06f1c0a..57c2292 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -922,7 +922,14 @@ notmuch_database_compact (const char* path,
 
 if (old_xapian_path != NULL) {
if (rename(xapian_path, old_xapian_path)) {
-   fprintf (stderr, Error moving old database out of the way\n);
+   fprintf (stderr, Error moving old database out of the way: %s\n,
+strerror(errno));
+   fprintf (stderr, \n);
+   fprintf (stderr, Old database: %s\n, xapian_path);
+   fprintf (stderr, Compacted database: %s\n, compact_xapian_path);
+   fprintf (stderr, \n);
+   fprintf (stderr, At this point it's probably best to remove the 
compacted database,\n);
+   fprintf (stderr, find the cause of this error, and try compacting 
again.\n);
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
}
-- 
1.8.1.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH] Compact tests and error handling

2013-10-14 Thread Ben Gamari
Here are a few patches adding a test case and some more user feedback on
rename failure during compaction.

Cheers,

- Ben


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] compact: Provide user with more error feedback

2013-10-14 Thread Ben Gamari
Provide instructions on what to do when we couldn't move the compacted
database into place.

Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 lib/database.cc | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 57c2292..6f9fed1 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -938,7 +938,23 @@ notmuch_database_compact (const char* path,
 }
 
 if (rename(compact_xapian_path, xapian_path)) {
-   fprintf (stderr, Error moving compacted database\n);
+   fprintf (stderr, Error moving compacted database into place: %s\n, 
strerror(errno));
+   fprintf (stderr, \n);
+   fprintf (stderr, Encountered error %s while moving the compacted 
database,\n,
+strerror(errno));
+   fprintf (stderr, \n);
+   fprintf (stderr, %s\n, compact_xapian_path);
+   fprintf (stderr, \n);
+   fprintf (stderr, to\n);
+   fprintf (stderr, \n);
+   fprintf (stderr, %s\n, xapian_path);
+   fprintf (stderr, \n);
+   fprintf (stderr, Please identify the reason for this and move the 
compacted database into place manually.\n);
+   if (backup_path != NULL) {
+   fprintf (stderr, Otherwise, you can revert to the backup database 
located at,\n);
+   fprintf (stderr, \n);
+   fprintf (stderr, %s\n, backup_path);
+   }
ret = NOTMUCH_STATUS_FILE_ERROR;
goto DONE;
 }
-- 
1.8.1.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH] notmuch compact support (v4)

2013-10-10 Thread Ben Gamari
David Bremner da...@tethera.net writes:

 Ben Gamari bgamari.f...@gmail.com writes:

 Here is yet another iteration of my notmuch compact patchset. It has been
 rebased on top of master and the interface reworked as suggested by Jani.


 Sorry, first version of this reply didn't go to the list.

 I haven't had time to do a review, but I did try them out, and they
 seemed not to eat my database.

 What about a test for the test suite? 

I'll implement something along these lines and send a patch shortly.

Cheers,

- Ben


pgp2OBEWreqE1.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] notmuch-compact: Add man page

2013-10-02 Thread Ben Gamari
Signed-off-by: Ben Gamari 
---
 man/Makefile.local |  1 +
 man/man1/notmuch-compact.1 | 36 
 2 files changed, 37 insertions(+)
 create mode 100644 man/man1/notmuch-compact.1

diff --git a/man/Makefile.local b/man/Makefile.local
index 216aaa0..57910b7 100644
--- a/man/Makefile.local
+++ b/man/Makefile.local
@@ -8,6 +8,7 @@ MAIN_PAGE := $(dir)/man1/notmuch.1

 MAN1 := \
$(MAIN_PAGE) \
+   $(dir)/man1/notmuch-compact.1 \
$(dir)/man1/notmuch-config.1 \
$(dir)/man1/notmuch-count.1 \
$(dir)/man1/notmuch-dump.1 \
diff --git a/man/man1/notmuch-compact.1 b/man/man1/notmuch-compact.1
new file mode 100644
index 000..1aeed22
--- /dev/null
+++ b/man/man1/notmuch-compact.1
@@ -0,0 +1,36 @@
+.TH NOTMUCH-COMPACT 1 2013-08-23 "Notmuch 0.16"
+.SH NAME
+notmuch-compact \- compact the notmuch database
+.SH SYNOPSIS
+
+.B notmuch compact
+
+.SH DESCRIPTION
+
+The
+.B compact
+command can be used to compact the notmuch database. This can both reduce
+the space required by the database and improve lookup performance.
+
+The compacted database is built in a temporary directory and is later
+moved into the place of the origin database. The original uncompacted
+database is preserved to be deleted by the user as desired.
+
+Note that the database write lock will be held during the compaction
+process (which may be quite long) to protect data integrity.
+
+.RE
+.SH ENVIRONMENT
+The following environment variables can be used to control the
+behavior of notmuch.
+.TP
+.B NOTMUCH_CONFIG
+Specifies the location of the notmuch configuration file. Notmuch will
+use ${HOME}/.notmuch\-config if this variable is not set.
+.SH SEE ALSO
+
+\fBnotmuch\fR(1), \fBnotmuch-count\fR(1), \fBnotmuch-dump\fR(1),
+\fBnotmuch-hooks\fR(5), \fBnotmuch-insert\fR(1), \fBnotmuch-new\fR(1),
+\fBnotmuch-reply\fR(1), \fBnotmuch-restore\fR(1), \fBnotmuch-search\fR(1),
+\fBnotmuch-search-terms\fR(7), \fBnotmuch-show\fR(1),
+\fBnotmuch-tag\fR(1)
-- 
1.8.1.2



[PATCH 1/3] database: Add notmuch_database_compact_close

2013-10-02 Thread Ben Gamari
This function uses Xapian's Compactor machinery to compact the notmuch
database. The compacted database is built in a temporary directory and
later moved into place while the original uncompacted database is
preserved.

Signed-off-by: Ben Gamari 
---
 configure   |  27 --
 lib/database.cc | 151 
 lib/notmuch.h   |  21 +++-
 3 files changed, 195 insertions(+), 4 deletions(-)

diff --git a/configure b/configure
index 6166917..1a8e939 100755
--- a/configure
+++ b/configure
@@ -277,7 +277,8 @@ printf "Checking for Xapian development files... "
 have_xapian=0
 for xapian_config in ${XAPIAN_CONFIG}; do
 if ${xapian_config} --version > /dev/null 2>&1; then
-   printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //')
+   xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
+   printf "Yes (%s).\n" ${xapian_version}
have_xapian=1
xapian_cxxflags=$(${xapian_config} --cxxflags)
xapian_ldflags=$(${xapian_config} --libs)
@@ -289,6 +290,21 @@ if [ ${have_xapian} = "0" ]; then
 errors=$((errors + 1))
 fi

+# Compaction is only supported on Xapian > 1.2.6
+have_xapian_compact=0
+if [ ${have_xapian} = "1" ]; then
+printf "Checking for Xapian compaction support... "
+case "${xapian_version}" in
+0.*|1.[01].*|1.2.[0-5])
+printf "No (only available with Xapian > 1.2.6).\n" ;;
+[1-9]*.[0-9]*.[0-9]*)
+have_xapian_compact=1
+printf "Yes.\n" ;;
+*)
+printf "Unknown version.\n" ;;
+esac
+fi
+
 printf "Checking for GMime development files... "
 have_gmime=0
 IFS=';'
@@ -729,6 +745,9 @@ HAVE_STRCASESTR = ${have_strcasestr}
 # build its own version)
 HAVE_STRSEP = ${have_strsep}

+# Whether the Xapian version in use supports compaction
+HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
+
 # Whether the getpwuid_r function is standards-compliant
 # (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS
 # to enable the standards-compliant version -- needed for Solaris)
@@ -787,13 +806,15 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
\$(GMIME_CFLAGS)  \\
   -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\
   -DHAVE_STRSEP=\$(HAVE_STRSEP) \\
   -DSTD_GETPWUID=\$(STD_GETPWUID)   \\
-  -DSTD_ASCTIME=\$(STD_ASCTIME)
+  -DSTD_ASCTIME=\$(STD_ASCTIME) \\
+  -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
 \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
 -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)   \\
 -DHAVE_STRSEP=\$(HAVE_STRSEP)   \\
 -DSTD_GETPWUID=\$(STD_GETPWUID) \\
--DSTD_ASCTIME=\$(STD_ASCTIME)
+-DSTD_ASCTIME=\$(STD_ASCTIME)   \\
+-DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
diff --git a/lib/database.cc b/lib/database.cc
index bb4f180..06f1c0a 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -24,7 +24,9 @@
 #include 

 #include 
+#include 
 #include 
+#include 

 #include  /* g_free, GPtrArray, GHashTable */
 #include  /* g_type_init */
@@ -268,6 +270,8 @@ notmuch_status_to_string (notmuch_status_t status)
return "Unbalanced number of calls to notmuch_message_freeze/thaw";
 case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
return "Unbalanced number of calls to 
notmuch_database_begin_atomic/end_atomic";
+case NOTMUCH_STATUS_UNSUPPORTED_OPERATION:
+   return "Unsupported operation";
 default:
 case NOTMUCH_STATUS_LAST_STATUS:
return "Unknown error status value";
@@ -800,6 +804,153 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch->date_range_processor = NULL;
 }

+#if HAVE_XAPIAN_COMPACT
+static int unlink_cb (const char *path,
+ unused (const struct stat *sb),
+ unused (int type),
+ unused (struct FTW *ftw))
+{
+return remove(path);
+}
+
+static int rmtree (const char *path)
+{
+return nftw(path, unlink_cb, 64, FTW_DEPTH | FTW_PHYS);
+}
+
+class NotmuchCompactor : public Xapian::Compactor
+{
+notmuch_compact_status_cb_t status_cb;
+
+public:
+NotmuchCompactor(notmuch_compact_status_cb_t cb) : status_cb(cb) { }
+
+virtual void
+set_status (const std::string , const std::string )
+{

[PATCH] notmuch compact support (v4)

2013-10-02 Thread Ben Gamari

Here is yet another iteration of my "notmuch compact" patchset. It has been
rebased on top of master and the interface reworked as suggested by Jani.

Cheers,

- Ben



[PATCH] notmuch compact support (v4)

2013-10-02 Thread Ben Gamari

Here is yet another iteration of my notmuch compact patchset. It has been
rebased on top of master and the interface reworked as suggested by Jani.

Cheers,

- Ben

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] notmuch-compact: Add man page

2013-10-02 Thread Ben Gamari
Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 man/Makefile.local |  1 +
 man/man1/notmuch-compact.1 | 36 
 2 files changed, 37 insertions(+)
 create mode 100644 man/man1/notmuch-compact.1

diff --git a/man/Makefile.local b/man/Makefile.local
index 216aaa0..57910b7 100644
--- a/man/Makefile.local
+++ b/man/Makefile.local
@@ -8,6 +8,7 @@ MAIN_PAGE := $(dir)/man1/notmuch.1
 
 MAN1 := \
$(MAIN_PAGE) \
+   $(dir)/man1/notmuch-compact.1 \
$(dir)/man1/notmuch-config.1 \
$(dir)/man1/notmuch-count.1 \
$(dir)/man1/notmuch-dump.1 \
diff --git a/man/man1/notmuch-compact.1 b/man/man1/notmuch-compact.1
new file mode 100644
index 000..1aeed22
--- /dev/null
+++ b/man/man1/notmuch-compact.1
@@ -0,0 +1,36 @@
+.TH NOTMUCH-COMPACT 1 2013-08-23 Notmuch 0.16
+.SH NAME
+notmuch-compact \- compact the notmuch database
+.SH SYNOPSIS
+
+.B notmuch compact
+
+.SH DESCRIPTION
+
+The
+.B compact
+command can be used to compact the notmuch database. This can both reduce
+the space required by the database and improve lookup performance.
+
+The compacted database is built in a temporary directory and is later
+moved into the place of the origin database. The original uncompacted
+database is preserved to be deleted by the user as desired.
+
+Note that the database write lock will be held during the compaction
+process (which may be quite long) to protect data integrity.
+
+.RE
+.SH ENVIRONMENT
+The following environment variables can be used to control the
+behavior of notmuch.
+.TP
+.B NOTMUCH_CONFIG
+Specifies the location of the notmuch configuration file. Notmuch will
+use ${HOME}/.notmuch\-config if this variable is not set.
+.SH SEE ALSO
+
+\fBnotmuch\fR(1), \fBnotmuch-count\fR(1), \fBnotmuch-dump\fR(1),
+\fBnotmuch-hooks\fR(5), \fBnotmuch-insert\fR(1), \fBnotmuch-new\fR(1),
+\fBnotmuch-reply\fR(1), \fBnotmuch-restore\fR(1), \fBnotmuch-search\fR(1),
+\fBnotmuch-search-terms\fR(7), \fBnotmuch-show\fR(1),
+\fBnotmuch-tag\fR(1)
-- 
1.8.1.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/3] notmuch-compact: Initial commit of CLI

2013-10-02 Thread Ben Gamari
Introduce the user command exposing the new compaction facility.

Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 Makefile.local|  1 +
 notmuch-client.h  |  3 +++
 notmuch-compact.c | 56 +++
 notmuch.c |  2 ++
 4 files changed, 62 insertions(+)
 create mode 100644 notmuch-compact.c

diff --git a/Makefile.local b/Makefile.local
index b7cd266..0464a50 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -257,6 +257,7 @@ notmuch_client_srcs =   \
gmime-filter-reply.c\
hooks.c \
notmuch.c   \
+   notmuch-compact.c   \
notmuch-config.c\
notmuch-count.c \
notmuch-dump.c  \
diff --git a/notmuch-client.h b/notmuch-client.h
index afb0ddf..0bfa4da 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -204,6 +204,9 @@ notmuch_tag_command (notmuch_config_t *config, int argc, 
char *argv[]);
 int
 notmuch_config_command (notmuch_config_t *config, int argc, char *argv[]);
 
+int
+notmuch_compact_command (notmuch_config_t *config, int argc, char *argv[]);
+
 const char *
 notmuch_time_relative_date (const void *ctx, time_t then);
 
diff --git a/notmuch-compact.c b/notmuch-compact.c
new file mode 100644
index 000..bfda40e
--- /dev/null
+++ b/notmuch-compact.c
@@ -0,0 +1,56 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright © 2013 Ben Gamari
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Ben Gamari bgamari.f...@gmail.com
+ */
+
+#include notmuch-client.h
+
+void status_update_cb (const char *msg);
+
+void
+status_update_cb (const char *msg)
+{
+printf(%s\n, msg);
+}
+
+int
+notmuch_compact_command (notmuch_config_t *config,
+unused (int argc),
+unused (char *argv[]))
+{
+const char *path = notmuch_config_get_database_path (config);
+const char *backup_path = path;
+notmuch_status_t ret;
+
+printf (Compacting database...\n);
+ret = notmuch_database_compact (path, backup_path, status_update_cb);
+if (ret) {
+   fprintf (stderr, Compaction failed: %s\n, 
notmuch_status_to_string(ret));
+} else {
+   printf (\n);
+   printf (\n);
+   printf (The old database has been moved to %s/xapian.old, 
backup_path);
+   printf (\n);
+   printf (To delete run,\n);
+   printf (\n);
+   printf (rm -R %s/xapian.old\n, backup_path);
+   printf (\n);
+}
+
+return 0;
+}
diff --git a/notmuch.c b/notmuch.c
index 7300c21..8d303a1 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -60,6 +60,8 @@ static command_t commands[] = {
   Create a plain-text dump of the tags for each message. },
 { restore, notmuch_restore_command, FALSE,
   Restore the tags from the given dump file (see 'dump'). },
+{ compact, notmuch_compact_command, FALSE,
+  Compact the notmuch database. },
 { config, notmuch_config_command, FALSE,
   Get or set settings in the notmuch configuration file. },
 { help, notmuch_help_command, TRUE, /* create but don't save config */
-- 
1.8.1.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/3] database: Add notmuch_database_compact_close

2013-09-03 Thread Ben Gamari
Jani Nikula  writes:

> On Sat, 24 Aug 2013, Ben Gamari  wrote:
>> This function uses Xapian's Compactor machinery to compact the notmuch
>> database. The compacted database is built in a temporary directory and
>> later moved into place while the original uncompacted database is
>> preserved.
>>

snip

>>  
>> +class NotmuchCompactor : public Xapian::Compactor
>> +{
>> +public:
>> +virtual void
>> +set_status (const std::string , const std::string )
>> +{
>> +if (status.length() == 0)
>> +printf ("compacting table %s:\n", table.c_str());
>> +else
>> +printf (" %s\n", status.c_str());
>> +}
>
> We're trying to reduce the amount of prints directly from libnotmuch,
> not increase. This applies here as well as below.
>
Fair enough. That being said, I think that status updates are fairly
important given that the compaction process can be rather long.  Would
the preferred interface be to provide notmuch_database_compact_close
with a progress callback?

>> +};
>> +
>> +#if HAVE_XAPIAN_COMPACT
>> +notmuch_status_t
>> +notmuch_database_compact_close (notmuch_database_t *notmuch)
>> +{
>> +void *local = talloc_new (NULL);
>> +NotmuchCompactor compactor;
>> +char *notmuch_path, *xapian_path, *compact_xapian_path, 
>> *old_xapian_path;
>> +notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
>> +
>> +if (! (notmuch_path = talloc_asprintf (local, "%s/%s", notmuch->path, 
>> ".notmuch"))) {
>> +ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
>> +goto DONE;
>> +}
>> +
>> +if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, 
>> "xapian"))) {
>> +ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
>> +goto DONE;
>> +}
>> +
>> +if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", 
>> xapian_path))) {
>> +ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
>> +goto DONE;
>> +}
>> +
>> +if (! (old_xapian_path = talloc_asprintf (local, "%s.old", 
>> xapian_path))) {
>> +ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
>> +goto DONE;
>> +}
>> +
>> +try {
>> +compactor.set_renumber(false);
>> +compactor.add_source(xapian_path);
>> +compactor.set_destdir(compact_xapian_path);
>> +compactor.compact();
>> +
>> +if (rename(xapian_path, old_xapian_path)) {
>> +fprintf (stderr, "Error moving old database out of the way\n");
>> +ret = NOTMUCH_STATUS_FILE_ERROR;
>> +goto DONE;
>> +}
>
> This fails if old_xapian_path exists.
>
Ouch, yes, you are right. I suspect the right way forward here will be
to check whether old_xapian_path exists before beginning compaction,
allowing the user to fix the situation before it fails after finishing
what might be a pretty long process.

>> +
>> +if (rename(compact_xapian_path, xapian_path)) {
>> +fprintf (stderr, "Error moving compacted database\n");
>> +ret = NOTMUCH_STATUS_FILE_ERROR;
>> +goto DONE;
>> +}
>> +} catch (Xapian::InvalidArgumentError e) {
>> +fprintf (stderr, "Error while compacting: %s", e.get_msg().c_str());
>> +ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
>> +goto DONE;
>> +}
>> +
>> +fprintf (stderr, "\n");
>> +fprintf (stderr, "\n");
>> +fprintf (stderr, "Old database has been moved to %s", old_xapian_path);
>> +fprintf (stderr, "\n");
>> +fprintf (stderr, "To delete run,\n");
>> +fprintf (stderr, "\n");
>> +fprintf (stderr, "rm -R %s\n", old_xapian_path);
>> +fprintf (stderr, "\n");
>> +
>> +notmuch_database_close(notmuch);
>> +
>> +DONE:
>> +talloc_free(local);
>
> The database does not get closed on errors. If that's intentional, it
> should be documented.
>
I had reasons for this but they have long fled my memory. Regardless of
what it does, this behavior should be documented. I'll take care of this.

>> +return ret;
>> +}
>> +#else
>> +notmuch_status_t
>> +notmuch_database_compact_close (unused (notmuch_database_t *notmuch))
>> +{
>> +fprintf (stderr, "notmuch was compiled against a xapian version lacking 
>> compaction support.\n");
>> +return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;
>> +}
>> +#endif
>> +

Re: [PATCH 1/3] database: Add notmuch_database_compact_close

2013-09-03 Thread Ben Gamari
Jani Nikula j...@nikula.org writes:

 On Sat, 24 Aug 2013, Ben Gamari bgamari.f...@gmail.com wrote:
 This function uses Xapian's Compactor machinery to compact the notmuch
 database. The compacted database is built in a temporary directory and
 later moved into place while the original uncompacted database is
 preserved.


snip

  
 +class NotmuchCompactor : public Xapian::Compactor
 +{
 +public:
 +virtual void
 +set_status (const std::string table, const std::string status)
 +{
 +if (status.length() == 0)
 +printf (compacting table %s:\n, table.c_str());
 +else
 +printf ( %s\n, status.c_str());
 +}

 We're trying to reduce the amount of prints directly from libnotmuch,
 not increase. This applies here as well as below.

Fair enough. That being said, I think that status updates are fairly
important given that the compaction process can be rather long.  Would
the preferred interface be to provide notmuch_database_compact_close
with a progress callback?

 +};
 +
 +#if HAVE_XAPIAN_COMPACT
 +notmuch_status_t
 +notmuch_database_compact_close (notmuch_database_t *notmuch)
 +{
 +void *local = talloc_new (NULL);
 +NotmuchCompactor compactor;
 +char *notmuch_path, *xapian_path, *compact_xapian_path, 
 *old_xapian_path;
 +notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
 +
 +if (! (notmuch_path = talloc_asprintf (local, %s/%s, notmuch-path, 
 .notmuch))) {
 +ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
 +goto DONE;
 +}
 +
 +if (! (xapian_path = talloc_asprintf (local, %s/%s, notmuch_path, 
 xapian))) {
 +ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
 +goto DONE;
 +}
 +
 +if (! (compact_xapian_path = talloc_asprintf (local, %s.compact, 
 xapian_path))) {
 +ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
 +goto DONE;
 +}
 +
 +if (! (old_xapian_path = talloc_asprintf (local, %s.old, 
 xapian_path))) {
 +ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
 +goto DONE;
 +}
 +
 +try {
 +compactor.set_renumber(false);
 +compactor.add_source(xapian_path);
 +compactor.set_destdir(compact_xapian_path);
 +compactor.compact();
 +
 +if (rename(xapian_path, old_xapian_path)) {
 +fprintf (stderr, Error moving old database out of the way\n);
 +ret = NOTMUCH_STATUS_FILE_ERROR;
 +goto DONE;
 +}

 This fails if old_xapian_path exists.

Ouch, yes, you are right. I suspect the right way forward here will be
to check whether old_xapian_path exists before beginning compaction,
allowing the user to fix the situation before it fails after finishing
what might be a pretty long process.

 +
 +if (rename(compact_xapian_path, xapian_path)) {
 +fprintf (stderr, Error moving compacted database\n);
 +ret = NOTMUCH_STATUS_FILE_ERROR;
 +goto DONE;
 +}
 +} catch (Xapian::InvalidArgumentError e) {
 +fprintf (stderr, Error while compacting: %s, e.get_msg().c_str());
 +ret = NOTMUCH_STATUS_XAPIAN_EXCEPTION;
 +goto DONE;
 +}
 +
 +fprintf (stderr, \n);
 +fprintf (stderr, \n);
 +fprintf (stderr, Old database has been moved to %s, old_xapian_path);
 +fprintf (stderr, \n);
 +fprintf (stderr, To delete run,\n);
 +fprintf (stderr, \n);
 +fprintf (stderr, rm -R %s\n, old_xapian_path);
 +fprintf (stderr, \n);
 +
 +notmuch_database_close(notmuch);
 +
 +DONE:
 +talloc_free(local);

 The database does not get closed on errors. If that's intentional, it
 should be documented.

I had reasons for this but they have long fled my memory. Regardless of
what it does, this behavior should be documented. I'll take care of this.

 +return ret;
 +}
 +#else
 +notmuch_status_t
 +notmuch_database_compact_close (unused (notmuch_database_t *notmuch))
 +{
 +fprintf (stderr, notmuch was compiled against a xapian version lacking 
 compaction support.\n);
 +return NOTMUCH_STATUS_UNSUPPORTED_OPERATION;
 +}
 +#endif
 +
  void
  notmuch_database_destroy (notmuch_database_t *notmuch)
  {
 diff --git a/lib/notmuch.h b/lib/notmuch.h
 index 998a4ae..e9abd90 100644
 --- a/lib/notmuch.h
 +++ b/lib/notmuch.h
 @@ -101,6 +101,7 @@ typedef enum _notmuch_status {
  NOTMUCH_STATUS_TAG_TOO_LONG,
  NOTMUCH_STATUS_UNBALANCED_FREEZE_THAW,
  NOTMUCH_STATUS_UNBALANCED_ATOMIC,
 +NOTMUCH_STATUS_UNSUPPORTED_OPERATION,
  
  NOTMUCH_STATUS_LAST_STATUS
  } notmuch_status_t;
 @@ -215,6 +216,20 @@ notmuch_database_open (const char *path,
  void
  notmuch_database_close (notmuch_database_t *database);
  
 +/* Close the given notmuch database and then compact it.

 The implementation first compacts then closes.


 + * After notmuch_database_close_compact has been called, calls to
 + * other functions on objects derived from this database may either
 + * behave as if the database had not been closed (e.g., if the
 + * required data has been cached) or may fail with a
 + * NOTMUCH_STATUS_XAPIAN_EXCEPTION.
 + *
 + * notmuch_database_close_compact can be called

[PATCH 3/3] notmuch-compact: Add man page

2013-08-23 Thread Ben Gamari
Signed-off-by: Ben Gamari 
---
 man/Makefile.local |  1 +
 man/man1/notmuch-compact.1 | 33 +
 2 files changed, 34 insertions(+)
 create mode 100644 man/man1/notmuch-compact.1

diff --git a/man/Makefile.local b/man/Makefile.local
index 216aaa0..57910b7 100644
--- a/man/Makefile.local
+++ b/man/Makefile.local
@@ -8,6 +8,7 @@ MAIN_PAGE := $(dir)/man1/notmuch.1

 MAN1 := \
$(MAIN_PAGE) \
+   $(dir)/man1/notmuch-compact.1 \
$(dir)/man1/notmuch-config.1 \
$(dir)/man1/notmuch-count.1 \
$(dir)/man1/notmuch-dump.1 \
diff --git a/man/man1/notmuch-compact.1 b/man/man1/notmuch-compact.1
new file mode 100644
index 000..48f76c3
--- /dev/null
+++ b/man/man1/notmuch-compact.1
@@ -0,0 +1,33 @@
+.TH NOTMUCH-COMPACT 1 2013-08-23 "Notmuch 0.16"
+.SH NAME
+notmuch-compact \- compact the notmuch database
+.SH SYNOPSIS
+
+.B notmuch compact
+
+.SH DESCRIPTION
+
+The
+.B compact
+command can be used to compact the notmuch database. This can both reduce
+the space required by the database and improve performance.
+
+The compacted database is built in a temporary directory and is later
+moved into the place of the origin database. The original uncompacted
+database is preserved to be deleted by the user as desired.
+
+.RE
+.SH ENVIRONMENT
+The following environment variables can be used to control the
+behavior of notmuch.
+.TP
+.B NOTMUCH_CONFIG
+Specifies the location of the notmuch configuration file. Notmuch will
+use ${HOME}/.notmuch\-config if this variable is not set.
+.SH SEE ALSO
+
+\fBnotmuch\fR(1), \fBnotmuch-count\fR(1), \fBnotmuch-dump\fR(1),
+\fBnotmuch-hooks\fR(5), \fBnotmuch-insert\fR(1), \fBnotmuch-new\fR(1),
+\fBnotmuch-reply\fR(1), \fBnotmuch-restore\fR(1), \fBnotmuch-search\fR(1),
+\fBnotmuch-search-terms\fR(7), \fBnotmuch-show\fR(1),
+\fBnotmuch-tag\fR(1)
-- 
1.8.1.2



[PATCH 2/3] notmuch-compact: Initial commit

2013-08-23 Thread Ben Gamari
Introduce the user command exposing the new compaction facility.

Signed-off-by: Ben Gamari 
---
 Makefile.local|  1 +
 notmuch-client.h  |  6 ++
 notmuch-compact.c | 43 +++
 notmuch.c |  2 ++
 4 files changed, 52 insertions(+)
 create mode 100644 notmuch-compact.c

diff --git a/Makefile.local b/Makefile.local
index b7cd266..0464a50 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -257,6 +257,7 @@ notmuch_client_srcs =   \
gmime-filter-reply.c\
hooks.c \
notmuch.c   \
+   notmuch-compact.c   \
notmuch-config.c\
notmuch-count.c \
notmuch-dump.c  \
diff --git a/notmuch-client.h b/notmuch-client.h
index da332f3..5b8a076 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -203,6 +203,12 @@ notmuch_tag_command (notmuch_config_t *config, int argc, 
char *argv[]);
 int
 notmuch_config_command (notmuch_config_t *config, int argc, char *argv[]);

+int
+notmuch_compact_command (notmuch_config_t *config, int argc, char *argv[]);
+
+int
+notmuch_config_command (notmuch_config_t *config, int argc, char *argv[]);
+
 const char *
 notmuch_time_relative_date (const void *ctx, time_t then);

diff --git a/notmuch-compact.c b/notmuch-compact.c
new file mode 100644
index 000..72b5c1e
--- /dev/null
+++ b/notmuch-compact.c
@@ -0,0 +1,43 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright ? 2013 Ben Gmaari
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Ben Gamari 
+ */
+
+#include "notmuch-client.h"
+
+int
+notmuch_compact_command (notmuch_config_t *config, unused (int argc), unused 
(char *argv[]))
+{
+notmuch_database_t *notmuch;
+notmuch_status_t ret;
+
+if (notmuch_database_open (notmuch_config_get_database_path (config),
+  NOTMUCH_DATABASE_MODE_READ_WRITE, ))
+   return 1;
+
+printf ("Compacting database...\n");
+ret = notmuch_database_compact_close (notmuch);
+if (ret) {
+   fprintf (stderr, "Compaction failed: %s\n", 
notmuch_status_to_string(ret));
+} else {
+   printf ("Done.\n");
+}
+notmuch_database_destroy (notmuch);
+
+return 0;
+}
diff --git a/notmuch.c b/notmuch.c
index 78d29a8..ab0b1cf 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -60,6 +60,8 @@ static command_t commands[] = {
   "Create a plain-text dump of the tags for each message." },
 { "restore", notmuch_restore_command, FALSE,
   "Restore the tags from the given dump file (see 'dump')." },
+{ "compact", notmuch_compact_command, FALSE,
+  "Compact the notmuch database." },
 { "config", notmuch_config_command, FALSE,
   "Get or set settings in the notmuch configuration file." },
 { "help", notmuch_help_command, TRUE, /* create but don't save config */
-- 
1.8.1.2



[PATCH 1/3] database: Add notmuch_database_compact_close

2013-08-23 Thread Ben Gamari
This function uses Xapian's Compactor machinery to compact the notmuch
database. The compacted database is built in a temporary directory and
later moved into place while the original uncompacted database is
preserved.

Signed-off-by: Ben Gamari 
---
 configure   | 27 +++--
 lib/database.cc | 91 +
 lib/notmuch.h   | 15 ++
 3 files changed, 130 insertions(+), 3 deletions(-)

diff --git a/configure b/configure
index 6166917..ee9e887 100755
--- a/configure
+++ b/configure
@@ -277,7 +277,8 @@ printf "Checking for Xapian development files... "
 have_xapian=0
 for xapian_config in ${XAPIAN_CONFIG}; do
 if ${xapian_config} --version > /dev/null 2>&1; then
-   printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //')
+   xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
+   printf "Yes (%s).\n" ${xapian_version}
have_xapian=1
xapian_cxxflags=$(${xapian_config} --cxxflags)
xapian_ldflags=$(${xapian_config} --libs)
@@ -289,6 +290,21 @@ if [ ${have_xapian} = "0" ]; then
 errors=$((errors + 1))
 fi

+# Compaction is only supported on Xapian > 1.2.6
+have_xapian_compact=0
+if [ ${have_xapian} = "1" ]; then
+printf "Checking for Xapian compact support... "
+case "${xapian_version}" in
+0.*|1.[01].*|1.2.[0-5])
+printf "No (only available with Xapian > 1.2.6).\n" ;;
+[1-9]*.[0-9]*.[0-9]*)
+have_xapian_compact=1
+printf "Yes.\n" ;;
+*)
+printf "Unknown version.\n" ;;
+esac
+fi
+
 printf "Checking for GMime development files... "
 have_gmime=0
 IFS=';'
@@ -729,6 +745,9 @@ HAVE_STRCASESTR = ${have_strcasestr}
 # build its own version)
 HAVE_STRSEP = ${have_strsep}

+# Whether the Xapian version in use supports compaction
+HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
+
 # Whether the getpwuid_r function is standards-compliant
 # (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS
 # to enable the standards-compliant version -- needed for Solaris)
@@ -787,13 +806,15 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
\$(GMIME_CFLAGS)  \\
   -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\
   -DHAVE_STRSEP=\$(HAVE_STRSEP) \\
   -DSTD_GETPWUID=\$(STD_GETPWUID)   \\
-  -DSTD_ASCTIME=\$(STD_ASCTIME)
+  -DSTD_ASCTIME=\$(STD_ASCTIME) \\
+   -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
 \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
 -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)   \\
 -DHAVE_STRSEP=\$(HAVE_STRSEP)   \\
 -DSTD_GETPWUID=\$(STD_GETPWUID) \\
--DSTD_ASCTIME=\$(STD_ASCTIME)
+-DSTD_ASCTIME=\$(STD_ASCTIME)   \\
+ -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
diff --git a/lib/database.cc b/lib/database.cc
index 5cc0765..b71751d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -268,6 +268,8 @@ notmuch_status_to_string (notmuch_status_t status)
return "Unbalanced number of calls to notmuch_message_freeze/thaw";
 case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
return "Unbalanced number of calls to 
notmuch_database_begin_atomic/end_atomic";
+case NOTMUCH_STATUS_UNSUPPORTED_OPERATION:
+   return "Unsupported operation";
 default:
 case NOTMUCH_STATUS_LAST_STATUS:
return "Unknown error status value";
@@ -800,6 +802,95 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch->date_range_processor = NULL;
 }

+class NotmuchCompactor : public Xapian::Compactor
+{
+public:
+virtual void
+set_status (const std::string , const std::string )
+{
+   if (status.length() == 0)
+   printf ("compacting table %s:\n", table.c_str());
+   else
+   printf (" %s\n", status.c_str());
+}
+};
+
+#if HAVE_XAPIAN_COMPACT
+notmuch_status_t
+notmuch_database_compact_close (notmuch_database_t *notmuch)
+{
+void *local = talloc_new (NULL);
+NotmuchCompactor compactor;
+char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;
+notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
+
+if (! (notmuch_path = talloc_asprintf (local, "%s/%s", notmuch->path, 
".notmuch&q

No subject

2013-08-23 Thread Ben Gamari
From: Ben Gamari <bgamari.f...@gmail.com>
Subject: [PATCH] notmuch compact support (v3)
In-Reply-To: 

Here is the latest (and hopefully last) iteration of my patchset introducing a
compact command. The set has been rebased on top of master, a manpage has been
added, and error handling has been greatly improved.

Cheers,

- Ben



[Olivier Berger] Bug#720543: notmuch: Please provide a script allowing to compact the xapian base

2013-08-23 Thread Ben Gamari
Olivier Berger said:

> I think it would be great to provide a script allowing to do some
> maintenance on the xapian DB to improve performance, as explained in
> http://notmuchmail.org/performance/ (Compact your database).

A few months^H^H^H nearly a year ago I had a patch[1] adding a `notmuch
compact` command.  I forgot about it until you mentioned this but I
think I'll try to finish the fix it up and resubmit for merge tonight.

Cheers,

- Ben


[1] http://notmuchmail.org/pipermail/notmuch/2012/012213.html
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: 



Re: [Olivier Berger] Bug#720543: notmuch: Please provide a script allowing to compact the xapian base

2013-08-23 Thread Ben Gamari
Olivier Berger said:

 I think it would be great to provide a script allowing to do some
 maintenance on the xapian DB to improve performance, as explained in
 http://notmuchmail.org/performance/ (Compact your database).

A few months^H^H^H nearly a year ago I had a patch[1] adding a `notmuch
compact` command.  I forgot about it until you mentioned this but I
think I'll try to finish the fix it up and resubmit for merge tonight.

Cheers,

- Ben


[1] http://notmuchmail.org/pipermail/notmuch/2012/012213.html


pgpLiimngLecC.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[no subject]

2013-08-23 Thread Ben Gamari
From: Ben Gamari bgamari.f...@gmail.com
Subject: [PATCH] notmuch compact support (v3)
In-Reply-To: 

Here is the latest (and hopefully last) iteration of my patchset introducing a
compact command. The set has been rebased on top of master, a manpage has been
added, and error handling has been greatly improved.

Cheers,

- Ben

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] notmuch-compact: Add man page

2013-08-23 Thread Ben Gamari
Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 man/Makefile.local |  1 +
 man/man1/notmuch-compact.1 | 33 +
 2 files changed, 34 insertions(+)
 create mode 100644 man/man1/notmuch-compact.1

diff --git a/man/Makefile.local b/man/Makefile.local
index 216aaa0..57910b7 100644
--- a/man/Makefile.local
+++ b/man/Makefile.local
@@ -8,6 +8,7 @@ MAIN_PAGE := $(dir)/man1/notmuch.1
 
 MAN1 := \
$(MAIN_PAGE) \
+   $(dir)/man1/notmuch-compact.1 \
$(dir)/man1/notmuch-config.1 \
$(dir)/man1/notmuch-count.1 \
$(dir)/man1/notmuch-dump.1 \
diff --git a/man/man1/notmuch-compact.1 b/man/man1/notmuch-compact.1
new file mode 100644
index 000..48f76c3
--- /dev/null
+++ b/man/man1/notmuch-compact.1
@@ -0,0 +1,33 @@
+.TH NOTMUCH-COMPACT 1 2013-08-23 Notmuch 0.16
+.SH NAME
+notmuch-compact \- compact the notmuch database
+.SH SYNOPSIS
+
+.B notmuch compact
+
+.SH DESCRIPTION
+
+The
+.B compact
+command can be used to compact the notmuch database. This can both reduce
+the space required by the database and improve performance.
+
+The compacted database is built in a temporary directory and is later
+moved into the place of the origin database. The original uncompacted
+database is preserved to be deleted by the user as desired.
+
+.RE
+.SH ENVIRONMENT
+The following environment variables can be used to control the
+behavior of notmuch.
+.TP
+.B NOTMUCH_CONFIG
+Specifies the location of the notmuch configuration file. Notmuch will
+use ${HOME}/.notmuch\-config if this variable is not set.
+.SH SEE ALSO
+
+\fBnotmuch\fR(1), \fBnotmuch-count\fR(1), \fBnotmuch-dump\fR(1),
+\fBnotmuch-hooks\fR(5), \fBnotmuch-insert\fR(1), \fBnotmuch-new\fR(1),
+\fBnotmuch-reply\fR(1), \fBnotmuch-restore\fR(1), \fBnotmuch-search\fR(1),
+\fBnotmuch-search-terms\fR(7), \fBnotmuch-show\fR(1),
+\fBnotmuch-tag\fR(1)
-- 
1.8.1.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/3] database: Add notmuch_database_compact_close

2013-08-23 Thread Ben Gamari
This function uses Xapian's Compactor machinery to compact the notmuch
database. The compacted database is built in a temporary directory and
later moved into place while the original uncompacted database is
preserved.

Signed-off-by: Ben Gamari bgamari.f...@gmail.com
---
 configure   | 27 +++--
 lib/database.cc | 91 +
 lib/notmuch.h   | 15 ++
 3 files changed, 130 insertions(+), 3 deletions(-)

diff --git a/configure b/configure
index 6166917..ee9e887 100755
--- a/configure
+++ b/configure
@@ -277,7 +277,8 @@ printf Checking for Xapian development files... 
 have_xapian=0
 for xapian_config in ${XAPIAN_CONFIG}; do
 if ${xapian_config} --version  /dev/null 21; then
-   printf Yes (%s).\n $(${xapian_config} --version | sed -e 's/.* //')
+   xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
+   printf Yes (%s).\n ${xapian_version}
have_xapian=1
xapian_cxxflags=$(${xapian_config} --cxxflags)
xapian_ldflags=$(${xapian_config} --libs)
@@ -289,6 +290,21 @@ if [ ${have_xapian} = 0 ]; then
 errors=$((errors + 1))
 fi
 
+# Compaction is only supported on Xapian  1.2.6
+have_xapian_compact=0
+if [ ${have_xapian} = 1 ]; then
+printf Checking for Xapian compact support... 
+case ${xapian_version} in
+0.*|1.[01].*|1.2.[0-5])
+printf No (only available with Xapian  1.2.6).\n ;;
+[1-9]*.[0-9]*.[0-9]*)
+have_xapian_compact=1
+printf Yes.\n ;;
+*)
+printf Unknown version.\n ;;
+esac
+fi
+
 printf Checking for GMime development files... 
 have_gmime=0
 IFS=';'
@@ -729,6 +745,9 @@ HAVE_STRCASESTR = ${have_strcasestr}
 # build its own version)
 HAVE_STRSEP = ${have_strsep}
 
+# Whether the Xapian version in use supports compaction
+HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
+
 # Whether the getpwuid_r function is standards-compliant
 # (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS
 # to enable the standards-compliant version -- needed for Solaris)
@@ -787,13 +806,15 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
\$(GMIME_CFLAGS)  \\
   -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\
   -DHAVE_STRSEP=\$(HAVE_STRSEP) \\
   -DSTD_GETPWUID=\$(STD_GETPWUID)   \\
-  -DSTD_ASCTIME=\$(STD_ASCTIME)
+  -DSTD_ASCTIME=\$(STD_ASCTIME) \\
+   -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
 \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
 -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)   \\
 -DHAVE_STRSEP=\$(HAVE_STRSEP)   \\
 -DSTD_GETPWUID=\$(STD_GETPWUID) \\
--DSTD_ASCTIME=\$(STD_ASCTIME)
+-DSTD_ASCTIME=\$(STD_ASCTIME)   \\
+ -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
diff --git a/lib/database.cc b/lib/database.cc
index 5cc0765..b71751d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -268,6 +268,8 @@ notmuch_status_to_string (notmuch_status_t status)
return Unbalanced number of calls to notmuch_message_freeze/thaw;
 case NOTMUCH_STATUS_UNBALANCED_ATOMIC:
return Unbalanced number of calls to 
notmuch_database_begin_atomic/end_atomic;
+case NOTMUCH_STATUS_UNSUPPORTED_OPERATION:
+   return Unsupported operation;
 default:
 case NOTMUCH_STATUS_LAST_STATUS:
return Unknown error status value;
@@ -800,6 +802,95 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch-date_range_processor = NULL;
 }
 
+class NotmuchCompactor : public Xapian::Compactor
+{
+public:
+virtual void
+set_status (const std::string table, const std::string status)
+{
+   if (status.length() == 0)
+   printf (compacting table %s:\n, table.c_str());
+   else
+   printf ( %s\n, status.c_str());
+}
+};
+
+#if HAVE_XAPIAN_COMPACT
+notmuch_status_t
+notmuch_database_compact_close (notmuch_database_t *notmuch)
+{
+void *local = talloc_new (NULL);
+NotmuchCompactor compactor;
+char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;
+notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
+
+if (! (notmuch_path = talloc_asprintf (local, %s/%s, notmuch-path, 
.notmuch))) {
+   ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE;
+}
+
+if (! (xapian_path = talloc_asprintf (local, %s/%s, notmuch_path, 
xapian))) {
+   ret = NOTMUCH_STATUS_OUT_OF_MEMORY;
+   goto DONE

[ANNOUNCE] notmuch-web 0.1.0

2013-03-03 Thread Ben Gamari
John Lenz  writes:

> I have been recently working on a web interface to notmuch.  The past two
> weeks, I have used it as my only email client and since the trial period
> was a great success (I had no major issues or problems and never had to
> revert to another client), I have decided to release an initial version.  I
> am planning on continuing to use this web client as my sole notmuch client
> and will continue developing features (most identified by my two week trial
> period :)  Combined with the It's All Text Firefox plugin for compose, I am
> really happy with the result.
>
Very nice! Am I correct in assuming that you are scraping the output of
the command line utility? Last I knew there were some memory management
issues[1] that needed to be resolved. I've been hoping to fix this at
some point but there's always something else to do.

Cheers,

- Ben

[1] See id:87aaa0l1tc.fsf at gmail.com and id:8739h1pbaq.fsf at gmail.com


Re: [ANNOUNCE] notmuch-web 0.1.0

2013-03-03 Thread Ben Gamari
John Lenz l...@math.uic.edu writes:

 I have been recently working on a web interface to notmuch.  The past two
 weeks, I have used it as my only email client and since the trial period
 was a great success (I had no major issues or problems and never had to
 revert to another client), I have decided to release an initial version.  I
 am planning on continuing to use this web client as my sole notmuch client
 and will continue developing features (most identified by my two week trial
 period :)  Combined with the It's All Text Firefox plugin for compose, I am
 really happy with the result.

Very nice! Am I correct in assuming that you are scraping the output of
the command line utility? Last I knew there were some memory management
issues[1] that needed to be resolved. I've been hoping to fix this at
some point but there's always something else to do.

Cheers,

- Ben

[1] See id:87aaa0l1tc@gmail.com and id:8739h1pbaq@gmail.com
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] Add notmuch compact command

2012-10-17 Thread Ben Gamari
---
 Makefile.local|1 +
 notmuch-client.h  |3 +++
 notmuch-compact.c |   43 +++
 notmuch.c |3 +++
 4 files changed, 50 insertions(+)
 create mode 100644 notmuch-compact.c

diff --git a/Makefile.local b/Makefile.local
index 7f2d4f1..e050ba6 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -258,6 +258,7 @@ notmuch_client_srcs =   \
gmime-filter-headers.c  \
hooks.c \
notmuch.c   \
+   notmuch-compact.c   \
notmuch-config.c\
notmuch-count.c \
notmuch-dump.c  \
diff --git a/notmuch-client.h b/notmuch-client.h
index ae9344b..a6c624b 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -157,6 +157,9 @@ int
 notmuch_cat_command (void *ctx, int argc, char *argv[]);

 int
+notmuch_compact_command (void *ctx, int argc, char *argv[]);
+
+int
 notmuch_config_command (void *ctx, int argc, char *argv[]);

 const char *
diff --git a/notmuch-compact.c b/notmuch-compact.c
new file mode 100644
index 000..6deb2ec
--- /dev/null
+++ b/notmuch-compact.c
@@ -0,0 +1,43 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright ? 2009 Carl Worth
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Carl Worth 
+ */
+
+#include "notmuch-client.h"
+
+int
+notmuch_compact_command (void *ctx, unused (int argc), unused (char *argv[]))
+{
+notmuch_config_t *config;
+notmuch_database_t *notmuch;
+
+config = notmuch_config_open (ctx, NULL, NULL);
+if (config == NULL)
+   return 1;
+
+if (notmuch_database_open (notmuch_config_get_database_path (config),
+  NOTMUCH_DATABASE_MODE_READ_WRITE, ))
+   return 1;
+
+printf ("Compacting database... ");
+notmuch_database_close_compact (notmuch);
+printf ("Done.\n");
+notmuch_database_destroy (notmuch);
+
+return 0;
+}
diff --git a/notmuch.c b/notmuch.c
index 477a09c..7b6c5ae 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -74,6 +74,9 @@ static command_t commands[] = {
 { "restore", notmuch_restore_command,
   "[--accumulate] []",
   "Restore the tags from the given dump file (see 'dump')." },
+{ "compact", notmuch_compact_command,
+  NULL,
+  "Compacts the database." },
 { "config", notmuch_config_command,
   "[get|set] . [value ...]",
   "Get or set settings in the notmuch configuration file." },
-- 
1.7.10.4



[PATCH 2/3] Produce status messages during compacting

2012-10-17 Thread Ben Gamari
---
 lib/database.cc |   15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 6e83a61..49aa36d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -780,11 +780,24 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch->value_range_processor = NULL;
 }

+class NotmuchCompactor : public Xapian::Compactor
+{
+public:
+virtual void
+set_status (const std::string , const std::string )
+{
+   if (status.length() == 0)
+   fprintf (stderr, "compacting table %s:\n", table.c_str());
+   else
+   fprintf (stderr, " %s\n", status.c_str());
+}
+};
+
 void
 notmuch_database_close_compact (notmuch_database_t *notmuch)
 {
 void *local = talloc_new (NULL);
-Xapian::Compactor compactor;
+NotmuchCompactor compactor;
 char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;

 #if HAVE_XAPIAN_COMPACT
-- 
1.7.10.4



[PATCH 1/3] Add notmuch_database_close_compact

2012-10-17 Thread Ben Gamari
---
 configure   |   21 -
 lib/database.cc |   54 ++
 lib/notmuch.h   |   14 ++
 3 files changed, 88 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index acb90a8..6551b13 100755
--- a/configure
+++ b/configure
@@ -270,7 +270,8 @@ printf "Checking for Xapian development files... "
 have_xapian=0
 for xapian_config in ${XAPIAN_CONFIG}; do
 if ${xapian_config} --version > /dev/null 2>&1; then
-   printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //')
+   xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
+   printf "Yes (%s).\n" ${xapian_version}
have_xapian=1
xapian_cxxflags=$(${xapian_config} --cxxflags)
xapian_ldflags=$(${xapian_config} --libs)
@@ -282,6 +283,20 @@ if [ ${have_xapian} = "0" ]; then
 errors=$((errors + 1))
 fi

+have_xapian_compact=0
+if [ ${have_xapian} = "1" ]; then
+printf "Checking for Xapian compact support... "
+case "${xapian_version}" in
+0.*|1.[01].*|1.2.[0-5])
+printf "No.\n" ;;
+[1-9]*.[0-9]*.[0-9]*) 
+have_xapian_compact=1
+printf "Yes.\n" ;;
+*)
+printf "Unknown version.\n" ;;
+esac
+fi
+
 printf "Checking for GMime development files... "
 have_gmime=0
 IFS=';'
@@ -679,6 +694,9 @@ LINKER_RESOLVES_LIBRARY_DEPENDENCIES = 
${linker_resolves_library_dependencies}
 XAPIAN_CXXFLAGS = ${xapian_cxxflags}
 XAPIAN_LDFLAGS = ${xapian_ldflags}

+# Whether compact is supported by this version of Xapian
+HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
+
 # Flags needed to compile and link against GMime-2.4
 GMIME_CFLAGS = ${gmime_cflags}
 GMIME_LDFLAGS = ${gmime_ldflags}
@@ -715,6 +733,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
\$(GMIME_CFLAGS)  \\
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
 \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
+ -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)   \\
  -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
diff --git a/lib/database.cc b/lib/database.cc
index 761dc1a..6e83a61 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -781,6 +781,60 @@ notmuch_database_close (notmuch_database_t *notmuch)
 }

 void
+notmuch_database_close_compact (notmuch_database_t *notmuch)
+{
+void *local = talloc_new (NULL);
+Xapian::Compactor compactor;
+char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;
+
+#if HAVE_XAPIAN_COMPACT
+if (! (notmuch_path = talloc_asprintf (local, "%s/%s", notmuch->path, 
".notmuch"))) {
+   fprintf (stderr, "Out of memory\n");
+   goto DONE;
+}
+
+if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, 
"xapian"))) {
+   fprintf (stderr, "Out of memory\n");
+   goto DONE;
+}
+
+if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", 
xapian_path))) {
+   fprintf (stderr, "Out of memory\n");
+   goto DONE;
+}
+
+if (! (old_xapian_path = talloc_asprintf (local, "%s.old", xapian_path))) {
+   fprintf (stderr, "Out of memory\n");
+   goto DONE;
+}
+
+try {
+   compactor.set_renumber(false);
+   compactor.add_source(xapian_path);
+   compactor.set_destdir(compact_xapian_path);
+   compactor.compact();
+
+   if (rename(xapian_path, old_xapian_path)) {
+   fprintf (stderr, "Error moving old database out of the way\n");
+   goto DONE;
+   }
+
+   if (rename(compact_xapian_path, xapian_path)) {
+   fprintf (stderr, "Error moving compacted database\n");
+   goto DONE;
+   }
+} catch (Xapian::InvalidArgumentError e) {
+   fprintf (stderr, "Error while compacting: %s", e.get_msg().c_str());
+}
+
+#endif
+
+notmuch_database_close(notmuch);
+DONE:
+talloc_free(local);
+}
+
+void
 notmuch_database_destroy (notmuch_database_t *notmuch)
 {
 notmuch_database_close (notmuch);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 3633bed..50babfb 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -215,6 +215,20 @@ notmuch_database_open (const char *path,
 void
 notmuch_database_close (notmuch_database_t *database);

+/* Close the given notmuch database and then compact it.
+ *
+ * After notmuch_database_close_compact has been called, calls to
+ * other functions on objects derived from this database may either
+ * behave as if the database had not been closed (e.g., if the
+ * required data has been cached) or may fail with a
+ * NOTMUCH_STATUS_XAPIAN_EXCEPTION.
+ *
+ * notmuch_database_close_compact can be called multiple times.  Later
+ * calls have no effect.
+ */
+void
+notmuch_database_close_compact (notmuch_database_t *notmuch);
+
 

[PATCH] notmuch compact support

2012-10-17 Thread Ben Gamari
Here is a new spin of my patchset introduced in late August
(id:"1345476704-17091-1-git-send-email-bgamari.foss at gmail.com") adding a
compact command to libnotmuch and the command line frontend.

I believe the concerns raised in August have been addressed, but correct me if
I'm wrong.

Cheers,

- Ben




[PATCH] notmuch compact support

2012-10-17 Thread Ben Gamari
Here is a new spin of my patchset introduced in late August
(id:1345476704-17091-1-git-send-email-bgamari.f...@gmail.com) adding a
compact command to libnotmuch and the command line frontend.

I believe the concerns raised in August have been addressed, but correct me if
I'm wrong.

Cheers,

- Ben


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 3/3] Add notmuch compact command

2012-10-17 Thread Ben Gamari
---
 Makefile.local|1 +
 notmuch-client.h  |3 +++
 notmuch-compact.c |   43 +++
 notmuch.c |3 +++
 4 files changed, 50 insertions(+)
 create mode 100644 notmuch-compact.c

diff --git a/Makefile.local b/Makefile.local
index 7f2d4f1..e050ba6 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -258,6 +258,7 @@ notmuch_client_srcs =   \
gmime-filter-headers.c  \
hooks.c \
notmuch.c   \
+   notmuch-compact.c   \
notmuch-config.c\
notmuch-count.c \
notmuch-dump.c  \
diff --git a/notmuch-client.h b/notmuch-client.h
index ae9344b..a6c624b 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -157,6 +157,9 @@ int
 notmuch_cat_command (void *ctx, int argc, char *argv[]);
 
 int
+notmuch_compact_command (void *ctx, int argc, char *argv[]);
+
+int
 notmuch_config_command (void *ctx, int argc, char *argv[]);
 
 const char *
diff --git a/notmuch-compact.c b/notmuch-compact.c
new file mode 100644
index 000..6deb2ec
--- /dev/null
+++ b/notmuch-compact.c
@@ -0,0 +1,43 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright © 2009 Carl Worth
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Carl Worth cwo...@cworth.org
+ */
+
+#include notmuch-client.h
+
+int
+notmuch_compact_command (void *ctx, unused (int argc), unused (char *argv[]))
+{
+notmuch_config_t *config;
+notmuch_database_t *notmuch;
+
+config = notmuch_config_open (ctx, NULL, NULL);
+if (config == NULL)
+   return 1;
+
+if (notmuch_database_open (notmuch_config_get_database_path (config),
+  NOTMUCH_DATABASE_MODE_READ_WRITE, notmuch))
+   return 1;
+
+printf (Compacting database... );
+notmuch_database_close_compact (notmuch);
+printf (Done.\n);
+notmuch_database_destroy (notmuch);
+
+return 0;
+}
diff --git a/notmuch.c b/notmuch.c
index 477a09c..7b6c5ae 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -74,6 +74,9 @@ static command_t commands[] = {
 { restore, notmuch_restore_command,
   [--accumulate] [filename],
   Restore the tags from the given dump file (see 'dump'). },
+{ compact, notmuch_compact_command,
+  NULL,
+  Compacts the database. },
 { config, notmuch_config_command,
   [get|set] section.item [value ...],
   Get or set settings in the notmuch configuration file. },
-- 
1.7.10.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/3] Produce status messages during compacting

2012-10-17 Thread Ben Gamari
---
 lib/database.cc |   15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 6e83a61..49aa36d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -780,11 +780,24 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch-value_range_processor = NULL;
 }
 
+class NotmuchCompactor : public Xapian::Compactor
+{
+public:
+virtual void
+set_status (const std::string table, const std::string status)
+{
+   if (status.length() == 0)
+   fprintf (stderr, compacting table %s:\n, table.c_str());
+   else
+   fprintf (stderr,  %s\n, status.c_str());
+}
+};
+
 void
 notmuch_database_close_compact (notmuch_database_t *notmuch)
 {
 void *local = talloc_new (NULL);
-Xapian::Compactor compactor;
+NotmuchCompactor compactor;
 char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;
 
 #if HAVE_XAPIAN_COMPACT
-- 
1.7.10.4

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/3] Add notmuch_database_close_compact

2012-08-21 Thread Ben Gamari
Tomi Ollila  writes:

> On Mon, Aug 20 2012, Ben Gamari  wrote:
>
>> ---
>>  configure   |   25 -
>>  lib/database.cc |   54 
>> ++
>>  lib/notmuch.h   |   14 ++
>>  3 files changed, 92 insertions(+), 1 deletion(-)
>>
>> diff --git a/configure b/configure
>> index dc0dba4..370fedd 100755
>> --- a/configure
>> +++ b/configure
>> @@ -270,7 +270,8 @@ printf "Checking for Xapian development files... "
>>  have_xapian=0
>>  for xapian_config in ${XAPIAN_CONFIG}; do
>>  if ${xapian_config} --version > /dev/null 2>&1; then
>> -printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //')
>> +xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
>> +printf "Yes (%s).\n" ${xapian_version}
>>  have_xapian=1
>>  xapian_cxxflags=$(${xapian_config} --cxxflags)
>>  xapian_ldflags=$(${xapian_config} --libs)
>> @@ -282,6 +283,24 @@ if [ ${have_xapian} = "0" ]; then
>>  errors=$((errors + 1))
>>  fi
>>  
>> +have_xapian_compact=0
>> +if [ ${have_xapian} = "1" ]; then
>> +printf "Checking for Xapian compact support... "
>> +IFS='.'
>> +old_args=$@
>> +set -- ${xapian_version}
>> +xapian_major_version=$1
>> +xapian_minor_version=$2
>> +xapian_subminor_version=$3
>> +set -- ${old_args}
>
> The part above breaks the argument vector in case there are spaces in 
> args (I though $IFS chars, but try the script), execute:
>
Hmmm, I suppose so.

> $ echo '#!/bin/bash
> IFS=.
> for x in "$@"; do echo $x; done; echo
> foo=$@
> for x in $foo; do echo $x; done; echo
> set -- $foo
> for x in "$@"; do echo $x; done; echo
> ' > foo.bash
>
> $ bash foo.bash a "b c" d
>
> Also, after processing, IFS is not restored (to $DEFAULT_IFS)
>
I assumed this would be alright since IFS is set in the next 

> an alternative is to put the code in function like the following
> way:
>
Sounds good to me.

> Hmm, I guess the check above is to determine whether xapian version is
> 1.2.6 or newer, but is there xapian version 1.1.6 or 1.0.6 or 0.3.0 or so ?
>
Eh? 1.2.6 is the first Xapian release to have Compactor exposed in the
public API.

> I am not qualified to comment about compaction itself :)
>
Nor am I really. I just noticed that this functionality was blocking on
library support which is now in place. It seemed that a pretty
straightforward thing to implement and it hasn't broken my index yet.

> In the last patch you give copyright to Carl (which is OK). However I'd
> debate whether it is good practise to declare Carl as author; I'd say that
> is OK if he agrees to claim authorship to your potentially shi^H^H^Hperfect
> code ;)
>
Yikes. That certainly wasn't intentional. I'll fix this in the next
iteration.

> There are at least a few style issues below: e.g. no space between function
> name and opening parenthesis.
>
Duly noted.

Thanks!

Cheers,

- Ben


[PATCH RFC?] Compactification support

2012-08-21 Thread Ben Gamari
Jani Nikula  writes:

> On Mon, 20 Aug 2012, Ben Gamari  wrote:
>> Here is a small patchset adding support for database compaction when
>> supported by Xapian. Here I add a function which both compacts and
>> closes the database.
>
> What are the benefits of this compared to xapian-compact [1]?
>
I think it's a bit of a shame you are currently forced to interact with
Xapian directly (an implementation detail) to compact the database. This
set seeks to abstract this out. This is especially important since it's
not too difficult to get the invocation of xapian-compact wrong
(e.g. forget --no-renumber).

Also, in the past Carl (id:"87ocl1lut1.fsf at yoom.home.cworth.org") has
mentioned it would be nice if compact would be run after the initial run
of "notmuch new". This patchset sets the groundwork to enable this.

Cheers,

- Ben



[PATCH 3/3] Add notmuch compact command

2012-08-20 Thread Ben Gamari
---
 Makefile.local|1 +
 notmuch-client.h  |3 +++
 notmuch-compact.c |   43 +++
 notmuch.c |3 +++
 4 files changed, 50 insertions(+)
 create mode 100644 notmuch-compact.c

diff --git a/Makefile.local b/Makefile.local
index de984ab..848605d 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -278,6 +278,7 @@ notmuch_client_srcs =   \
gmime-filter-headers.c  \
hooks.c \
notmuch.c   \
+   notmuch-compact.c   \
notmuch-config.c\
notmuch-count.c \
notmuch-dump.c  \
diff --git a/notmuch-client.h b/notmuch-client.h
index ae9344b..a6c624b 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -157,6 +157,9 @@ int
 notmuch_cat_command (void *ctx, int argc, char *argv[]);

 int
+notmuch_compact_command (void *ctx, int argc, char *argv[]);
+
+int
 notmuch_config_command (void *ctx, int argc, char *argv[]);

 const char *
diff --git a/notmuch-compact.c b/notmuch-compact.c
new file mode 100644
index 000..6deb2ec
--- /dev/null
+++ b/notmuch-compact.c
@@ -0,0 +1,43 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright ? 2009 Carl Worth
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Carl Worth 
+ */
+
+#include "notmuch-client.h"
+
+int
+notmuch_compact_command (void *ctx, unused (int argc), unused (char *argv[]))
+{
+notmuch_config_t *config;
+notmuch_database_t *notmuch;
+
+config = notmuch_config_open (ctx, NULL, NULL);
+if (config == NULL)
+   return 1;
+
+if (notmuch_database_open (notmuch_config_get_database_path (config),
+  NOTMUCH_DATABASE_MODE_READ_WRITE, ))
+   return 1;
+
+printf ("Compacting database... ");
+notmuch_database_close_compact (notmuch);
+printf ("Done.\n");
+notmuch_database_destroy (notmuch);
+
+return 0;
+}
diff --git a/notmuch.c b/notmuch.c
index 477a09c..7b6c5ae 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -74,6 +74,9 @@ static command_t commands[] = {
 { "restore", notmuch_restore_command,
   "[--accumulate] []",
   "Restore the tags from the given dump file (see 'dump')." },
+{ "compact", notmuch_compact_command,
+  NULL,
+  "Compacts the database." },
 { "config", notmuch_config_command,
   "[get|set] . [value ...]",
   "Get or set settings in the notmuch configuration file." },
-- 
1.7.9.5



[PATCH 2/3] Produce status messages during compacting

2012-08-20 Thread Ben Gamari
---
 lib/database.cc |   15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 6e83a61..49aa36d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -780,11 +780,24 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch->value_range_processor = NULL;
 }

+class NotmuchCompactor : public Xapian::Compactor
+{
+public:
+virtual void
+set_status (const std::string , const std::string )
+{
+   if (status.length() == 0)
+   fprintf (stderr, "compacting table %s:\n", table.c_str());
+   else
+   fprintf (stderr, " %s\n", status.c_str());
+}
+};
+
 void
 notmuch_database_close_compact (notmuch_database_t *notmuch)
 {
 void *local = talloc_new (NULL);
-Xapian::Compactor compactor;
+NotmuchCompactor compactor;
 char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;

 #if HAVE_XAPIAN_COMPACT
-- 
1.7.9.5



[PATCH 1/3] Add notmuch_database_close_compact

2012-08-20 Thread Ben Gamari
---
 configure   |   25 -
 lib/database.cc |   54 ++
 lib/notmuch.h   |   14 ++
 3 files changed, 92 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index dc0dba4..370fedd 100755
--- a/configure
+++ b/configure
@@ -270,7 +270,8 @@ printf "Checking for Xapian development files... "
 have_xapian=0
 for xapian_config in ${XAPIAN_CONFIG}; do
 if ${xapian_config} --version > /dev/null 2>&1; then
-   printf "Yes (%s).\n" $(${xapian_config} --version | sed -e 's/.* //')
+   xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
+   printf "Yes (%s).\n" ${xapian_version}
have_xapian=1
xapian_cxxflags=$(${xapian_config} --cxxflags)
xapian_ldflags=$(${xapian_config} --libs)
@@ -282,6 +283,24 @@ if [ ${have_xapian} = "0" ]; then
 errors=$((errors + 1))
 fi

+have_xapian_compact=0
+if [ ${have_xapian} = "1" ]; then
+printf "Checking for Xapian compact support... "
+IFS='.'
+old_args=$@
+set -- ${xapian_version}
+xapian_major_version=$1
+xapian_minor_version=$2
+xapian_subminor_version=$3
+set -- ${old_args}
+if [ "${xapian_major_version}" -gt 1 ] || [ ${xapian_minor_version} -gt 2 
] || [ ${xapian_subminor_version} -ge 6 ]; then
+   printf "Yes.\n"
+   have_xapian_compact=1
+else
+   printf "No.\n"
+fi
+fi
+
 printf "Checking for GMime development files... "
 have_gmime=0
 IFS=';'
@@ -675,6 +694,9 @@ LINKER_RESOLVES_LIBRARY_DEPENDENCIES = 
${linker_resolves_library_dependencies}
 XAPIAN_CXXFLAGS = ${xapian_cxxflags}
 XAPIAN_LDFLAGS = ${xapian_ldflags}

+# Whether compact is supported by this version of Xapian
+HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
+
 # Flags needed to compile and link against GMime-2.4
 GMIME_CFLAGS = ${gmime_cflags}
 GMIME_LDFLAGS = ${gmime_ldflags}
@@ -711,6 +733,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
\$(GMIME_CFLAGS)  \\
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
 \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
+ -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)   \\
  -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
diff --git a/lib/database.cc b/lib/database.cc
index 761dc1a..6e83a61 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -781,6 +781,60 @@ notmuch_database_close (notmuch_database_t *notmuch)
 }

 void
+notmuch_database_close_compact (notmuch_database_t *notmuch)
+{
+void *local = talloc_new (NULL);
+Xapian::Compactor compactor;
+char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;
+
+#if HAVE_XAPIAN_COMPACT
+if (! (notmuch_path = talloc_asprintf (local, "%s/%s", notmuch->path, 
".notmuch"))) {
+   fprintf (stderr, "Out of memory\n");
+   goto DONE;
+}
+
+if (! (xapian_path = talloc_asprintf (local, "%s/%s", notmuch_path, 
"xapian"))) {
+   fprintf (stderr, "Out of memory\n");
+   goto DONE;
+}
+
+if (! (compact_xapian_path = talloc_asprintf (local, "%s.compact", 
xapian_path))) {
+   fprintf (stderr, "Out of memory\n");
+   goto DONE;
+}
+
+if (! (old_xapian_path = talloc_asprintf (local, "%s.old", xapian_path))) {
+   fprintf (stderr, "Out of memory\n");
+   goto DONE;
+}
+
+try {
+   compactor.set_renumber(false);
+   compactor.add_source(xapian_path);
+   compactor.set_destdir(compact_xapian_path);
+   compactor.compact();
+
+   if (rename(xapian_path, old_xapian_path)) {
+   fprintf (stderr, "Error moving old database out of the way\n");
+   goto DONE;
+   }
+
+   if (rename(compact_xapian_path, xapian_path)) {
+   fprintf (stderr, "Error moving compacted database\n");
+   goto DONE;
+   }
+} catch (Xapian::InvalidArgumentError e) {
+   fprintf (stderr, "Error while compacting: %s", e.get_msg().c_str());
+}
+
+#endif
+
+notmuch_database_close(notmuch);
+DONE:
+talloc_free(local);
+}
+
+void
 notmuch_database_destroy (notmuch_database_t *notmuch)
 {
 notmuch_database_close (notmuch);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 3633bed..50babfb 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -215,6 +215,20 @@ notmuch_database_open (const char *path,
 void
 notmuch_database_close (notmuch_database_t *database);

+/* Close the given notmuch database and then compact it.
+ *
+ * After notmuch_database_close_compact has been called, calls to
+ * other functions on objects derived from this database may either
+ * behave as if the database had not been closed (e.g., if the
+ * required data has been cached) or may fail with a
+ * NOTMUCH_STATUS_XAPIAN_EXCEPTION.
+ *
+ * notmuch_database_close_compact 

[PATCH RFC?] Compactification support

2012-08-20 Thread Ben Gamari

Here is a small patchset adding support for database compaction when supported
by Xapian. Here I add a function which both compacts and closes the database.
This is due to it being not entirely clear whether one is allowed to compact
the database while it is being held open for read/write.


Since it is nice to be able to hold the read/write lock while compacting, I
close the database only after the compaction has finished which Xapian seems to
be fine with. That being said, the database is moved after compaction, so it 
seems appropriate to lump compaction in with closing the database.

Lastly, I wasn't entirely sure whether/how Xapian's status messages should be
presented to the user. At the moment I simply spit them out on stderr. Perhaps
I should just pass a callback to notmuch_database_close_compact and leave this
to the caller?

Cheers,

- Ben



[PATCH RFC?] Compactification support

2012-08-20 Thread Ben Gamari

Here is a small patchset adding support for database compaction when supported
by Xapian. Here I add a function which both compacts and closes the database.
This is due to it being not entirely clear whether one is allowed to compact
the database while it is being held open for read/write.


Since it is nice to be able to hold the read/write lock while compacting, I
close the database only after the compaction has finished which Xapian seems to
be fine with. That being said, the database is moved after compaction, so it 
seems appropriate to lump compaction in with closing the database.

Lastly, I wasn't entirely sure whether/how Xapian's status messages should be
presented to the user. At the moment I simply spit them out on stderr. Perhaps
I should just pass a callback to notmuch_database_close_compact and leave this
to the caller?

Cheers,

- Ben

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 2/3] Produce status messages during compacting

2012-08-20 Thread Ben Gamari
---
 lib/database.cc |   15 ++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/lib/database.cc b/lib/database.cc
index 6e83a61..49aa36d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -780,11 +780,24 @@ notmuch_database_close (notmuch_database_t *notmuch)
 notmuch-value_range_processor = NULL;
 }
 
+class NotmuchCompactor : public Xapian::Compactor
+{
+public:
+virtual void
+set_status (const std::string table, const std::string status)
+{
+   if (status.length() == 0)
+   fprintf (stderr, compacting table %s:\n, table.c_str());
+   else
+   fprintf (stderr,  %s\n, status.c_str());
+}
+};
+
 void
 notmuch_database_close_compact (notmuch_database_t *notmuch)
 {
 void *local = talloc_new (NULL);
-Xapian::Compactor compactor;
+NotmuchCompactor compactor;
 char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;
 
 #if HAVE_XAPIAN_COMPACT
-- 
1.7.9.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


[PATCH 1/3] Add notmuch_database_close_compact

2012-08-20 Thread Ben Gamari
---
 configure   |   25 -
 lib/database.cc |   54 ++
 lib/notmuch.h   |   14 ++
 3 files changed, 92 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index dc0dba4..370fedd 100755
--- a/configure
+++ b/configure
@@ -270,7 +270,8 @@ printf Checking for Xapian development files... 
 have_xapian=0
 for xapian_config in ${XAPIAN_CONFIG}; do
 if ${xapian_config} --version  /dev/null 21; then
-   printf Yes (%s).\n $(${xapian_config} --version | sed -e 's/.* //')
+   xapian_version=$(${xapian_config} --version | sed -e 's/.* //')
+   printf Yes (%s).\n ${xapian_version}
have_xapian=1
xapian_cxxflags=$(${xapian_config} --cxxflags)
xapian_ldflags=$(${xapian_config} --libs)
@@ -282,6 +283,24 @@ if [ ${have_xapian} = 0 ]; then
 errors=$((errors + 1))
 fi
 
+have_xapian_compact=0
+if [ ${have_xapian} = 1 ]; then
+printf Checking for Xapian compact support... 
+IFS='.'
+old_args=$@
+set -- ${xapian_version}
+xapian_major_version=$1
+xapian_minor_version=$2
+xapian_subminor_version=$3
+set -- ${old_args}
+if [ ${xapian_major_version} -gt 1 ] || [ ${xapian_minor_version} -gt 2 
] || [ ${xapian_subminor_version} -ge 6 ]; then
+   printf Yes.\n
+   have_xapian_compact=1
+else
+   printf No.\n
+fi
+fi
+
 printf Checking for GMime development files... 
 have_gmime=0
 IFS=';'
@@ -675,6 +694,9 @@ LINKER_RESOLVES_LIBRARY_DEPENDENCIES = 
${linker_resolves_library_dependencies}
 XAPIAN_CXXFLAGS = ${xapian_cxxflags}
 XAPIAN_LDFLAGS = ${xapian_ldflags}
 
+# Whether compact is supported by this version of Xapian
+HAVE_XAPIAN_COMPACT = ${have_xapian_compact}
+
 # Flags needed to compile and link against GMime-2.4
 GMIME_CFLAGS = ${gmime_cflags}
 GMIME_LDFLAGS = ${gmime_ldflags}
@@ -711,6 +733,7 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
\$(GMIME_CFLAGS)  \\
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
 \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
+ -DHAVE_XAPIAN_COMPACT=\$(HAVE_XAPIAN_COMPACT)   \\
  -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
diff --git a/lib/database.cc b/lib/database.cc
index 761dc1a..6e83a61 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -781,6 +781,60 @@ notmuch_database_close (notmuch_database_t *notmuch)
 }
 
 void
+notmuch_database_close_compact (notmuch_database_t *notmuch)
+{
+void *local = talloc_new (NULL);
+Xapian::Compactor compactor;
+char *notmuch_path, *xapian_path, *compact_xapian_path, *old_xapian_path;
+
+#if HAVE_XAPIAN_COMPACT
+if (! (notmuch_path = talloc_asprintf (local, %s/%s, notmuch-path, 
.notmuch))) {
+   fprintf (stderr, Out of memory\n);
+   goto DONE;
+}
+
+if (! (xapian_path = talloc_asprintf (local, %s/%s, notmuch_path, 
xapian))) {
+   fprintf (stderr, Out of memory\n);
+   goto DONE;
+}
+
+if (! (compact_xapian_path = talloc_asprintf (local, %s.compact, 
xapian_path))) {
+   fprintf (stderr, Out of memory\n);
+   goto DONE;
+}
+
+if (! (old_xapian_path = talloc_asprintf (local, %s.old, xapian_path))) {
+   fprintf (stderr, Out of memory\n);
+   goto DONE;
+}
+
+try {
+   compactor.set_renumber(false);
+   compactor.add_source(xapian_path);
+   compactor.set_destdir(compact_xapian_path);
+   compactor.compact();
+
+   if (rename(xapian_path, old_xapian_path)) {
+   fprintf (stderr, Error moving old database out of the way\n);
+   goto DONE;
+   }
+
+   if (rename(compact_xapian_path, xapian_path)) {
+   fprintf (stderr, Error moving compacted database\n);
+   goto DONE;
+   }
+} catch (Xapian::InvalidArgumentError e) {
+   fprintf (stderr, Error while compacting: %s, e.get_msg().c_str());
+}
+
+#endif
+
+notmuch_database_close(notmuch);
+DONE:
+talloc_free(local);
+}
+
+void
 notmuch_database_destroy (notmuch_database_t *notmuch)
 {
 notmuch_database_close (notmuch);
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 3633bed..50babfb 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -215,6 +215,20 @@ notmuch_database_open (const char *path,
 void
 notmuch_database_close (notmuch_database_t *database);
 
+/* Close the given notmuch database and then compact it.
+ *
+ * After notmuch_database_close_compact has been called, calls to
+ * other functions on objects derived from this database may either
+ * behave as if the database had not been closed (e.g., if the
+ * required data has been cached) or may fail with a
+ * NOTMUCH_STATUS_XAPIAN_EXCEPTION.
+ *
+ * notmuch_database_close_compact can be called multiple times.  Later
+ * calls 

[PATCH 3/3] Add notmuch compact command

2012-08-20 Thread Ben Gamari
---
 Makefile.local|1 +
 notmuch-client.h  |3 +++
 notmuch-compact.c |   43 +++
 notmuch.c |3 +++
 4 files changed, 50 insertions(+)
 create mode 100644 notmuch-compact.c

diff --git a/Makefile.local b/Makefile.local
index de984ab..848605d 100644
--- a/Makefile.local
+++ b/Makefile.local
@@ -278,6 +278,7 @@ notmuch_client_srcs =   \
gmime-filter-headers.c  \
hooks.c \
notmuch.c   \
+   notmuch-compact.c   \
notmuch-config.c\
notmuch-count.c \
notmuch-dump.c  \
diff --git a/notmuch-client.h b/notmuch-client.h
index ae9344b..a6c624b 100644
--- a/notmuch-client.h
+++ b/notmuch-client.h
@@ -157,6 +157,9 @@ int
 notmuch_cat_command (void *ctx, int argc, char *argv[]);
 
 int
+notmuch_compact_command (void *ctx, int argc, char *argv[]);
+
+int
 notmuch_config_command (void *ctx, int argc, char *argv[]);
 
 const char *
diff --git a/notmuch-compact.c b/notmuch-compact.c
new file mode 100644
index 000..6deb2ec
--- /dev/null
+++ b/notmuch-compact.c
@@ -0,0 +1,43 @@
+/* notmuch - Not much of an email program, (just index and search)
+ *
+ * Copyright © 2009 Carl Worth
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: Carl Worth cwo...@cworth.org
+ */
+
+#include notmuch-client.h
+
+int
+notmuch_compact_command (void *ctx, unused (int argc), unused (char *argv[]))
+{
+notmuch_config_t *config;
+notmuch_database_t *notmuch;
+
+config = notmuch_config_open (ctx, NULL, NULL);
+if (config == NULL)
+   return 1;
+
+if (notmuch_database_open (notmuch_config_get_database_path (config),
+  NOTMUCH_DATABASE_MODE_READ_WRITE, notmuch))
+   return 1;
+
+printf (Compacting database... );
+notmuch_database_close_compact (notmuch);
+printf (Done.\n);
+notmuch_database_destroy (notmuch);
+
+return 0;
+}
diff --git a/notmuch.c b/notmuch.c
index 477a09c..7b6c5ae 100644
--- a/notmuch.c
+++ b/notmuch.c
@@ -74,6 +74,9 @@ static command_t commands[] = {
 { restore, notmuch_restore_command,
   [--accumulate] [filename],
   Restore the tags from the given dump file (see 'dump'). },
+{ compact, notmuch_compact_command,
+  NULL,
+  Compacts the database. },
 { config, notmuch_config_command,
   [get|set] section.item [value ...],
   Get or set settings in the notmuch configuration file. },
-- 
1.7.9.5

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Segmentation fault in notmuch search --format=json

2012-08-07 Thread Ben Gamari
It seems some messages trigger a segmentation fault in
`do_search_threads()`. It appears the problem occurs (at least) when
`authors` is NULL.

Program received signal SIGSEGV, Segmentation fault.
0x00415aa3 in json_string (sp=0x646c70, val=0x0) at 
sprinter-json.c:121
121 json_string_len (sp, val, strlen (val));
(gdb) bt
#0  0x00415aa3 in json_string (sp=0x646c70, val=0x0)
at sprinter-json.c:121
#1  0x0041084b in do_search_threads (format=0x646c70, 
query=0x64bb70, 
sort=NOTMUCH_SORT_NEWEST_FIRST, output=OUTPUT_SUMMARY, offset=0, 
limit=-1)
at notmuch-search.c:131
#2  0x00411361 in notmuch_search_command (ctx=0x6361b0, argc=3, 
argv=0x7fffdfb0) at notmuch-search.c:405
#3  0x00409e22 in main (argc=4, argv=0x7fffdfa8) at 
notmuch.c:294
(gdb) up
#1  0x0041084b in do_search_threads (format=0x646c70, 
query=0x64bb70, 
sort=NOTMUCH_SORT_NEWEST_FIRST, output=OUTPUT_SUMMARY, offset=0, 
limit=-1)
at notmuch-search.c:131
131 format->string (format, authors);
(gdb) list
126 format->map_key (format, "matched");
127 format->integer (format, matched);
128 format->map_key (format, "total");
129 format->integer (format, total);
130 format->map_key (format, "authors");
131 format->string (format, authors);
132 format->map_key (format, "subject");
133 format->string (format, subject);
134 }
135 
(gdb) 

It seems that the message in question was produced by a script I use to
feed RSS feeds into notmuch so while I wouldn't doubt that there are few
cases where `authors` should be NULL, it would be nice if notmuch
handled this case with a bit more grace.

Cheers,

- Ben



Segmentation fault in notmuch search --format=json

2012-08-06 Thread Ben Gamari
It seems some messages trigger a segmentation fault in
`do_search_threads()`. It appears the problem occurs (at least) when
`authors` is NULL.

Program received signal SIGSEGV, Segmentation fault.
0x00415aa3 in json_string (sp=0x646c70, val=0x0) at 
sprinter-json.c:121
121 json_string_len (sp, val, strlen (val));
(gdb) bt
#0  0x00415aa3 in json_string (sp=0x646c70, val=0x0)
at sprinter-json.c:121
#1  0x0041084b in do_search_threads (format=0x646c70, 
query=0x64bb70, 
sort=NOTMUCH_SORT_NEWEST_FIRST, output=OUTPUT_SUMMARY, offset=0, 
limit=-1)
at notmuch-search.c:131
#2  0x00411361 in notmuch_search_command (ctx=0x6361b0, argc=3, 
argv=0x7fffdfb0) at notmuch-search.c:405
#3  0x00409e22 in main (argc=4, argv=0x7fffdfa8) at 
notmuch.c:294
(gdb) up
#1  0x0041084b in do_search_threads (format=0x646c70, 
query=0x64bb70, 
sort=NOTMUCH_SORT_NEWEST_FIRST, output=OUTPUT_SUMMARY, offset=0, 
limit=-1)
at notmuch-search.c:131
131 format-string (format, authors);
(gdb) list
126 format-map_key (format, matched);
127 format-integer (format, matched);
128 format-map_key (format, total);
129 format-integer (format, total);
130 format-map_key (format, authors);
131 format-string (format, authors);
132 format-map_key (format, subject);
133 format-string (format, subject);
134 }
135 
(gdb) 

It seems that the message in question was produced by a script I use to
feed RSS feeds into notmuch so while I wouldn't doubt that there are few
cases where `authors` should be NULL, it would be nice if notmuch
handled this case with a bit more grace.

Cheers,

- Ben

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Ubuntu messaging menu integration

2012-05-29 Thread Ben Gamari
Hey all,

I recently dusted off a little hack I've been sitting on for quite a
while integrating notmuch into Ubuntu's messaging menu[1]. The code is now
on Github[2] in case someone finds it useful. Pull requests are of course
welcome.

Cheers,

- Ben


[1] https://wiki.ubuntu.com/MessagingMenu
[2] https://github.com/bgamari/notmuch-indicator


Ubuntu messaging menu integration

2012-05-29 Thread Ben Gamari
Hey all,

I recently dusted off a little hack I've been sitting on for quite a
while integrating notmuch into Ubuntu's messaging menu[1]. The code is now
on Github[2] in case someone finds it useful. Pull requests are of course
welcome.

Cheers,

- Ben


[1] https://wiki.ubuntu.com/MessagingMenu
[2] https://github.com/bgamari/notmuch-indicator
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Rethinking *_destroy()

2011-09-19 Thread Ben Gamari
On Mon, 19 Sep 2011 16:22:39 -0400, Ben Gamari  
wrote:
> I can see at least two ways of doing this,
> 
>   1) Acknowledging that we use talloc and allowing users to use
>  talloc_ref and talloc_unlink directly
> 
>   2) Wrapping talloc by adding a *_ref() and *_unref() to each object
> 
I should not that these aren't quite as trivial as they sound. As I
neglect to mention in this message, we currently use talloc_free in
*_destroy(). As of talloc-2.0, talloc_free() fails on objects with more
than one parent. When we allow library users to add their own references
to notmuch objects, this assumption will break. Sorry for the confusion,

Cheers,

- Ben


Re: Rethinking *_destroy()

2011-09-19 Thread Ben Gamari
On Mon, 19 Sep 2011 16:22:39 -0400, Ben Gamari bgamari.f...@gmail.com wrote:
 I can see at least two ways of doing this,
 
   1) Acknowledging that we use talloc and allowing users to use
  talloc_ref and talloc_unlink directly
 
   2) Wrapping talloc by adding a *_ref() and *_unref() to each object
 
I should not that these aren't quite as trivial as they sound. As I
neglect to mention in this message, we currently use talloc_free in
*_destroy(). As of talloc-2.0, talloc_free() fails on objects with more
than one parent. When we allow library users to add their own references
to notmuch objects, this assumption will break. Sorry for the confusion,

Cheers,

- Ben
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Memory management practices

2011-09-11 Thread Ben Gamari
Sorry I've been so quiet on this recently. I've been a little under the
weather.

On Fri, 9 Sep 2011 13:53:28 -0400, Austin Clements  wrote:
> Ah, the *Python* objects don't care, but the underlying C objects do.
> Suppose the Query were finalized first.  Python calls Query.__del__,
> which calls notmuch_query_destroy, which releases the underlying
> talloc references to the C notmuch_messages_t objects, causing talloc
> to free the notmuch_messages_t.  Messages._msgs now points to freed
> memory, so when Python then finalizes the Messages object,
> Messages.__del__ will pass this dangling pointer to
> notmuch_messages_destroy, which will crash.

Exactly. This is exactly what I suspect is happening in my case.

> 
> Hence my suggestion that, rather than trying to emulate C-style memory
> management in bindings, bindings should create an additional talloc
> reference to the underlying objects and rather than calling
> notmuch_*_destroy during finalization, they should simply unlink this
> additional reference.

Currently talloc's reference counting interface is hidden behind
_destroy. While this might be a fairly intrusive change, perhaps notmuch
wants to juse expose a pair of reference counting *_ref/unref functions
instead of the *_destroy. Most users would simply need to change
existing *_destroy()s to _unref()s. Furthermore, this would allow
bindings authors to easily ensure non-broken GC behavior.

Does this sound completely insane, somewhat insane, or reasonable?

Cheers,

- Ben


Re: Memory management practices

2011-09-11 Thread Ben Gamari
Sorry I've been so quiet on this recently. I've been a little under the
weather.

On Fri, 9 Sep 2011 13:53:28 -0400, Austin Clements amdra...@mit.edu wrote:
 Ah, the *Python* objects don't care, but the underlying C objects do.
 Suppose the Query were finalized first.  Python calls Query.__del__,
 which calls notmuch_query_destroy, which releases the underlying
 talloc references to the C notmuch_messages_t objects, causing talloc
 to free the notmuch_messages_t.  Messages._msgs now points to freed
 memory, so when Python then finalizes the Messages object,
 Messages.__del__ will pass this dangling pointer to
 notmuch_messages_destroy, which will crash.

Exactly. This is exactly what I suspect is happening in my case.

 
 Hence my suggestion that, rather than trying to emulate C-style memory
 management in bindings, bindings should create an additional talloc
 reference to the underlying objects and rather than calling
 notmuch_*_destroy during finalization, they should simply unlink this
 additional reference.

Currently talloc's reference counting interface is hidden behind
_destroy. While this might be a fairly intrusive change, perhaps notmuch
wants to juse expose a pair of reference counting *_ref/unref functions
instead of the *_destroy. Most users would simply need to change
existing *_destroy()s to _unref()s. Furthermore, this would allow
bindings authors to easily ensure non-broken GC behavior.

Does this sound completely insane, somewhat insane, or reasonable?

Cheers,

- Ben
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Memory management practices

2011-09-07 Thread Ben Gamari
On Mon, 29 Aug 2011 16:30:57 -0400, Ben Gamari  
wrote:
> [SNIP]
> 
> In general, it seems to me that memory management in notmuch bindings is
> a little bit harder than it needs to me due to the decision not to
> talloc_ref parent objects when a new child object is created. This means
> that a bindings author needs to recreate the ownership tree in their
> binding, a task which is fairly easily done (except in the case of
> Haskell due to the weak GC finalization guarantees) but seems quite
> unnecessary. Is there a reason this decision was made? Would a patch be
> accepted adding talloc_ref'ing parents in those functions creating
> children and talloc_frees in *_destroys?
> 
Any opinions concerning whether this is an acceptable idea? I wouldn't
mind putting together a patch-set, but I'd rather not waste my time if
the set would ultimately be rejected due to some technical objection I
have yet to think of.

Cheers,

- Ben


Memory management practices

2011-08-29 Thread Ben Gamari
Hey all,

Over the last few weeks I've been trying to fix some brokeness in the
notmuch-haskell bindings with respect to memory management.

In discussion included below, I describe the issue as I approached it.
In short, it appears that GHC's garbage collector is quite liberal with
the order in which it frees resources (which is apparently permitted by
Haskell's FFI specification), allowing, for instance, a Messages object to be
freed before a Query object despite my attempts to hold the proper references in
the Haskell wrapper objects to keep the Query reachable.

In general, it seems to me that memory management in notmuch bindings is
a little bit harder than it needs to me due to the decision not to
talloc_ref parent objects when a new child object is created. This means
that a bindings author needs to recreate the ownership tree in their
binding, a task which is fairly easily done (except in the case of
Haskell due to the weak GC finalization guarantees) but seems quite
unnecessary. Is there a reason this decision was made? Would a patch be
accepted adding talloc_ref'ing parents in those functions creating
children and talloc_frees in *_destroys?

Cheers,

- Ben



On Mon, 29 Aug 2011 20:30:10 +0200, Bertram Felgenhauer  wrote:
> Dear Ben,
> 
> Ben Gamari wrote:
> > After looking into this issue in a bit more depth, I'm even more
> > confused. In fact, I would not be surprised if I have stumbled into a
> > bug in the GC.
> [...]
> > MessagesMessage
> >   |   
> >   |  msmpp
> >   \/
> > QueryMessages
> >   |
> >   |  qmpp
> >   \/
> > Query
> > 
> > As we can see, each MessagesMessage object in the Messages list
> > resulting from queryMessages holds a reference to the Query object from
> > which it originated. For this reason, I fail to see how it is possible
> > that the RTS would attempt to free the Query before freeing the
> > MessagesPtr.
> 
> When a garbage collection is performed, the RTS determines which heap
> objects are still reachable. The rest is then freed _simultaneously_,
> and the corresponding finalizers are run in some random order.
> 
> So assuming the application holds a reference to the MessagesMessage
> object for a while and then drops it, the GC will detect unreachability
> of all the three objects at the same time and in the end, the finalizer
> for MessagesMessage may be run before that of Query.
> 
> So I think this is not a bug.
> 
> To solve this problem properly, libnotmuch should stop imposing order
> constraints on when objects are freed - this would mean tracking
> references using talloc_ref and talloc_unlink instead of
> talloc_free inside the library.
> 
> For a bindings author who does not want to touch the library, the best
> idea I have is to add a layer with the sole purpose of tracking those
> implicit references.
> 
> Best regards,
> 
> Bertram
> 
> ___
> Glasgow-haskell-users mailing list
> Glasgow-haskell-users at haskell.org
> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users


Bug in GC's ordering of ForeignPtr finalization?

2011-08-29 Thread Ben Gamari
On Sun, 28 Aug 2011 22:26:05 -0500, Antoine Latter  
wrote:
> One problem you might be running in to is that the optimization passes
> can notice that a function isn't using all of its arguments, and then
> it won't pass them. These even applies if the arguments are bound
> together in a record type.
> 
In this case I wouldn't be able to reproduce the problem with
optimization disabled, no? Unfortunately, this is not the case; the
problem persists even with -O0.

- Ben


Memory management practices

2011-08-29 Thread Ben Gamari
Hey all,

Over the last few weeks I've been trying to fix some brokeness in the
notmuch-haskell bindings with respect to memory management.

In discussion included below, I describe the issue as I approached it.
In short, it appears that GHC's garbage collector is quite liberal with
the order in which it frees resources (which is apparently permitted by
Haskell's FFI specification), allowing, for instance, a Messages object to be
freed before a Query object despite my attempts to hold the proper references in
the Haskell wrapper objects to keep the Query reachable.

In general, it seems to me that memory management in notmuch bindings is
a little bit harder than it needs to me due to the decision not to
talloc_ref parent objects when a new child object is created. This means
that a bindings author needs to recreate the ownership tree in their
binding, a task which is fairly easily done (except in the case of
Haskell due to the weak GC finalization guarantees) but seems quite
unnecessary. Is there a reason this decision was made? Would a patch be
accepted adding talloc_ref'ing parents in those functions creating
children and talloc_frees in *_destroys?

Cheers,

- Ben



On Mon, 29 Aug 2011 20:30:10 +0200, Bertram Felgenhauer 
bertram.felgenha...@googlemail.com wrote:
 Dear Ben,
 
 Ben Gamari wrote:
  After looking into this issue in a bit more depth, I'm even more
  confused. In fact, I would not be surprised if I have stumbled into a
  bug in the GC.
 [...]
  MessagesMessage
|   
|  msmpp
\/
  QueryMessages
|
|  qmpp
\/
  Query
  
  As we can see, each MessagesMessage object in the Messages list
  resulting from queryMessages holds a reference to the Query object from
  which it originated. For this reason, I fail to see how it is possible
  that the RTS would attempt to free the Query before freeing the
  MessagesPtr.
 
 When a garbage collection is performed, the RTS determines which heap
 objects are still reachable. The rest is then freed _simultaneously_,
 and the corresponding finalizers are run in some random order.
 
 So assuming the application holds a reference to the MessagesMessage
 object for a while and then drops it, the GC will detect unreachability
 of all the three objects at the same time and in the end, the finalizer
 for MessagesMessage may be run before that of Query.
 
 So I think this is not a bug.
 
 To solve this problem properly, libnotmuch should stop imposing order
 constraints on when objects are freed - this would mean tracking
 references using talloc_ref and talloc_unlink instead of
 talloc_free inside the library.
 
 For a bindings author who does not want to touch the library, the best
 idea I have is to add a layer with the sole purpose of tracking those
 implicit references.
 
 Best regards,
 
 Bertram
 
 ___
 Glasgow-haskell-users mailing list
 glasgow-haskell-us...@haskell.org
 http://www.haskell.org/mailman/listinfo/glasgow-haskell-users
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Bug in GC's ordering of ForeignPtr finalization?

2011-08-28 Thread Ben Gamari
On Tue, 16 Aug 2011 12:32:13 -0400, Ben Gamari  
wrote:
> It seems that the notmuch-haskell bindings (version 0.2.2 built against
> notmuch from git master; passes notmuch-test) aren't dealing with memory
> management properly. In particular, the attached test code[1] causes
> talloc to abort.  Unfortunately, while the issue is consistently
> reproducible, it only occurs with some queries (see source[1]). I have
> been unable to establish the exact criterion for failure.
> 
> It seems that the crash is caused by an invalid access to a freed Query
> object while freeing a Messages object (see Valgrind trace[3]). I've
> taken a brief look at the bindings themselves but, being only minimally
> familiar with the FFI, there's nothing obviously wrong (the finalizers
> passed to newForeignPtr look sane). I was under the impression that
> talloc was reference counted, so the Query object shouldn't have been
> freed unless if there was still a Messages object holding a
> reference. Any idea what might have gone wrong here?  Thanks!
> 
After looking into this issue in a bit more depth, I'm even more
confused. In fact, I would not be surprised if I have stumbled into a
bug in the GC. It seems that the notmuch-haskell bindings follow the
example of the python bindings in that child objects keep references to
their parents to prevent the garbage collector from releasing the
parent, which would in turn cause talloc to free the child objects,
resulting in odd behavior when the child objects were next accessed. For
instance, the Query and Messages objects are defined as follows,

type MessagesPtr = ForeignPtr S__notmuch_messages
type MessagePtr = ForeignPtr S__notmuch_message
newtype Query = Query (ForeignPtr S__notmuch_query)
data MessagesRef = QueryMessages { qmpp :: Query, msp :: MessagesPtr }
 | ThreadMessages { tmpp :: Thread, msp :: MessagesPtr }
 | MessageMessages { mmspp :: Message, msp :: MessagesPtr }
data Message = MessagesMessage { msmpp :: MessagesRef, mp :: MessagePtr }
 | Message { mp :: MessagePtr }
type Messages = [Message]

As seen in the Valgrind dump given in my previous message, it seems that
the Query object is being freed before the Messages object. Since the
Messages object is a child of the Query object, this fails.

In my case, I'm calling queryMessages which begins by issuing a given
notmuch Query, resulting in a MessagesPtr. This is then packaged into a
QueryMessages object which is then passed off to
unpackMessages. unpackMessages iterates over this collection, creating
MessagesMessage objects which themselves refer to the QueryMessages
object. Finally, these MessagesMessage objects are packed into a list,
resulting in a Messages object. Thus we have the following chain of
references,

MessagesMessage
  |   
  |  msmpp
  \/
QueryMessages
  |
  |  qmpp
  \/
Query

As we can see, each MessagesMessage object in the Messages list
resulting from queryMessages holds a reference to the Query object from
which it originated. For this reason, I fail to see how it is possible
that the RTS would attempt to free the Query before freeing the
MessagesPtr. Did I miss something in my analysis? Are there tools for
debugging issues such as this? Perhaps this is a bug in the GC?

Any help at all would be greatly appreciated.

Cheers,

- Ben


Bug in GC's ordering of ForeignPtr finalization?

2011-08-28 Thread Ben Gamari
On Tue, 16 Aug 2011 12:32:13 -0400, Ben Gamari bgamari.f...@gmail.com wrote:
 It seems that the notmuch-haskell bindings (version 0.2.2 built against
 notmuch from git master; passes notmuch-test) aren't dealing with memory
 management properly. In particular, the attached test code[1] causes
 talloc to abort.  Unfortunately, while the issue is consistently
 reproducible, it only occurs with some queries (see source[1]). I have
 been unable to establish the exact criterion for failure.
 
 It seems that the crash is caused by an invalid access to a freed Query
 object while freeing a Messages object (see Valgrind trace[3]). I've
 taken a brief look at the bindings themselves but, being only minimally
 familiar with the FFI, there's nothing obviously wrong (the finalizers
 passed to newForeignPtr look sane). I was under the impression that
 talloc was reference counted, so the Query object shouldn't have been
 freed unless if there was still a Messages object holding a
 reference. Any idea what might have gone wrong here?  Thanks!
 
After looking into this issue in a bit more depth, I'm even more
confused. In fact, I would not be surprised if I have stumbled into a
bug in the GC. It seems that the notmuch-haskell bindings follow the
example of the python bindings in that child objects keep references to
their parents to prevent the garbage collector from releasing the
parent, which would in turn cause talloc to free the child objects,
resulting in odd behavior when the child objects were next accessed. For
instance, the Query and Messages objects are defined as follows,

type MessagesPtr = ForeignPtr S__notmuch_messages
type MessagePtr = ForeignPtr S__notmuch_message
newtype Query = Query (ForeignPtr S__notmuch_query)
data MessagesRef = QueryMessages { qmpp :: Query, msp :: MessagesPtr }
 | ThreadMessages { tmpp :: Thread, msp :: MessagesPtr }
 | MessageMessages { mmspp :: Message, msp :: MessagesPtr }
data Message = MessagesMessage { msmpp :: MessagesRef, mp :: MessagePtr }
 | Message { mp :: MessagePtr }
type Messages = [Message]

As seen in the Valgrind dump given in my previous message, it seems that
the Query object is being freed before the Messages object. Since the
Messages object is a child of the Query object, this fails.

In my case, I'm calling queryMessages which begins by issuing a given
notmuch Query, resulting in a MessagesPtr. This is then packaged into a
QueryMessages object which is then passed off to
unpackMessages. unpackMessages iterates over this collection, creating
MessagesMessage objects which themselves refer to the QueryMessages
object. Finally, these MessagesMessage objects are packed into a list,
resulting in a Messages object. Thus we have the following chain of
references,

MessagesMessage
  |   
  |  msmpp
  \/
QueryMessages
  |
  |  qmpp
  \/
Query

As we can see, each MessagesMessage object in the Messages list
resulting from queryMessages holds a reference to the Query object from
which it originated. For this reason, I fail to see how it is possible
that the RTS would attempt to free the Query before freeing the
MessagesPtr. Did I miss something in my analysis? Are there tools for
debugging issues such as this? Perhaps this is a bug in the GC?

Any help at all would be greatly appreciated.

Cheers,

- Ben
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch


Memory management issue in notmuch-haskell bindings

2011-08-16 Thread Ben Gamari
It seems that the notmuch-haskell bindings (version 0.2.2 built against
notmuch from git master; passes notmuch-test) aren't dealing with memory
management properly. In particular, the attached test code[1] causes
talloc to abort.  Unfortunately, while the issue is consistently
reproducible, it only occurs with some queries (see source[1]). I have
been unable to establish the exact criterion for failure.

It seems that the crash is caused by an invalid access to a freed Query
object while freeing a Messages object (see Valgrind trace[3]). I've
taken a brief look at the bindings themselves but, being only minimally
familiar with the FFI, there's nothing obviously wrong (the finalizers
passed to newForeignPtr look sane). I was under the impression that
talloc was reference counted, so the Query object shouldn't have been
freed unless if there was still a Messages object holding a
reference. Any idea what might have gone wrong here?  Thanks!

Cheers,

- Ben



[1] Test case,

import Data.List
import Control.Monad
import System.Environment
import Foreign.Notmuch

dbpath = /home/ben/.mail

getAddresses :: Database - String - IO [String]
getAddresses db q = do
query - queryCreate db q
msgs - queryMessages query
addrs - mapM (flip messageGetHeader $ From) msgs
return addrs

main = do
db - databaseOpen dbpath DatabaseModeReadOnly
--addrs2 - getAddresses db tag:haskell -- This succeeds
addrs3 - getAddresses db to:dietz -- This fails

--print addrs2
--print addrs3

databaseClose db



[2] Crashed session and backtrace,

[1217 ben@ben-laptop ~] $ ghc test.hs -auto-all -rtsopts -prof  gdb ./test 
GNU gdb (Ubuntu/Linaro 7.2-1ubuntu11) 7.2
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type show copying
and show warranty for details.
This GDB was configured as x86_64-linux-gnu.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /home/ben/test...(no debugging symbols found)...done.
(gdb) run
Starting program: /home/ben/test 
[Thread debugging using libthread_db enabled]

Program received signal SIGABRT, Aborted.
0x75979d05 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
in ../nptl/sysdeps/unix/sysv/linux/raise.c
(gdb) bt
#0  0x75979d05 in raise (sig=6) at 
../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x7597dab6 in abort () at abort.c:92
#2  0x76de038c in talloc_abort (reason=0x76de56e8 Bad talloc magic 
value - access after free) at ../talloc.c:210
#3  0x76de0271 in talloc_abort_access_after_free (ptr=0x769190, 
location=0x77bd4e04 lib/messages.c:142) at ../talloc.c:229
#4  talloc_chunk_from_ptr (ptr=0x769190, location=0x77bd4e04 
lib/messages.c:142) at ../talloc.c:250
#5  _talloc_free (ptr=0x769190, location=0x77bd4e04 lib/messages.c:142) 
at ../talloc.c:1164
#6  0x77bc7e65 in notmuch_messages_destroy (messages=0x769190) at 
lib/messages.c:142
#7  0x004de1c9 in scheduleFinalizers ()
#8  0x004e013d in GarbageCollect ()
#9  0x004d9e40 in scheduleDoGC.clone.18 ()
#10 0x004db0e0 in exitScheduler ()
#11 0x004d9066 in hs_exit_ ()
#12 0x004d940a in shutdownHaskellAndExit ()
#13 0x004d8a91 in real_main ()
#14 0x004d8ade in hs_main ()
#15 0x75964eff in __libc_start_main (main=0x408ed0 main, argc=1, 
ubp_av=0x7fffe4f8, init=value optimized out, fini=value optimized out, 
rtld_fini=value optimized out, stack_end=0x7fffe4e8) at 
libc-start.c:226
#16 0x00407791 in _start ()
(gdb) 


[3] Valgrind output,

==25241== Memcheck, a memory error detector
==25241== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==25241== Using Valgrind-3.6.1 and LibVEX; rerun with -h for copyright info
==25241== Command: ./test
==25241== 
==25241== Conditional jump or move depends on uninitialised value(s)
==25241==at 0x52BB510: inflateReset2 (in 
/lib/x86_64-linux-gnu/libz.so.1.2.3.4)
==25241==by 0x52BB605: inflateInit2_ (in 
/lib/x86_64-linux-gnu/libz.so.1.2.3.4)
==25241==by 0x5F211BE: ChertTable::lazy_alloc_inflate_zstream() const 
(chert_table.cc:1672)
==25241==by 0x5F23B06: ChertTable::read_tag(Cursor*, std::string*, bool) 
const (chert_table.cc:1264)
==25241==by 0x5F260F9: ChertTable::get_exact_entry(std::string const, 
std::string) const (chert_table.cc:1210)
==25241==by 0x5F26DE2: 
ChertTermList::ChertTermList(Xapian::Internal::RefCntPtrChertDatabase const, 
unsigned int) (chert_termlist.cc:44)
==25241==by 0x5EFF2E5: ChertDatabase::open_term_list(unsigned int) const 
(chert_database.cc:891)
==25241==by 0x5E7E7FB: 

Notmuch scripts

2011-06-28 Thread Ben Gamari
On Tue, 28 Jun 2011 10:36:59 +0200, Pieter Praet  wrote:
> On Mon, 27 Jun 2011 21:31:26 -0400, Ben Gamari  
> wrote:
> > [SNIP] It
> > would be nice, however, if the emacs interface supported hiding tags
> > matching certain patterns (say /\..+/).
> 
> This should be possible as of last month (eb4e0ea2),
> by courtesy of Daniel Schoepe [1].
> 
Thanks for your note, althought it appears this only appears to apply to
the notmuch-hello all-tags list. I was hoping one could hide tag
patterns from all lists (or at least search results). This is definitely
a good first step, however.

Cheers,

- Ben


  1   2   3   >