Re: [PATCH v4 3/3] cli/reply: make --decrypt take a keyword

2017-12-29 Thread Daniel Kahn Gillmor
On Fri 2017-12-29 10:30:00 -0400, David Bremner wrote:
> Daniel Kahn Gillmor  writes:
>
>> No, we should not support --decrypt=nostash for show or reply.  The
>> semantics of the display commands (show, reply) are such that they
>> *never* modify the index or stash anything in there.  The equivalent for
>> the indexing (new, insert, reindex) commands' "--decrypt=nostash" in the
>> display commands is simply "--decrypt=true".
>
> I'm not sure I completely agree, but its a trivial matter to add nostash
> later if desired. And it's always easier to add API / command options
> than to take them away.

yes, true that we can always expand out, and it's more parsimonious to
start small.  :)

It sounds like you were suggesting "--decrypt=nostash" as a synonym for
"--decrypt=true" on show/reply, which i confess i didn't fully
understand when i wrote my response.  If it really makes you feel better
to add the alias/synonym, i wouldn't block such a change.

But I think the documentation is tricky to write (and trickier to read
and trickier still to understand!)  if "--decrypt=nostash" means the
same thing as "--decrypt=true" in one context, but they mean different
things in a different context.

I'm still trying to aim for that sweet spot where the smallest possible
API guides the user to sensible decisions while still understanding
what's going on. :)

   --dkg


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 0/2] nmbug: Bump to 0.3 and document in NEWS

2017-12-29 Thread Daniel Kahn Gillmor
On Fri 2017-12-29 17:33:45 +0200, Tomi Ollila wrote:
> On Thu, Dec 28 2017, W. Trevor King wrote:
>
>> As discussed previously in [1,2,3].  I've split this into two patches,
>> and only covered the changes since the last notmuch release.  There
>> have been additional changes since nmbug 0.2 (which went out with
>> notmuch 0.19), and I mention them in the commit message for patch 1/1,
>> but none of them seemed important enough to call out in NEWS.
>>
>> I'm also fine leaving these changes out of NEWS and just bumping the
>> nmbug version (which is why I split this series into two patches).
>
> Series LGTM

also LGTM.  I've removed the notmuch::needs-review tags on this.

 --dkg


signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: Update on python-cffi bindings

2017-12-29 Thread Floris Bruynooghe
Hi all,

Daniel Kahn Gillmor  writes:
> On Thu 2017-12-21 12:30:39 +0100, Floris Bruynooghe wrote:
>> The API changes a lot and there is no easy migration.  And history has
>> shown that's a terrible way to get something new adopted.  Last time I
>> suggested a possible multi-tiered approach (maybe not as explicit):
>>
>> 1 I think it's possible to move the memory management technique to the
>>   existing API without API breakage.  It would still allow users to call
>>   functions in the wrong order etc, but that's not any regression.
>>
>> 2 It's probably possible to either switch the existing API to use CFFI
>>   or create a drop-in replacement for it based on CFFI.  The benefit
>>   here is two-fold: users get better PyPy performance and I believe it
>>   becomes easier to maintain the bindings, e.g. all you need to do to
>>   call notmuch_database_get_path is
>>   capi.ffi.string(capi.lib.notmuch_database_get_path(dbptr)) (see
>>   
>> https://github.com/flub/notmuch/blob/cffi/bindings/python-cffi/notdb/_database.py#L263)
>>   for an actual example).  But this really depends on what everyone else
>>   here that maintains the Python bindings here thinks.  I'd encourage to
>>   have a look at the CFFI implementation to get an idea of this.
>
> these two steps on their own seem like they give us:
>
>   * better and safer memory management
>
>   * better performance on PyPy
>
>   * arguably, easier maintenance of the bindings.
>
> These seem like unambiguous wins, and there is no downside -- people
> using the notmuch module can just upgrade to the new version and it's
> done.  I'd love to see these changes happen.
>
> The only thing it doesn't do is provide more idiomatic bindings for new
> users, which you describe as:
>
>> 3 As last step I still think providing the more idiomatic bindings is
>>   useful, especially for new users.  It does take the burden of
>>   correctly calling the C functions somewhat more.  This could then
>>   either treat a notmuch_cffi as a lower level API which more directly
>>   maps the C API or it could call C directly as it does now.  I'm not
>>   currently sure on which is more feasible or better here.
>>
>>   An additional thing that could be done here to ease migration is to
>>   allow creating the new objects from the old ones allowing a codebase
>>   to gradually adopt the newer API where appropriate.  E.g.:
>>
>>  db = notmuch.Database(...)
>>  msg = db.find_message(...)
>>  new_db = notdb.Database.from_notmuch(db)
>>  new_msg = notdb.Message.from_notmuch(msg)
>>  print('Tags not on msg: {}'.format(new_db.tags - new_msg.tags))
>>
>>   This would rely on the existing API to migrate to CFFI, otherwise it
>>   could still be possible but would be very hairy.
>
> I'm not sure i understand this last sentence.  Are you saying that step
> 3 depends on step 2 happening?

The mixing of a new API with the existing API would be possible if we
can successfully migrate the existing notmuch API to use CFFI.  If not
it may theoretically be possible but could be a lot of pain.

> I'm not sure about the name "notdb"

I'm in no way attached to this name, just needed something to work with.
Naming things is hard, if anyone has a different idea please share.

> but i don't mind the idea of there
> being some other "more-pythonic" notmuch bindings.  New users would
> likely prefer it, and that'd be fine.
>
>> - For exposing completely new APIs, sure building the
>>   notdb.ImmutableTagSet and MutableTagSet was not straight forward,
>>   likewise for the PropertiesMap.  Many other things are much easier
>>   though.  One possible nice way to alleviate this with the idea of the
>>   existing notmuch API being the lower-level layer nearly mimicking the
>>   C-API directly.  That way adding new APIs there is more or less
>>   straight forward and there is less time pressure to add them to the
>>   higher level API.  Especially if mixing the APIs is supported.
>
> I think this is in line with the approach described above.  I like it.

I would really like to hear from some of the existing python binding
users on all of the above and what they thing of this approach.  Is this
work worth it to you?  Will it make you more or less likely to want to
migrate to more pythonic bindings?

In particular I'm not that convinced in the suitability of the existing
library as a low-level binding as it currently sits somewhere above
basic wrapping of libnotmuch.  Anyway, I would appreciate any feedback
on the proposed APIs and way forward.


Cheers,
Floris
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH 0/2] nmbug: Bump to 0.3 and document in NEWS

2017-12-29 Thread Tomi Ollila
On Thu, Dec 28 2017, W. Trevor King wrote:

> As discussed previously in [1,2,3].  I've split this into two patches,
> and only covered the changes since the last notmuch release.  There
> have been additional changes since nmbug 0.2 (which went out with
> notmuch 0.19), and I mention them in the commit message for patch 1/1,
> but none of them seemed important enough to call out in NEWS.
>
> I'm also fine leaving these changes out of NEWS and just bumping the
> nmbug version (which is why I split this series into two patches).

Series LGTM

Tomi

>
> Cheers,
> Trevor
>
> [1]: id:cover.1507675236.git.wk...@tremily.us
>  Subject: [PATCH 0/3] nmbug:
>  Date: Tue, 10 Oct 2017 15:49:48 -0700
> [2]: id: id:20171211174707.gx19...@valgrind.us
>  Subject: Re: [PATCH 3/4] nmbug: Auto-checkout in clone if it
>wouldn't clobber
>  Date: Mon, 11 Dec 2017 09:47:07 -0800
> [3]: id:87a7ypdm3p@tesseract.cs.unb.ca
>  Subject: Re: [PATCH 3/4] nmbug: Auto-checkout in clone if it
>wouldn't clobber
>  Date: Mon, 11 Dec 2017 16:51:38 -0400
>
> W. Trevor King (2):
>   nmbug: Bump to version 0.3
>   NEWS: Add nmbug 0.3 release notes to the notmuch 0.26 section
>
>  NEWS  | 13 +
>  devel/nmbug/nmbug |  2 +-
>  2 files changed, 14 insertions(+), 1 deletion(-)
>
> -- 
> 2.13.6
>
> ___
> notmuch mailing list
> notmuch@notmuchmail.org
> https://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: [PATCH v4 3/3] cli/reply: make --decrypt take a keyword

2017-12-29 Thread David Bremner
Daniel Kahn Gillmor  writes:

> No, we should not support --decrypt=nostash for show or reply.  The
> semantics of the display commands (show, reply) are such that they
> *never* modify the index or stash anything in there.  The equivalent for
> the indexing (new, insert, reindex) commands' "--decrypt=nostash" in the
> display commands is simply "--decrypt=true".
>

I'm not sure I completely agree, but its a trivial matter to add nostash
later if desired. And it's always easier to add API / command options
than to take them away.

d
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: Notmuch 0.26 release schedule

2017-12-29 Thread David Bremner
David Bremner  writes:

> David Bremner  writes:
>
>> I plan a feature freeze for December 25, and a release (all going well)
>> on or around January 1.
>
> I've tagged 0.26_rc0 (and uploaded to Debian experimental). I expect a
> few more changes before final release, but I want to start to shake out
> any build problems or API change issues.

Per tradition, we need to start collecting NEWS.

Here's a lightly edited output from git shortlog; some of these things
may already have NEWS items, and some may not need them.


Daniel Kahn Gillmor (68):
  lib: add notmuch_message_reindex
  add "notmuch reindex" subcommand
  database: add n_d_index_file (deprecates n_d_add_message)
  lib: index the content-type of the parts of encrypted messages
  crypto: move into libnotmuch_util
  properties: add notmuch_message_remove_all_properties_with_prefix()
  index: implement notmuch_indexopts_t with try_decrypt
  crypto: index encrypted parts when indexopts try_decrypt is set.
  config: test whether an item is stored in the database by name
  config: define new option index.try_decrypt
  cli/new: add --try-decrypt=(true|false)
  cli/insert: add --try-decrypt=(true|false)
  cli/reindex: add --try-decrypt=(true|false)
  crypto: use stashed session-key properties for decryption, if available
  crypto: new decryption policy "auto"
  cli/reply: use decryption policy "auto" by default.
  cli/show: use decryption policy "auto" by default.
  crypto: record whether an actual decryption attempt happened
  cli/new, insert, reindex: change index.decrypt to "auto" by default
  python: add decrypt_policy argument to Database.index_file()


David Bremner (55):
  lib: index message files with duplicate message-ids
  lib: add notmuch_message_count_files
  lib: add notmuch_thread_get_total_files
  cli/search: print total number of files matched in summary output.
  cli/new: improve error reporting
  Merge branch 'release'
  CLI/new: support maildir synced tags in new.tags
  build: add target to run cppcheck
  lib: return "" rather than NULL from notmuch_thread_get_authors
  python: remove obsolete debian directory

Florian Klink (2):
  python: open messages in binary mode

Gaute Hope (1):
  python: deprecated add_message calls index_file correctly and returns 
result

Jani Nikula (65):
  lib: index the content type of signature parts
  emacs: sanitize subject in replies
  cli: allow empty strings for notmuch insert --folder argument
  cli: add support for --no- prefixed boolean and keyword flag arguments
  cli: use the negating boolean support for new and insert --no-hooks
  cli: add support for only printing the addresses in notmuch address
  cli/new: support // in new.ignore

Tomi Ollila (3):
  make release archive: common (or no) timestamps

Vladimir Panteleev (13):
  emacs: Refactor subprocess stderr propagation
  emacs: Use make-process when available
  emacs: Refuse requests to refresh tree views while a refresh is running

Yuri Volchkov (6):
  database: move striping of trailing '/' into helper function
  insert: strip trailing / in folder path
  test: show id:<> works even if the first duplicate is deleted
  show: workaround for the missing file problem

l-...@web.de (7):
  python: add bindings to access config
  python: add default arg to get_config_list
  python: turn get_config_list into a generator
  python: Rename get_config_list to get_configs




signature.asc
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch


Re: Xapian exception leading to database corruption

2017-12-29 Thread David Edmondson
On Thursday, 2017-12-28 at 22:00:56 -0400, David Bremner wrote:

> David Edmondson  writes:
>
>> Using current git notmuch on Debian testing a rebuild from scratch of my
>> database fails:
>>
>>> agrajag-testing ~/s/notmuch % ./notmuch new
>>> Found 605510 total files (that's not much mail).
>>> add_file: A Xapian exception occurred 28m 37s remaining).
>>> A Xapian exception occurred adding message: Unexpected end of posting list 
>>> for 'G00014364'.
>>> Processed 137296 total files in 8h 4m 45s (4 files/sec.).
>>> Added 135950 new messages to the database.
>>> Note: A fatal error was encountered: A Xapian exception occurred
>>> agrajag-testing ~/s/notmuch %  
>>
>
> I can't duplicate this (probably not surprising) on debian testing. I
> also have about 600k files, and I'm running debian testing. My total
> index time is about 1h on a not-very-recent machine with an SSD. I'm
> guessing you have a hard disk (or something is deeply wrong to take
> that long). Even for a hard disk, 8h to index 137k files seems slow.
> Did you happen to check for disk errors?

That was running under valgrind (after seeing the discussion in
#xapian). valgrind only complains about some things in debugger.c, which
seems mildly ironic.

Without valgrind the time estimate for 600k files is around 2.5 hours,
though I suspect in reality it will be more than this.

The files are stored on a ZFS mirror of two SSDs. The machine is an AMD
X3216, so it doesn't have particularly quick CPUs and “only” 8G RAM.

When I said “Debian testing”, perhaps I should have been more explicit -
it's a Debian testing docker container running on Debian stable. I don't
think that this really matters - the failure happens with the version of
notmuch from stable backports on the host. The error messages are just
better from notmuch git and I'm trying to avoid installing all of the
development tools on the host.

>> I see that bremner reported something like this in #xapian, but not any
>> resolution.
>
> I guess your best bet is to write to
>
>   xapian-disc...@lists.xapian.org

I'll go there, thanks.

>>
>> Any suggestions? Is it possible to force a new chert database to be
>> created rather than glass? (Mostly I'd like to get back to work!)
>
> According to Olly Betts,
>
>With 1.4 you can pass Xapian::DB_BACKEND_CHERT in the flags when
>constructing the WritableDatabase object.
>
> So you'd have to hack the notmuch source, I'm guessing around line 55 of
> database.cc
>
> #define DB_ACTION (Xapian::DB_CREATE_OR_OPEN | Xapian::DB_RETRY_LOCK | 
> Xapian::DB_BACKEND_CHERT)
>
> Do move the existing database out of the way, or apparently xapian can
> get confused.

Thanks.

dme.
-- 
But whichever way I go, I come back to the place you are.
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch