Looking for the perfect mail client

2014-10-23 Thread Sepp Tannhuber 
Franz Fellner  schrieb am 12:58 Donnerstag, 
23.Oktober 2014:




> /usr/share\36/usr/loal/share/pixmaps/hicolor/index.theme
> \36 is $, isn't it?
> and "loal" should be "local"?
Any idea how this could happen? And how can I fix it?

> Probably you messed up your env? (.bashrc;/etc/env.d;...)

I don't have these files. There's a /etc/bash.bashrc which looks clean.

[PATCH v2] VIM: Fix header management and fold threads

2014-10-23 Thread Ian Main 
Yes, it's here, the patch you've all been waiting for!  Well, maybe:

- Add a variable to control which headers are displayed normally.
- Any header that is not displayed normally will be listed in a folded
  area in the message so you can access it at will (can be turned off).
- Add the (optional) ability to list threads using vim folding so that
  you can see at a glance the whole thread, which messages are new etc.

Ian
---

Fix a thinko where I was using the local variable name instead of the
global.

 vim/notmuch.vim | 61 ++---
 1 file changed, 54 insertions(+), 7 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index cad9517..21cfcae 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -54,12 +54,22 @@ let s:notmuch_folders_default = [
\ [ 'unread', 'tag:unread' ],
\ ]

+let s:notmuch_show_headers_default = [
+   \ 'Subject',
+   \ 'To',
+   \ 'Cc',
+   \ 'Date',
+   \ 'Message-ID',
+   \ ]
+
 let s:notmuch_date_format_default = '%d.%m.%y'
 let s:notmuch_datetime_format_default = '%d.%m.%y %H:%M:%S'
 let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
+let s:notmuch_show_folded_full_headers_default = 1
+let s:notmuch_show_folded_threads_default = 1

 function! s:new_file_buffer(type, fname)
exec printf('edit %s', a:fname)
@@ -311,6 +321,9 @@ function! s:show(thread_id)
call s:new_buffer('show')
setlocal modifiable
 ruby << EOF
+   show_full_headers = VIM::evaluate('g:notmuch_show_folded_full_headers')
+   show_threads_folded = VIM::evaluate('g:notmuch_show_folded_threads')
+
thread_id = VIM::evaluate('a:thread_id')
$cur_thread = thread_id
$messages.clear
@@ -326,11 +339,22 @@ ruby << EOF
date_fmt = VIM::evaluate('g:notmuch_datetime_format')
date = Time.at(msg.date).strftime(date_fmt)
nm_m.start = b.count
-   b << "%s %s (%s)" % [msg['from'], date, msg.tags]
-   b << "Subject: %s" % [msg['subject']]
-   b << "To: %s" % msg['to']
-   b << "Cc: %s" % msg['cc']
-   b << "Date: %s" % msg['date']
+   b << "From: %s %s (%s)" % [msg['from'], date, msg.tags]
+   showheaders = VIM::evaluate('g:notmuch_show_headers')
+   showheaders.each do |h|
+   b << "%s: %s" % [h, m.header[h]]
+   end
+   if show_full_headers
+   # Now show the rest in a folded area.
+   nm_m.full_header_start = b.count
+   m.header.fields.each do |k|
+   # Only show the ones we haven't already 
printed out.
+   if not showheaders.include?(k.name)
+   b << '%s: %s' % [k.name, k.to_s]
+   end
+   end
+   nm_m.full_header_end = b.count
+   end
nm_m.body_start = b.count
b << "--- %s ---" % part.mime_type
part.convert.each_line do |l|
@@ -343,11 +367,19 @@ ruby << EOF
end
$messages.each_with_index do |msg, i|
VIM::command("syntax region nmShowMsg#{i}Desc start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgDesc" % [msg.start, msg.start + 1])
-   VIM::command("syntax region nmShowMsg#{i}Head start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgHead" % [msg.start + 1, msg.body_start])
+   VIM::command("syntax region nmShowMsg#{i}Head start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgHead" % [msg.start + 1, msg.full_header_start])
VIM::command("syntax region nmShowMsg#{i}Body start='\\%%%il' 
end='\\%%%dl' contains=@nmShowMsgBody" % [msg.body_start, msg.end])
+   if show_full_headers
+   VIM::command("syntax region nmFold#{i}Headers 
start='\\%%%il' end='\\%%%il' fold transparent contains=@nmShowMsgHead" % 
[msg.full_header_start, msg.full_header_end])
+   end
+   # Only fold the whole message if there are multiple emails in 
this thread.
+   if $messages.count > 1 and show_threads_folded
+   VIM::command("syntax region nmShowMsgFold#{i} 
start='\\%%%il' end='\\%%%il' fold transparent contains=ALL" % [msg.start, 
msg.end])
+   end
end
 EOF
setlocal nomodifiable
+   setlocal foldmethod=syntax
call s:set_map(g:notmuch_show_maps)
 endfunction

@@ -460,6 +492,19 @@ function! s:set_defaults()

Looking for the perfect mail client

2014-10-23 Thread Ian Main 
Sepp Tannhuber wrote:
> Dear all,
> 
> I'm wondering whether there's a notmuch mail client
> - with a handling similar to alot
> - which is capable to show html content without html2ascii conversion
> - which can use vim as editor
> - which doesn't need a web browser
> - which doesn't need a mouse
> - which is fast
> 
> In principle, alot is the perfect mail client for me except that I've
> many problems with all the html mails people send to me. So I'm wondering
> whether anybody knows a webkit based alot fork or something similar. It's
> strange that after so many years I'm still looking for the perfect mail
> client.

I wonder if you'd be willing to give the vim client a go.  We've done a lot
of work to it lately and I think it would handle all you listed here.

Try the github version as I have all the patches applied there:

https://github.com/imain/notmuch-vim

Ian

[PATCH] VIM: Fix header management and fold threads

2014-10-23 Thread Ian Main 
Yes, it's here, the patch you've all been waiting for!  Well, maybe:

- Add a variable to control which headers are displayed normally.
- Any header that is not displayed normally will be listed in a folded
  area in the message so you can access it at will (can be turned off).
- Add the (optional) ability to list threads using vim folding so that
  you can see at a glance the whole thread, which messages are new etc.

Ian
---
 vim/notmuch.vim | 61 ++---
 1 file changed, 54 insertions(+), 7 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index cad9517..affd1e2 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -54,12 +54,22 @@ let s:notmuch_folders_default = [
\ [ 'unread', 'tag:unread' ],
\ ]

+let s:notmuch_show_headers_default = [
+   \ 'Subject',
+   \ 'To',
+   \ 'Cc',
+   \ 'Date',
+   \ 'Message-ID',
+   \ ]
+
 let s:notmuch_date_format_default = '%d.%m.%y'
 let s:notmuch_datetime_format_default = '%d.%m.%y %H:%M:%S'
 let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
+let s:notmuch_show_folded_full_headers_default = 1
+let s:notmuch_show_folded_threads_default = 1

 function! s:new_file_buffer(type, fname)
exec printf('edit %s', a:fname)
@@ -311,6 +321,9 @@ function! s:show(thread_id)
call s:new_buffer('show')
setlocal modifiable
 ruby << EOF
+   show_full_headers = VIM::evaluate('g:notmuch_show_folded_full_headers')
+   show_threads_folded = 
VIM::evaluate('g:notmuch_show_folded_threads_default')
+
thread_id = VIM::evaluate('a:thread_id')
$cur_thread = thread_id
$messages.clear
@@ -326,11 +339,22 @@ ruby << EOF
date_fmt = VIM::evaluate('g:notmuch_datetime_format')
date = Time.at(msg.date).strftime(date_fmt)
nm_m.start = b.count
-   b << "%s %s (%s)" % [msg['from'], date, msg.tags]
-   b << "Subject: %s" % [msg['subject']]
-   b << "To: %s" % msg['to']
-   b << "Cc: %s" % msg['cc']
-   b << "Date: %s" % msg['date']
+   b << "From: %s %s (%s)" % [msg['from'], date, msg.tags]
+   showheaders = VIM::evaluate('g:notmuch_show_headers')
+   showheaders.each do |h|
+   b << "%s: %s" % [h, m.header[h]]
+   end
+   if show_full_headers
+   # Now show the rest in a folded area.
+   nm_m.full_header_start = b.count
+   m.header.fields.each do |k|
+   # Only show the ones we haven't already 
printed out.
+   if not showheaders.include?(k.name)
+   b << '%s: %s' % [k.name, k.to_s]
+   end
+   end
+   nm_m.full_header_end = b.count
+   end
nm_m.body_start = b.count
b << "--- %s ---" % part.mime_type
part.convert.each_line do |l|
@@ -343,11 +367,19 @@ ruby << EOF
end
$messages.each_with_index do |msg, i|
VIM::command("syntax region nmShowMsg#{i}Desc start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgDesc" % [msg.start, msg.start + 1])
-   VIM::command("syntax region nmShowMsg#{i}Head start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgHead" % [msg.start + 1, msg.body_start])
+   VIM::command("syntax region nmShowMsg#{i}Head start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgHead" % [msg.start + 1, msg.full_header_start])
VIM::command("syntax region nmShowMsg#{i}Body start='\\%%%il' 
end='\\%%%dl' contains=@nmShowMsgBody" % [msg.body_start, msg.end])
+   if show_full_headers
+   VIM::command("syntax region nmFold#{i}Headers 
start='\\%%%il' end='\\%%%il' fold transparent contains=@nmShowMsgHead" % 
[msg.full_header_start, msg.full_header_end])
+   end
+   # Only fold the whole message if there are multiple emails in 
this thread.
+   if $messages.count > 1 and show_threads_folded
+   VIM::command("syntax region nmShowMsgFold#{i} 
start='\\%%%il' end='\\%%%il' fold transparent contains=ALL" % [msg.start, 
msg.end])
+   end
end
 EOF
setlocal nomodifiable
+   setlocal foldmethod=syntax
call s:set_map(g:notmuch_show_maps)
 endfunction

@@ -460,6 +492,19 @@ function! s:set_defaults()
let g:notmuch_folders = s:notmuch_folders_default

[PATCH v2] VIM: Add URI handling

2014-10-23 Thread Ian Main 
Add URI handling to the vim client.  You can now press 'enter' by default and
the client will parse the current line and find any 'Part's or URIs available
for opening.  If there are more than one it opens the one under the cursor or
else it opens the only one available.  It also supports mailto: URI's and will
compose a new message when activated.

By default xdg-open is used for everything but mailto: which generally
does the right thing afaict.

Note that this is now dependant on the attachment patch in order to make
the nice 'enter' behavior work for both.

Ian
---
 vim/notmuch.txt |  3 ++-
 vim/notmuch.vim | 76 +++--
 2 files changed, 70 insertions(+), 9 deletions(-)

diff --git a/vim/notmuch.txt b/vim/notmuch.txt
index 838a904..5d84fde 100644
--- a/vim/notmuch.txt
+++ b/vim/notmuch.txt
@@ -74,7 +74,8 @@ I Mark as read (-unread)
 t  Tag (prompted)
 e   Extract attachment on the current 'Attachment' line or all
attachments if the cursor is elsewhere.
-v   View attachment on the current 'Attachment' line.
+ View email part on the current 'Part' line, or open URI under cursor
+or on line.
 s  Search
 p  Save patches
 r  Reply
diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 1466e50..2f76f55 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -12,7 +12,7 @@ let g:notmuch_folders_maps = {
\ '':'folders_show_search()',
\ 's':  'folders_search_prompt()',
\ '=':  'folders_refresh()',
-   \ 'c':  'compose()',
+   \ 'c':  'compose("")',
\ }

 let g:notmuch_search_maps = {
@@ -25,7 +25,7 @@ let g:notmuch_search_maps = {
\ 's':  'search_search_prompt()',
\ '=':  'search_refresh()',
\ '?':  'search_info()',
-   \ 'c':  'compose()',
+   \ 'c':  'compose("")',
\ }

 let g:notmuch_show_maps = {
@@ -35,13 +35,13 @@ let g:notmuch_show_maps = {
\ 't':  'show_tag("")',
\ 'o':  'show_open_msg()',
\ 'e':  'show_extract_msg()',
-   \ '':'show_view_attachment()',
+   \ '':'show_view_magic()',
\ 's':  'show_save_msg()',
\ 'p':  'show_save_patches()',
\ 'r':  'show_reply()',
\ '?':  'show_info()',
\ '':  'show_next_msg()',
-   \ 'c':  'compose()',
+   \ 'c':  'compose("")',
\ }

 let g:notmuch_compose_maps = {
@@ -63,6 +63,7 @@ let s:notmuch_view_attachment_default = 'xdg-open'
 let s:notmuch_attachment_tmpdir_default = '~/.notmuch/tmp'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
+let s:notmuch_open_uri_default = 'xdg-open'

 function! s:new_file_buffer(type, fname)
exec printf('edit %s', a:fname)
@@ -141,8 +142,8 @@ function! s:show_reply()
end
 endfunction

-function! s:compose()
-   ruby open_compose
+function! s:compose(to_email)
+   ruby open_compose(VIM::evaluate('a:to_email'))
let b:compose_done = 0
call s:set_map(g:notmuch_compose_maps)
autocmd BufDelete  call s:on_compose_delete()
@@ -155,6 +156,22 @@ function! s:show_info()
ruby vim_puts get_message.inspect
 endfunction

+function! s:show_view_magic()
+   let line = getline(".")
+
+ruby << EOF
+   line = VIM::evaluate('line')
+
+   # Easiest to check for 'Part' types first..
+   match = line.match(/^Part (\d*):/)
+   if match and match.length == 2
+   VIM::command('call s:show_view_attachment()')
+   else
+   VIM::command('call s:show_open_uri()')
+   end
+EOF
+endfunction
+
 function! s:show_view_attachment()
let line = getline(".")
 ruby << EOF
@@ -226,6 +243,45 @@ ruby << EOF
 EOF
 endfunction

+function! s:show_open_uri()
+   let line = getline(".")
+   let pos = getpos(".")
+   let col = pos[2]
+ruby << EOF
+   m = get_message
+   line = VIM::evaluate('line')
+   col = VIM::evaluate('col') - 1
+   uris = URI.extract(line)
+   wanted_uri = nil
+   if uris.length == 1
+   wanted_uri = uris[0]
+   else
+   uris.each do |uri|
+   # Check to see the URI is at the present cursor location
+   idx = line.index(uri)
+   if col >= idx and col <= idx + uri.length
+   wanted_uri = uri
+   break
+   end
+   end
+   end
+
+   if wanted_uri
+   uri = URI.parse(wanted_uri)
+   if uri.class == URI::MailTo
+   vim_puts("Composing new email to #{uri.to}.")
+   VIM::command("call s:compose('#{uri.to}')")
+   else
+   vim_puts("Opening #{uri.to_s}.")
+   cmd =

[PATCH] VIM: Add better attachment support

2014-10-23 Thread Ian Main 
This patch changes how the notmuch vim client supports attachments:

- For each message part a 'Part : ' is added
  to the header.
- You can then use 'e' to extract the attachment under the cursor
  or use it elsewhere to extract all attachments (the prior behavior)
- You can use 'v' to 'view' the attachment/part using xdg-open by default.
- If the message is 'text/html' we include a 'Part:' for the body of the
  message so you can easily view it in a web browser if you so choose.

Ian
---
 vim/notmuch.txt |  8 +-
 vim/notmuch.vim | 84 +++--
 2 files changed, 89 insertions(+), 3 deletions(-)

diff --git a/vim/notmuch.txt b/vim/notmuch.txt
index 4374102..838a904 100644
--- a/vim/notmuch.txt
+++ b/vim/notmuch.txt
@@ -72,6 +72,9 @@ q Quit view
 A  Archive (-inbox -unread)
 I  Mark as read (-unread)
 t  Tag (prompted)
+e   Extract attachment on the current 'Attachment' line or all
+   attachments if the cursor is elsewhere.
+v   View attachment on the current 'Attachment' line.
 s  Search
 p  Save patches
 r  Reply
@@ -148,6 +151,9 @@ You can also configure your externail mail reader and 
sendemail program:
 >
let g:notmuch_reader = 'mutt -f %s'
let g:notmuch_sendmail = 'sendmail'
-<
+
+You can also configure what probram is used to view attachments:
+
+   let g:notmuch_view_attachment = 'xdg-open'

 vim:tw=78:ts=8:noet:ft=help:
diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 331e930..5f73dce 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -35,6 +35,7 @@ let g:notmuch_show_maps = {
\ 't':  'show_tag("")',
\ 'o':  'show_open_msg()',
\ 'e':  'show_extract_msg()',
+   \ '':'show_view_attachment()',
\ 's':  'show_save_msg()',
\ 'p':  'show_save_patches()',
\ 'r':  'show_reply()',
@@ -58,6 +59,8 @@ let s:notmuch_date_format_default = '%d.%m.%y'
 let s:notmuch_datetime_format_default = '%d.%m.%y %H:%M:%S'
 let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
+let s:notmuch_view_attachment_default = 'xdg-open'
+let s:notmuch_attachment_tmpdir_default = '~/.notmuch/tmp'
 let s:notmuch_folders_count_threads_default = 0

 function! s:new_file_buffer(type, fname)
@@ -147,13 +150,72 @@ function! s:show_info()
ruby vim_puts get_message.inspect
 endfunction

+function! s:show_view_attachment()
+   let line = getline(".")
+ruby << EOF
+   m = get_message
+   line = VIM::evaluate('line')
+
+   match = line.match(/^Part (\d*):/)
+   if match and match.length == 2
+   # Set up the tmpdir
+   tmpdir = VIM::evaluate('g:notmuch_attachment_tmpdir')
+   tmpdir = File.expand_path(tmpdir)
+   Dir.mkdir(tmpdir) unless Dir.exists?(tmpdir)
+
+   p = m.mail.parts[match[1].to_i - 1]
+   if p == nil
+   # Not a multipart message, use the message itself.
+   p = m.mail
+   end
+   if p.filename and p.filename.length > 0
+   filename = p.filename
+   else
+   suffix = ''
+   if p.mime_type == 'text/html'
+   suffix = '.html'
+   end
+   filename = "part-#{match[1]}#{suffix}"
+   end
+
+   # Sanitize just in case..
+   filename.gsub!(/[^0-9A-Za-z.\-]/, '_')
+
+   fullpath = File.expand_path("#{tmpdir}/#{filename}")
+   vim_puts "Viewing attachment #{fullpath}"
+   File.open(fullpath, 'w') do |f|
+   f.write p.body.decoded
+   cmd = VIM::evaluate('g:notmuch_view_attachment')
+   system(cmd, fullpath)
+   end
+   else
+   vim_puts "No attachment on this line."
+   end
+EOF
+endfunction
+
 function! s:show_extract_msg()
+   let line = getline(".")
 ruby << EOF
m = get_message
-   m.mail.attachments.each do |a|
+   line = VIM::evaluate('line')
+
+   # If the user is on a line that has an 'Part'
+   # line, we just extract the one attachment.
+   match = line.match(/^Part (\d*):/)
+   if match and match.length == 2
+   a = m.mail.parts[match[1].to_i - 1]
File.open(a.filename, 'w') do |f|
f.write a.body.decoded
-   print "Extracted '#{a.filename}'"
+   vim_puts "Extracted #{a.filename}"
+   end
+   else
+   # Extract them all..
+   m.mail.attachments.each do |a|
+   File.open(a.filename, 'w') do |f|
+   f.write a.body.decoded
+   vim_puts "Extracted #{a.filename}"
+

Looking for the perfect mail client

2014-10-23 Thread Franz Fellner 
Justus Winter wrote:
> Quoting Sepp Tannhuber (2014-10-23 12:47:10)
> > Peter Feigl  schrieb am 19:38 Mittwoch, 22.Oktober 2014:
> > > Try strace or
> > > gdb to find out what really crashes notmuch.
> > Good idea. Can anybody help me to interprete the output:
> > https://gist.github.com/tannhuber/c7cae862f897efccd3cb
> 
> Your process is killed by SIGABRT ;)
Yes, that's how exceptions kill your program.
throw -> (exception not caught) -> std::terminate -> std::terminate_handler -> 
(default) std::abort

> 
> Try running it under gdb and obtain a stack trace.
> 
> Justus
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Looking for the perfect mail client

2014-10-23 Thread Franz Fellner 
Sepp Tannhuber wrote:
> Peter Feigl  schrieb am 19:38 Mittwoch, 22.Oktober 2014:
> 
> 
> 
> 
> > Try strace or
> > gdb to find out what really crashes notmuch.
> Good idea. Can anybody help me to interprete the output:
> https://gist.github.com/tannhuber/c7cae862f897efccd3cb
HUH?
/usr/share\36/usr/loal/share/pixmaps/hicolor/index.theme
\36 is $, isn't it?
and "loal" should be "local"?
Probably you messed up your env? (.bashrc;/etc/env.d;...)

> 
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Looking for the perfect mail client

2014-10-23 Thread Justus Winter 
Quoting Sepp Tannhuber (2014-10-23 12:47:10)
> Peter Feigl  schrieb am 19:38 Mittwoch, 22.Oktober 2014:
> > Try strace or
> > gdb to find out what really crashes notmuch.
> Good idea. Can anybody help me to interprete the output:
> https://gist.github.com/tannhuber/c7cae862f897efccd3cb

Your process is killed by SIGABRT ;)

Try running it under gdb and obtain a stack trace.

Justus

Looking for the perfect mail client

2014-10-23 Thread Sepp Tannhuber 
Peter Feigl  schrieb am 19:38 Mittwoch, 22.Oktober 2014:




> Try strace or
> gdb to find out what really crashes notmuch.
Good idea. Can anybody help me to interprete the output:
https://gist.github.com/tannhuber/c7cae862f897efccd3cb

[PATCH v3 3/4] cli: Extend the search command for --output={sender, recipients}

2014-10-23 Thread Mark Walters 
On Sun, 12 Oct 2014, Michal Sojka  wrote:
> The new outputs allow printing senders, recipients or both of matching
> messages. The --output option is converted from "keyword" argument to
> "flags" argument, which means that the user can use --output=sender and
> --output=recipients simultaneously, to print both. Other combinations
> produce an error.
>
> This code based on a patch from Jani Nikula.
> ---
>  completion/notmuch-completion.bash |   2 +-
>  completion/notmuch-completion.zsh  |   3 +-
>  doc/man1/notmuch-search.rst|  22 +++-
>  notmuch-search.c   | 110 
> ++---
>  test/T090-search-output.sh |  64 +
>  5 files changed, 189 insertions(+), 12 deletions(-)
>
> diff --git a/completion/notmuch-completion.bash 
> b/completion/notmuch-completion.bash
> index 0571dc9..cfbd389 100644
> --- a/completion/notmuch-completion.bash
> +++ b/completion/notmuch-completion.bash
> @@ -294,7 +294,7 @@ _notmuch_search()
>   return
>   ;;
>   --output)
> - COMPREPLY=( $( compgen -W "summary threads messages files tags" -- 
> "${cur}" ) )
> + COMPREPLY=( $( compgen -W "summary threads messages files tags 
> sender recipients" -- "${cur}" ) )
>   return
>   ;;
>   --sort)
> diff --git a/completion/notmuch-completion.zsh 
> b/completion/notmuch-completion.zsh
> index 67a9aba..3e52a00 100644
> --- a/completion/notmuch-completion.zsh
> +++ b/completion/notmuch-completion.zsh
> @@ -52,7 +52,8 @@ _notmuch_search()
>_arguments -s : \
>  '--max-threads=[display only the first x threads from the search 
> results]:number of threads to show: ' \
>  '--first=[omit the first x threads from the search results]:number of 
> threads to omit: ' \
> -'--sort=[sort results]:sorting:((newest-first\:"reverse chronological 
> order" oldest-first\:"chronological order"))'
> +'--sort=[sort results]:sorting:((newest-first\:"reverse chronological 
> order" oldest-first\:"chronological order"))' \
> +'--output=[select what to output]:output:((summary threads messages 
> files tags sender recipients))'
>  }
>  
>  _notmuch()
> diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
> index 90160f2..c9d38b1 100644
> --- a/doc/man1/notmuch-search.rst
> +++ b/doc/man1/notmuch-search.rst
> @@ -35,7 +35,7 @@ Supported options for **search** include
>  intended for programs that invoke **notmuch(1)** internally. If
>  omitted, the latest supported version will be used.
>  
> -``--output=(summary|threads|messages|files|tags)``
> +``--output=(summary|threads|messages|files|tags|sender|recipients)``
>  
>  **summary**
>  Output a summary of each thread with any message matching
> @@ -78,6 +78,26 @@ Supported options for **search** include
>  by null characters (--format=text0), as a JSON array
>  (--format=json), or as an S-Expression list (--format=sexp).
>  
> + **sender**
> +Output all addresses from the *From* header that appear on
> +any message matching the search terms, either one per line
> +(--format=text), separated by null characters
> +(--format=text0), as a JSON array (--format=json), or as
> +an S-Expression list (--format=sexp).
> +
> + Note: Searching for **sender** should be much faster than
> + searching for **recipients**, because sender addresses are
> + cached directly in the database whereas other addresses
> + need to be fetched from message files.
> +
> + **recipients**
> +Like **sender** but for addresses from *To*, *Cc* and
> + *Bcc* headers.
> +
> + This option can be given multiple times to combine different
> + outputs. Curently, this is only supported for **sender** and
> + **recipients** outputs.
> +
>  ``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
>  This option can be used to present results in either
>  chronological order (**oldest-first**) or reverse chronological
> diff --git a/notmuch-search.c b/notmuch-search.c
> index 5ac2a26..74588f8 100644
> --- a/notmuch-search.c
> +++ b/notmuch-search.c
> @@ -23,11 +23,14 @@
>  #include "string-util.h"
>  
>  typedef enum {
> -OUTPUT_SUMMARY,
> -OUTPUT_THREADS,
> -OUTPUT_MESSAGES,
> -OUTPUT_FILES,
> -OUTPUT_TAGS
> +OUTPUT_SUMMARY   = 1 << 0,
> +OUTPUT_THREADS   = 1 << 1,
> +OUTPUT_MESSAGES  = 1 << 2,
> +OUTPUT_FILES = 1 << 3,
> +OUTPUT_TAGS  = 1 << 4,
> +OUTPUT_SENDER= 1 << 5,
> +OUTPUT_RECIPIENTS= 1 << 6,
> +OUTPUT_ADDRESSES = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
>  } output_t;
>  
>  typedef struct {
> @@ -220,6 +223,67 @@ do_search_threads (search_options_t *o)
>  return 0;
>  }
>  
> +static void
> +print_address_list (const search_options_t *o, InternetAddressList *list)
> +{
> +

[PATCH v3 9/9] lib: Remove unnecessary thread linking steps when using ghost messages

2014-10-23 Thread Austin Clements 
From: Austin Clements 

Previously, it was necessary to link new messages to children to work
around some (though not all) problems with the old metadata-based
approach to stored thread IDs.  With ghost messages, this is no longer
necessary, so don't bother with child linking when ghost messages are
in use.
---
 lib/database.cc | 29 +
 1 file changed, 21 insertions(+), 8 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index b673718..3601f9d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -2136,11 +2136,11 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
  * reference 'message'.
  *
  * In all cases, we assign to the current message the first thread ID
- * found (through either parent or child). We will also merge any
- * existing, distinct threads where this message belongs to both,
- * (which is not uncommon when messages are processed out of order).
+ * found. We will also merge any existing, distinct threads where this
+ * message belongs to both, (which is not uncommon when messages are
+ * processed out of order).
  *
- * Finally, if no thread ID has been found through parent or child, we
+ * Finally, if no thread ID has been found through referenced messages, we
  * call _notmuch_message_generate_thread_id to generate a new thread
  * ID. This should only happen for new, top-level messages, (no
  * References or In-Reply-To header in this message, and no previously
@@ -2172,10 +2172,23 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
 if (status)
goto DONE;

-status = _notmuch_database_link_message_to_children (notmuch, message,
-_id);
-if (status)
-   goto DONE;
+if (! (notmuch->features & NOTMUCH_FEATURE_GHOSTS)) {
+   /* In general, it shouldn't be necessary to link children,
+* since the earlier indexing of those children will have
+* stored a thread ID for the missing parent.  However, prior
+* to ghost messages, these stored thread IDs were NOT
+* rewritten during thread merging (and there was no
+* performant way to do so), so if indexed children were
+* pulled into a different thread ID by a merge, it was
+* necessary to pull them *back* into the stored thread ID of
+* the parent.  With ghost messages, we just rewrite the
+* stored thread IDs during merging, so this workaround isn't
+* necessary. */
+   status = _notmuch_database_link_message_to_children (notmuch, message,
+_id);
+   if (status)
+   goto DONE;
+}

 /* If not part of any existing thread, generate a new thread ID. */
 if (thread_id == NULL) {
-- 
2.1.0

[PATCH v3 8/9] test: Test upgrade to ghost messages feature

2014-10-23 Thread Austin Clements 
---
 test/T530-upgrade.sh | 21 +
 1 file changed, 21 insertions(+)

diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
index c4c4ac8..6b42a69 100755
--- a/test/T530-upgrade.sh
+++ b/test/T530-upgrade.sh
@@ -116,4 +116,25 @@ MAIL_DIR/bar/new/21:2,
 MAIL_DIR/bar/new/22:2,
 MAIL_DIR/cur/51:2,"

+# Ghost messages are difficult to test since they're nearly invisible.
+# However, if the upgrade works correctly, the ghost message should
+# retain the right thread ID even if all of the original messages in
+# the thread are deleted.  That's what we test.  This won't detect if
+# the upgrade just plain didn't happen, but it should detect if
+# something went wrong.
+test_begin_subtest "ghost message retains thread ID"
+# Upgrade database
+notmuch new
+# Get thread ID of real message
+thread=$(notmuch search --output=threads id:4EFC743A.3060609 at april.org)
+# Delete all real messages in that thread
+rm $(notmuch search --output=files $thread)
+notmuch new
+# "Deliver" ghost message
+add_message '[subject]=Ghost' '[id]=4EFC3931.6030007 at april.org'
+# If the ghost upgrade worked, the new message should be attached to
+# the existing thread ID.
+nthread=$(notmuch search --output=threads id:4EFC3931.6030007 at april.org)
+test_expect_equal "$thread" "$nthread"
+
 test_done
-- 
2.1.0

[PATCH v3 7/9] lib: Enable ghost messages feature

2014-10-23 Thread Austin Clements 
From: Austin Clements 

This fixes the broken thread order test.
---
 lib/database-private.h| 2 +-
 test/T260-thread-order.sh | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index e2e4bc8..15e03cc 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -166,7 +166,7 @@ struct _notmuch_database {
  * databases will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
 (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
- NOTMUCH_FEATURE_BOOL_FOLDER)
+ NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)

 /* Return the list of terms from the given iterator matching a prefix.
  * The prefix will be stripped from the strings in the returned list.
diff --git a/test/T260-thread-order.sh b/test/T260-thread-order.sh
index b435d79..99f5833 100755
--- a/test/T260-thread-order.sh
+++ b/test/T260-thread-order.sh
@@ -30,7 +30,6 @@ expected=$(for ((i = 0; i < $nthreads; i++)); do
 test_expect_equal "$output" "$expected"

 test_begin_subtest "Messages with all parents get linked in all delivery 
orders"
-test_subtest_known_broken
 # Here we do the same thing as the previous test, but each message
 # references all of its parents.  Since every message references the
 # root of the thread, each thread should always be fully joined.  This
-- 
2.1.0

[PATCH v3 6/9] lib: Implement upgrade to ghost messages feature

2014-10-23 Thread Austin Clements 
From: Austin Clements 

Somehow this is the first upgrade pass that actually does *any* error
checking, so this also adds the bit of necessary infrastructure to
handle that.
---
 lib/database.cc | 66 +++--
 1 file changed, 64 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 92a92d9..b673718 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1231,6 +1231,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 notmuch_bool_t timer_is_active = FALSE;
 enum _notmuch_features target_features, new_features;
 notmuch_status_t status;
+notmuch_private_status_t private_status;
 unsigned int count = 0, total = 0;

 status = _notmuch_database_ensure_writable (notmuch);
@@ -1275,6 +1276,13 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
for (t = db->allterms_begin ("XTIMESTAMP"); t != t_end; t++)
++total;
 }
+if (new_features & NOTMUCH_FEATURE_GHOSTS) {
+   /* The ghost message upgrade converts all thread_id_*
+* metadata values into ghost message documents. */
+   t_end = db->metadata_keys_end ("thread_id_");
+   for (t = db->metadata_keys_begin ("thread_id_"); t != t_end; ++t)
+   ++total;
+}

 /* Perform the upgrade in a transaction. */
 db->begin_transaction (true);
@@ -1378,10 +1386,64 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
}
 }

+/* Perform metadata upgrades. */
+
+/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
+ * messages were stored as database metadata. Change these to
+ * ghost messages.
+ */
+if (new_features & NOTMUCH_FEATURE_GHOSTS) {
+   notmuch_message_t *message;
+   std::string message_id, thread_id;
+
+   t_end = db->metadata_keys_end (NOTMUCH_METADATA_THREAD_ID_PREFIX);
+   for (t = db->metadata_keys_begin (NOTMUCH_METADATA_THREAD_ID_PREFIX);
+t != t_end; ++t) {
+   if (do_progress_notify) {
+   progress_notify (closure, (double) count / total);
+   do_progress_notify = 0;
+   }
+
+   message_id = (*t).substr (
+   strlen (NOTMUCH_METADATA_THREAD_ID_PREFIX));
+   thread_id = db->get_metadata (*t);
+
+   /* Create ghost message */
+   message = _notmuch_message_create_for_message_id (
+   notmuch, message_id.c_str (), _status);
+   if (private_status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+   /* Document already exists; ignore the stored thread ID */
+   } else if (private_status ==
+  NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+   private_status = _notmuch_message_initialize_ghost (
+   message, thread_id.c_str ());
+   if (! private_status)
+   _notmuch_message_sync (message);
+   }
+
+   if (private_status) {
+   fprintf (stderr,
+"Upgrade failed while creating ghost messages.\n");
+   status = COERCE_STATUS (private_status, "Unexpected status from 
_notmuch_message_initialize_ghost");
+   goto DONE;
+   }
+
+   /* Clear saved metadata thread ID */
+   db->set_metadata (*t, "");
+
+   ++count;
+   }
+}
+
+status = NOTMUCH_STATUS_SUCCESS;
 db->set_metadata ("features", _print_features (local, notmuch->features));
 db->set_metadata ("version", STRINGIFY (NOTMUCH_DATABASE_VERSION));

-db->commit_transaction ();
+ DONE:
+if (status == NOTMUCH_STATUS_SUCCESS)
+   db->commit_transaction ();
+else
+   db->cancel_transaction ();

 if (timer_is_active) {
/* Now stop the timer. */
@@ -1397,7 +1459,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 }

 talloc_free (local);
-return NOTMUCH_STATUS_SUCCESS;
+return status;
 }

 notmuch_status_t
-- 
2.1.0

[PATCH v3 5/9] lib: Implement ghost-based thread linking

2014-10-23 Thread Austin Clements 
From: Austin Clements 

This updates the thread linking code to use ghost messages instead of
user metadata to link messages into threads.

In contrast with the old approach, this is actually correct.
Previously, thread merging updated only the thread IDs of message
documents, not thread IDs stored in user metadata.  As originally
diagnosed by Mark Walters [1] and as demonstrated by the broken
T260-thread-order test, this can cause notmuch to fail to link
messages even though they're in the same thread.  In principle the old
approach could have been fixed by updating the user metadata thread
IDs as well, but these are not indexed and hence this would have
required a full scan of all stored thread IDs.  Ghost messages solve
this problem naturally by reusing the exact same thread ID and message
ID representation and indexing as regular messages.

Furthermore, thanks to this greater symmetry, ghost messages are also
algorithmically simpler.  We continue to support the old user metadata
format, so this patch can't delete any code, but when we do remove
support for the old format, several functions can simply be deleted.

[1] id:8738h7kv2q.fsf at qmul.ac.uk
---
 lib/database.cc | 99 +++--
 1 file changed, 83 insertions(+), 16 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index c641bcd..92a92d9 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1752,6 +1752,12 @@ _get_metadata_thread_id_key (void *ctx, const char 
*message_id)
message_id);
 }

+static notmuch_status_t
+_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
+ void *ctx,
+ const char *message_id,
+ const char **thread_id_ret);
+
 /* Find the thread ID to which the message with 'message_id' belongs.
  *
  * Note: 'thread_id_ret' must not be NULL!
@@ -1760,9 +1766,9 @@ _get_metadata_thread_id_key (void *ctx, const char 
*message_id)
  *
  * Note: If there is no message in the database with the given
  * 'message_id' then a new thread_id will be allocated for this
- * message and stored in the database metadata, (where this same
+ * message ID and stored in the database metadata so that the
  * thread ID can be looked up if the message is added to the database
- * later).
+ * later.
  */
 static notmuch_status_t
 _resolve_message_id_to_thread_id (notmuch_database_t *notmuch,
@@ -1770,6 +1776,49 @@ _resolve_message_id_to_thread_id (notmuch_database_t 
*notmuch,
  const char *message_id,
  const char **thread_id_ret)
 {
+notmuch_private_status_t status;
+notmuch_message_t *message;
+
+if (! (notmuch->features & NOTMUCH_FEATURE_GHOSTS))
+   return _resolve_message_id_to_thread_id_old (notmuch, ctx, message_id,
+thread_id_ret);
+
+/* Look for this message (regular or ghost) */
+message = _notmuch_message_create_for_message_id (
+   notmuch, message_id, );
+if (status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+   /* Message exists */
+   *thread_id_ret = talloc_steal (
+   ctx, notmuch_message_get_thread_id (message));
+} else if (status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+   /* Message did not exist.  Give it a fresh thread ID and
+* populate this message as a ghost message. */
+   *thread_id_ret = talloc_strdup (
+   ctx, _notmuch_database_generate_thread_id (notmuch));
+   if (! *thread_id_ret) {
+   status = NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+   } else {
+   status = _notmuch_message_initialize_ghost (message, 
*thread_id_ret);
+   if (status == 0)
+   /* Commit the new ghost message */
+   _notmuch_message_sync (message);
+   }
+} else {
+   /* Create failed. Fall through. */
+}
+
+notmuch_message_destroy (message);
+
+return COERCE_STATUS (status, "Error creating ghost message");
+}
+
+/* Pre-ghost messages _resolve_message_id_to_thread_id */
+static notmuch_status_t
+_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
+ void *ctx,
+ const char *message_id,
+ const char **thread_id_ret)
+{
 notmuch_status_t status;
 notmuch_message_t *message;
 string thread_id_string;
@@ -2007,13 +2056,16 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
 }
 }

-/* Given a (mostly empty) 'message' and its corresponding
+/* Given a blank or ghost 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
- * The first check is in the metadata of the database to see if we
- * have pre-allocated a thread_id in advance for this message, (which
- * would have

[PATCH v3 4/9] lib: Internal support for querying and creating ghost messages

2014-10-23 Thread Austin Clements 
From: Austin Clements 

This updates the message abstraction to support ghost messages: it
adds a message flag that distinguishes regular messages from ghost
messages, and an internal function for initializing a newly created
(blank) message as a ghost message.
---
 lib/message.cc| 52 +--
 lib/notmuch-private.h |  4 
 lib/notmuch.h |  9 -
 3 files changed, 62 insertions(+), 3 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 55d2ff6..a7a13cc 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -39,6 +39,9 @@ struct visible _notmuch_message {
 notmuch_message_file_t *message_file;
 notmuch_message_list_t *replies;
 unsigned long flags;
+/* For flags that are initialized on-demand, lazy_flags indicates
+ * if each flag has been initialized. */
+unsigned long lazy_flags;

 Xapian::Document doc;
 Xapian::termcount termpos;
@@ -99,6 +102,7 @@ _notmuch_message_create_for_document (const void 
*talloc_owner,

 message->frozen = 0;
 message->flags = 0;
+message->lazy_flags = 0;

 /* Each of these will be lazily created as needed. */
 message->message_id = NULL;
@@ -192,7 +196,7 @@ _notmuch_message_create (const void *talloc_owner,
  *
  * There is already a document with message ID 'message_id' in the
  * database. The returned message can be used to query/modify the
- * document.
+ * document. The message may be a ghost message.
  *
  *   NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND:
  *
@@ -305,6 +309,7 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message)
 const char *thread_prefix = _find_prefix ("thread"),
*tag_prefix = _find_prefix ("tag"),
*id_prefix = _find_prefix ("id"),
+   *type_prefix = _find_prefix ("type"),
*filename_prefix = _find_prefix ("file-direntry"),
*replyto_prefix = _find_prefix ("replyto");

@@ -337,10 +342,25 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message)
message->message_id =
_notmuch_message_get_term (message, i, end, id_prefix);

+/* Get document type */
+assert (strcmp (id_prefix, type_prefix) < 0);
+if (! NOTMUCH_TEST_BIT (message->lazy_flags, NOTMUCH_MESSAGE_FLAG_GHOST)) {
+   i.skip_to (type_prefix);
+   /* "T" is the prefix "type" fields.  See
+* BOOLEAN_PREFIX_INTERNAL. */
+   if (*i == "Tmail")
+   NOTMUCH_CLEAR_BIT (>flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+   else if (*i == "Tghost")
+   NOTMUCH_SET_BIT (>flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+   else
+   INTERNAL_ERROR ("Message without type term");
+   NOTMUCH_SET_BIT (>lazy_flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+}
+
 /* Get filename list.  Here we get only the terms.  We lazily
  * expand them to full file names when needed in
  * _notmuch_message_ensure_filename_list. */
-assert (strcmp (id_prefix, filename_prefix) < 0);
+assert (strcmp (type_prefix, filename_prefix) < 0);
 if (!message->filename_term_list && !message->filename_list)
message->filename_term_list =
_notmuch_database_get_terms_with_prefix (message, i, end,
@@ -371,6 +391,11 @@ _notmuch_message_invalidate_metadata (notmuch_message_t 
*message,
message->tag_list = NULL;
 }

+if (strcmp ("type", prefix_name) == 0) {
+   NOTMUCH_CLEAR_BIT (>flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+   NOTMUCH_CLEAR_BIT (>lazy_flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+}
+
 if (strcmp ("file-direntry", prefix_name) == 0) {
talloc_free (message->filename_term_list);
talloc_free (message->filename_list);
@@ -869,6 +894,10 @@ notmuch_bool_t
 notmuch_message_get_flag (notmuch_message_t *message,
  notmuch_message_flag_t flag)
 {
+if (flag == NOTMUCH_MESSAGE_FLAG_GHOST &&
+   ! NOTMUCH_TEST_BIT (message->lazy_flags, flag))
+   _notmuch_message_ensure_metadata (message);
+
 return NOTMUCH_TEST_BIT (message->flags, flag);
 }

@@ -880,6 +909,7 @@ notmuch_message_set_flag (notmuch_message_t *message,
NOTMUCH_SET_BIT (>flags, flag);
 else
NOTMUCH_CLEAR_BIT (>flags, flag);
+NOTMUCH_SET_BIT (>lazy_flags, flag);
 }

 time_t
@@ -989,6 +1019,24 @@ _notmuch_message_delete (notmuch_message_t *message)
 return NOTMUCH_STATUS_SUCCESS;
 }

+/* Transform a blank message into a ghost message.  The caller must
+ * _notmuch_message_sync the message. */
+notmuch_private_status_t
+_notmuch_message_initialize_ghost (notmuch_message_t *message,
+  const char *thread_id)
+{
+notmuch_private_status_t status;
+
+status = _notmuch_message_add_term (message, "type", "ghost");
+if (status)
+   return status;
+status = _notmuch_message_add_term (message, "thread", thread_id);
+if (status)
+   return status;
+
+return NOTMUCH_PRIVATE_STATUS_SUCCESS;
+}
+
 /* Ensure that 'message' is not

[PATCH v3 3/9] lib: Introduce macros for bit operations

2014-10-23 Thread Austin Clements 
These macros help clarify basic bit-twiddling code and are written to
be robust against C undefined behavior of shift operators.
---
 lib/message.cc|  6 +++---
 lib/notmuch-private.h | 11 +++
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 38bc929..55d2ff6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -869,7 +869,7 @@ notmuch_bool_t
 notmuch_message_get_flag (notmuch_message_t *message,
  notmuch_message_flag_t flag)
 {
-return message->flags & (1 << flag);
+return NOTMUCH_TEST_BIT (message->flags, flag);
 }

 void
@@ -877,9 +877,9 @@ notmuch_message_set_flag (notmuch_message_t *message,
  notmuch_message_flag_t flag, notmuch_bool_t enable)
 {
 if (enable)
-   message->flags |= (1 << flag);
+   NOTMUCH_SET_BIT (>flags, flag);
 else
-   message->flags &= ~(1 << flag);
+   NOTMUCH_CLEAR_BIT (>flags, flag);
 }

 time_t
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 36cc12b..7250291 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -63,6 +63,17 @@ NOTMUCH_BEGIN_DECLS
 #define STRNCMP_LITERAL(var, literal) \
 strncmp ((var), (literal), sizeof (literal) - 1)

+/* Robust bit test/set/reset macros */
+#define NOTMUCH_TEST_BIT(val, bit) \
+((bit < 0 || bit >= CHAR_BIT * sizeof (unsigned long long)) ? 0\
+ : !!((val) & (1ull << bit)))
+#define NOTMUCH_SET_BIT(valp, bit) \
+((bit < 0 || bit >= CHAR_BIT * sizeof (unsigned long long)) ? *(valp) \
+ : (*(valp) |= (1ull << bit)))
+#define NOTMUCH_CLEAR_BIT(valp,  bit) \
+((bit < 0 || bit >= CHAR_BIT * sizeof (unsigned long long)) ? *(valp) \
+ : (*(valp) &= ~(1ull << bit)))
+
 #define unused(x) x __attribute__ ((unused))

 #ifdef __cplusplus
-- 
2.1.0

[PATCH v3 2/9] lib: Update database schema doc for ghost messages

2014-10-23 Thread Austin Clements 
From: Austin Clements 

This describes the structure of ghost mail documents.  Ghost messages
are not yet implemented.
---
 lib/database.cc | 20 ++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 8fd7fad..c641bcd 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -50,8 +50,8 @@ typedef struct {

 /* Here's the current schema for our database (for NOTMUCH_DATABASE_VERSION):
  *
- * We currently have two different types of documents (mail and
- * directory) and also some metadata.
+ * We currently have three different types of documents (mail, ghost,
+ * and directory) and also some metadata.
  *
  * Mail document
  * -
@@ -109,6 +109,15 @@ typedef struct {
  *
  * The data portion of a mail document is empty.
  *
+ * Ghost mail document [if NOTMUCH_FEATURE_GHOSTS]
+ * ---
+ * A ghost mail document is like a mail document, but where we don't
+ * have the message content.  These are used to track thread reference
+ * information for messages we haven't received.
+ *
+ * A ghost mail document has type: ghost; id and thread fields that
+ * are identical to the mail document fields; and a MESSAGE_ID value.
+ *
  * Directory document
  * --
  * A directory document is used by a client of the notmuch library to
@@ -172,6 +181,13 @@ typedef struct {
  * generated is 1 and the value will be
  * incremented for each thread ID.
  *
+ * Obsolete metadata
+ * -
+ *
+ * If ! NOTMUCH_FEATURE_GHOSTS, there are no ghost mail documents.
+ * Instead, the database has the following additional database
+ * metadata:
+ *
  * thread_id_* A pre-allocated thread ID for a particular
  * message. This is actually an arbitrarily large
  * family of metadata name. Any particular name is
-- 
2.1.0

[PATCH v3 1/9] lib: Add a ghost messages database feature

2014-10-23 Thread Austin Clements 
From: Austin Clements 

This will be implemented over the next several patches.  The feature
is not yet "enabled" (this does not add it to
NOTMUCH_FEATURES_CURRENT).
---
 lib/database-private.h | 7 +++
 lib/database.cc| 2 ++
 2 files changed, 9 insertions(+)

diff --git a/lib/database-private.h b/lib/database-private.h
index ca0751c..e2e4bc8 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -85,6 +85,13 @@ enum _notmuch_features {
  *
  * Introduced: version 2. */
 NOTMUCH_FEATURE_BOOL_FOLDER = 1 << 3,
+
+/* If set, missing messages are stored in ghost mail documents.
+ * If unset, thread IDs of ghost messages are stored as database
+ * metadata instead of in ghost documents.
+ *
+ * Introduced: version 3. */
+NOTMUCH_FEATURE_GHOSTS = 1 << 4,
 };

 /* In C++, a named enum is its own type, so define bitwise operators
diff --git a/lib/database.cc b/lib/database.cc
index 1c6ffc5..8fd7fad 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -286,6 +286,8 @@ static const struct {
   "from/subject/message-ID in database", "w" },
 { NOTMUCH_FEATURE_BOOL_FOLDER,
   "exact folder:/path: search", "rw" },
+{ NOTMUCH_FEATURE_GHOSTS,
+  "mail documents for missing messages", "w"},
 };

 const char *
-- 
2.1.0

[PATCH v3 0/9] Add ghost messages and fix thread linking

2014-10-23 Thread Austin Clements 
This is v3 of
id:1412637438-4821-1-git-send-email-aclements at csail.mit.edu.
This fixes some comments, as suggested by Mark.  There are no
code changes relative to v2.  The diff from v2 is below.

diff --git a/lib/database.cc b/lib/database.cc
index 6e51a72..3601f9d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -2121,10 +2121,13 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
 /* Given a blank or ghost 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
- * The first check is in the metadata of the database to see if we
- * have pre-allocated a thread_id in advance for this message, (which
- * would have happened if a message was previously added that
- * referenced this one).
+ * First, if is_ghost, this retrieves the thread ID already stored in
+ * the message (which will be the case if a message was previously
+ * added that referenced this one).  If the message is blank
+ * (!is_ghost), it doesn't have a thread ID yet (we'll generate one
+ * later in this function).  If the database does not support ghost
+ * messages, this checks for a thread ID stored in database metadata
+ * for this message ID.
  *
  * Second, we look at 'message_file' and its link-relevant headers
  * (References and In-Reply-To) for message IDs.
@@ -2132,12 +2135,12 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
  * Finally, we look in the database for existing message that
  * reference 'message'.
  *
- * In all cases, we assign to the current message the first thread_id
- * found (through either parent or child). We will also merge any
- * existing, distinct threads where this message belongs to both,
- * (which is not uncommon when messages are processed out of order).
+ * In all cases, we assign to the current message the first thread ID
+ * found. We will also merge any existing, distinct threads where this
+ * message belongs to both, (which is not uncommon when messages are
+ * processed out of order).
  *
- * Finally, if no thread ID has been found through parent or child, we
+ * Finally, if no thread ID has been found through referenced messages, we
  * call _notmuch_message_generate_thread_id to generate a new thread
  * ID. This should only happen for new, top-level messages, (no
  * References or In-Reply-To header in this message, and no previously

Re: [PATCH v3 3/4] cli: Extend the search command for --output={sender, recipients}

2014-10-23 Thread Mark Walters 
On Sun, 12 Oct 2014, Michal Sojka sojk...@fel.cvut.cz wrote:
 The new outputs allow printing senders, recipients or both of matching
 messages. The --output option is converted from keyword argument to
 flags argument, which means that the user can use --output=sender and
 --output=recipients simultaneously, to print both. Other combinations
 produce an error.

 This code based on a patch from Jani Nikula.
 ---
  completion/notmuch-completion.bash |   2 +-
  completion/notmuch-completion.zsh  |   3 +-
  doc/man1/notmuch-search.rst|  22 +++-
  notmuch-search.c   | 110 
 ++---
  test/T090-search-output.sh |  64 +
  5 files changed, 189 insertions(+), 12 deletions(-)

 diff --git a/completion/notmuch-completion.bash 
 b/completion/notmuch-completion.bash
 index 0571dc9..cfbd389 100644
 --- a/completion/notmuch-completion.bash
 +++ b/completion/notmuch-completion.bash
 @@ -294,7 +294,7 @@ _notmuch_search()
   return
   ;;
   --output)
 - COMPREPLY=( $( compgen -W summary threads messages files tags -- 
 ${cur} ) )
 + COMPREPLY=( $( compgen -W summary threads messages files tags 
 sender recipients -- ${cur} ) )
   return
   ;;
   --sort)
 diff --git a/completion/notmuch-completion.zsh 
 b/completion/notmuch-completion.zsh
 index 67a9aba..3e52a00 100644
 --- a/completion/notmuch-completion.zsh
 +++ b/completion/notmuch-completion.zsh
 @@ -52,7 +52,8 @@ _notmuch_search()
_arguments -s : \
  '--max-threads=[display only the first x threads from the search 
 results]:number of threads to show: ' \
  '--first=[omit the first x threads from the search results]:number of 
 threads to omit: ' \
 -'--sort=[sort results]:sorting:((newest-first\:reverse chronological 
 order oldest-first\:chronological order))'
 +'--sort=[sort results]:sorting:((newest-first\:reverse chronological 
 order oldest-first\:chronological order))' \
 +'--output=[select what to output]:output:((summary threads messages 
 files tags sender recipients))'
  }
  
  _notmuch()
 diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
 index 90160f2..c9d38b1 100644
 --- a/doc/man1/notmuch-search.rst
 +++ b/doc/man1/notmuch-search.rst
 @@ -35,7 +35,7 @@ Supported options for **search** include
  intended for programs that invoke **notmuch(1)** internally. If
  omitted, the latest supported version will be used.
  
 -``--output=(summary|threads|messages|files|tags)``
 +``--output=(summary|threads|messages|files|tags|sender|recipients)``
  
  **summary**
  Output a summary of each thread with any message matching
 @@ -78,6 +78,26 @@ Supported options for **search** include
  by null characters (--format=text0), as a JSON array
  (--format=json), or as an S-Expression list (--format=sexp).
  
 + **sender**
 +Output all addresses from the *From* header that appear on
 +any message matching the search terms, either one per line
 +(--format=text), separated by null characters
 +(--format=text0), as a JSON array (--format=json), or as
 +an S-Expression list (--format=sexp).
 +
 + Note: Searching for **sender** should be much faster than
 + searching for **recipients**, because sender addresses are
 + cached directly in the database whereas other addresses
 + need to be fetched from message files.
 +
 + **recipients**
 +Like **sender** but for addresses from *To*, *Cc* and
 + *Bcc* headers.
 +
 + This option can be given multiple times to combine different
 + outputs. Curently, this is only supported for **sender** and
 + **recipients** outputs.
 +
  ``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
  This option can be used to present results in either
  chronological order (**oldest-first**) or reverse chronological
 diff --git a/notmuch-search.c b/notmuch-search.c
 index 5ac2a26..74588f8 100644
 --- a/notmuch-search.c
 +++ b/notmuch-search.c
 @@ -23,11 +23,14 @@
  #include string-util.h
  
  typedef enum {
 -OUTPUT_SUMMARY,
 -OUTPUT_THREADS,
 -OUTPUT_MESSAGES,
 -OUTPUT_FILES,
 -OUTPUT_TAGS
 +OUTPUT_SUMMARY   = 1  0,
 +OUTPUT_THREADS   = 1  1,
 +OUTPUT_MESSAGES  = 1  2,
 +OUTPUT_FILES = 1  3,
 +OUTPUT_TAGS  = 1  4,
 +OUTPUT_SENDER= 1  5,
 +OUTPUT_RECIPIENTS= 1  6,
 +OUTPUT_ADDRESSES = OUTPUT_SENDER | OUTPUT_RECIPIENTS,
  } output_t;
  
  typedef struct {
 @@ -220,6 +223,67 @@ do_search_threads (search_options_t *o)
  return 0;
  }
  
 +static void
 +print_address_list (const search_options_t *o, InternetAddressList *list)
 +{
 +InternetAddress *address;
 +int i;
 +
 +for (i = 0; i  internet_address_list_length (list); i++) {
 + address =

Re: Looking for the perfect mail client

2014-10-23 Thread Sepp Tannhuber 
Peter Feigl cra...@gmx.net schrieb am 19:38 Mittwoch, 22.Oktober 2014:




 Try strace or
 gdb to find out what really crashes notmuch.
Good idea. Can anybody help me to interprete the output:
https://gist.github.com/tannhuber/c7cae862f897efccd3cb

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Looking for the perfect mail client

2014-10-23 Thread Justus Winter 
Quoting Sepp Tannhuber (2014-10-23 12:47:10)
 Peter Feigl cra...@gmx.net schrieb am 19:38 Mittwoch, 22.Oktober 2014:
  Try strace or
  gdb to find out what really crashes notmuch.
 Good idea. Can anybody help me to interprete the output:
 https://gist.github.com/tannhuber/c7cae862f897efccd3cb

Your process is killed by SIGABRT ;)

Try running it under gdb and obtain a stack trace.

Justus
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Looking for the perfect mail client

2014-10-23 Thread Franz Fellner 
Sepp Tannhuber wrote:
 Peter Feigl cra...@gmx.net schrieb am 19:38 Mittwoch, 22.Oktober 2014:
 
 
 
 
  Try strace or
  gdb to find out what really crashes notmuch.
 Good idea. Can anybody help me to interprete the output:
 https://gist.github.com/tannhuber/c7cae862f897efccd3cb
HUH?
/usr/share\36/usr/loal/share/pixmaps/hicolor/index.theme
\36 is $, isn't it?
and loal should be local?
Probably you messed up your env? (.bashrc;/etc/env.d;...)

 
 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Looking for the perfect mail client

2014-10-23 Thread Franz Fellner 
Justus Winter wrote:
 Quoting Sepp Tannhuber (2014-10-23 12:47:10)
  Peter Feigl cra...@gmx.net schrieb am 19:38 Mittwoch, 22.Oktober 2014:
   Try strace or
   gdb to find out what really crashes notmuch.
  Good idea. Can anybody help me to interprete the output:
  https://gist.github.com/tannhuber/c7cae862f897efccd3cb
 
 Your process is killed by SIGABRT ;)
Yes, that's how exceptions kill your program.
throw - (exception not caught) - std::terminate - std::terminate_handler - 
(default) std::abort

 
 Try running it under gdb and obtain a stack trace.
 
 Justus
 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 0/9] Add ghost messages and fix thread linking

2014-10-23 Thread Austin Clements 
This is v3 of
id:1412637438-4821-1-git-send-email-acleme...@csail.mit.edu.
This fixes some comments, as suggested by Mark.  There are no
code changes relative to v2.  The diff from v2 is below.

diff --git a/lib/database.cc b/lib/database.cc
index 6e51a72..3601f9d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -2121,10 +2121,13 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
 /* Given a blank or ghost 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
- * The first check is in the metadata of the database to see if we
- * have pre-allocated a thread_id in advance for this message, (which
- * would have happened if a message was previously added that
- * referenced this one).
+ * First, if is_ghost, this retrieves the thread ID already stored in
+ * the message (which will be the case if a message was previously
+ * added that referenced this one).  If the message is blank
+ * (!is_ghost), it doesn't have a thread ID yet (we'll generate one
+ * later in this function).  If the database does not support ghost
+ * messages, this checks for a thread ID stored in database metadata
+ * for this message ID.
  *
  * Second, we look at 'message_file' and its link-relevant headers
  * (References and In-Reply-To) for message IDs.
@@ -2132,12 +2135,12 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
  * Finally, we look in the database for existing message that
  * reference 'message'.
  *
- * In all cases, we assign to the current message the first thread_id
- * found (through either parent or child). We will also merge any
- * existing, distinct threads where this message belongs to both,
- * (which is not uncommon when messages are processed out of order).
+ * In all cases, we assign to the current message the first thread ID
+ * found. We will also merge any existing, distinct threads where this
+ * message belongs to both, (which is not uncommon when messages are
+ * processed out of order).
  *
- * Finally, if no thread ID has been found through parent or child, we
+ * Finally, if no thread ID has been found through referenced messages, we
  * call _notmuch_message_generate_thread_id to generate a new thread
  * ID. This should only happen for new, top-level messages, (no
  * References or In-Reply-To header in this message, and no previously

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 3/9] lib: Introduce macros for bit operations

2014-10-23 Thread Austin Clements 
These macros help clarify basic bit-twiddling code and are written to
be robust against C undefined behavior of shift operators.
---
 lib/message.cc|  6 +++---
 lib/notmuch-private.h | 11 +++
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 38bc929..55d2ff6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -869,7 +869,7 @@ notmuch_bool_t
 notmuch_message_get_flag (notmuch_message_t *message,
  notmuch_message_flag_t flag)
 {
-return message-flags  (1  flag);
+return NOTMUCH_TEST_BIT (message-flags, flag);
 }
 
 void
@@ -877,9 +877,9 @@ notmuch_message_set_flag (notmuch_message_t *message,
  notmuch_message_flag_t flag, notmuch_bool_t enable)
 {
 if (enable)
-   message-flags |= (1  flag);
+   NOTMUCH_SET_BIT (message-flags, flag);
 else
-   message-flags = ~(1  flag);
+   NOTMUCH_CLEAR_BIT (message-flags, flag);
 }
 
 time_t
diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
index 36cc12b..7250291 100644
--- a/lib/notmuch-private.h
+++ b/lib/notmuch-private.h
@@ -63,6 +63,17 @@ NOTMUCH_BEGIN_DECLS
 #define STRNCMP_LITERAL(var, literal) \
 strncmp ((var), (literal), sizeof (literal) - 1)
 
+/* Robust bit test/set/reset macros */
+#define NOTMUCH_TEST_BIT(val, bit) \
+((bit  0 || bit = CHAR_BIT * sizeof (unsigned long long)) ? 0\
+ : !!((val)  (1ull  bit)))
+#define NOTMUCH_SET_BIT(valp, bit) \
+((bit  0 || bit = CHAR_BIT * sizeof (unsigned long long)) ? *(valp) \
+ : (*(valp) |= (1ull  bit)))
+#define NOTMUCH_CLEAR_BIT(valp,  bit) \
+((bit  0 || bit = CHAR_BIT * sizeof (unsigned long long)) ? *(valp) \
+ : (*(valp) = ~(1ull  bit)))
+
 #define unused(x) x __attribute__ ((unused))
 
 #ifdef __cplusplus
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 2/9] lib: Update database schema doc for ghost messages

2014-10-23 Thread Austin Clements 
From: Austin Clements amdra...@mit.edu

This describes the structure of ghost mail documents.  Ghost messages
are not yet implemented.
---
 lib/database.cc | 20 ++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 8fd7fad..c641bcd 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -50,8 +50,8 @@ typedef struct {
 
 /* Here's the current schema for our database (for NOTMUCH_DATABASE_VERSION):
  *
- * We currently have two different types of documents (mail and
- * directory) and also some metadata.
+ * We currently have three different types of documents (mail, ghost,
+ * and directory) and also some metadata.
  *
  * Mail document
  * -
@@ -109,6 +109,15 @@ typedef struct {
  *
  * The data portion of a mail document is empty.
  *
+ * Ghost mail document [if NOTMUCH_FEATURE_GHOSTS]
+ * ---
+ * A ghost mail document is like a mail document, but where we don't
+ * have the message content.  These are used to track thread reference
+ * information for messages we haven't received.
+ *
+ * A ghost mail document has type: ghost; id and thread fields that
+ * are identical to the mail document fields; and a MESSAGE_ID value.
+ *
  * Directory document
  * --
  * A directory document is used by a client of the notmuch library to
@@ -172,6 +181,13 @@ typedef struct {
  * generated is 1 and the value will be
  * incremented for each thread ID.
  *
+ * Obsolete metadata
+ * -
+ *
+ * If ! NOTMUCH_FEATURE_GHOSTS, there are no ghost mail documents.
+ * Instead, the database has the following additional database
+ * metadata:
+ *
  * thread_id_* A pre-allocated thread ID for a particular
  * message. This is actually an arbitrarily large
  * family of metadata name. Any particular name is
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 1/9] lib: Add a ghost messages database feature

2014-10-23 Thread Austin Clements 
From: Austin Clements amdra...@mit.edu

This will be implemented over the next several patches.  The feature
is not yet enabled (this does not add it to
NOTMUCH_FEATURES_CURRENT).
---
 lib/database-private.h | 7 +++
 lib/database.cc| 2 ++
 2 files changed, 9 insertions(+)

diff --git a/lib/database-private.h b/lib/database-private.h
index ca0751c..e2e4bc8 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -85,6 +85,13 @@ enum _notmuch_features {
  *
  * Introduced: version 2. */
 NOTMUCH_FEATURE_BOOL_FOLDER = 1  3,
+
+/* If set, missing messages are stored in ghost mail documents.
+ * If unset, thread IDs of ghost messages are stored as database
+ * metadata instead of in ghost documents.
+ *
+ * Introduced: version 3. */
+NOTMUCH_FEATURE_GHOSTS = 1  4,
 };
 
 /* In C++, a named enum is its own type, so define bitwise operators
diff --git a/lib/database.cc b/lib/database.cc
index 1c6ffc5..8fd7fad 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -286,6 +286,8 @@ static const struct {
   from/subject/message-ID in database, w },
 { NOTMUCH_FEATURE_BOOL_FOLDER,
   exact folder:/path: search, rw },
+{ NOTMUCH_FEATURE_GHOSTS,
+  mail documents for missing messages, w},
 };
 
 const char *
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 6/9] lib: Implement upgrade to ghost messages feature

2014-10-23 Thread Austin Clements 
From: Austin Clements amdra...@mit.edu

Somehow this is the first upgrade pass that actually does *any* error
checking, so this also adds the bit of necessary infrastructure to
handle that.
---
 lib/database.cc | 66 +++--
 1 file changed, 64 insertions(+), 2 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 92a92d9..b673718 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1231,6 +1231,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 notmuch_bool_t timer_is_active = FALSE;
 enum _notmuch_features target_features, new_features;
 notmuch_status_t status;
+notmuch_private_status_t private_status;
 unsigned int count = 0, total = 0;
 
 status = _notmuch_database_ensure_writable (notmuch);
@@ -1275,6 +1276,13 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
for (t = db-allterms_begin (XTIMESTAMP); t != t_end; t++)
++total;
 }
+if (new_features  NOTMUCH_FEATURE_GHOSTS) {
+   /* The ghost message upgrade converts all thread_id_*
+* metadata values into ghost message documents. */
+   t_end = db-metadata_keys_end (thread_id_);
+   for (t = db-metadata_keys_begin (thread_id_); t != t_end; ++t)
+   ++total;
+}
 
 /* Perform the upgrade in a transaction. */
 db-begin_transaction (true);
@@ -1378,10 +1386,64 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
}
 }
 
+/* Perform metadata upgrades. */
+
+/* Prior to NOTMUCH_FEATURE_GHOSTS, thread IDs for missing
+ * messages were stored as database metadata. Change these to
+ * ghost messages.
+ */
+if (new_features  NOTMUCH_FEATURE_GHOSTS) {
+   notmuch_message_t *message;
+   std::string message_id, thread_id;
+
+   t_end = db-metadata_keys_end (NOTMUCH_METADATA_THREAD_ID_PREFIX);
+   for (t = db-metadata_keys_begin (NOTMUCH_METADATA_THREAD_ID_PREFIX);
+t != t_end; ++t) {
+   if (do_progress_notify) {
+   progress_notify (closure, (double) count / total);
+   do_progress_notify = 0;
+   }
+
+   message_id = (*t).substr (
+   strlen (NOTMUCH_METADATA_THREAD_ID_PREFIX));
+   thread_id = db-get_metadata (*t);
+
+   /* Create ghost message */
+   message = _notmuch_message_create_for_message_id (
+   notmuch, message_id.c_str (), private_status);
+   if (private_status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+   /* Document already exists; ignore the stored thread ID */
+   } else if (private_status ==
+  NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+   private_status = _notmuch_message_initialize_ghost (
+   message, thread_id.c_str ());
+   if (! private_status)
+   _notmuch_message_sync (message);
+   }
+
+   if (private_status) {
+   fprintf (stderr,
+Upgrade failed while creating ghost messages.\n);
+   status = COERCE_STATUS (private_status, Unexpected status from 
_notmuch_message_initialize_ghost);
+   goto DONE;
+   }
+
+   /* Clear saved metadata thread ID */
+   db-set_metadata (*t, );
+
+   ++count;
+   }
+}
+
+status = NOTMUCH_STATUS_SUCCESS;
 db-set_metadata (features, _print_features (local, notmuch-features));
 db-set_metadata (version, STRINGIFY (NOTMUCH_DATABASE_VERSION));
 
-db-commit_transaction ();
+ DONE:
+if (status == NOTMUCH_STATUS_SUCCESS)
+   db-commit_transaction ();
+else
+   db-cancel_transaction ();
 
 if (timer_is_active) {
/* Now stop the timer. */
@@ -1397,7 +1459,7 @@ notmuch_database_upgrade (notmuch_database_t *notmuch,
 }
 
 talloc_free (local);
-return NOTMUCH_STATUS_SUCCESS;
+return status;
 }
 
 notmuch_status_t
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 8/9] test: Test upgrade to ghost messages feature

2014-10-23 Thread Austin Clements 
---
 test/T530-upgrade.sh | 21 +
 1 file changed, 21 insertions(+)

diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
index c4c4ac8..6b42a69 100755
--- a/test/T530-upgrade.sh
+++ b/test/T530-upgrade.sh
@@ -116,4 +116,25 @@ MAIL_DIR/bar/new/21:2,
 MAIL_DIR/bar/new/22:2,
 MAIL_DIR/cur/51:2,
 
+# Ghost messages are difficult to test since they're nearly invisible.
+# However, if the upgrade works correctly, the ghost message should
+# retain the right thread ID even if all of the original messages in
+# the thread are deleted.  That's what we test.  This won't detect if
+# the upgrade just plain didn't happen, but it should detect if
+# something went wrong.
+test_begin_subtest ghost message retains thread ID
+# Upgrade database
+notmuch new
+# Get thread ID of real message
+thread=$(notmuch search --output=threads id:4efc743a.3060...@april.org)
+# Delete all real messages in that thread
+rm $(notmuch search --output=files $thread)
+notmuch new
+# Deliver ghost message
+add_message '[subject]=Ghost' '[id]=4efc3931.6030...@april.org'
+# If the ghost upgrade worked, the new message should be attached to
+# the existing thread ID.
+nthread=$(notmuch search --output=threads id:4efc3931.6030...@april.org)
+test_expect_equal $thread $nthread
+
 test_done
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 4/9] lib: Internal support for querying and creating ghost messages

2014-10-23 Thread Austin Clements 
From: Austin Clements amdra...@mit.edu

This updates the message abstraction to support ghost messages: it
adds a message flag that distinguishes regular messages from ghost
messages, and an internal function for initializing a newly created
(blank) message as a ghost message.
---
 lib/message.cc| 52 +--
 lib/notmuch-private.h |  4 
 lib/notmuch.h |  9 -
 3 files changed, 62 insertions(+), 3 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 55d2ff6..a7a13cc 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -39,6 +39,9 @@ struct visible _notmuch_message {
 notmuch_message_file_t *message_file;
 notmuch_message_list_t *replies;
 unsigned long flags;
+/* For flags that are initialized on-demand, lazy_flags indicates
+ * if each flag has been initialized. */
+unsigned long lazy_flags;
 
 Xapian::Document doc;
 Xapian::termcount termpos;
@@ -99,6 +102,7 @@ _notmuch_message_create_for_document (const void 
*talloc_owner,
 
 message-frozen = 0;
 message-flags = 0;
+message-lazy_flags = 0;
 
 /* Each of these will be lazily created as needed. */
 message-message_id = NULL;
@@ -192,7 +196,7 @@ _notmuch_message_create (const void *talloc_owner,
  *
  * There is already a document with message ID 'message_id' in the
  * database. The returned message can be used to query/modify the
- * document.
+ * document. The message may be a ghost message.
  *
  *   NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND:
  *
@@ -305,6 +309,7 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message)
 const char *thread_prefix = _find_prefix (thread),
*tag_prefix = _find_prefix (tag),
*id_prefix = _find_prefix (id),
+   *type_prefix = _find_prefix (type),
*filename_prefix = _find_prefix (file-direntry),
*replyto_prefix = _find_prefix (replyto);
 
@@ -337,10 +342,25 @@ _notmuch_message_ensure_metadata (notmuch_message_t 
*message)
message-message_id =
_notmuch_message_get_term (message, i, end, id_prefix);
 
+/* Get document type */
+assert (strcmp (id_prefix, type_prefix)  0);
+if (! NOTMUCH_TEST_BIT (message-lazy_flags, NOTMUCH_MESSAGE_FLAG_GHOST)) {
+   i.skip_to (type_prefix);
+   /* T is the prefix type fields.  See
+* BOOLEAN_PREFIX_INTERNAL. */
+   if (*i == Tmail)
+   NOTMUCH_CLEAR_BIT (message-flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+   else if (*i == Tghost)
+   NOTMUCH_SET_BIT (message-flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+   else
+   INTERNAL_ERROR (Message without type term);
+   NOTMUCH_SET_BIT (message-lazy_flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+}
+
 /* Get filename list.  Here we get only the terms.  We lazily
  * expand them to full file names when needed in
  * _notmuch_message_ensure_filename_list. */
-assert (strcmp (id_prefix, filename_prefix)  0);
+assert (strcmp (type_prefix, filename_prefix)  0);
 if (!message-filename_term_list  !message-filename_list)
message-filename_term_list =
_notmuch_database_get_terms_with_prefix (message, i, end,
@@ -371,6 +391,11 @@ _notmuch_message_invalidate_metadata (notmuch_message_t 
*message,
message-tag_list = NULL;
 }
 
+if (strcmp (type, prefix_name) == 0) {
+   NOTMUCH_CLEAR_BIT (message-flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+   NOTMUCH_CLEAR_BIT (message-lazy_flags, NOTMUCH_MESSAGE_FLAG_GHOST);
+}
+
 if (strcmp (file-direntry, prefix_name) == 0) {
talloc_free (message-filename_term_list);
talloc_free (message-filename_list);
@@ -869,6 +894,10 @@ notmuch_bool_t
 notmuch_message_get_flag (notmuch_message_t *message,
  notmuch_message_flag_t flag)
 {
+if (flag == NOTMUCH_MESSAGE_FLAG_GHOST 
+   ! NOTMUCH_TEST_BIT (message-lazy_flags, flag))
+   _notmuch_message_ensure_metadata (message);
+
 return NOTMUCH_TEST_BIT (message-flags, flag);
 }
 
@@ -880,6 +909,7 @@ notmuch_message_set_flag (notmuch_message_t *message,
NOTMUCH_SET_BIT (message-flags, flag);
 else
NOTMUCH_CLEAR_BIT (message-flags, flag);
+NOTMUCH_SET_BIT (message-lazy_flags, flag);
 }
 
 time_t
@@ -989,6 +1019,24 @@ _notmuch_message_delete (notmuch_message_t *message)
 return NOTMUCH_STATUS_SUCCESS;
 }
 
+/* Transform a blank message into a ghost message.  The caller must
+ * _notmuch_message_sync the message. */
+notmuch_private_status_t
+_notmuch_message_initialize_ghost (notmuch_message_t *message,
+  const char *thread_id)
+{
+notmuch_private_status_t status;
+
+status = _notmuch_message_add_term (message, type, ghost);
+if (status)
+   return status;
+status = _notmuch_message_add_term (message, thread, thread_id);
+if (status)
+   return status;
+
+return NOTMUCH_PRIVATE_STATUS_SUCCESS;
+}
+
 /* Ensure that

[PATCH v3 5/9] lib: Implement ghost-based thread linking

2014-10-23 Thread Austin Clements 
From: Austin Clements amdra...@mit.edu

This updates the thread linking code to use ghost messages instead of
user metadata to link messages into threads.

In contrast with the old approach, this is actually correct.
Previously, thread merging updated only the thread IDs of message
documents, not thread IDs stored in user metadata.  As originally
diagnosed by Mark Walters [1] and as demonstrated by the broken
T260-thread-order test, this can cause notmuch to fail to link
messages even though they're in the same thread.  In principle the old
approach could have been fixed by updating the user metadata thread
IDs as well, but these are not indexed and hence this would have
required a full scan of all stored thread IDs.  Ghost messages solve
this problem naturally by reusing the exact same thread ID and message
ID representation and indexing as regular messages.

Furthermore, thanks to this greater symmetry, ghost messages are also
algorithmically simpler.  We continue to support the old user metadata
format, so this patch can't delete any code, but when we do remove
support for the old format, several functions can simply be deleted.

[1] id:8738h7kv2q@qmul.ac.uk
---
 lib/database.cc | 99 +++--
 1 file changed, 83 insertions(+), 16 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index c641bcd..92a92d9 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1752,6 +1752,12 @@ _get_metadata_thread_id_key (void *ctx, const char 
*message_id)
message_id);
 }
 
+static notmuch_status_t
+_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
+ void *ctx,
+ const char *message_id,
+ const char **thread_id_ret);
+
 /* Find the thread ID to which the message with 'message_id' belongs.
  *
  * Note: 'thread_id_ret' must not be NULL!
@@ -1760,9 +1766,9 @@ _get_metadata_thread_id_key (void *ctx, const char 
*message_id)
  *
  * Note: If there is no message in the database with the given
  * 'message_id' then a new thread_id will be allocated for this
- * message and stored in the database metadata, (where this same
+ * message ID and stored in the database metadata so that the
  * thread ID can be looked up if the message is added to the database
- * later).
+ * later.
  */
 static notmuch_status_t
 _resolve_message_id_to_thread_id (notmuch_database_t *notmuch,
@@ -1770,6 +1776,49 @@ _resolve_message_id_to_thread_id (notmuch_database_t 
*notmuch,
  const char *message_id,
  const char **thread_id_ret)
 {
+notmuch_private_status_t status;
+notmuch_message_t *message;
+
+if (! (notmuch-features  NOTMUCH_FEATURE_GHOSTS))
+   return _resolve_message_id_to_thread_id_old (notmuch, ctx, message_id,
+thread_id_ret);
+
+/* Look for this message (regular or ghost) */
+message = _notmuch_message_create_for_message_id (
+   notmuch, message_id, status);
+if (status == NOTMUCH_PRIVATE_STATUS_SUCCESS) {
+   /* Message exists */
+   *thread_id_ret = talloc_steal (
+   ctx, notmuch_message_get_thread_id (message));
+} else if (status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+   /* Message did not exist.  Give it a fresh thread ID and
+* populate this message as a ghost message. */
+   *thread_id_ret = talloc_strdup (
+   ctx, _notmuch_database_generate_thread_id (notmuch));
+   if (! *thread_id_ret) {
+   status = NOTMUCH_PRIVATE_STATUS_OUT_OF_MEMORY;
+   } else {
+   status = _notmuch_message_initialize_ghost (message, 
*thread_id_ret);
+   if (status == 0)
+   /* Commit the new ghost message */
+   _notmuch_message_sync (message);
+   }
+} else {
+   /* Create failed. Fall through. */
+}
+
+notmuch_message_destroy (message);
+
+return COERCE_STATUS (status, Error creating ghost message);
+}
+
+/* Pre-ghost messages _resolve_message_id_to_thread_id */
+static notmuch_status_t
+_resolve_message_id_to_thread_id_old (notmuch_database_t *notmuch,
+ void *ctx,
+ const char *message_id,
+ const char **thread_id_ret)
+{
 notmuch_status_t status;
 notmuch_message_t *message;
 string thread_id_string;
@@ -2007,13 +2056,16 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
 }
 }
 
-/* Given a (mostly empty) 'message' and its corresponding
+/* Given a blank or ghost 'message' and its corresponding
  * 'message_file' link it to existing threads in the database.
  *
- * The first check is in the metadata of the database to see if we
- * have pre-allocated a thread_id in advance for this message, (which
- * would have

[PATCH v3 7/9] lib: Enable ghost messages feature

2014-10-23 Thread Austin Clements 
From: Austin Clements amdra...@mit.edu

This fixes the broken thread order test.
---
 lib/database-private.h| 2 +-
 test/T260-thread-order.sh | 1 -
 2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/lib/database-private.h b/lib/database-private.h
index e2e4bc8..15e03cc 100644
--- a/lib/database-private.h
+++ b/lib/database-private.h
@@ -166,7 +166,7 @@ struct _notmuch_database {
  * databases will have it). */
 #define NOTMUCH_FEATURES_CURRENT \
 (NOTMUCH_FEATURE_FILE_TERMS | NOTMUCH_FEATURE_DIRECTORY_DOCS | \
- NOTMUCH_FEATURE_BOOL_FOLDER)
+ NOTMUCH_FEATURE_BOOL_FOLDER | NOTMUCH_FEATURE_GHOSTS)
 
 /* Return the list of terms from the given iterator matching a prefix.
  * The prefix will be stripped from the strings in the returned list.
diff --git a/test/T260-thread-order.sh b/test/T260-thread-order.sh
index b435d79..99f5833 100755
--- a/test/T260-thread-order.sh
+++ b/test/T260-thread-order.sh
@@ -30,7 +30,6 @@ expected=$(for ((i = 0; i  $nthreads; i++)); do
 test_expect_equal $output $expected
 
 test_begin_subtest Messages with all parents get linked in all delivery 
orders
-test_subtest_known_broken
 # Here we do the same thing as the previous test, but each message
 # references all of its parents.  Since every message references the
 # root of the thread, each thread should always be fully joined.  This
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v3 9/9] lib: Remove unnecessary thread linking steps when using ghost messages

2014-10-23 Thread Austin Clements 
From: Austin Clements amdra...@mit.edu

Previously, it was necessary to link new messages to children to work
around some (though not all) problems with the old metadata-based
approach to stored thread IDs.  With ghost messages, this is no longer
necessary, so don't bother with child linking when ghost messages are
in use.
---
 lib/database.cc | 29 +
 1 file changed, 21 insertions(+), 8 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index b673718..3601f9d 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -2136,11 +2136,11 @@ _consume_metadata_thread_id (void *ctx, 
notmuch_database_t *notmuch,
  * reference 'message'.
  *
  * In all cases, we assign to the current message the first thread ID
- * found (through either parent or child). We will also merge any
- * existing, distinct threads where this message belongs to both,
- * (which is not uncommon when messages are processed out of order).
+ * found. We will also merge any existing, distinct threads where this
+ * message belongs to both, (which is not uncommon when messages are
+ * processed out of order).
  *
- * Finally, if no thread ID has been found through parent or child, we
+ * Finally, if no thread ID has been found through referenced messages, we
  * call _notmuch_message_generate_thread_id to generate a new thread
  * ID. This should only happen for new, top-level messages, (no
  * References or In-Reply-To header in this message, and no previously
@@ -2172,10 +2172,23 @@ _notmuch_database_link_message (notmuch_database_t 
*notmuch,
 if (status)
goto DONE;
 
-status = _notmuch_database_link_message_to_children (notmuch, message,
-thread_id);
-if (status)
-   goto DONE;
+if (! (notmuch-features  NOTMUCH_FEATURE_GHOSTS)) {
+   /* In general, it shouldn't be necessary to link children,
+* since the earlier indexing of those children will have
+* stored a thread ID for the missing parent.  However, prior
+* to ghost messages, these stored thread IDs were NOT
+* rewritten during thread merging (and there was no
+* performant way to do so), so if indexed children were
+* pulled into a different thread ID by a merge, it was
+* necessary to pull them *back* into the stored thread ID of
+* the parent.  With ghost messages, we just rewrite the
+* stored thread IDs during merging, so this workaround isn't
+* necessary. */
+   status = _notmuch_database_link_message_to_children (notmuch, message,
+thread_id);
+   if (status)
+   goto DONE;
+}
 
 /* If not part of any existing thread, generate a new thread ID. */
 if (thread_id == NULL) {
-- 
2.1.0

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Looking for the perfect mail client

2014-10-23 Thread Sepp Tannhuber 
Franz Fellner alpine.art...@gmail.com schrieb am 12:58 Donnerstag, 23.Oktober 
2014:




 /usr/share\36/usr/loal/share/pixmaps/hicolor/index.theme
 \36 is $, isn't it?
 and loal should be local?
Any idea how this could happen? And how can I fix it?

 Probably you messed up your env? (.bashrc;/etc/env.d;...)

I don't have these files. There's a /etc/bash.bashrc which looks clean.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] VIM: Add better attachment support

2014-10-23 Thread Ian Main 
This patch changes how the notmuch vim client supports attachments:

- For each message part a 'Part number: filename' is added
  to the header.
- You can then use 'e' to extract the attachment under the cursor
  or use it elsewhere to extract all attachments (the prior behavior)
- You can use 'v' to 'view' the attachment/part using xdg-open by default.
- If the message is 'text/html' we include a 'Part:' for the body of the
  message so you can easily view it in a web browser if you so choose.

Ian
---
 vim/notmuch.txt |  8 +-
 vim/notmuch.vim | 84 +++--
 2 files changed, 89 insertions(+), 3 deletions(-)

diff --git a/vim/notmuch.txt b/vim/notmuch.txt
index 4374102..838a904 100644
--- a/vim/notmuch.txt
+++ b/vim/notmuch.txt
@@ -72,6 +72,9 @@ q Quit view
 A  Archive (-inbox -unread)
 I  Mark as read (-unread)
 t  Tag (prompted)
+e   Extract attachment on the current 'Attachment' line or all
+   attachments if the cursor is elsewhere.
+v   View attachment on the current 'Attachment' line.
 s  Search
 p  Save patches
 r  Reply
@@ -148,6 +151,9 @@ You can also configure your externail mail reader and 
sendemail program:
 
let g:notmuch_reader = 'mutt -f %s'
let g:notmuch_sendmail = 'sendmail'
-
+
+You can also configure what probram is used to view attachments:
+
+   let g:notmuch_view_attachment = 'xdg-open'
 
 vim:tw=78:ts=8:noet:ft=help:
diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 331e930..5f73dce 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -35,6 +35,7 @@ let g:notmuch_show_maps = {
\ 't':  'show_tag()',
\ 'o':  'show_open_msg()',
\ 'e':  'show_extract_msg()',
+   \ 'Enter':'show_view_attachment()',
\ 's':  'show_save_msg()',
\ 'p':  'show_save_patches()',
\ 'r':  'show_reply()',
@@ -58,6 +59,8 @@ let s:notmuch_date_format_default = '%d.%m.%y'
 let s:notmuch_datetime_format_default = '%d.%m.%y %H:%M:%S'
 let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
+let s:notmuch_view_attachment_default = 'xdg-open'
+let s:notmuch_attachment_tmpdir_default = '~/.notmuch/tmp'
 let s:notmuch_folders_count_threads_default = 0
 
 function! s:new_file_buffer(type, fname)
@@ -147,13 +150,72 @@ function! s:show_info()
ruby vim_puts get_message.inspect
 endfunction
 
+function! s:show_view_attachment()
+   let line = getline(.)
+ruby  EOF
+   m = get_message
+   line = VIM::evaluate('line')
+
+   match = line.match(/^Part (\d*):/)
+   if match and match.length == 2
+   # Set up the tmpdir
+   tmpdir = VIM::evaluate('g:notmuch_attachment_tmpdir')
+   tmpdir = File.expand_path(tmpdir)
+   Dir.mkdir(tmpdir) unless Dir.exists?(tmpdir)
+
+   p = m.mail.parts[match[1].to_i - 1]
+   if p == nil
+   # Not a multipart message, use the message itself.
+   p = m.mail
+   end
+   if p.filename and p.filename.length  0
+   filename = p.filename
+   else
+   suffix = ''
+   if p.mime_type == 'text/html'
+   suffix = '.html'
+   end
+   filename = part-#{match[1]}#{suffix}
+   end
+
+   # Sanitize just in case..
+   filename.gsub!(/[^0-9A-Za-z.\-]/, '_')
+
+   fullpath = File.expand_path(#{tmpdir}/#{filename})
+   vim_puts Viewing attachment #{fullpath}
+   File.open(fullpath, 'w') do |f|
+   f.write p.body.decoded
+   cmd = VIM::evaluate('g:notmuch_view_attachment')
+   system(cmd, fullpath)
+   end
+   else
+   vim_puts No attachment on this line.
+   end
+EOF
+endfunction
+
 function! s:show_extract_msg()
+   let line = getline(.)
 ruby  EOF
m = get_message
-   m.mail.attachments.each do |a|
+   line = VIM::evaluate('line')
+
+   # If the user is on a line that has an 'Part'
+   # line, we just extract the one attachment.
+   match = line.match(/^Part (\d*):/)
+   if match and match.length == 2
+   a = m.mail.parts[match[1].to_i - 1]
File.open(a.filename, 'w') do |f|
f.write a.body.decoded
-   print Extracted '#{a.filename}'
+   vim_puts Extracted #{a.filename}
+   end
+   else
+   # Extract them all..
+   m.mail.attachments.each do |a|
+   File.open(a.filename, 'w') do |f|
+   f.write a.body.decoded
+   vim_puts Extracted #{a.filename}
+

[PATCH v2] VIM: Add URI handling

2014-10-23 Thread Ian Main 
Add URI handling to the vim client.  You can now press 'enter' by default and
the client will parse the current line and find any 'Part's or URIs available
for opening.  If there are more than one it opens the one under the cursor or
else it opens the only one available.  It also supports mailto: URI's and will
compose a new message when activated.

By default xdg-open is used for everything but mailto: which generally
does the right thing afaict.

Note that this is now dependant on the attachment patch in order to make
the nice 'enter' behavior work for both.

Ian
---
 vim/notmuch.txt |  3 ++-
 vim/notmuch.vim | 76 +++--
 2 files changed, 70 insertions(+), 9 deletions(-)

diff --git a/vim/notmuch.txt b/vim/notmuch.txt
index 838a904..5d84fde 100644
--- a/vim/notmuch.txt
+++ b/vim/notmuch.txt
@@ -74,7 +74,8 @@ I Mark as read (-unread)
 t  Tag (prompted)
 e   Extract attachment on the current 'Attachment' line or all
attachments if the cursor is elsewhere.
-v   View attachment on the current 'Attachment' line.
+enter View email part on the current 'Part' line, or open URI under cursor
+or on line.
 s  Search
 p  Save patches
 r  Reply
diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index 1466e50..2f76f55 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -12,7 +12,7 @@ let g:notmuch_folders_maps = {
\ 'Enter':'folders_show_search()',
\ 's':  'folders_search_prompt()',
\ '=':  'folders_refresh()',
-   \ 'c':  'compose()',
+   \ 'c':  'compose()',
\ }
 
 let g:notmuch_search_maps = {
@@ -25,7 +25,7 @@ let g:notmuch_search_maps = {
\ 's':  'search_search_prompt()',
\ '=':  'search_refresh()',
\ '?':  'search_info()',
-   \ 'c':  'compose()',
+   \ 'c':  'compose()',
\ }
 
 let g:notmuch_show_maps = {
@@ -35,13 +35,13 @@ let g:notmuch_show_maps = {
\ 't':  'show_tag()',
\ 'o':  'show_open_msg()',
\ 'e':  'show_extract_msg()',
-   \ 'Enter':'show_view_attachment()',
+   \ 'Enter':'show_view_magic()',
\ 's':  'show_save_msg()',
\ 'p':  'show_save_patches()',
\ 'r':  'show_reply()',
\ '?':  'show_info()',
\ 'Tab':  'show_next_msg()',
-   \ 'c':  'compose()',
+   \ 'c':  'compose()',
\ }
 
 let g:notmuch_compose_maps = {
@@ -63,6 +63,7 @@ let s:notmuch_view_attachment_default = 'xdg-open'
 let s:notmuch_attachment_tmpdir_default = '~/.notmuch/tmp'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
+let s:notmuch_open_uri_default = 'xdg-open'
 
 function! s:new_file_buffer(type, fname)
exec printf('edit %s', a:fname)
@@ -141,8 +142,8 @@ function! s:show_reply()
end
 endfunction
 
-function! s:compose()
-   ruby open_compose
+function! s:compose(to_email)
+   ruby open_compose(VIM::evaluate('a:to_email'))
let b:compose_done = 0
call s:set_map(g:notmuch_compose_maps)
autocmd BufDelete buffer call s:on_compose_delete()
@@ -155,6 +156,22 @@ function! s:show_info()
ruby vim_puts get_message.inspect
 endfunction
 
+function! s:show_view_magic()
+   let line = getline(.)
+
+ruby  EOF
+   line = VIM::evaluate('line')
+
+   # Easiest to check for 'Part' types first..
+   match = line.match(/^Part (\d*):/)
+   if match and match.length == 2
+   VIM::command('call s:show_view_attachment()')
+   else
+   VIM::command('call s:show_open_uri()')
+   end
+EOF
+endfunction
+
 function! s:show_view_attachment()
let line = getline(.)
 ruby  EOF
@@ -226,6 +243,45 @@ ruby  EOF
 EOF
 endfunction
 
+function! s:show_open_uri()
+   let line = getline(.)
+   let pos = getpos(.)
+   let col = pos[2]
+ruby  EOF
+   m = get_message
+   line = VIM::evaluate('line')
+   col = VIM::evaluate('col') - 1
+   uris = URI.extract(line)
+   wanted_uri = nil
+   if uris.length == 1
+   wanted_uri = uris[0]
+   else
+   uris.each do |uri|
+   # Check to see the URI is at the present cursor location
+   idx = line.index(uri)
+   if col = idx and col = idx + uri.length
+   wanted_uri = uri
+   break
+   end
+   end
+   end
+
+   if wanted_uri
+   uri = URI.parse(wanted_uri)
+   if uri.class == URI::MailTo
+   vim_puts(Composing new email to #{uri.to}.)
+   VIM::command(call s:compose('#{uri.to}'))
+   else
+   vim_puts(Opening #{uri.to_s}.)
+

[PATCH] VIM: Fix header management and fold threads

2014-10-23 Thread Ian Main 
Yes, it's here, the patch you've all been waiting for!  Well, maybe:

- Add a variable to control which headers are displayed normally.
- Any header that is not displayed normally will be listed in a folded
  area in the message so you can access it at will (can be turned off).
- Add the (optional) ability to list threads using vim folding so that
  you can see at a glance the whole thread, which messages are new etc.

Ian
---
 vim/notmuch.vim | 61 ++---
 1 file changed, 54 insertions(+), 7 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index cad9517..affd1e2 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -54,12 +54,22 @@ let s:notmuch_folders_default = [
\ [ 'unread', 'tag:unread' ],
\ ]
 
+let s:notmuch_show_headers_default = [
+   \ 'Subject',
+   \ 'To',
+   \ 'Cc',
+   \ 'Date',
+   \ 'Message-ID',
+   \ ]
+
 let s:notmuch_date_format_default = '%d.%m.%y'
 let s:notmuch_datetime_format_default = '%d.%m.%y %H:%M:%S'
 let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
+let s:notmuch_show_folded_full_headers_default = 1
+let s:notmuch_show_folded_threads_default = 1
 
 function! s:new_file_buffer(type, fname)
exec printf('edit %s', a:fname)
@@ -311,6 +321,9 @@ function! s:show(thread_id)
call s:new_buffer('show')
setlocal modifiable
 ruby  EOF
+   show_full_headers = VIM::evaluate('g:notmuch_show_folded_full_headers')
+   show_threads_folded = 
VIM::evaluate('g:notmuch_show_folded_threads_default')
+
thread_id = VIM::evaluate('a:thread_id')
$cur_thread = thread_id
$messages.clear
@@ -326,11 +339,22 @@ ruby  EOF
date_fmt = VIM::evaluate('g:notmuch_datetime_format')
date = Time.at(msg.date).strftime(date_fmt)
nm_m.start = b.count
-   b  %s %s (%s) % [msg['from'], date, msg.tags]
-   b  Subject: %s % [msg['subject']]
-   b  To: %s % msg['to']
-   b  Cc: %s % msg['cc']
-   b  Date: %s % msg['date']
+   b  From: %s %s (%s) % [msg['from'], date, msg.tags]
+   showheaders = VIM::evaluate('g:notmuch_show_headers')
+   showheaders.each do |h|
+   b  %s: %s % [h, m.header[h]]
+   end
+   if show_full_headers
+   # Now show the rest in a folded area.
+   nm_m.full_header_start = b.count
+   m.header.fields.each do |k|
+   # Only show the ones we haven't already 
printed out.
+   if not showheaders.include?(k.name)
+   b  '%s: %s' % [k.name, k.to_s]
+   end
+   end
+   nm_m.full_header_end = b.count
+   end
nm_m.body_start = b.count
b  --- %s --- % part.mime_type
part.convert.each_line do |l|
@@ -343,11 +367,19 @@ ruby  EOF
end
$messages.each_with_index do |msg, i|
VIM::command(syntax region nmShowMsg#{i}Desc start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgDesc % [msg.start, msg.start + 1])
-   VIM::command(syntax region nmShowMsg#{i}Head start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgHead % [msg.start + 1, msg.body_start])
+   VIM::command(syntax region nmShowMsg#{i}Head start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgHead % [msg.start + 1, msg.full_header_start])
VIM::command(syntax region nmShowMsg#{i}Body start='\\%%%il' 
end='\\%%%dl' contains=@nmShowMsgBody % [msg.body_start, msg.end])
+   if show_full_headers
+   VIM::command(syntax region nmFold#{i}Headers 
start='\\%%%il' end='\\%%%il' fold transparent contains=@nmShowMsgHead % 
[msg.full_header_start, msg.full_header_end])
+   end
+   # Only fold the whole message if there are multiple emails in 
this thread.
+   if $messages.count  1 and show_threads_folded
+   VIM::command(syntax region nmShowMsgFold#{i} 
start='\\%%%il' end='\\%%%il' fold transparent contains=ALL % [msg.start, 
msg.end])
+   end
end
 EOF
setlocal nomodifiable
+   setlocal foldmethod=syntax
call s:set_map(g:notmuch_show_maps)
 endfunction
 
@@ -460,6 +492,19 @@ function! s:set_defaults()
let g:notmuch_folders = s:notmuch_folders_default
endif
endif
+
+   if

RE: Looking for the perfect mail client

2014-10-23 Thread Ian Main 
Sepp Tannhuber wrote:
 Dear all,
 
 I'm wondering whether there's a notmuch mail client
 - with a handling similar to alot
 - which is capable to show html content without html2ascii conversion
 - which can use vim as editor
 - which doesn't need a web browser
 - which doesn't need a mouse
 - which is fast
 
 In principle, alot is the perfect mail client for me except that I've
 many problems with all the html mails people send to me. So I'm wondering
 whether anybody knows a webkit based alot fork or something similar. It's
 strange that after so many years I'm still looking for the perfect mail
 client.

I wonder if you'd be willing to give the vim client a go.  We've done a lot
of work to it lately and I think it would handle all you listed here.

Try the github version as I have all the patches applied there:

https://github.com/imain/notmuch-vim

Ian
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH v3 3/9] lib: Introduce macros for bit operations

2014-10-23 Thread Jani Nikula 
On Oct 23, 2014 3:31 PM, Austin Clements acleme...@csail.mit.edu wrote:

 These macros help clarify basic bit-twiddling code and are written to
 be robust against C undefined behavior of shift operators.
 ---
  lib/message.cc|  6 +++---
  lib/notmuch-private.h | 11 +++
  2 files changed, 14 insertions(+), 3 deletions(-)

 diff --git a/lib/message.cc b/lib/message.cc
 index 38bc929..55d2ff6 100644
 --- a/lib/message.cc
 +++ b/lib/message.cc
 @@ -869,7 +869,7 @@ notmuch_bool_t
  notmuch_message_get_flag (notmuch_message_t *message,
   notmuch_message_flag_t flag)
  {
 -return message-flags  (1  flag);
 +return NOTMUCH_TEST_BIT (message-flags, flag);
  }

  void
 @@ -877,9 +877,9 @@ notmuch_message_set_flag (notmuch_message_t *message,
   notmuch_message_flag_t flag, notmuch_bool_t
enable)
  {
  if (enable)
 -   message-flags |= (1  flag);
 +   NOTMUCH_SET_BIT (message-flags, flag);
  else
 -   message-flags = ~(1  flag);
 +   NOTMUCH_CLEAR_BIT (message-flags, flag);
  }

  time_t
 diff --git a/lib/notmuch-private.h b/lib/notmuch-private.h
 index 36cc12b..7250291 100644
 --- a/lib/notmuch-private.h
 +++ b/lib/notmuch-private.h
 @@ -63,6 +63,17 @@ NOTMUCH_BEGIN_DECLS
  #define STRNCMP_LITERAL(var, literal) \
  strncmp ((var), (literal), sizeof (literal) - 1)

 +/* Robust bit test/set/reset macros */
 +#define NOTMUCH_TEST_BIT(val, bit) \
 +((bit  0 || bit = CHAR_BIT * sizeof (unsigned long long)) ? 0\
 + : !!((val)  (1ull  bit)))
 +#define NOTMUCH_SET_BIT(valp, bit) \
 +((bit  0 || bit = CHAR_BIT * sizeof (unsigned long long)) ?
*(valp) \
 + : (*(valp) |= (1ull  bit)))
 +#define NOTMUCH_CLEAR_BIT(valp,  bit) \
 +((bit  0 || bit = CHAR_BIT * sizeof (unsigned long long)) ?
*(valp) \
 + : (*(valp) = ~(1ull  bit)))

bit should be in braces in the above, like valp.

Jani.

 +
  #define unused(x) x __attribute__ ((unused))

  #ifdef __cplusplus
 --
 2.1.0

 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2] VIM: Fix header management and fold threads

2014-10-23 Thread Ian Main 
Yes, it's here, the patch you've all been waiting for!  Well, maybe:

- Add a variable to control which headers are displayed normally.
- Any header that is not displayed normally will be listed in a folded
  area in the message so you can access it at will (can be turned off).
- Add the (optional) ability to list threads using vim folding so that
  you can see at a glance the whole thread, which messages are new etc.

Ian
---

Fix a thinko where I was using the local variable name instead of the
global.

 vim/notmuch.vim | 61 ++---
 1 file changed, 54 insertions(+), 7 deletions(-)

diff --git a/vim/notmuch.vim b/vim/notmuch.vim
index cad9517..21cfcae 100644
--- a/vim/notmuch.vim
+++ b/vim/notmuch.vim
@@ -54,12 +54,22 @@ let s:notmuch_folders_default = [
\ [ 'unread', 'tag:unread' ],
\ ]
 
+let s:notmuch_show_headers_default = [
+   \ 'Subject',
+   \ 'To',
+   \ 'Cc',
+   \ 'Date',
+   \ 'Message-ID',
+   \ ]
+
 let s:notmuch_date_format_default = '%d.%m.%y'
 let s:notmuch_datetime_format_default = '%d.%m.%y %H:%M:%S'
 let s:notmuch_reader_default = 'mutt -f %s'
 let s:notmuch_sendmail_default = 'sendmail'
 let s:notmuch_folders_count_threads_default = 0
 let s:notmuch_compose_start_insert_default = 1
+let s:notmuch_show_folded_full_headers_default = 1
+let s:notmuch_show_folded_threads_default = 1
 
 function! s:new_file_buffer(type, fname)
exec printf('edit %s', a:fname)
@@ -311,6 +321,9 @@ function! s:show(thread_id)
call s:new_buffer('show')
setlocal modifiable
 ruby  EOF
+   show_full_headers = VIM::evaluate('g:notmuch_show_folded_full_headers')
+   show_threads_folded = VIM::evaluate('g:notmuch_show_folded_threads')
+
thread_id = VIM::evaluate('a:thread_id')
$cur_thread = thread_id
$messages.clear
@@ -326,11 +339,22 @@ ruby  EOF
date_fmt = VIM::evaluate('g:notmuch_datetime_format')
date = Time.at(msg.date).strftime(date_fmt)
nm_m.start = b.count
-   b  %s %s (%s) % [msg['from'], date, msg.tags]
-   b  Subject: %s % [msg['subject']]
-   b  To: %s % msg['to']
-   b  Cc: %s % msg['cc']
-   b  Date: %s % msg['date']
+   b  From: %s %s (%s) % [msg['from'], date, msg.tags]
+   showheaders = VIM::evaluate('g:notmuch_show_headers')
+   showheaders.each do |h|
+   b  %s: %s % [h, m.header[h]]
+   end
+   if show_full_headers
+   # Now show the rest in a folded area.
+   nm_m.full_header_start = b.count
+   m.header.fields.each do |k|
+   # Only show the ones we haven't already 
printed out.
+   if not showheaders.include?(k.name)
+   b  '%s: %s' % [k.name, k.to_s]
+   end
+   end
+   nm_m.full_header_end = b.count
+   end
nm_m.body_start = b.count
b  --- %s --- % part.mime_type
part.convert.each_line do |l|
@@ -343,11 +367,19 @@ ruby  EOF
end
$messages.each_with_index do |msg, i|
VIM::command(syntax region nmShowMsg#{i}Desc start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgDesc % [msg.start, msg.start + 1])
-   VIM::command(syntax region nmShowMsg#{i}Head start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgHead % [msg.start + 1, msg.body_start])
+   VIM::command(syntax region nmShowMsg#{i}Head start='\\%%%il' 
end='\\%%%il' contains=@nmShowMsgHead % [msg.start + 1, msg.full_header_start])
VIM::command(syntax region nmShowMsg#{i}Body start='\\%%%il' 
end='\\%%%dl' contains=@nmShowMsgBody % [msg.body_start, msg.end])
+   if show_full_headers
+   VIM::command(syntax region nmFold#{i}Headers 
start='\\%%%il' end='\\%%%il' fold transparent contains=@nmShowMsgHead % 
[msg.full_header_start, msg.full_header_end])
+   end
+   # Only fold the whole message if there are multiple emails in 
this thread.
+   if $messages.count  1 and show_threads_folded
+   VIM::command(syntax region nmShowMsgFold#{i} 
start='\\%%%il' end='\\%%%il' fold transparent contains=ALL % [msg.start, 
msg.end])
+   end
end
 EOF
setlocal nomodifiable
+   setlocal foldmethod=syntax
call s:set_map(g:notmuch_show_maps)
 endfunction
 
@@ -460,6 +492,19 @@ function! s:set_defaults()
let g:notmuch_folders =