Re: Avoiding the "huge INBOX of death"

[Vladimir Marek]

I can't speak for everyone, but here's what I am using.

I do have 'messages' directory which contains 'imap' and 'archive'.

messages
 \-imap
 \-archive

The imap directory is being synchronized via offlineimap to the server.

In the offlineimap postsync hook I call a script which does some
bookkeeping on the newly received messages and it also marks old and
already read and not flagged emails by the 'old-tread' tag.

N=notmuch
while : ; do
   AGE=100d # 100 days
   # Find all threads containing old messages
   OLD_MESSAGES=$( $N search --output=threads not tag:archive and date:..$AGE 
and not tag:unread and not tag:flagged )
   [ -n "$OLD_MESSAGES" ] || break
   # Find threads containing newer messages
   NEW_MESSAGES=$( $N search --output=threads not tag:archive and not '(' 
date:..$AGE or tag:unread or tag:flagged ')' )
   # remove the threads containing newer messages
   OLD_THREADS=$( /usr/xpg4/bin/fgrep -v -x -f <(echo "$NEW_MESSAGES") < <(echo 
"$OLD_MESSAGES") )
   [ -n "$OLD_THREADS" ] || break
   $N tag +old-thread -- $OLD_THREADS
   break
done

# If old thread received newer message, unmark it too
$N tag -old-thread -- tag:old-thread and '(' not date:..$AGE or tag:unread or 
tag:flagged or tag:archive ')'


Once in a while I review the old-thread messages and assign them by hand
'archive' flag. The postsync hook is looking for this flag and moves all
messages to be archived into archive folder. You can list files to be
archived by

$N search --output=files 'path:imap/**' tag:archive

I won't list the commands to move files from messages/imap to
messages/archive since I have a bit more complex scenario. But it should
be straightforward.

On next run offlineimap sees that the messages disappeared from 'imap'
folder and removes them from imap server.

$ notmuch count '*'
734346
$ notmuch count 'path:imap/**'
18419


In my case I am server side filtering the emails into several folders so
that I have at least some order in emails when reading email via phone,
so 'path:imap/**' contains all the messages in whole imap tree.

$ notmuch count 'path:imap/** and tag:inbox and tag:unread'
11

I'm trying to have my virtual INBOX empty most of the time.

I'm interested what tricks others use!
Cheers
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch

Deduplication ?

[Vladimir Marek]

Hi,


So I wrote some code which works for me well. I have erased ~40k
messages out of 500k. It does not try to be complete solution, it only
detects and removes the obvious cases. The idea is to help me control
the number of duplicates when I import big mail archives which surely
contain many duplicates into my mail database.

> Thinking about this a bit...

> The headers are likely to be different, so you could remove them (get
> rid of everything up to the first empty line).

Yes, that's what I ended up doing. And I delete the files which have
less 'Received:' headers.


> Various mailing lists add footers, so you would need to remove them (a
> regular expression based approach would catch most of them easily).

I defined a list of known footers. Then I take the two mails with the
same message-id, create diff between them and  compare it to the list of
footers.


> The remaining content should be the same for identical messages, so a
> sensible hash (md5) could be used to compare.
> 
> Although, some MTAs modify the body of the message when manipulating
> encoding. I don't know how to address this.

I'm attaching my perl script if anyone is interested. It's in no way
complete solution. It is supposed to be used as

notmuch search --output=files --duplicate=2 '*' > dups
./dedup # It opens the file 'dups'

The attached version does not remove anyting (the 'unlink' command is
commented out).


Interestingly this does not work (it seems to return all messages):
notmuch search --output=messages --duplicate=2 '*'

Also I have found that if I run 'notmuch search' and 'notmuch new' at
the same time, the notmuch search crashes sometimes. That's why I don't
use

notmuch search ... | ./dedup

Use with care :)

Thank you for your help
-- 
Vlad
-- next part --
#!/usr/bin/perl

use Data::Dumper;
use List::Util;


@TO_IGNORE= (

<<'EOT'
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
EOT

,

<<'EOT'
> ___
> Userland-perl mailing list
> Userland-perl at userland.us.oracle.com
> http://userland.us.oracle.com/mailman/listinfo/userland-perl
EOT

,

<<'EOT'
> ___
> Mercurial mailing list
> Mercurial at selenic.com
> http://selenic.com/mailman/listinfo/mercurial
EOT

,

<<'EOT'
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
EOT

,

<<'EOT'
> 
EOT

);

sub rm($$) {
my ($file, $comment) = @_;
print "-> $file\n";
print $comment;
# unlink $file;
}

sub check_mail_id($) {
$ID = $_[0];

unless (open ID, "-|", "./notmuch", "search", "--output=files", 
"id:$ID") {
warn "Can not fork: $!";
return;
}
chomp(@FILES = );
close ID;

if (scalar @FILES <= 1) {
warn "Not enough files for ID:$ID\n";
return;
}

my ($F1, $F2) = @FILES;

unless (-r $F1) {
warn "Can not read $F1 in ID:$ID\n";
return;
}
unless (-r $F2) {
warn "Can not read $F2 in ID:$ID\n";
return;
}
if ($F1 eq $F2) {
warn "Same filename $F1\n in ID:$ID\n";
return;
}

unless (open DIFF_WHOLE, "-|", $diff, $F1, $F2) {
warn "Can not fork $diff: $!\n";
return;
}
$DIFF_WHOLE = join "", ;
close DIFF_WHOLE;

if ( length($DIFF_WHOLE) == 0 ) {
rm $F2, "deleting_1\nID:$ID\n\n";
return;
}

# 35a36
# > Content-Length: 893
if (
$DIFF_WHOLE =~ /^\d+a\d+\n> Content-Length: \d+$/
or
$DIFF_WHOLE =~ /^\d+d\d+\n< Content-Length: \d+$/
) {
rm $F2, "deleting_2\nID:$ID\n\n";
return;
}



# $r="[a-zA-Z0-9 ()[\]\.\+:/=;,\t-]+";
# if (
#   $DIFF_WHOLE =~ /1,7d0\n< Received:$r\n< \t$r\n< \t$r\n< 
Received:$r\n< \t$r\n< \t$r\n< \t$r\n\d+a\d+,\d+\n> Content-Length:$r\n> 
Lines:$r/
# ) {
#   printf "deleting_3\nID:$ID\n$DIFF_WHOLE\n\n";
#   return;
# }

unless (open DIFF_BODY, "-|", "bash", "-c", "$diff <(sed -e 1,/^\$/d 
\"\$1\" ) <(sed -e 1,/^\$/d \"\$2\" )", "", $F1, $F2) {
warn "Can not fork $diff (2): $!\n";
return;
}
$DIFF_BODY = join "", ;
close DIFF_BODY;

if ( length($DIFF_BODY) == 0 ) {
# The bodies are the same - let's find which one has less
# Received: headers and delete that
unless (open F, $F1) 
{
warn "Can't open F1 '$F1': $!";
  

Re: Deduplication ?

[Vladimir Marek]

Hi,


So I wrote some code which works for me well. I have erased ~40k
messages out of 500k. It does not try to be complete solution, it only
detects and removes the obvious cases. The idea is to help me control
the number of duplicates when I import big mail archives which surely
contain many duplicates into my mail database.

 Thinking about this a bit...

 The headers are likely to be different, so you could remove them (get
 rid of everything up to the first empty line).

Yes, that's what I ended up doing. And I delete the files which have
less 'Received:' headers.


 Various mailing lists add footers, so you would need to remove them (a
 regular expression based approach would catch most of them easily).

I defined a list of known footers. Then I take the two mails with the
same message-id, create diff between them and  compare it to the list of
footers.


 The remaining content should be the same for identical messages, so a
 sensible hash (md5) could be used to compare.
 
 Although, some MTAs modify the body of the message when manipulating
 encoding. I don't know how to address this.

I'm attaching my perl script if anyone is interested. It's in no way
complete solution. It is supposed to be used as

notmuch search --output=files --duplicate=2 '*'  dups
./dedup # It opens the file 'dups'

The attached version does not remove anyting (the 'unlink' command is
commented out).


Interestingly this does not work (it seems to return all messages):
notmuch search --output=messages --duplicate=2 '*'

Also I have found that if I run 'notmuch search' and 'notmuch new' at
the same time, the notmuch search crashes sometimes. That's why I don't
use

notmuch search ... | ./dedup

Use with care :)

Thank you for your help
-- 
Vlad
#!/usr/bin/perl

use Data::Dumper;
use List::Util;


@TO_IGNORE= (

'EOT'
 ___
 notmuch mailing list
 notmuch@notmuchmail.org
 http://notmuchmail.org/mailman/listinfo/notmuch
EOT

,

'EOT'
 ___
 Userland-perl mailing list
 userland-p...@userland.us.oracle.com
 http://userland.us.oracle.com/mailman/listinfo/userland-perl
EOT

,

'EOT'
 ___
 Mercurial mailing list
 mercur...@selenic.com
 http://selenic.com/mailman/listinfo/mercurial
EOT

,

'EOT'
 --
 To unsubscribe from this list go to the following URL and read the
 instructions:  https://lists.samba.org/mailman/options/samba
EOT

,

'EOT'
 
EOT

);

sub rm($$) {
my ($file, $comment) = @_;
print - $file\n;
print $comment;
# unlink $file;
}

sub check_mail_id($) {
$ID = $_[0];

unless (open ID, -|, ./notmuch, search, --output=files, 
id:$ID) {
warn Can not fork: $!;
return;
}
chomp(@FILES = ID);
close ID;

if (scalar @FILES = 1) {
warn Not enough files for ID:$ID\n;
return;
}

my ($F1, $F2) = @FILES;

unless (-r $F1) {
warn Can not read $F1 in ID:$ID\n;
return;
}
unless (-r $F2) {
warn Can not read $F2 in ID:$ID\n;
return;
}
if ($F1 eq $F2) {
warn Same filename $F1\n in ID:$ID\n;
return;
}

unless (open DIFF_WHOLE, -|, $diff, $F1, $F2) {
warn Can not fork $diff: $!\n;
return;
}
$DIFF_WHOLE = join , DIFF_WHOLE;
close DIFF_WHOLE;

if ( length($DIFF_WHOLE) == 0 ) {
rm $F2, deleting_1\nID:$ID\n\n;
return;
}

# 35a36
#  Content-Length: 893
if (
$DIFF_WHOLE =~ /^\d+a\d+\n Content-Length: \d+$/
or
$DIFF_WHOLE =~ /^\d+d\d+\n Content-Length: \d+$/
) {
rm $F2, deleting_2\nID:$ID\n\n;
return;
}



# $r=[a-zA-Z0-9 ()[\]\.\+:/=;,\t-]+;
# if (
#   $DIFF_WHOLE =~ /1,7d0\n Received:$r\n \t$r\n \t$r\n 
Received:$r\n \t$r\n \t$r\n \t$r\n\d+a\d+,\d+\n Content-Length:$r\n 
Lines:$r/
# ) {
#   printf deleting_3\nID:$ID\n$DIFF_WHOLE\n\n;
#   return;
# }

unless (open DIFF_BODY, -|, bash, -c, $diff (sed -e 1,/^\$/d 
\\$1\ ) (sed -e 1,/^\$/d \\$2\ ), , $F1, $F2) {
warn Can not fork $diff (2): $!\n;
return;
}
$DIFF_BODY = join , DIFF_BODY;
close DIFF_BODY;

if ( length($DIFF_BODY) == 0 ) {
# The bodies are the same - let's find which one has less
# Received: headers and delete that
unless (open F, $F1) 
{
warn Can't open F1 '$F1': $!;
return;
}
my $count1 = grep { /^Received: / } F;
close F;
 

Deduplication ?

[Vladimir Marek]

> > I want to import bigger chunk of archived messages into my notmuch
> > database. It's about 100k messages. The problem is, that I most probably
> > have quite a lot of those messages in the DB. Basically I would like to
> > add only those I don't have already.
> >
> > There are two possibilities
> >
> > a) I will add all the 100k messages and then remove the duplicities.
> >
> > b) I will write a script which will parse the message ID's of the
> >to-be-added messages and try to match them to the notmuch DB. Adding
> >only files I can't find already.
> >
> > Ad b) might be better option, but I started to play with the idea of
> > deduplication. I'm thinking about listing all the message IDs stored in
> > DB, listing all files belonging to the IDs and deleting all but one.
> > Also I'm thinking about implementing some simple algorithm telling me
> > whether the messages are really very similar. Just to be sure I don't
> > delete something I don't want to.
> >
> > Was anyone playing with the idea?
> 
> notsync[1] used the (lack of) existence of a message id in the store to
> decide whether to add something from an IMAP server, but it is old,
> crufty, unused and unloved code.

I see, that's close to my b) solution, thanks!
-- 
Vlad

Re: Deduplication ?

[Vladimir Marek]

  I want to import bigger chunk of archived messages into my notmuch
  database. It's about 100k messages. The problem is, that I most probably
  have quite a lot of those messages in the DB. Basically I would like to
  add only those I don't have already.
 
  There are two possibilities
 
  a) I will add all the 100k messages and then remove the duplicities.
 
  b) I will write a script which will parse the message ID's of the
 to-be-added messages and try to match them to the notmuch DB. Adding
 only files I can't find already.
 
  Ad b) might be better option, but I started to play with the idea of
  deduplication. I'm thinking about listing all the message IDs stored in
  DB, listing all files belonging to the IDs and deleting all but one.
  Also I'm thinking about implementing some simple algorithm telling me
  whether the messages are really very similar. Just to be sure I don't
  delete something I don't want to.
 
  Was anyone playing with the idea?
 
 notsync[1] used the (lack of) existence of a message id in the store to
 decide whether to add something from an IMAP server, but it is old,
 crufty, unused and unloved code.

I see, that's close to my b) solution, thanks!
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2] new: Don't scan unchanged directories with no sub-directories

[Vladimir Marek]

> > This can substantially reduce the cost of notmuch new in some
> > situations, such as when the file system cache is cold or when the
> > Maildir is on NFS.
> > ---
> 
> pushed as commit 516efb7807

Muchas gracias!
-- 
Vlad

Re: Re: [PATCH v2] new: Don't scan unchanged directories with no sub-directories

[Vladimir Marek]

  This can substantially reduce the cost of notmuch new in some
  situations, such as when the file system cache is cold or when the
  Maildir is on NFS.
  ---
 
 pushed as commit 516efb7807

Muchas gracias!
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH v2] new: Don't scan unchanged directories with no sub-directories

[Vladimir Marek]

Thank you both for your help!
-- 
Vlad

Re: Re: [PATCH v2] new: Don't scan unchanged directories with no sub-directories

[Vladimir Marek]

Thank you both for your help!
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Possible addtions to notmuch new ?

[Vladimir Marek]

Hi,

> > > My mail setup is a directory containing several subdirectories each
> > > subdirectory corresponds to one real mail account I am using. Each mail
> > > account is synchronized differently - I am using offlineimap, fetchmeail
> > > or even synthetically created emails (I am writing very simple jabber<->
> > > mail gate).Every now and then I am running 'notmuch new' to discover new
> > > emails and make them available in my MUA.
> > > 
> > > That works pretty well, but has some disadvantages too
> > >  - notmuch new takes very long time (30s) during which the notmuch
> > >database seems to be locked for any other updates from my MUA
> > >  - notmuch new takes long time because it always processes my archive
> > >dir containing many files. That's mostly un-necessary as typically
> > >there's no new mail delivered
> > 
> > Could you try this patch?  It's basically untested other than passing
> > the test suite, though in principle the worst harm it could do is make
> > notmuch new miss new messages or think renames are deletions.  If it
> > helps significantly with your performance problems, I'll clean it up
> > and add a test.
> > 
> > diff --git a/notmuch-new.c b/notmuch-new.c
> > index faa33f1..196c5cb 100644
> > --- a/notmuch-new.c
> > +++ b/notmuch-new.c
> > @@ -323,6 +323,9 @@ add_files (notmuch_database_t *notmuch,
> >  }
> >  db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;
> >  
> > +if (directory && db_mtime == fs_mtime && st.st_nlink == 2)
> > +   goto DONE;
> > +
> >  /* If the database knows about this directory, then we sort based
> >   * on strcmp to match the database sorting. Otherwise, we can do
> >   * inode-based sorting for faster filesystem operation. */
> 
> 
> I'm sorry for my late reply. It cuts the average time of 'notmuch new'
> from 25s to 0.2s . Which is a bit scary :) But understandable. I have
> the notmuch database on NFS mount which hopefully won't make any
> difference here.

Is there anything missing before including the patch into notmuch
sources? Today I rebuilt notmuch and forgot to include this patch and
"notmuch new" is dead slow again.

Cheers
-- 
Vlad

Re: Re: Re: Possible addtions to notmuch new ?

[Vladimir Marek]

Hi,

   My mail setup is a directory containing several subdirectories each
   subdirectory corresponds to one real mail account I am using. Each mail
   account is synchronized differently - I am using offlineimap, fetchmeail
   or even synthetically created emails (I am writing very simple jabber-
   mail gate).Every now and then I am running 'notmuch new' to discover new
   emails and make them available in my MUA.
   
   That works pretty well, but has some disadvantages too
- notmuch new takes very long time (30s) during which the notmuch
  database seems to be locked for any other updates from my MUA
- notmuch new takes long time because it always processes my archive
  dir containing many files. That's mostly un-necessary as typically
  there's no new mail delivered
  
  Could you try this patch?  It's basically untested other than passing
  the test suite, though in principle the worst harm it could do is make
  notmuch new miss new messages or think renames are deletions.  If it
  helps significantly with your performance problems, I'll clean it up
  and add a test.
  
  diff --git a/notmuch-new.c b/notmuch-new.c
  index faa33f1..196c5cb 100644
  --- a/notmuch-new.c
  +++ b/notmuch-new.c
  @@ -323,6 +323,9 @@ add_files (notmuch_database_t *notmuch,
   }
   db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;
   
  +if (directory  db_mtime == fs_mtime  st.st_nlink == 2)
  +   goto DONE;
  +
   /* If the database knows about this directory, then we sort based
* on strcmp to match the database sorting. Otherwise, we can do
* inode-based sorting for faster filesystem operation. */
 
 
 I'm sorry for my late reply. It cuts the average time of 'notmuch new'
 from 25s to 0.2s . Which is a bit scary :) But understandable. I have
 the notmuch database on NFS mount which hopefully won't make any
 difference here.

Is there anything missing before including the patch into notmuch
sources? Today I rebuilt notmuch and forgot to include this patch and
notmuch new is dead slow again.

Cheers
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/1] test: unset 'xpg_echo' bash shell option

[Vladimir Marek]

Yes, that fixes the tests on Solaris which were broken by the 'echo'
performing differently. If my vote counts, put it in, please!

Thank you
-- 
Vlad

> When 'xpg_echo' bash shell option is unset (usually the default)
> echo builtin does not expand backslash-escape sequences by default
> (i.e. '\n' is echoed as '\n' instead of newline). Not all bash
> installations have this feature we depend on activated by default.
> 
> Note that the feature is bash (and GNU /bin/echo) specific. It is used
> as it is convenient. If portability is needed (elsewhere) use printf(1)
> (also often available as a shell builtin).
> ---
>  test/test-lib.sh | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/test/test-lib.sh b/test/test-lib.sh
> index ffab1bb..d3a8c58 100644
> --- a/test/test-lib.sh
> +++ b/test/test-lib.sh
> @@ -22,6 +22,9 @@ if [ ${BASH_VERSINFO[0]} -lt 4 ]; then
>  exit 1
>  fi
>  
> +# Make sure echo builtin does not expand backslash-escape sequences by 
> default.
> +shopt -u xpg_echo
> +
>  # if --tee was passed, write the output not only to the terminal, but
>  # additionally to the file test-results/$BASENAME.out, too.
>  case "$GIT_TEST_TEE_STARTED, $* " in
> -- 
> 1.8.0
> 
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

Re: Re: Possible addtions to notmuch new ?

[Vladimir Marek]

  My mail setup is a directory containing several subdirectories each
  subdirectory corresponds to one real mail account I am using. Each mail
  account is synchronized differently - I am using offlineimap, fetchmeail
  or even synthetically created emails (I am writing very simple jabber-
  mail gate).Every now and then I am running 'notmuch new' to discover new
  emails and make them available in my MUA.
  
  That works pretty well, but has some disadvantages too
   - notmuch new takes very long time (30s) during which the notmuch
 database seems to be locked for any other updates from my MUA
   - notmuch new takes long time because it always processes my archive
 dir containing many files. That's mostly un-necessary as typically
 there's no new mail delivered
 
 Could you try this patch?  It's basically untested other than passing
 the test suite, though in principle the worst harm it could do is make
 notmuch new miss new messages or think renames are deletions.  If it
 helps significantly with your performance problems, I'll clean it up
 and add a test.
 
 diff --git a/notmuch-new.c b/notmuch-new.c
 index faa33f1..196c5cb 100644
 --- a/notmuch-new.c
 +++ b/notmuch-new.c
 @@ -323,6 +323,9 @@ add_files (notmuch_database_t *notmuch,
  }
  db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;
  
 +if (directory  db_mtime == fs_mtime  st.st_nlink == 2)
 +   goto DONE;
 +
  /* If the database knows about this directory, then we sort based
   * on strcmp to match the database sorting. Otherwise, we can do
   * inode-based sorting for faster filesystem operation. */


I'm sorry for my late reply. It cuts the average time of 'notmuch new'
from 25s to 0.2s . Which is a bit scary :) But understandable. I have
the notmuch database on NFS mount which hopefully won't make any
difference here.

Thank you for the tip!
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Possible addtions to notmuch new ?

[Vladimir Marek]

> > My mail setup is a directory containing several subdirectories each
> > subdirectory corresponds to one real mail account I am using. Each mail
> > account is synchronized differently - I am using offlineimap, fetchmeail
> > or even synthetically created emails (I am writing very simple jabber<->
> > mail gate).Every now and then I am running 'notmuch new' to discover new
> > emails and make them available in my MUA.
> > 
> > That works pretty well, but has some disadvantages too
> >  - notmuch new takes very long time (30s) during which the notmuch
> >database seems to be locked for any other updates from my MUA
> >  - notmuch new takes long time because it always processes my archive
> >dir containing many files. That's mostly un-necessary as typically
> >there's no new mail delivered
> 
> Could you try this patch?  It's basically untested other than passing
> the test suite, though in principle the worst harm it could do is make
> notmuch new miss new messages or think renames are deletions.  If it
> helps significantly with your performance problems, I'll clean it up
> and add a test.
> 
> diff --git a/notmuch-new.c b/notmuch-new.c
> index faa33f1..196c5cb 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -323,6 +323,9 @@ add_files (notmuch_database_t *notmuch,
>  }
>  db_mtime = directory ? notmuch_directory_get_mtime (directory) : 0;
>  
> +if (directory && db_mtime == fs_mtime && st.st_nlink == 2)
> +   goto DONE;
> +
>  /* If the database knows about this directory, then we sort based
>   * on strcmp to match the database sorting. Otherwise, we can do
>   * inode-based sorting for faster filesystem operation. */


I'm sorry for my late reply. It cuts the average time of 'notmuch new'
from 25s to 0.2s . Which is a bit scary :) But understandable. I have
the notmuch database on NFS mount which hopefully won't make any
difference here.

Thank you for the tip!
-- 
Vlad

[PATCH 4/4] timegm: add portable implementation (Solaris support)

[Vladimir . Marek]

From: Blake Jones bla...@foo.net

The timegm(3) function is a non-standard extension to libc which is
available in GNU libc and on some BSDs.  Although SunOS had this
function in its libc, Solaris (unfortunately) removed it.  This patch
implements a very simple version of timegm() which is good enough for
parse-time-string.c.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 compat/Makefile.local |4 +++
 compat/compat.h   |5 +++
 compat/have_timegm.c  |7 
 compat/timegm.c   |   57 +
 configure |   11 +++
 parse-time-string/parse-time-string.c |1 +
 6 files changed, 85 insertions(+)
 create mode 100644 compat/have_timegm.c
 create mode 100644 compat/timegm.c

diff --git a/compat/Makefile.local b/compat/Makefile.local
index 2c4f65f..b0d5417 100644
--- a/compat/Makefile.local
+++ b/compat/Makefile.local
@@ -17,4 +17,8 @@ ifneq ($(HAVE_STRSEP),1)
 notmuch_compat_srcs += $(dir)/strsep.c
 endif
 
+ifneq ($(HAVE_TIMEGM),1)
+notmuch_compat_srcs += $(dir)/timegm.c
+endif
+
 SRCS := $(SRCS) $(notmuch_compat_srcs)
diff --git a/compat/compat.h b/compat/compat.h
index ae762c3..5a402d5 100644
--- a/compat/compat.h
+++ b/compat/compat.h
@@ -57,6 +57,11 @@ char* strcasestr(const char *haystack, const char *needle);
 char *strsep(char **stringp, const char *delim);
 #endif /* !HAVE_STRSEP */
 
+#if !HAVE_TIMEGM
+#include time.h
+time_t timegm (struct tm *tm);
+#endif /* !HAVE_TIMEGM */
+
 /* Silence gcc warnings about unused results.  These warnings exist
  * for a reason; any use of this needs to be justified. */
 #ifdef __GNUC__
diff --git a/compat/have_timegm.c b/compat/have_timegm.c
new file mode 100644
index 000..b62b793
--- /dev/null
+++ b/compat/have_timegm.c
@@ -0,0 +1,7 @@
+#include time.h
+#include compat.h
+
+int main()
+{
+return (int) timegm((struct tm *)0);
+}
diff --git a/compat/timegm.c b/compat/timegm.c
new file mode 100644
index 000..213963b
--- /dev/null
+++ b/compat/timegm.c
@@ -0,0 +1,57 @@
+/* timegm.c --- Implementation of replacement timegm function.
+   Copyright (C) 2012 Free Software Foundation, Inc.
+
+   This program is free software; you can redistribute it and/or
+   modify it under the terms of the GNU General Public License as
+   published by the Free Software Foundation; either version 3, or (at
+   your option) any later version.
+
+   This program is distributed in the hope that it will be useful, but
+   WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+   02110-1301, USA.  */
+
+/* Written by Blake Jones. */
+
+#include time.h
+#include compat.h
+
+static int
+leapyear (int year)
+{
+return ((year % 4) == 0  ((year % 100) != 0 || (year % 400) == 0));
+}
+
+/*
+ * This is a simple implementation of timegm() which does what is needed
+ * by create_output() -- just turns the struct tm into a GMT time_t.
+ * It does not normalize any of the fields of the struct tm, nor does
+ * it set tm_wday or tm_yday.
+ */
+time_t
+timegm (struct tm *tm)
+{
+intmonthlen[2][12] = {
+   { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 },
+   { 31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 },
+};
+intyear, month, days;
+
+days = 365 * (tm-tm_year - 70);
+for (year = 70; year  tm-tm_year; year++) {
+   if (leapyear(1900 + year)) {
+   days++;
+   }
+}
+for (month = 0; month  tm-tm_mon; month++) {
+   days += monthlen[leapyear(1900 + year)][month];
+}
+days += tm-tm_mday - 1;
+
+return days * 24) + tm-tm_hour) * 60 + tm-tm_min) * 60 + tm-tm_sec);
+}
diff --git a/configure b/configure
index ac44857..6166917 100755
--- a/configure
+++ b/configure
@@ -530,6 +530,17 @@ else
 fi
 rm -f compat/have_strsep
 
+printf Checking for timegm... 
+if ${CC} -o compat/have_timegm $srcdir/compat/have_timegm.c  /dev/null 21
+then
+printf Yes.\n
+have_timegm=1
+else
+printf No (will use our own instead).\n
+have_timegm=0
+fi
+rm -f compat/have_timegm
+
 printf Checking for standard version of getpwuid_r... 
 if ${CC} -o compat/check_getpwuid $srcdir/compat/check_getpwuid.c  
/dev/null 21
 then
diff --git a/parse-time-string/parse-time-string.c 
b/parse-time-string/parse-time-string.c
index 584067d..ccad422 100644
--- a/parse-time-string/parse-time-string.c
+++ b/parse-time-string/parse-time-string.c
@@ -32,6 +32,7 @@
 #include sys/time.h
 #include sys/types.h
 
+#include compat.h
 #include parse-time-string.h
 
 /*
-- 
1.7.9.2

___
notmuch mailing list
notmuch@notmuchmail.org
http

Solaris support - missing or incompatible functions

[Vladimir Marek]

Hi,

> > > I didn't pay too much attention to the test output as it's broken on
> > > linux too :)
> > 
> > It shouldn't be. All tests pass for me, and we're pretty strict about
> > maintaining that.
> > 
> > There may be some prerequisites to running tests that you're missing,
> > and the test setup fails to tell you about. A pastebin of the 'make
> > test' output might be helpful.
> 
> Ok, in that case please let me take a look at the tests first.

I'm sorry it took me so long, but finally I got as far as possible with
tests. At the moment the status is

Notmuch test suite complete.
548/557 tests passed.
9 tests failed.

1 test fails because there are new exported symbols in notmuch binary
2 test fails because I don't have gpg
6 test fails because my emacs somehow refuses to send email

With that I would like to ask for including the patches into notmuch
tree. Should I rebase them to the latest git tree or can you push them
as they are?

Thank you
-- 
Vlad

notmuch tests issue on solaris

[Vladimir Marek]

Hi,

I returned back to making the test suite work on Solaris. But now I
stumbled upon a problem with emacs which I don't know how to solve ...

Emacs seems to throw some kind of error while trying to send email

+ emacsclient --socket-name=notmuch-test-suite-9620 --eval '(progn (let 
((message-send-mail-function '\''message-smtpmail-send-it)
   (smtpmail-smtp-server "localhost")
   (smtpmail-smtp-service "25025"))
   (setq smtpmail-debug-info t)
   (notmuch-hello)
   (notmuch-mua-mail)
   (message-goto-to)
   (insert "test_suite at notmuchmail.org\nDate: 01 Jan 2000 12:00:00 
-")
   (message-goto-subject)
   (insert "Testing message sent via SMTP")
   (message-goto-body)
   (insert "This is a test that messages are sent via SMTP")
   (message-goto-to)
 (kill-whole-line)
 (insert "To: user at example.com\n")
   (message-send-and-exit)))'
*ERROR*: Sending failed; SMTP protocol error




If I try to instrument the smtp-dummy binary, I can see:

11183/1: 0.1832 write(5, 0x0806550C, 26)= 26
11183/1:   2 2 0   l o c a l h o s t   s m t p - d u m m y\r\n
11183/1: 0.1839 read(5, 0x0806550C, 8192)   = 14
11183/1:   E H L O   S 1 1 . 1 _ F C S
11183/1: 0.1841 read(5, "\r\n", 8192)   = 2
11183/1: 0.1842 write(5, "\r\n", 2) = 2
11183/1: 0.1842 write(5, 0x0806550C, 21)= 21
11183/1:   5 0 2   n o t   i m p l e m e n t e d\r\n
11183/1: 0.1845 read(5, 0x0806550C, 8192)   = 14
11183/1:   H E L O   S 1 1 . 1 _ F C S
11183/1: 0.1847 read(5, "\r\n", 8192)   = 2
11183/1: 0.1847 write(5, "\r\n", 2) = 2
11183/1: 0.1847 write(5, 0x0806550C, 15)= 15
11183/1:   2 5 0   l o c a l h o s t\r\n
11183/1: 0.1850 read(5, " Q U I T", 8192)   = 4
11183/1: 0.1852 read(5, "\r\n", 8192)   = 2
11183/1: 0.1853 write(5, "\r\n", 2) = 2
11183/1: 0.1853 write(5, " 2 2 1   B Y E\r\n", 9)   = 9
11183/1: 0.1854 close(3)= 0
11183/1: 0.1855 llseek(5, 0, SEEK_CUR)  Err#29 
ESPIPE
11183/1: 0.1855 close(5)= 0
11183/1: 0.1855 close(4)= 0
11183/1: 0.1856 _exit(0)

Which is

-> 220 localhost smtp-dummy
<- EHLO S11.1_FCS
-> 502 not implemented
<- HELO S11.1_FCS
-> 250 localhost
<- QUIT
-> 221 BYE

So it seems that emacs just quits after HELO command? Is there a way to
debug emacs? Several tests are using emacs for sending email ...

Thank you
-- 
Vlad

Possible addtions to notmuch new ?

[Vladimir Marek]

Hi,

My mail setup is a directory containing several subdirectories each
subdirectory corresponds to one real mail account I am using. Each mail
account is synchronized differently - I am using offlineimap, fetchmeail
or even synthetically created emails (I am writing very simple jabber<->
mail gate).Every now and then I am running 'notmuch new' to discover new
emails and make them available in my MUA.

That works pretty well, but has some disadvantages too
 - notmuch new takes very long time (30s) during which the notmuch
   database seems to be locked for any other updates from my MUA
 - notmuch new takes long time because it always processes my archive
   dir containing many files. That's mostly un-necessary as typically
   there's no new mail delivered
 - I don't have the possibility of passing new mails through procmail.
   That would be useful for example for changing cron mail subjects,
   putting related automated mails into threads (bugzilla, etc.).


I was thinking that if we could split the new mail discovery from
it's processing, it would solve the disadvantages I'm facing. Something
like

notmuch new --verbose --dry-run [dir] | my_filter | notmuch insert -

It would work
 - --dry-run would not lock and change the database
 - --verbose would print the changes to stdout/stderr. Something like:

new mail/file.1
new mail/file.2
deleted mail/file.3
renamed mail/file.4 mail/file.5
...

[dir] would limit the scope of 'notmuch' new search to dir and it's
subdirectories. Alternatively we could have set of include or exclude
rules similarly to rsync (for example), but [dir] is easier to
implement.

'my_filter' would be my script which could change the contents of emails
before they are inserted into notmuch database.

Notmuch insert would be able to not only add new mails, but also remove
old ones or note that the file was renamed.

How would this sound?

I'm not saying I would implement this, I'm rather curious where would
you like to see notmuch in the future.

Cheers
-- 
Vlad

[PATCH 0/3] Introduce the add command

[Vladimir Marek]

> > On Sat, Jul 13 2013, Adam Wolfe Gordon  wrote:
> >> The recent introduction of insert inspired me to finally add a feature I've
> >> been wanting: a command to index a specific file in the maildir. My usecase
> >> for this is that I have an inotify-based script that watches for new mail
> >> and calls notmuch new when new mail shows up. Running notmuch new in this
> >> situation is overkill, since I know exactly what's changed. A faster 
> >> command
> >> that just adds a single file reduces contention on the database lock.
> >>
> >> This series introduces a new command, "notmuch add", which indexes a file
> >> that already exists in the maildir. It is implemented in notmuch-insert.c
> >> because it uses the basic infrastructure introduced for the insert command.
> >>
> >> Missing man page for now - wanted to get the code out first for review.
> >
> > Hey, Adam.  This feature seems fine, but it seems similar enought to
> > insert that I wonder if they can just be unified.  What if insert just
> > took an optional path argument as well, e.g.:
> >
> >   notmuch insert [options] [ +|- ... ] [-- /path/to/file]
> >
> > If the path is not in the db, it would insert it the same as if it had
> > come in via stdin.  If the path *is* in the db, it could just do the add
> > part that you're looking for.  That seems like it might be a more
> > intuitive UI experience to me.
> 
> I sort of agree but wonder if it would be more natural under "new" than
> "insert". so notmuch new /path/to/file just adds that file (provided it
> is in the database; perhaps a relative path?) This would also be
> extensible to do a whole mail sub-directory which seems like it might
> also be a useful feature. (I am definitely not saying that this needs to
> be implemented now!)

That would be mostly welcome addition indeed ...

Cheers
-- 
Vlad

how to work with entire threads

[Vladimir Marek]

> > I would like to create the following process:
> >
> >  1) if I tag any message with the keyword 'kill', it means that I don't
> >want to see any message from whole thread in my inbox
> >
> > At the moment I do something like
> >
> > THREADS=$(notmuch search --output=threads tag:kill)
> > [ -z "$THREADS" ] || notmuch tag -unread -- '(' $THREADS ')'
> >
> >  2) if there is such a 'killed' thread and it won't receive message for
> > 30 days, I want to remove the kill tag from the thread
> >
> > That I achieved by something like
> >
> > N=notmuch
> > KILL=$( $N search --output=threads tag:kill )   
> > 
> >
> > if [ -n "$KILL" ]; then
> ># Then find threads which contain messages younger than 30 days
> >KILL_RECENT=$( $N search --output=threads '(' $KILL ')' and 
> > date:30days.. )
> >KILL_RECENT=${KILL_RECENT:-'*'}
> ># Then find 'kill' threads not matching those recent
> >KILL_COUNT=$( $N count --output=threads '(' $KILL ')' and not '(' 
> > $KILL_RECENT ')' )
> >if [ $KILL_COUNT -gt 0 ]; then
> >   echo "Threads to remove 'kill': $KILL"
> >   $N tag -kill -- $( $N search --output=threads '(' $KILL ')' and not 
> > '(' $KILL_RECENT ')' )
> >fi
> > fi
> >
> >
> >
> > I was thinking if it could not be simplified. Without knowing anything about
> > notmuch or xapian internals, it seems that it would be handy to change the
> > search pattern to return threads. My two previous examples would then read 
> > like
> >
> > 1) notmuch tag -unread -- 'threads(tag:kill)'
> > and
> > 2) notmuch tag -kill -- 'threads(tag:kill)' and not 'threads(date:30days..)'
> >
> > Given that this would be possible in the first place, wouldn't it be a nice
> > addition to notmuch?
> 
> Some form of general thread based search would definitely be a nice
> addition. This seems to be non-trivial as we would need to parse the
> query ourselves rather than leaving that to xapian.
> 
> However, I think thread bases tagging would be quite easy:
> 
> eg
> 
> notmuch tag --thread +/-tags 
> 
> to tag all messages in all threads that contain a message match the
> search terms. Your first case would become notmuch tag --thread
> -unread tag:kill but your second case would still not be directly
> possible.
> 
> Would this be a useful addition by itself?


As you said, marking whole threads as read would be a case I would use
regularly. If anyone is willing to add that feature, I would be glad.

Even the second case would be possible, just not as elegant (but still
better than my shell construct)

notmuch tag --thread +kill -- tag:kill
notmuch tag -kill -- tag:kill and not date:30days..

Cheers
-- 
Vlad

How to find mails which are sent to 'undisclosed-recipients' ?

[Vladimir Marek]

> > > > Since my mail works in a way that it hides everything but what I
> > > > selected to be shown to me, I was overlooking all mails which where sent
> > > > to undisclosed-recipients. I tried to match such mails by
> > > > to:undisclosed-recipients but that does not seem to work. Is there any
> > > > workaround to find such mails?
> > > 
> > > I have some mail with this To header:
> > > 
> > > To: Undisclosed recipients 
> > > 
> > > I can find them with the search to:"Undisclosed recipients". Note the 
> > > quotes.
> > 
> > The To: line in my case looks like:
> > 
> > To: undisclosed-recipients:;
> 
> RFC822 group syntax!  Fascinating.  You're right that notmuch doesn't
> index group names, though I think it could with a small addition to
> _index_address_group.  It already descends into group addresses, it
> just currently ignores the group name.
> 
> > And I'm not able to find it in any way. Maybe notmuch won't parse it as an
> > email and so won't store the To header to the database?
> 
> I suspect it is indexing it.  The only thing notmuch requires is that
> the message be at least slightly well-formed and have either a From,
> Subject, or To header.  You can find the message's Message-ID header
> and try a search like
> 
>   notmuch search id:
> 
> > Any idea if there is a way to dump what notmuch knows about given email?
> 
> There's no easy way.  You can use Xapian's quest and delve tools to
> find the document ID and get the term list of a message, but this is a
> *very* low-level view of what notmuch knows.


Thank you for looking at this. At the moment I plugged procmail to my
setup to tag any such messages (sent to undisclosed recipients) with
appropriate tag.

Thank you
-- 
Vlad

How to find mails which are sent to 'undisclosed-recipients' ?

[Vladimir Marek]

> > Since my mail works in a way that it hides everything but what I
> > selected to be shown to me, I was overlooking all mails which where sent
> > to undisclosed-recipients. I tried to match such mails by
> > to:undisclosed-recipients but that does not seem to work. Is there any
> > workaround to find such mails?
> 
> I have some mail with this To header:
> 
> To: Undisclosed recipients 
> 
> I can find them with the search to:"Undisclosed recipients". Note the quotes.

The To: line in my case looks like:

To: undisclosed-recipients:;

And I'm not able to find it in any way. Maybe notmuch won't parse it as an
email and so won't store the To header to the database?

Any idea if there is a way to dump what notmuch knows about given email?

Thank you
-- 
Vlad

How to find mails which are sent to 'undisclosed-recipients' ?

[Vladimir Marek]

Hi,

Since my mail works in a way that it hides everything but what I
selected to be shown to me, I was overlooking all mails which where sent
to undisclosed-recipients. I tried to match such mails by
to:undisclosed-recipients but that does not seem to work. Is there any
workaround to find such mails?

Thank you
-- 
Vlad

how to work with entire threads

[Vladimir Marek]

Hi,

I would like to create the following process:

 1) if I tag any message with the keyword 'kill', it means that I don't
   want to see any message from whole thread in my inbox

At the moment I do something like

THREADS=$(notmuch search --output=threads tag:kill)
[ -z "$THREADS" ] || notmuch tag -unread -- '(' $THREADS ')'

 2) if there is such a 'killed' thread and it won't receive message for
30 days, I want to remove the kill tag from the thread

That I achieved by something like

N=notmuch
KILL=$( $N search --output=threads tag:kill )   

   
if [ -n "$KILL" ]; then
   # Then find threads which contain messages younger than 30 days
   KILL_RECENT=$( $N search --output=threads '(' $KILL ')' and date:30days.. )
   KILL_RECENT=${KILL_RECENT:-'*'}
   # Then find 'kill' threads not matching those recent
   KILL_COUNT=$( $N count --output=threads '(' $KILL ')' and not '(' 
$KILL_RECENT ')' )
   if [ $KILL_COUNT -gt 0 ]; then
  echo "Threads to remove 'kill': $KILL"
  $N tag -kill -- $( $N search --output=threads '(' $KILL ')' and not '(' 
$KILL_RECENT ')' )
   fi
fi



I was thinking if it could not be simplified. Without knowing anything about
notmuch or xapian internals, it seems that it would be handy to change the
search pattern to return threads. My two previous examples would then read like

1) notmuch tag -unread -- 'threads(tag:kill)'
and
2) notmuch tag -kill -- 'threads(tag:kill)' and not 'threads(date:30days..)'

Given that this would be possible in the first place, wouldn't it be a nice
addition to notmuch?

Thank you
-- 
Vlad

Solaris support - missing or incompatible functions

[Vladimir Marek]

[...]

> That must be some setting in (your?) bash:
> 
> $ sh -c 'echo '\''foo\nbar'\'''
> foo
> bar
> 
> $ bash -c 'echo '\''foo\nbar'\'''
> foo\nbar
> 
> $ dash -c 'echo '\''foo\nbar'\'''
> foo
> bar
> 
> ... after a bit digging ...
> 
> $ bash -c 'shopt -s xpg_echo; echo '\''foo\nbar'\'''
> foo
> bar
> 
> $ bash -c 'shopt -u xpg_echo; echo '\''foo\nbar'\'''
> foo\nbar
> 
> you could try adding  shopt -u xpg_echo  into test-lib.sh


Nice, that's the one! By quickly looking around Solaris it seems that
it's the default compiled in.

Thank you
-- 
Vlad

Solaris support - missing or incompatible functions

[Vladimir Marek]

> >> There may be some prerequisites to running tests that you're missing,
> >> and the test setup fails to tell you about. A pastebin of the 'make
> >> test' output might be helpful.
> >
> > Ok, in that case please let me take a look at the tests first.
> >
> 
> For what it's worth, I can confirm no test failures for me either on
> linux (Debian wheezy).

Thank you for looking into that. I'm still not done with fixing the test
suite on Solaris, but form ~70 broken tests I'm at 28. Mostly it's
because Solaris is trying to be posix correct while Linux does not.

There was one surprise though:

A='12\n34'
echo $A

prints two lines on Solaris bug only one line on Linux ('\n' is not
interpreted). That breaks most of json tests as they contain '\n').

Anyway once I'm done I'll have another patch.

-- 
Vlad

Re: Re: Solaris support - missing or incompatible functions

[Vladimir Marek]

  There may be some prerequisites to running tests that you're missing,
  and the test setup fails to tell you about. A pastebin of the 'make
  test' output might be helpful.
 
  Ok, in that case please let me take a look at the tests first.
 
 
 For what it's worth, I can confirm no test failures for me either on
 linux (Debian wheezy).

Thank you for looking into that. I'm still not done with fixing the test
suite on Solaris, but form ~70 broken tests I'm at 28. Mostly it's
because Solaris is trying to be posix correct while Linux does not.

There was one surprise though:

A='12\n34'
echo $A

prints two lines on Solaris bug only one line on Linux ('\n' is not
interpreted). That breaks most of json tests as they contain '\n').

Anyway once I'm done I'll have another patch.

-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Re: Re: Solaris support - missing or incompatible functions

[Vladimir Marek]

[...]

 That must be some setting in (your?) bash:
 
 $ sh -c 'echo '\''foo\nbar'\'''
 foo
 bar
 
 $ bash -c 'echo '\''foo\nbar'\'''
 foo\nbar
 
 $ dash -c 'echo '\''foo\nbar'\'''
 foo
 bar
 
 ... after a bit digging ...
 
 $ bash -c 'shopt -s xpg_echo; echo '\''foo\nbar'\'''
 foo
 bar
 
 $ bash -c 'shopt -u xpg_echo; echo '\''foo\nbar'\'''
 foo\nbar
 
 you could try adding  shopt -u xpg_echo  into test-lib.sh


Nice, that's the one! By quickly looking around Solaris it seems that
it's the default compiled in.

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Joining messages into threads

[Vladimir Marek]

> > I am getting mails from some automated tools (let's imagine bugzilla for
> > example), which concern several specific topics (like bugs).
> > Unfortunately those mails are not 'threaded' (don't have In-Reply-To
> > header). Once the mails are indexed, is there a way make them threaded?
> > I don't think that just editing the mail file and running 'notmuch new'
> > would find out.
> 
> Currently there is not a way to force two messages to be (or not to be)
> in the same thread in notmuch.  Threads are not (explicit) objects in
> the database so this complicates things a bit.

Ok, good to know!

Thank you
-- 
Vlad

Re: Re: Joining messages into threads

[Vladimir Marek]

  I am getting mails from some automated tools (let's imagine bugzilla for
  example), which concern several specific topics (like bugs).
  Unfortunately those mails are not 'threaded' (don't have In-Reply-To
  header). Once the mails are indexed, is there a way make them threaded?
  I don't think that just editing the mail file and running 'notmuch new'
  would find out.
 
 Currently there is not a way to force two messages to be (or not to be)
 in the same thread in notmuch.  Threads are not (explicit) objects in
 the database so this complicates things a bit.

Ok, good to know!

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Joining messages into threads

[Vladimir Marek]

Hi,

I am getting mails from some automated tools (let's imagine bugzilla for
example), which concern several specific topics (like bugs).
Unfortunately those mails are not 'threaded' (don't have In-Reply-To
header). Once the mails are indexed, is there a way make them threaded?
I don't think that just editing the mail file and running 'notmuch new'
would find out.

Similar question would be, can I join two separate message threads into
one by making the 'head' of one thread to be 'child' of the other
thread?

Thank you
-- 
Vlad

Solaris support - missing or incompatible functions

[Vladimir Marek]

> > I didn't pay too much attention to the test output as it's broken on
> > linux too :)
> 
> It shouldn't be. All tests pass for me, and we're pretty strict about
> maintaining that.
> 
> There may be some prerequisites to running tests that you're missing,
> and the test setup fails to tell you about. A pastebin of the 'make
> test' output might be helpful.

Ok, in that case please let me take a look at the tests first.

-- 
Vlad

Joining messages into threads

[Vladimir Marek]

Hi,

I am getting mails from some automated tools (let's imagine bugzilla for
example), which concern several specific topics (like bugs).
Unfortunately those mails are not 'threaded' (don't have In-Reply-To
header). Once the mails are indexed, is there a way make them threaded?
I don't think that just editing the mail file and running 'notmuch new'
would find out.

Similar question would be, can I join two separate message threads into
one by making the 'head' of one thread to be 'child' of the other
thread?

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Solaris support - missing or incompatible functions

[Vladimir Marek]

> > I can't run the notmuch 'make test' on Solaris (too old bash) and on linux 
> > the
> > results are
> >
> > 421/525 tests passed.
> > 25 tests failed.
> > 79 tests skipped.
> >
> > but they are the same without my changes too ...
> 
> The series LGTM, except I didn't pay much attention to the timegm()
> implementation. I hope the time parsing tests pass though. If they do,
> and you think the implementation is good enough, then who am I to
> argue. ;)


I didn't pay too much attention to the test output as it's broken on
linux too :)

Thank you
-- 
Vlad

Re: Solaris support - missing or incompatible functions

[Vladimir Marek]

  I can't run the notmuch 'make test' on Solaris (too old bash) and on linux 
  the
  results are
 
  421/525 tests passed.
  25 tests failed.
  79 tests skipped.
 
  but they are the same without my changes too ...
 
 The series LGTM, except I didn't pay much attention to the timegm()
 implementation. I hope the time parsing tests pass though. If they do,
 and you think the implementation is good enough, then who am I to
 argue. ;)


I didn't pay too much attention to the test output as it's broken on
linux too :)

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Solaris support - missing or incompatible functions

[Vladimir Marek]

  I didn't pay too much attention to the test output as it's broken on
  linux too :)
 
 It shouldn't be. All tests pass for me, and we're pretty strict about
 maintaining that.
 
 There may be some prerequisites to running tests that you're missing,
 and the test setup fails to tell you about. A pastebin of the 'make
 test' output might be helpful.

Ok, in that case please let me take a look at the tests first.

-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Solaris support - missing or incompatible functions

[Vladimir . Marek]

Hi,

This is next series of patches adding Solaris support to notmuch. I am mostly
using what Blake Jones sent some time ago + I fixed all the comments he
received back then. Since it's mostly his work, it's IMO fair to keep
him as an author. But since I made sure the patches work, I'm signing
them. I'm really git newbie, so hopefully this is right thing to do ...

I tested the build on Solaris using Oracle Studio and on Linux using
gcc (to make sure I haven't break anything).

I can't run the notmuch 'make test' on Solaris (too old bash) and on linux the
results are

421/525 tests passed.
25 tests failed.
79 tests skipped.

but they are the same without my changes too ...

Cheers
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/4] asctime: check for standards compliance (Solaris support)

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz

Add checks to configure to see whether _POSIX_PTHREAD_SEMANTICS needs
to be defined to get the right number of arguments in the prototypes for
asctime_r().  Solaris' default implementation conforms to POSIX.1c
Draft 6, rather than the final POSIX.1c spec.  The standards-compliant
version can be used by defining _POSIX_PTHREAD_SEMANTICS.

This change also adds the file compat/check_asctime.c, which
configure uses to perform its check, and modifies compat/compat.h to
define _POSIX_PTHREAD_SEMANTICS if configure detected it was needed.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 compat/check_asctime.c |   11 +++
 compat/compat.h|3 +++
 configure  |   22 --
 3 files changed, 34 insertions(+), 2 deletions(-)
 create mode 100644 compat/check_asctime.c

diff --git a/compat/check_asctime.c b/compat/check_asctime.c
new file mode 100644
index 000..b0e56f0
--- /dev/null
+++ b/compat/check_asctime.c
@@ -0,0 +1,11 @@
+#include time.h
+#include stdio.h
+
+int main()
+{
+struct tm tm;
+
+(void) asctime_r (tm, NULL);
+
+return (0);
+}
diff --git a/compat/compat.h b/compat/compat.h
index c1ee0f9..0c4ac66 100644
--- a/compat/compat.h
+++ b/compat/compat.h
@@ -33,6 +33,9 @@ extern C {
 #if !STD_GETPWUID
 #define _POSIX_PTHREAD_SEMANTICS 1
 #endif
+#if !STD_ASCTIME
+#define _POSIX_PTHREAD_SEMANTICS 1
+#endif
 
 #if !HAVE_GETLINE
 #include stdio.h
diff --git a/configure b/configure
index 8ef7bac..99ec71a 100755
--- a/configure
+++ b/configure
@@ -528,6 +528,17 @@ else
 fi
 rm -f compat/check_getpwuid
 
+printf Checking for standard version of asctime_r... 
+if ${CC} -o compat/check_asctime $srcdir/compat/check_asctime.c  /dev/null 
21
+then
+printf Yes.\n
+std_asctime=1
+else
+printf No (will define _POSIX_PTHREAD_SEMANTICS to get it).\n
+std_asctime=0
+fi
+rm -f compat/check_asctime
+
 printf int main(void){return 0;}\n  minimal.c
 
 printf Checking for rpath support... 
@@ -692,6 +703,11 @@ HAVE_STRCASESTR = ${have_strcasestr}
 # to enable the standards-compliant version -- needed for Solaris)
 STD_GETPWUID = ${std_getpwuid}
 
+# Whether the asctime_r function is standards-compliant
+# (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS
+# to enable the standards-compliant version -- needed for Solaris)
+STD_ASCTIME = ${std_asctime}
+
 # Supported platforms (so far) are: LINUX, MACOSX, SOLARIS, FREEBSD, OPENBSD
 PLATFORM = ${platform}
 
@@ -738,11 +754,13 @@ CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) 
\$(GMIME_CFLAGS)  \\
   \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND)   \\
   \$(VALGRIND_CFLAGS)   \\
   -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\
-  -DSTD_GETPWUID=\$(STD_GETPWUID)
+  -DSTD_GETPWUID=\$(STD_GETPWUID)   \\
+  -DSTD_ASCTIME=\$(STD_ASCTIME)
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
 \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
 -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)   \\
--DSTD_GETPWUID=\$(STD_GETPWUID)
+-DSTD_GETPWUID=\$(STD_GETPWUID) \\
+-DSTD_ASCTIME=\$(STD_ASCTIME)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
-- 
1.7.9.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/4] getpwuid: check for standards compliance (Solaris support)

[Vladimir . Marek]

From: Blake Jones bla...@foo.net

Add checks to configure to see whether _POSIX_PTHREAD_SEMANTICS needs
to be defined to get the right number of arguments in the prototypes for
getpwuid_r().  Solaris' default implementation conforms to POSIX.1c
Draft 6, rather than the final POSIX.1c spec.  The standards-compliant
version can be used by defining _POSIX_PTHREAD_SEMANTICS.

This change also adds the file compat/check_getpwuid.c, which
configure uses to perform its check, and modifies compat/compat.h to
define _POSIX_PTHREAD_SEMANTICS if configure detected it was needed.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 compat/check_getpwuid.c |   11 +++
 compat/compat.h |4 
 configure   |   23 +--
 3 files changed, 36 insertions(+), 2 deletions(-)
 create mode 100644 compat/check_getpwuid.c

diff --git a/compat/check_getpwuid.c b/compat/check_getpwuid.c
new file mode 100644
index 000..c435eb8
--- /dev/null
+++ b/compat/check_getpwuid.c
@@ -0,0 +1,11 @@
+#include stdio.h
+#include pwd.h
+
+int main()
+{
+struct passwd passwd, *ignored;
+
+(void) getpwuid_r (0, passwd, NULL, 0, ignored);
+
+return (0);
+}
diff --git a/compat/compat.h b/compat/compat.h
index b2e2736..c1ee0f9 100644
--- a/compat/compat.h
+++ b/compat/compat.h
@@ -30,6 +30,10 @@
 extern C {
 #endif
 
+#if !STD_GETPWUID
+#define _POSIX_PTHREAD_SEMANTICS 1
+#endif
+
 #if !HAVE_GETLINE
 #include stdio.h
 #include unistd.h
diff --git a/configure b/configure
index 460fcfc..8ef7bac 100755
--- a/configure
+++ b/configure
@@ -517,6 +517,17 @@ else
 fi
 rm -f compat/have_strcasestr
 
+printf Checking for standard version of getpwuid_r... 
+if ${CC} -o compat/check_getpwuid $srcdir/compat/check_getpwuid.c  
/dev/null 21
+then
+printf Yes.\n
+std_getpwuid=1
+else
+printf No (will define _POSIX_PTHREAD_SEMANTICS to get it).\n
+std_getpwuid=0
+fi
+rm -f compat/check_getpwuid
+
 printf int main(void){return 0;}\n  minimal.c
 
 printf Checking for rpath support... 
@@ -676,6 +687,11 @@ HAVE_GETLINE = ${have_getline}
 # build its own version)
 HAVE_STRCASESTR = ${have_strcasestr}
 
+# Whether the getpwuid_r function is standards-compliant
+# (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS
+# to enable the standards-compliant version -- needed for Solaris)
+STD_GETPWUID = ${std_getpwuid}
+
 # Supported platforms (so far) are: LINUX, MACOSX, SOLARIS, FREEBSD, OPENBSD
 PLATFORM = ${platform}
 
@@ -720,10 +736,13 @@ WITH_ZSH = ${WITH_ZSH}
 # Combined flags for compiling and linking against all of the above
 CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)  \\
   \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND)   \\
-  \$(VALGRIND_CFLAGS) -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
+  \$(VALGRIND_CFLAGS)   \\
+  -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR) \\
+  -DSTD_GETPWUID=\$(STD_GETPWUID)
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
 \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) \\
- -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
+-DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)   \\
+-DSTD_GETPWUID=\$(STD_GETPWUID)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
-- 
1.7.9.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 4/4] timegm: add portable implementation (Solaris support)

[Vladimir . Marek]

From: Blake Jones bla...@foo.net

The timegm(3) function is a non-standard extension to libc which is
available in GNU libc and on some BSDs.  Although SunOS had this
function in its libc, Solaris (unfortunately) removed it.  This patch
implements a very simple version of timegm() which is good enough for
parse-time-string.c.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 compat/Makefile.local |4 
 compat/compat.h   |5 +
 compat/have_timegm.c  |7 +++
 compat/timegm.c   |   37 +
 configure |   11 ++
 parse-time-string/parse-time-string.c |1 +
 6 files changed, 65 insertions(+)
 create mode 100644 compat/have_timegm.c
 create mode 100644 compat/timegm.c

diff --git a/compat/Makefile.local b/compat/Makefile.local
index 2c4f65f..b0d5417 100644
--- a/compat/Makefile.local
+++ b/compat/Makefile.local
@@ -17,4 +17,8 @@ ifneq ($(HAVE_STRSEP),1)
 notmuch_compat_srcs += $(dir)/strsep.c
 endif
 
+ifneq ($(HAVE_TIMEGM),1)
+notmuch_compat_srcs += $(dir)/timegm.c
+endif
+
 SRCS := $(SRCS) $(notmuch_compat_srcs)
diff --git a/compat/compat.h b/compat/compat.h
index ae762c3..5a402d5 100644
--- a/compat/compat.h
+++ b/compat/compat.h
@@ -57,6 +57,11 @@ char* strcasestr(const char *haystack, const char *needle);
 char *strsep(char **stringp, const char *delim);
 #endif /* !HAVE_STRSEP */
 
+#if !HAVE_TIMEGM
+#include time.h
+time_t timegm (struct tm *tm);
+#endif /* !HAVE_TIMEGM */
+
 /* Silence gcc warnings about unused results.  These warnings exist
  * for a reason; any use of this needs to be justified. */
 #ifdef __GNUC__
diff --git a/compat/have_timegm.c b/compat/have_timegm.c
new file mode 100644
index 000..b62b793
--- /dev/null
+++ b/compat/have_timegm.c
@@ -0,0 +1,7 @@
+#include time.h
+#include compat.h
+
+int main()
+{
+return (int) timegm((struct tm *)0);
+}
diff --git a/compat/timegm.c b/compat/timegm.c
new file mode 100644
index 000..6d76164
--- /dev/null
+++ b/compat/timegm.c
@@ -0,0 +1,37 @@
+#include time.h
+#include compat.h
+
+static int
+leapyear (int year)
+{
+return ((year % 4) == 0  ((year % 100) != 0 || (year % 400) == 0));
+}
+
+/*
+ * This is a simple implementation of timegm() which does what is needed
+ * by create_output() -- just turns the struct tm into a GMT time_t.
+ * It does not normalize any of the fields of the struct tm, nor does
+ * it set tm_wday or tm_yday.
+ */
+time_t
+timegm (struct tm *tm)
+{
+intmonthlen[2][12] = {
+   { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 },
+   { 31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 },
+};
+intyear, month, days;
+
+days = 365 * (tm-tm_year - 70);
+for (year = 70; year  tm-tm_year; year++) {
+   if (leapyear(1900 + year)) {
+   days++;
+   }
+}
+for (month = 0; month  tm-tm_mon; month++) {
+   days += monthlen[leapyear(1900 + year)][month];
+}
+days += tm-tm_mday - 1;
+
+return days * 24) + tm-tm_hour) * 60 + tm-tm_min) * 60 + tm-tm_sec);
+}
diff --git a/configure b/configure
index 5243cac..ee037d8 100755
--- a/configure
+++ b/configure
@@ -528,6 +528,17 @@ else
 fi
 rm -f compat/have_strsep
 
+printf Checking for timegm... 
+if ${CC} -o compat/have_timegm $srcdir/compat/have_timegm.c  /dev/null 21
+then
+printf Yes.\n
+have_timegm=1
+else
+printf No (will use our own instead).\n
+have_timegm=0
+fi
+rm -f compat/have_timegm
+
 printf Checking for standard version of getpwuid_r... 
 if ${CC} -o compat/check_getpwuid $srcdir/compat/check_getpwuid.c  
/dev/null 21
 then
diff --git a/parse-time-string/parse-time-string.c 
b/parse-time-string/parse-time-string.c
index 584067d..ccad422 100644
--- a/parse-time-string/parse-time-string.c
+++ b/parse-time-string/parse-time-string.c
@@ -32,6 +32,7 @@
 #include sys/time.h
 #include sys/types.h
 
+#include compat.h
 #include parse-time-string.h
 
 /*
-- 
1.7.9.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 3/4] strsep: check for availability (Solaris support)

[Vladimir . Marek]

From: Blake Jones bla...@foo.net

Solaris does not ship a version of the strsep() function.  This change
adds a check to configure to see whether notmuch needs to provide its
own implementation, and if so, it uses the new version in
compat/strsep.c (which was copied from Mutt, and apparently before
that from glibc).

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 compat/Makefile.local |4 +++
 compat/compat.h   |4 +++
 compat/have_strsep.c  |   11 +
 compat/strsep.c   |   65 +
 configure |   17 +
 5 files changed, 101 insertions(+)
 create mode 100644 compat/have_strsep.c
 create mode 100644 compat/strsep.c

diff --git a/compat/Makefile.local b/compat/Makefile.local
index 13f16cd..2c4f65f 100644
--- a/compat/Makefile.local
+++ b/compat/Makefile.local
@@ -13,4 +13,8 @@ ifneq ($(HAVE_STRCASESTR),1)
 notmuch_compat_srcs += $(dir)/strcasestr.c
 endif
 
+ifneq ($(HAVE_STRSEP),1)
+notmuch_compat_srcs += $(dir)/strsep.c
+endif
+
 SRCS := $(SRCS) $(notmuch_compat_srcs)
diff --git a/compat/compat.h b/compat/compat.h
index 0c4ac66..ae762c3 100644
--- a/compat/compat.h
+++ b/compat/compat.h
@@ -53,6 +53,10 @@ getdelim (char **lineptr, size_t *n, int delimiter, FILE 
*fp);
 char* strcasestr(const char *haystack, const char *needle);
 #endif /* !HAVE_STRCASESTR */
 
+#if !HAVE_STRSEP
+char *strsep(char **stringp, const char *delim);
+#endif /* !HAVE_STRSEP */
+
 /* Silence gcc warnings about unused results.  These warnings exist
  * for a reason; any use of this needs to be justified. */
 #ifdef __GNUC__
diff --git a/compat/have_strsep.c b/compat/have_strsep.c
new file mode 100644
index 000..2abab81
--- /dev/null
+++ b/compat/have_strsep.c
@@ -0,0 +1,11 @@
+#define _GNU_SOURCE
+#include string.h
+
+int main()
+{
+char *found;
+char **stringp;
+const char *delim;
+
+found = strsep(stringp, delim);
+}
diff --git a/compat/strsep.c b/compat/strsep.c
new file mode 100644
index 000..78ab9e7
--- /dev/null
+++ b/compat/strsep.c
@@ -0,0 +1,65 @@
+/* Copyright (C) 1992, 93, 96, 97, 98, 99, 2004 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, write to the Free
+   Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
+   02111-1307 USA.  */
+
+#include string.h
+
+/* Taken from glibc 2.6.1 */
+
+char *strsep (char **stringp, const char *delim)
+{
+  char *begin, *end;
+
+  begin = *stringp;
+  if (begin == NULL)
+return NULL;
+
+  /* A frequent case is when the delimiter string contains only one
+ character.  Here we don't need to call the expensive `strpbrk'
+ function and instead work using `strchr'.  */
+  if (delim[0] == '\0' || delim[1] == '\0')
+{
+  char ch = delim[0];
+
+  if (ch == '\0')
+   end = NULL;
+  else
+   {
+ if (*begin == ch)
+   end = begin;
+ else if (*begin == '\0')
+   end = NULL;
+ else
+   end = strchr (begin + 1, ch);
+   }
+}
+  else
+/* Find the end of the token.  */
+end = strpbrk (begin, delim);
+
+  if (end)
+{
+  /* Terminate the token and set *STRINGP past NUL character.  */
+  *end++ = '\0';
+  *stringp = end;
+}
+  else
+/* No more delimiters; this is the last token.  */
+*stringp = NULL;
+
+  return begin;
+}
diff --git a/configure b/configure
index 99ec71a..5243cac 100755
--- a/configure
+++ b/configure
@@ -517,6 +517,17 @@ else
 fi
 rm -f compat/have_strcasestr
 
+printf Checking for strsep... 
+if ${CC} -o compat/have_strsep $srcdir/compat/have_strsep.c  /dev/null 21
+then
+printf Yes.\n
+have_strsep=1
+else
+printf No (will use our own instead).\n
+have_strsep=0
+fi
+rm -f compat/have_strsep
+
 printf Checking for standard version of getpwuid_r... 
 if ${CC} -o compat/check_getpwuid $srcdir/compat/check_getpwuid.c  
/dev/null 21
 then
@@ -698,6 +709,10 @@ HAVE_GETLINE = ${have_getline}
 # build its own version)
 HAVE_STRCASESTR = ${have_strcasestr}
 
+# Whether the strsep function is available (if not, then notmuch will
+# build its own version)
+HAVE_STRSEP = ${have_strsep}
+
 # Whether the getpwuid_r function is standards-compliant
 # (if not, then notmuch will #define _POSIX_PTHREAD_SEMANTICS
 # to enable the standards-compliant version

[PATCH 01/24] lib/message.cc: stale pointer bug

[Vladimir Marek]

> >  int prefix_len = strlen (prefix);
> > -const char *term = NULL;
> > +std::string term;
> >  char *value;
> >  
> >  i.skip_to (prefix);
> >  
> > -if (i != end)
> > -   term = (*i).c_str ();
> > +if (i == end)
> > +   return NULL;
> >  
> > -if (!term || strncmp (term, prefix, prefix_len))
> > +term = *i;
> 
> ... hmm, a raii(?) solution above would be std::string term = *i;

I'm not sure what's raii (I'm not very good at c++ ...), but I guess you
mean to use 'std::string term = *i;' to avoid copy constructor. That
surely is a good idea. Let me rework the patch!

-- 
Vlad

[PATCH 01/24] lib/message.cc: stale pointer bug

[Vladimir Marek]

Uh, oh, it's patch 01/01 actually. There's 23 other patches waiting but
will be submitted separately.

> Xapian::TermIterator::operator* returns std::string which is destroyed
> as soon as (*i).c_str() finishes. The remembered pointer 'term' then
> references invalid memory.

I reworded the comment and I changed the fix slightly.

Thank you
-- 
Vlad

[PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

> Would you mind updating your commit message to reflect the best guesses
> from this discussion? 

Hi,

Sorry it took me a while. I just sent new version of the patch for
consideration.

-- 
Vlad

Re: [PATCH 01/24] lib/message.cc: stale pointer bug

[Vladimir Marek]

   int prefix_len = strlen (prefix);
  -const char *term = NULL;
  +std::string term;
   char *value;
   
   i.skip_to (prefix);
   
  -if (i != end)
  -   term = (*i).c_str ();
  +if (i == end)
  +   return NULL;
   
  -if (!term || strncmp (term, prefix, prefix_len))
  +term = *i;
 
 ... hmm, a raii(?) solution above would be std::string term = *i;

I'm not sure what's raii (I'm not very good at c++ ...), but I guess you
mean to use 'std::string term = *i;' to avoid copy constructor. That
surely is a good idea. Let me rework the patch!

-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] lib/message.cc: stale pointer bug (v3)

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz

Xapian::TermIterator::operator* returns std::string which is destroyed
as soon as (*i).c_str() finishes. The remembered pointer 'term' then
references invalid memory.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 lib/message.cc |   10 +-
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 8720c1b..c4261e6 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -266,18 +266,18 @@ _notmuch_message_get_term (notmuch_message_t *message,
   const char *prefix)
 {
 int prefix_len = strlen (prefix);
-const char *term = NULL;
 char *value;
 
 i.skip_to (prefix);
 
-if (i != end)
-   term = (*i).c_str ();
+if (i == end)
+   return NULL;
 
-if (!term || strncmp (term, prefix, prefix_len))
+std::string term = *i;
+if (strncmp (term.c_str(), prefix, prefix_len))
return NULL;
 
-value = talloc_strdup (message, term + prefix_len);
+value = talloc_strdup (message, term.c_str() + prefix_len);
 
 #if DEBUG_DATABASE_SANITY
 i++;
-- 
1.7.9.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

> You have to somehow instantiate a temporary object, using a function
> call for example.
> 
> #include 
> #include 
> struct example {
>   char data[10];
>   char *c_str() { return data; }
>  example() { strcpy(data, "test"); }
>   ~example() { strcpy(data, "destroyed"); }
> };
> example foo()
> {
>   example res;
>   return res;
> }
> main()
> {
>   printf("%s\n", foo().c_str());
>   char *x = foo().c_str();
>   printf("%s\n", x);
> }

Right. After more tests I think I'm getting hang of it. In my examples I
had function similar to

char *foo() {
  example res;
  return res.c_str();
}

In this case the reference returned is to _char pointer_ and not to
example struct. So the struct is destroyed before returning from foo to
caller. Compiler does not understand that the returned char pointer does
not have any meaning without the example struct.

This is where I was doing the mistake, I believe.

Thank you
-- 
Vlad

Re: [PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

 You have to somehow instantiate a temporary object, using a function
 call for example.
 
 #include string.h
 #include stdio.h
 struct example {
   char data[10];
   char *c_str() { return data; }
  example() { strcpy(data, test); }
   ~example() { strcpy(data, destroyed); }
 };
 example foo()
 {
   example res;
   return res;
 }
 main()
 {
   printf(%s\n, foo().c_str());
   char *x = foo().c_str();
   printf(%s\n, x);
 }

Right. After more tests I think I'm getting hang of it. In my examples I
had function similar to

char *foo() {
  example res;
  return res.c_str();
}

In this case the reference returned is to _char pointer_ and not to
example struct. So the struct is destroyed before returning from foo to
caller. Compiler does not understand that the returned char pointer does
not have any meaning without the example struct.

This is where I was doing the mistake, I believe.

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 01/24] lib/message.cc: stale pointer bug

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz

Xapian::TermIterator::operator* returns std::string which is destroyed
as soon as (*i).c_str() finishes. The remembered pointer 'term' then
references invalid memory.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 lib/message.cc |   11 ++-
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 8720c1b..a890550 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -266,18 +266,19 @@ _notmuch_message_get_term (notmuch_message_t *message,
   const char *prefix)
 {
 int prefix_len = strlen (prefix);
-const char *term = NULL;
+std::string term;
 char *value;
 
 i.skip_to (prefix);
 
-if (i != end)
-   term = (*i).c_str ();
+if (i == end)
+   return NULL;
 
-if (!term || strncmp (term, prefix, prefix_len))
+term = *i;
+if (strncmp (term.c_str(), prefix, prefix_len))
return NULL;
 
-value = talloc_strdup (message, term + prefix_len);
+value = talloc_strdup (message, term.c_str() + prefix_len);
 
 #if DEBUG_DATABASE_SANITY
 i++;
-- 
1.7.9.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

 Would you mind updating your commit message to reflect the best guesses
 from this discussion? 

Hi,

Sorry it took me a while. I just sent new version of the patch for
consideration.

-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 01/24] lib/message.cc: stale pointer bug

[Vladimir Marek]

Uh, oh, it's patch 01/01 actually. There's 23 other patches waiting but
will be submitted separately.

 Xapian::TermIterator::operator* returns std::string which is destroyed
 as soon as (*i).c_str() finishes. The remembered pointer 'term' then
 references invalid memory.

I reworded the comment and I changed the fix slightly.

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

Hi,

Thank you, I found it eventually too. But I wrote little test program
(attached) which confused me. I haven't had much time to take a look
into it since weekend.

The idea is to have temporary object where I can detect whether
destructor was called.

I thought that

printf ("%s\n", s.c_str());
will print "test"

and

x=s.c_str();
printf ("%s\n", x);

will print "destroyed"

On my machine both prints "destroyed".


I still believe my fix is correct, but I'm not at the position to be
able to defend it at the moment :)

Thank you
-- 
Vlad

> The january 2012 working draft: 
> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3337.pdf
> 
> 12.2 Temporary objects [class.temporary]
> 
> 1 Temporaries of class type are created in various contexts: binding a
> reference to a prvalue (8.5.3), returning a prvalue (6.6.3) [?]
> 
> 3 When an implementation introduces a temporary object of a class that
> has a non-trivial constructor (12.1, 12.8), it shall ensure that a
> constructor is called for the temporary object. Similarly, the
> destructor shall be called for a temporary with a non-trivial destructor
> (12.4). Temporary objects are destroyed as the last step in evaluating
> the full-expression (1.9) that (lexically) contains the point where they
> were created.
-- next part --
#include 
#include 
#include 

using namespace std;

class array {
vector impl;

public:

array(int size):impl(size) { }
array(array ):impl(in.impl) { }

char*
operator[](size_t i) {
return [i];
};

~array() {
strcpy([0], "destroyed");
}
};

class str {
array tmp;

public:

str(const char *input):tmp(100) {
strcpy(tmp[0], input);
};

const char* c_str() {
return (array(tmp))[0];
};
};


int
main (int argc, char **argv)
{
str s("test");
const char *x;
printf ("%s\n", s.c_str());
x=s.c_str();
printf ("%s\n", x);
return 0;
}

Re: [PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

Hi,

Thank you, I found it eventually too. But I wrote little test program
(attached) which confused me. I haven't had much time to take a look
into it since weekend.

The idea is to have temporary object where I can detect whether
destructor was called.

I thought that

printf (%s\n, s.c_str());
will print test

and

x=s.c_str();
printf (%s\n, x);

will print destroyed

On my machine both prints destroyed.


I still believe my fix is correct, but I'm not at the position to be
able to defend it at the moment :)

Thank you
-- 
Vlad

 The january 2012 working draft: 
 http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2012/n3337.pdf
 
 12.2 Temporary objects [class.temporary]
 
 1 Temporaries of class type are created in various contexts: binding a
 reference to a prvalue (8.5.3), returning a prvalue (6.6.3) […]
 
 3 When an implementation introduces a temporary object of a class that
 has a non-trivial constructor (12.1, 12.8), it shall ensure that a
 constructor is called for the temporary object. Similarly, the
 destructor shall be called for a temporary with a non-trivial destructor
 (12.4). Temporary objects are destroyed as the last step in evaluating
 the full-expression (1.9) that (lexically) contains the point where they
 were created.
#include stdio.h
#include vector
#include string.h

using namespace std;

class array {
vectorchar impl;

public:

array(int size):impl(size) { }
array(array in):impl(in.impl) { }

char*
operator[](size_t i) {
return impl[i];
};

~array() {
strcpy(impl[0], destroyed);
}
};

class str {
array tmp;

public:

str(const char *input):tmp(100) {
strcpy(tmp[0], input);
};

const char* c_str() {
return (array(tmp))[0];
};
};


int
main (int argc, char **argv)
{
str s(test);
const char *x;
printf (%s\n, s.c_str());
x=s.c_str();
printf (%s\n, x);
return 0;
}
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

> It's okay to use the result of .c_str() as long as the string object
> stays in scope, and none of the non-const member functions are
> called. Here, I think the problem is that TermIterator's overloaded
> operator*() returns a string object within the if block's scope, and it
> goes immediately out of scope. You could check this by adding

Right, I overlooked that TermIterator creates temporary string (if I
understand you correctly).

> 
>   string s = *i;
>
> in function scope, and replacing (*i) with s in the if block. This might
> also be more obvious than the presented patch, but I think the patch is
> fine too.

I would prefer my change as it avoids creating another std::string. At
least I think.

Thank you
-- 
Vlad

[PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

> > const char*
> > string::c_str(void) {
> > char buf[100];
> >
> > strcpy (buf, this->internal_representation);
> > return buf;
> > }
> 
> Isn't that undefined behavior, returning a pointer to a (non-static)
> local variable?

Right, I was trying to bring up an example and this one is not very
good. Maybe something like that:


const char*
c_str() {
vector X(my_string);
return [0];
};

char buf[];
strcpy(buf, c_str());

X destructor is called after strcpy is done.

char buf[];
char *tmp=c_str();
strcpy(buf, tmp);

X destructor is called before strcpy.

At least this is how I understand it.

-- 
Vlad

[PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

> > From: Vladimir Marek 
> >
> > This is causing problems when compiled by Oracle Studio. Memory pointed
> > by (const char*)term was already changed once talloc_strdup was called.
> 
> If that changes, I'd like to understand why (and stated in the commit
> message). If that is clear to everyone else I will withdraw the question --
> I am not too familiar with these iterators magic... :D

Well, a) standards says that

A temporary bound to a reference parameter in a function call (5.2.2)
persists until the completion of the full expression containing the call

(you can find the message all over the net, but I can't find actual link
to the standard :-/)


b) Imagine the function c_str() looks like that:

const char*
string::c_str(void) {
char buf[100];

strcpy (buf, this->internal_representation);
return buf;
}

you can't do:
char *my_tmp = string.c_str();
printf(my_tmp);

you can do:
printf(string.c_str());

Hopefully I made the example right ...

Cheers
-- 
Vlad

Re: [PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

  From: Vladimir Marek vlma...@volny.cz
 
  This is causing problems when compiled by Oracle Studio. Memory pointed
  by (const char*)term was already changed once talloc_strdup was called.
 
 If that changes, I'd like to understand why (and stated in the commit
 message). If that is clear to everyone else I will withdraw the question --
 I am not too familiar with these iterators magic... :D

Well, a) standards says that

A temporary bound to a reference parameter in a function call (5.2.2)
persists until the completion of the full expression containing the call

(you can find the message all over the net, but I can't find actual link
to the standard :-/)


b) Imagine the function c_str() looks like that:

const char*
string::c_str(void) {
char buf[100];

strcpy (buf, this-internal_representation);
return buf;
}

you can't do:
char *my_tmp = string.c_str();
printf(my_tmp);

you can do:
printf(string.c_str());

Hopefully I made the example right ...

Cheers
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

  const char*
  string::c_str(void) {
  char buf[100];
 
  strcpy (buf, this-internal_representation);
  return buf;
  }
 
 Isn't that undefined behavior, returning a pointer to a (non-static)
 local variable?

Right, I was trying to bring up an example and this one is not very
good. Maybe something like that:


const char*
c_str() {
vectorchar X(my_string);
return X[0];
};

char buf[];
strcpy(buf, c_str());

X destructor is called after strcpy is done.

char buf[];
char *tmp=c_str();
strcpy(buf, tmp);

X destructor is called before strcpy.

At least this is how I understand it.

-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] don't store temporary value returned from c_str()

[Vladimir Marek]

 It's okay to use the result of .c_str() as long as the string object
 stays in scope, and none of the non-const member functions are
 called. Here, I think the problem is that TermIterator's overloaded
 operator*() returns a string object within the if block's scope, and it
 goes immediately out of scope. You could check this by adding

Right, I overlooked that TermIterator creates temporary string (if I
understand you correctly).

 
   string s = *i;

 in function scope, and replacing (*i) with s in the if block. This might
 also be more obvious than the presented patch, but I think the patch is
 fine too.

I would prefer my change as it avoids creating another std::string. At
least I think.

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: crash during saving

[Vladimir Marek]

  Right, so the problem really seems to be in throwing/catching
  exception.  Function _notmuch_message_remove_term is supposed to
  catch the exception and ignore it. Which does not happen in my case.
 
 Yep, that was exactly what I was seeing.

At the end, the issue happens only when I try to link all the libraries
statically. I believe especially the problem is with linking STL library
statically. So I stopped doing that :)

-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] don't store temporary value returned from c_str()

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz

This is causing problems when compiled by Oracle Studio. Memory pointed
by (const char*)term was already changed once talloc_strdup was called.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 lib/message.cc |9 -
 1 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/lib/message.cc b/lib/message.cc
index 8720c1b..8d329d1 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -266,18 +266,17 @@ _notmuch_message_get_term (notmuch_message_t *message,
   const char *prefix)
 {
 int prefix_len = strlen (prefix);
-const char *term = NULL;
 char *value;
 
 i.skip_to (prefix);
 
-if (i != end)
-   term = (*i).c_str ();
+if (i == end)
+   return NULL;
 
-if (!term || strncmp (term, prefix, prefix_len))
+if (strncmp ((*i).c_str(), prefix, prefix_len))
return NULL;
 
-value = talloc_strdup (message, term + prefix_len);
+value = talloc_strdup (message, (*i).c_str() + prefix_len);
 
 #if DEBUG_DATABASE_SANITY
 i++;
-- 
1.7.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

crash during saving

[Vladimir Marek]

> > I just indexed my mail archive by notmuch and I'm starting to play
> > with mutt-kz. The biggest stopper right now is that mutt cores when
> > set already read mail to new (toggle-new in mutt). Once I try to leave
> > the virtual folder (be it to another folder or because of quitting
> > mutt) it crashes.
> > 
> > I haven't had the time yet to investigate deeper, so I'll just post
> > whatever info I have and hope that it will be something obvious for
> > you :)
> 
> I saw something like that when I was first using mutt-kz as well...
> it ended up that I had compiled libxapian with one version of gcc (3.4.3
> maybe?) and notmuch with 4.5.2, and the C++ runtime libraries were
> incompatible between the two versions.  In my case, any time it tried to
> throw an exception (e.g. when I removed a tag that was not present) it
> would die with a similar stack trace.  (And, of course, that was on
> Solaris :) )

Right, so the problem really seems to be in throwing/catching exception.
Function "_notmuch_message_remove_term" is supposed to catch the
exception and ignore it. Which does not happen in my case.

On a side note, I wonder, is catching exception faster than going
through list of tags to see if given tag exists? Might be interesting to
compare.

Cheers
-- 
Vlad

Re: crash during saving

[Vladimir Marek]

  I just indexed my mail archive by notmuch and I'm starting to play
  with mutt-kz. The biggest stopper right now is that mutt cores when
  set already read mail to new (toggle-new in mutt). Once I try to leave
  the virtual folder (be it to another folder or because of quitting
  mutt) it crashes.
  
  I haven't had the time yet to investigate deeper, so I'll just post
  whatever info I have and hope that it will be something obvious for
  you :)
 
 I saw something like that when I was first using mutt-kz as well...
 it ended up that I had compiled libxapian with one version of gcc (3.4.3
 maybe?) and notmuch with 4.5.2, and the C++ runtime libraries were
 incompatible between the two versions.  In my case, any time it tried to
 throw an exception (e.g. when I removed a tag that was not present) it
 would die with a similar stack trace.  (And, of course, that was on
 Solaris :) )

Right, so the problem really seems to be in throwing/catching exception.
Function _notmuch_message_remove_term is supposed to catch the
exception and ignore it. Which does not happen in my case.

On a side note, I wonder, is catching exception faster than going
through list of tags to see if given tag exists? Might be interesting to
compare.

Cheers
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

crash during saving

[Vladimir Marek]

> > I just indexed my mail archive by notmuch and I'm starting to play
> > with mutt-kz. The biggest stopper right now is that mutt cores when
> > set already read mail to new (toggle-new in mutt). Once I try to leave
> > the virtual folder (be it to another folder or because of quitting
> > mutt) it crashes.
> > 
> > I haven't had the time yet to investigate deeper, so I'll just post
> > whatever info I have and hope that it will be something obvious for
> > you :)
> 
> I saw something like that when I was first using mutt-kz as well...
> it ended up that I had compiled libxapian with one version of gcc (3.4.3
> maybe?) and notmuch with 4.5.2, and the C++ runtime libraries were
> incompatible between the two versions.  In my case, any time it tried to
> throw an exception (e.g. when I removed a tag that was not present) it
> would die with a similar stack trace.  (And, of course, that was on
> Solaris :) )

Hmm, I tried to be extra cautions to compile everything with stlport4,
but I'll double check that!

Thank you
-- 
Vlad

crash during saving

[Vladimir Marek]

Hi,

I just indexed my mail archive by notmuch and I'm starting to play with
mutt-kz. The biggest stopper right now is that mutt cores when set
already read mail to new (toggle-new in mutt). Once I try to leave the
virtual folder (be it to another folder or because of quitting mutt) it
crashes.

I haven't had the time yet to investigate deeper, so I'll just post
whatever info I have and hope that it will be something obvious for you
:)

It seems that mutt is calling "notmuch_message_maildir_flags_to_tags"
which in turn tries to remove flag "draft" which probably is not present
and thus raises an exception. Does
"notmuch_message_maildir_flags_to_tags" expect the mail to be in certain
state (have certain flags set) or it should not crash no matter which
flags were and are now set?

Now to be fair I have to say that this is on Solaris with handful of
patches applied to make notmuch compilable there by Oracle Studio
compilers. (I'm working on finalizing the patches to have them suitable
for review, which I want to post soon).

Any tips welcome :)

Thank you
-- 
Vlad

The stack looks like:

(dbx) where
  [1] _lwp_kill(0x1, 0x6, 0xfeffd028, 0xfe5f2168), at 0xfe5f9265
  [2] thr_kill(0x1, 0x6, 0xfeffd028, 0xfe5a0fe1), at 0xfe5f218a
  [3] raise(0x6, 0x0, 0xfeffd084, 0xfe57873d), at 0xfe5a0fed
  [4] abort(0x0), at 0xfe57875d
  [5] __Cimpl::default_terminate(0x0), at 0x8705898
  [6] std::terminate(0x0, 0x89290a8, 0xfeffd0ac, 0x89290a8, 0x0, 0x89290a8), at 
0x87055cb
  [7] __Cimpl::ex_terminate(0x884afac, 0x0), at 0x8705736
   hidden frames, use 'where -h' to see them all 
=>[10] Xapian::Document::Internal::remove_term(this = 0x8b14cb8, tname = 
CLASS), line 404 in "omdocument.cc"
  [11] Xapian::Document::remove_term(this = 0x98e3684, tname = CLASS), line 156 
in "omdocument.cc"
  [12] _notmuch_message_remove_term(message = 0x98e3650, prefix_name = 
0x884149b "tag", value = 0x88e9f09 "draft"), line 960 in "message.cc"
  [13] notmuch_message_remove_tag(message = 0x98e3650, tag = 0x88e9f09 
"draft"), line 1019 in "message.cc"
  [14] notmuch_message_maildir_flags_to_tags(message = 0x98e3650), line 1126 in 
"message.cc"
  [15] rename_filename(db = 0x98e26c0, old = 0xfeffd48d 
"/tank/vm/vmarek/mail/archive/064/cur/1366056222.6981_56079.vi64-x3-2e-prg06:2,S",
 new = 0xfeffd38e 
"/tank/vm/vmarek/mail/archive/064/cur/1366056222.6981_56079.vi64-x3-2e-prg06:2,",
 h = 0x8b1b880), line 1192 in "mutt_notmuch.c"
  [16] nm_sync(ctx = 0x8ac77f0, index_hint = 0xfeffdb68), line 1277 in 
"mutt_notmuch.c"
  [17] sync_mailbox(ctx = 0x8ac77f0, index_hint = 0xfeffdb68), line 822 in 
"mx.c"
  [18] mx_close_mailbox(ctx = 0x8ac77f0, index_hint = 0xfeffdb68), line 1020 in 
"mx.c"
  [19] mutt_index_menu(), line 954 in "curs_main.c"
  [20] main(argc = 1, argv = 0xfeffea38), line 1056 in "main.c"




(dbx) print -r tname
tname = {
basic_string::_String_base::_M_start  = 0x8b2b900 
"Kdraft"
basic_string::_String_base::_M_finish = 0x8b2b906 ""
basic_string::_String_base::_M_end_of_storage = {
basic_string::_String_base::_STLP_alloc_proxy::_M_data = 0x8b2b907 
"T^O"
} 
}



(dbx) list
  404   throw Xapian::InvalidArgumentError("Term `" + tname +
  405   "' is not present in document, in "
  406   "Xapian::Document::Internal::remove_term()");
  407   }
  408   positions_modified = !i->second.positions.empty();
  409   terms.erase(i);
  410   }
  411   
  412   void
  413   Xapian::Document::Internal::clear_terms()




(dbx) frame 15
Current function is rename_filename
 1192   notmuch_message_maildir_flags_to_tags(msg);
(dbx) list
 1192   notmuch_message_maildir_flags_to_tags(msg);
 1193   update_tags(msg, nm_header_get_tags(h));
 1194   }
 1195   
 1196   rc = 0;
 1197   done:
 1198   if (msg)
 1199   notmuch_message_destroy(msg);
 1200   notmuch_database_end_atomic(db);
 1201   return rc;


(dbx) frame 14
Current function is notmuch_message_maildir_flags_to_tags
 1126   status = notmuch_message_remove_tag (message, 
flag2tag[i].tag);

(dbx) print filename
filename = 0x8b24e10 
"/tank/vm/vmarek/mail/archive/064/cur/1366056222.6981_56079.vi64-x3-2e-prg06:2,"

(dbx) print *flags  
*flags = '\0'

(dbx) print *filenames
*filenames = {
iterator = (nil)
}

crash during saving

[Vladimir Marek]

Hi,

I just indexed my mail archive by notmuch and I'm starting to play with
mutt-kz. The biggest stopper right now is that mutt cores when set
already read mail to new (toggle-new in mutt). Once I try to leave the
virtual folder (be it to another folder or because of quitting mutt) it
crashes.

I haven't had the time yet to investigate deeper, so I'll just post
whatever info I have and hope that it will be something obvious for you
:)

It seems that mutt is calling notmuch_message_maildir_flags_to_tags
which in turn tries to remove flag draft which probably is not present
and thus raises an exception. Does
notmuch_message_maildir_flags_to_tags expect the mail to be in certain
state (have certain flags set) or it should not crash no matter which
flags were and are now set?

Now to be fair I have to say that this is on Solaris with handful of
patches applied to make notmuch compilable there by Oracle Studio
compilers. (I'm working on finalizing the patches to have them suitable
for review, which I want to post soon).

Any tips welcome :)

Thank you
-- 
Vlad

The stack looks like:

(dbx) where
  [1] _lwp_kill(0x1, 0x6, 0xfeffd028, 0xfe5f2168), at 0xfe5f9265
  [2] thr_kill(0x1, 0x6, 0xfeffd028, 0xfe5a0fe1), at 0xfe5f218a
  [3] raise(0x6, 0x0, 0xfeffd084, 0xfe57873d), at 0xfe5a0fed
  [4] abort(0x0), at 0xfe57875d
  [5] __Cimpl::default_terminate(0x0), at 0x8705898
  [6] std::terminate(0x0, 0x89290a8, 0xfeffd0ac, 0x89290a8, 0x0, 0x89290a8), at 
0x87055cb
  [7] __Cimpl::ex_terminate(0x884afac, 0x0), at 0x8705736
   hidden frames, use 'where -h' to see them all 
=[10] Xapian::Document::Internal::remove_term(this = 0x8b14cb8, tname = 
CLASS), line 404 in omdocument.cc
  [11] Xapian::Document::remove_term(this = 0x98e3684, tname = CLASS), line 156 
in omdocument.cc
  [12] _notmuch_message_remove_term(message = 0x98e3650, prefix_name = 
0x884149b tag, value = 0x88e9f09 draft), line 960 in message.cc
  [13] notmuch_message_remove_tag(message = 0x98e3650, tag = 0x88e9f09 
draft), line 1019 in message.cc
  [14] notmuch_message_maildir_flags_to_tags(message = 0x98e3650), line 1126 in 
message.cc
  [15] rename_filename(db = 0x98e26c0, old = 0xfeffd48d 
/tank/vm/vmarek/mail/archive/064/cur/1366056222.6981_56079.vi64-x3-2e-prg06:2,S,
 new = 0xfeffd38e 
/tank/vm/vmarek/mail/archive/064/cur/1366056222.6981_56079.vi64-x3-2e-prg06:2,,
 h = 0x8b1b880), line 1192 in mutt_notmuch.c
  [16] nm_sync(ctx = 0x8ac77f0, index_hint = 0xfeffdb68), line 1277 in 
mutt_notmuch.c
  [17] sync_mailbox(ctx = 0x8ac77f0, index_hint = 0xfeffdb68), line 822 in 
mx.c
  [18] mx_close_mailbox(ctx = 0x8ac77f0, index_hint = 0xfeffdb68), line 1020 in 
mx.c
  [19] mutt_index_menu(), line 954 in curs_main.c
  [20] main(argc = 1, argv = 0xfeffea38), line 1056 in main.c




(dbx) print -r tname
tname = {
basic_stringchar,std::char_traitschar,std::allocatorchar 
::_String_basechar,std::allocatorchar ::_M_start  = 0x8b2b900 
Kdraft
basic_stringchar,std::char_traitschar,std::allocatorchar 
::_String_basechar,std::allocatorchar ::_M_finish = 0x8b2b906 
basic_stringchar,std::char_traitschar,std::allocatorchar 
::_String_basechar,std::allocatorchar ::_M_end_of_storage = {
basic_stringchar,std::char_traitschar,std::allocatorchar 
::_String_basechar,std::allocatorchar 
::_STLP_alloc_proxychar*,char,std::allocatorchar ::_M_data = 0x8b2b907 
T^O
} 
}



(dbx) list
  404   throw Xapian::InvalidArgumentError(Term ` + tname +
  405   ' is not present in document, in 
  406   Xapian::Document::Internal::remove_term());
  407   }
  408   positions_modified = !i-second.positions.empty();
  409   terms.erase(i);
  410   }
  411   
  412   void
  413   Xapian::Document::Internal::clear_terms()




(dbx) frame 15
Current function is rename_filename
 1192   notmuch_message_maildir_flags_to_tags(msg);
(dbx) list
 1192   notmuch_message_maildir_flags_to_tags(msg);
 1193   update_tags(msg, nm_header_get_tags(h));
 1194   }
 1195   
 1196   rc = 0;
 1197   done:
 1198   if (msg)
 1199   notmuch_message_destroy(msg);
 1200   notmuch_database_end_atomic(db);
 1201   return rc;


(dbx) frame 14
Current function is notmuch_message_maildir_flags_to_tags
 1126   status = notmuch_message_remove_tag (message, 
flag2tag[i].tag);

(dbx) print filename
filename = 0x8b24e10 
/tank/vm/vmarek/mail/archive/064/cur/1366056222.6981_56079.vi64-x3-2e-prg06:2,

(dbx) print *flags  
*flags = '\0'

(dbx) print *filenames
*filenames = {
iterator = (nil)
}


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: crash during saving

[Vladimir Marek]

  I just indexed my mail archive by notmuch and I'm starting to play
  with mutt-kz. The biggest stopper right now is that mutt cores when
  set already read mail to new (toggle-new in mutt). Once I try to leave
  the virtual folder (be it to another folder or because of quitting
  mutt) it crashes.
  
  I haven't had the time yet to investigate deeper, so I'll just post
  whatever info I have and hope that it will be something obvious for
  you :)
 
 I saw something like that when I was first using mutt-kz as well...
 it ended up that I had compiled libxapian with one version of gcc (3.4.3
 maybe?) and notmuch with 4.5.2, and the C++ runtime libraries were
 incompatible between the two versions.  In my case, any time it tried to
 throw an exception (e.g. when I removed a tag that was not present) it
 would die with a similar stack trace.  (And, of course, that was on
 Solaris :) )

Hmm, I tried to be extra cautions to compile everything with stlport4,
but I'll double check that!

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[ANNOUNCE] notmuchfs - A virtual maildir file system for notmuch queries

[Vladimir Marek]

> > > Why?
> 
> I know - proceed to the next paragraph for my explanation. Summary -
> mutt-kz wasn't done when I did this.

Uh, oh, overlooked that. Sorry for the noise
-- 
Vlad

[ANNOUNCE] notmuchfs - A virtual maildir file system for notmuch queries

[Vladimir Marek]

Hi,

[...]

> Why?
> 
> I wanted to use notmuch with mutt. I didn't feel that ploughing though mutt's
> codebase to add tight integration was the quickest way to a result, so
> notmuchfs was born.

Someone else already created mutt's fork using notmuch directly.

https://github.com/karelzak/mutt-kz/wiki

Just saying ...

-- 
Vlad

Re: [ANNOUNCE] notmuchfs - A virtual maildir file system for notmuch queries

[Vladimir Marek]

Hi,

[...]

 Why?
 
 I wanted to use notmuch with mutt. I didn't feel that ploughing though mutt's
 codebase to add tight integration was the quickest way to a result, so
 notmuchfs was born.

Someone else already created mutt's fork using notmuch directly.

https://github.com/karelzak/mutt-kz/wiki

Just saying ...

-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [ANNOUNCE] notmuchfs - A virtual maildir file system for notmuch queries

[Vladimir Marek]

   Why?
 
 I know - proceed to the next paragraph for my explanation. Summary -
 mutt-kz wasn't done when I did this.

Uh, oh, overlooked that. Sorry for the noise
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Alternative (raw) message store (i.e. instead of maildir)

[Vladimir Marek]

Hi,

I have objections against maildir too, but I tried to tackle it from
different perspective. Store the maildir in zip file and use fuse-zip to
manage it. It works sort of but it has two major disadvantages:

 - fuse zip stores all changes in memory until unmounted
 - fuse zip (and libzip for that matter) creates new temporary file when
   updating archive, which takes considerable time when the archive is
   very big.

Looking at the zip file format, it could be made so that all
modifications would result only in appending new data to it (deleting is
writing new directory index not containing the deleted file).

I even made proof of concept libzip modification.

Of course this solution would have some disadvantages too, but for me
the advantages would win. At the moment I'm not sure if I want to
continue working on that. Maybe if there would be more interested guys
...

Cheers
-- 
Vlad

Re: Alternative (raw) message store (i.e. instead of maildir)

[Vladimir Marek]

Hi,

I have objections against maildir too, but I tried to tackle it from
different perspective. Store the maildir in zip file and use fuse-zip to
manage it. It works sort of but it has two major disadvantages:

 - fuse zip stores all changes in memory until unmounted
 - fuse zip (and libzip for that matter) creates new temporary file when
   updating archive, which takes considerable time when the archive is
   very big.

Looking at the zip file format, it could be made so that all
modifications would result only in appending new data to it (deleting is
writing new directory index not containing the deleted file).

I even made proof of concept libzip modification.

Of course this solution would have some disadvantages too, but for me
the advantages would win. At the moment I'm not sure if I want to
continue working on that. Maybe if there would be more interested guys
...

Cheers
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 1/4] Make configure use /bin/bash instead of /bin/sh

[Vladimir Marek]

Hi,

> >> > Posix /bin/sh is not capable of running this configure and fails.
> >> 
> >> What fails? What would it take to make this work on posix sh instead?
> >> 
> >> The tests do require bash, but generally I think it would be preferable to
> >> not depend on bash to build.
> >
> > Well I gave it a quick stab. This is not posix:
> >
> > BLAH=$( ... )
> > BLAH=$(( ... ))
> > ${option%=*}
> > ${option%%=*}
> > ${option#=*}
> > ${option##=*}
> 
> According to 
> 
> http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html
> 
> all of those are part of Shell Command Language...

Hmm, you are right. The solaris /bin/sh is older revision of posix.



> Does the configure script work if you replace /bin/sh with /bin/ksh
> in your Solaris box

yes, it does work if executed by /bin/bash or /bin/ksh


> If yes, something like the following could be added to the beginning
> of 'configure'
> 
> option=option=value
> if test ! x"${option$*=}" = x"value" 2>/dev/null; then
>   if test x"${PREVENT_LOOPING-}" = x; then
>   PREVENT_LOOPING=true; export PREVENT_LOOPING
> test ! -x /bin/ksh || exec /bin/ksh "$0" "$@"
> test ! -x /bin/bash || exec /bin/bash "$0" "$@"
> fi
> echo "Cannot find compatible shell for '$0'" >&2
> exit 1
> fi

Unfortunately, no. The /bin/sh says "bad substitution" and does not run
the script at all. I also tried

eval 'echo ${A%%1}'; echo ok

but that does not run the 'echo ok' and fails also.


I can see three possible solutions

1) use bash or ksh in the shebang line
2) rewrite the script as I gave the overview
3) declare that solaris 10 /bin/sh is not compatible with configure
script


Frankly even 3) is viable option, one just have to remember to run
'bash configure'. If everything else would work, I would be happy :)

Thank you
-- 
Vlad

Re: [PATCH 1/4] Make configure use /bin/bash instead of /bin/sh

[Vladimir Marek]

Hi,

   Posix /bin/sh is not capable of running this configure and fails.
  
  What fails? What would it take to make this work on posix sh instead?
  
  The tests do require bash, but generally I think it would be preferable to
  not depend on bash to build.
 
  Well I gave it a quick stab. This is not posix:
 
  BLAH=$( ... )
  BLAH=$(( ... ))
  ${option%=*}
  ${option%%=*}
  ${option#=*}
  ${option##=*}
 
 According to 
 
 http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html
 
 all of those are part of Shell Command Language...

Hmm, you are right. The solaris /bin/sh is older revision of posix.



 Does the configure script work if you replace /bin/sh with /bin/ksh
 in your Solaris box

yes, it does work if executed by /bin/bash or /bin/ksh


 If yes, something like the following could be added to the beginning
 of 'configure'
 
 option=option=value
 if test ! x${option$*=} = xvalue 2/dev/null; then
   if test x${PREVENT_LOOPING-} = x; then
   PREVENT_LOOPING=true; export PREVENT_LOOPING
 test ! -x /bin/ksh || exec /bin/ksh $0 $@
 test ! -x /bin/bash || exec /bin/bash $0 $@
 fi
 echo Cannot find compatible shell for '$0' 2
 exit 1
 fi

Unfortunately, no. The /bin/sh says bad substitution and does not run
the script at all. I also tried

eval 'echo ${A%%1}'; echo ok

but that does not run the 'echo ok' and fails also.


I can see three possible solutions

1) use bash or ksh in the shebang line
2) rewrite the script as I gave the overview
3) declare that solaris 10 /bin/sh is not compatible with configure
script


Frankly even 3) is viable option, one just have to remember to run
'bash configure'. If everything else would work, I would be happy :)

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 4/4] Explicitly type void* pointers

[Vladimir Marek]

[...]

> I'm throwing in a third alternative below. Does it work for you? I think
> it's both prettier and uglier than the above at the same time! ;)
> 
> A middle ground would be to change the callers to use
> "notmuch_talloc_steal", and just #define notmuch_talloc_steal
> talloc_steal if __GNUC__ >= 3.
> 
> One could argue upstream talloc should have this, but OTOH it's a C
> library.

That's a nice trick, and it indeed works for me. And I like it more than
what I suggested.

Thank you
-- 
Vlad

Re: [PATCH 4/4] Explicitly type void* pointers

[Vladimir Marek]

[...]

 I'm throwing in a third alternative below. Does it work for you? I think
 it's both prettier and uglier than the above at the same time! ;)
 
 A middle ground would be to change the callers to use
 notmuch_talloc_steal, and just #define notmuch_talloc_steal
 talloc_steal if __GNUC__ = 3.
 
 One could argue upstream talloc should have this, but OTOH it's a C
 library.

That's a nice trick, and it indeed works for me. And I like it more than
what I suggested.

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 4/4] Explicitly type void* pointers

[Vladimir Marek]

Hi,

> Hi, does notmuch not compile without this? IIRC talloc_steal is a macro
> that's supposed to provide type safety (at least with GCC), and I'd be
> hesitant about adding the casts. Please look in your talloc.h.

It does not compile. It might be that I'm using Sun/Oracle CC instead of
gcc. The error looks like this:

"lib/database.cc", line 1368: Error: Cannot assign void* to const char*.


When looking into talloc documentation, the definition seems to be:

void* talloc_steal ( const void * new_ctx, const void * ptr )

http://talloc.samba.org/talloc/doc/html/group__talloc.html#gaccc66139273e727183fb5bdda11ef82c


When looking into talloc.h, it says:

/* try to make talloc_set_destructor() and talloc_steal() type safe,
   if we have a recent gcc */

So, maybe the way to satisfy everyone would be:

notmuch_message_t *tmp_message = message;
talloc_steal(notmuch, tmp_message);
return(tmp_message);

Or alternatively,

#if (__GNUC__ >= 3)
   return talloc_steal (notmuch, message);
#else
   return (notmuch_message_t*) talloc_steal (notmuch, message);
#fi


Of course I'm happy either way :)

Thank you
-- 
Vlad

[PATCH 2/4] dirent->d_type not available on Soalris

[Vladimir Marek]

> Notmuch new is possibly one of the most performance critical bits for
> people with, uh, much mail. The performance impact of the new syscalls
> should be measured. (Can't do this myself atm.)

Fair enough. Is there some performance test suite? Another way would be
to make this compile time option set by configure. Used only when the
system in question does not have dirent->d_type member.

-- 
Vlad

[PATCH 1/4] Make configure use /bin/bash instead of /bin/sh

[Vladimir Marek]

> > Posix /bin/sh is not capable of running this configure and fails.
> 
> What fails? What would it take to make this work on posix sh instead?
> 
> The tests do require bash, but generally I think it would be preferable to
> not depend on bash to build.

Well I gave it a quick stab. This is not posix:

BLAH=$( ... )
BLAH=$(( ... ))
${option%=*}
${option%%=*}
${option#=*}
${option##=*}

First two cases are easy to replace by `...` resp `expr ...`. The rest
leads to external utility like sed. The dirtiest part of configure is
parsing the commandline arguments, but that could be replaced by
/usr/bin/getopts.

If it is appealing way of doing that, I can rework my patch and submit
it for consideration.

Thank you
-- 
Vlad

[PATCH 1/4] Make configure use /bin/bash instead of /bin/sh

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz

Posix /bin/sh is not capable of running this configure and fails.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 configure |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/configure b/configure
index 71981b7..6870341 100755
--- a/configure
+++ b/configure
@@ -1,4 +1,4 @@
-#! /bin/sh
+#! /bin/bash
 
 # Store original IFS value so it can be changed (and restored) in many places.
 readonly DEFAULT_IFS=$IFS
-- 
1.7.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/4] dirent-d_type not available on Soalris

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz

The inspiration was taken from similar issue in mutt:
http://does-not-exist.org/mail-archives/mutt-dev/msg11290.html

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 notmuch-new.c |   19 +--
 1 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 4f13535..20bc580 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -21,6 +21,7 @@
 #include notmuch-client.h
 
 #include unistd.h
+#include sys/types.h
 
 typedef struct _filename_node {
 char *filename;
@@ -165,9 +166,12 @@ static int
 _entries_resemble_maildir (struct dirent **entries, int count)
 {
 int i, found = 0;
+struct stat statbuf;
 
 for (i = 0; i  count; i++) {
-   if (entries[i]-d_type != DT_DIR  entries[i]-d_type != DT_UNKNOWN)
+   if (stat(entries[i]-d_name, statbuf) == -1)
+   continue;
+   if (! S_ISDIR(statbuf.st_mode))
continue;
 
if (strcmp(entries[i]-d_name, new) == 0 ||
@@ -258,6 +262,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 struct stat st;
 notmuch_bool_t is_maildir, new_directory;
 const char **tag;
+struct stat statbuf;
 
 if (stat (path, st)) {
fprintf (stderr, Error reading directory %s: %s\n,
@@ -321,6 +326,9 @@ add_files_recursive (notmuch_database_t *notmuch,
 
entry = fs_entries[i];
 
+   if (stat(entry-d_name, statbuf) == -1)
+   continue;
+
/* We only want to descend into directories.
 * But symlinks can be to directories too, of course.
 *
@@ -328,9 +336,8 @@ add_files_recursive (notmuch_database_t *notmuch,
 * scandir results, then it might be a directory (and if not,
 * then we'll stat and return immediately in the next level of
 * recursion). */
-   if (entry-d_type != DT_DIR 
-   entry-d_type != DT_LNK 
-   entry-d_type != DT_UNKNOWN)
+   if (!(statbuf.st_mode  S_IFDIR) 
+   !(statbuf.st_mode  S_IFLNK))
{
continue;
}
@@ -427,7 +434,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 *
 * In either case, a stat does the trick.
 */
-   if (entry-d_type == DT_LNK || entry-d_type == DT_UNKNOWN) {
+   if (stat(entry-d_name, statbuf) == -1 || statbuf.st_mode  S_IFLNK) {
int err;
 
next = talloc_asprintf (notmuch, %s/%s, path, entry-d_name);
@@ -443,7 +450,7 @@ add_files_recursive (notmuch_database_t *notmuch,
 
if (! S_ISREG (st.st_mode))
continue;
-   } else if (entry-d_type != DT_REG) {
+   } else if ( statbuf.st_mode  S_IFREG) {
continue;
}
 
-- 
1.7.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 4/4] Explicitly type void* pointers

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz


Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 lib/database.cc |2 +-
 lib/message.cc  |2 +-
 lib/thread.cc   |2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/database.cc b/lib/database.cc
index 16c4354..3c82632 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -1361,7 +1361,7 @@ _resolve_message_id_to_thread_id (notmuch_database_t 
*notmuch,
return status;
 
 if (message) {
-   *thread_id_ret = talloc_steal (ctx,
+   *thread_id_ret = (const char*)talloc_steal (ctx,
   notmuch_message_get_thread_id (message));
 
notmuch_message_destroy (message);
diff --git a/lib/message.cc b/lib/message.cc
index 0075425..d56d442 100644
--- a/lib/message.cc
+++ b/lib/message.cc
@@ -220,7 +220,7 @@ _notmuch_message_create_for_message_id (notmuch_database_t 
*notmuch,

message_id,

message);
 if (message)
-   return talloc_steal (notmuch, message);
+   return (notmuch_message_t*) talloc_steal (notmuch, message);
 else if (*status_ret)
return NULL;
 
diff --git a/lib/thread.cc b/lib/thread.cc
index e976d64..d41ff3e 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -225,7 +225,7 @@ _thread_add_message (notmuch_thread_t *thread,
 char *clean_author;
 
 _notmuch_message_list_add_message (thread-message_list,
-  talloc_steal (thread, message));
+  (_notmuch_message*)talloc_steal (thread, 
message));
 thread-total_messages++;
 
 g_hash_table_insert (thread-message_hash,
-- 
1.7.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

notmuch on Solaris

[Vladimir . Marek]

Hi,

In order to make notmuch compilable on Solaris, I had to make some
changes in the source code. Some changes are not yet ready to be
included to official tree, some I believe are. So I took first few to
see if I'm doing anything which could be accepted and hopefully decrease
number of local patches I have to hold. Comments are mostly welcome.

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 3/4] Private strsep implementation

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz

strsep is not available on Solaris 10, so we stole the one used by mutt.

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 compat/Makefile.local |4 +++
 compat/compat.h   |4 +++
 compat/have_strsep.c  |   10 +++
 compat/strsep.c   |   65 +
 configure |   21 ++-
 5 files changed, 102 insertions(+), 2 deletions(-)
 create mode 100644 compat/have_strsep.c
 create mode 100644 compat/strsep.c

diff --git a/compat/Makefile.local b/compat/Makefile.local
index 13f16cd..2c4f65f 100644
--- a/compat/Makefile.local
+++ b/compat/Makefile.local
@@ -13,4 +13,8 @@ ifneq ($(HAVE_STRCASESTR),1)
 notmuch_compat_srcs += $(dir)/strcasestr.c
 endif
 
+ifneq ($(HAVE_STRSEP),1)
+notmuch_compat_srcs += $(dir)/strsep.c
+endif
+
 SRCS := $(SRCS) $(notmuch_compat_srcs)
diff --git a/compat/compat.h b/compat/compat.h
index b2e2736..cf8d56d 100644
--- a/compat/compat.h
+++ b/compat/compat.h
@@ -46,6 +46,10 @@ getdelim (char **lineptr, size_t *n, int delimiter, FILE 
*fp);
 char* strcasestr(const char *haystack, const char *needle);
 #endif /* !HAVE_STRCASESTR */
 
+#if !HAVE_STRSEP
+char *strsep(char **stringp, const char *delim);
+#endif /* !HAVE_STRSEP */
+
 /* Silence gcc warnings about unused results.  These warnings exist
  * for a reason; any use of this needs to be justified. */
 #ifdef __GNUC__
diff --git a/compat/have_strsep.c b/compat/have_strsep.c
new file mode 100644
index 000..5bd396c
--- /dev/null
+++ b/compat/have_strsep.c
@@ -0,0 +1,10 @@
+#define _GNU_SOURCE
+#include string.h
+
+int main()
+{
+char *found;
+char **stringp, const char *delim;
+
+found = strsep(stringp, delim);
+}
diff --git a/compat/strsep.c b/compat/strsep.c
new file mode 100644
index 000..78ab9e7
--- /dev/null
+++ b/compat/strsep.c
@@ -0,0 +1,65 @@
+/* Copyright (C) 1992, 93, 96, 97, 98, 99, 2004 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, write to the Free
+   Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
+   02111-1307 USA.  */
+
+#include string.h
+
+/* Taken from glibc 2.6.1 */
+
+char *strsep (char **stringp, const char *delim)
+{
+  char *begin, *end;
+
+  begin = *stringp;
+  if (begin == NULL)
+return NULL;
+
+  /* A frequent case is when the delimiter string contains only one
+ character.  Here we don't need to call the expensive `strpbrk'
+ function and instead work using `strchr'.  */
+  if (delim[0] == '\0' || delim[1] == '\0')
+{
+  char ch = delim[0];
+
+  if (ch == '\0')
+   end = NULL;
+  else
+   {
+ if (*begin == ch)
+   end = begin;
+ else if (*begin == '\0')
+   end = NULL;
+ else
+   end = strchr (begin + 1, ch);
+   }
+}
+  else
+/* Find the end of the token.  */
+end = strpbrk (begin, delim);
+
+  if (end)
+{
+  /* Terminate the token and set *STRINGP past NUL character.  */
+  *end++ = '\0';
+  *stringp = end;
+}
+  else
+/* No more delimiters; this is the last token.  */
+*stringp = NULL;
+
+  return begin;
+}
diff --git a/configure b/configure
index 6870341..a06df7c 100755
--- a/configure
+++ b/configure
@@ -486,6 +486,17 @@ else
 fi
 rm -f compat/have_strcasestr
 
+printf Checking for strsep... 
+if ${CC} -o compat/have_strsep $srcdir/compat/have_strsep  /dev/null 21
+then
+printf Yes.\n
+have_strsep=1
+else
+printf No (will use our own instead).\n
+have_strsep=0
+fi
+rm -f compat/have_strsep
+
 printf int main(void){return 0;}\n  minimal.c
 
 printf Checking for rpath support... 
@@ -645,6 +656,10 @@ HAVE_GETLINE = ${have_getline}
 # build its own version)
 HAVE_STRCASESTR = ${have_strcasestr}
 
+# Whether the strsep function is available (if not, then notmuch will
+# build its own version)
+HAVE_STRSEP = ${have_strsep}
+
 # Supported platforms (so far) are: LINUX, MACOSX, SOLARIS
 PLATFORM = ${platform}
 
@@ -689,10 +704,12 @@ WITH_ZSH = ${WITH_ZSH}
 # Combined flags for compiling and linking against all of the above
 CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)  \\
   \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND)   \\
-  \$(VALGRIND_CFLAGS) -DHAVE_STRCASESTR

Re: [PATCH 1/4] Make configure use /bin/bash instead of /bin/sh

[Vladimir Marek]

  Posix /bin/sh is not capable of running this configure and fails.
 
 What fails? What would it take to make this work on posix sh instead?
 
 The tests do require bash, but generally I think it would be preferable to
 not depend on bash to build.

Well I gave it a quick stab. This is not posix:

BLAH=$( ... )
BLAH=$(( ... ))
${option%=*}
${option%%=*}
${option#=*}
${option##=*}

First two cases are easy to replace by `...` resp `expr ...`. The rest
leads to external utility like sed. The dirtiest part of configure is
parsing the commandline arguments, but that could be replaced by
/usr/bin/getopts.

If it is appealing way of doing that, I can rework my patch and submit
it for consideration.

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 2/4] dirent-d_type not available on Soalris

[Vladimir Marek]

 Notmuch new is possibly one of the most performance critical bits for
 people with, uh, much mail. The performance impact of the new syscalls
 should be measured. (Can't do this myself atm.)

Fair enough. Is there some performance test suite? Another way would be
to make this compile time option set by configure. Used only when the
system in question does not have dirent-d_type member.

-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH 2/4] dirent-d_type not available on Soalris

[Vladimir . Marek]

From: Vladimir Marek vlma...@volny.cz

The inspiration was taken from similar issue in mutt:
http://does-not-exist.org/mail-archives/mutt-dev/msg11290.html

Signed-off-by: Vladimir Marek vlma...@volny.cz
---
 notmuch-new.c |   28 
 1 files changed, 28 insertions(+), 0 deletions(-)

diff --git a/notmuch-new.c b/notmuch-new.c
index 4f13535..3d265bd 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -21,6 +21,9 @@
 #include notmuch-client.h
 
 #include unistd.h
+#ifndef _DIRENT_HAVE_D_TYPE
+#include sys/types.h
+#endif
 
 typedef struct _filename_node {
 char *filename;
@@ -167,7 +170,14 @@ _entries_resemble_maildir (struct dirent **entries, int 
count)
 int i, found = 0;
 
 for (i = 0; i  count; i++) {
+#ifdef _DIRENT_HAVE_D_TYPE
if (entries[i]-d_type != DT_DIR  entries[i]-d_type != DT_UNKNOWN)
+#else
+   struct stat statbuf;
+   if (stat(entries[i]-d_name, statbuf) == -1)
+   continue;
+   if (! S_ISDIR(statbuf.st_mode))
+#endif
continue;
 
if (strcmp(entries[i]-d_name, new) == 0 ||
@@ -258,6 +268,9 @@ add_files_recursive (notmuch_database_t *notmuch,
 struct stat st;
 notmuch_bool_t is_maildir, new_directory;
 const char **tag;
+#ifndef _DIRENT_HAVE_D_TYPE
+struct stat statbuf;
+#endif
 
 if (stat (path, st)) {
fprintf (stderr, Error reading directory %s: %s\n,
@@ -328,9 +341,16 @@ add_files_recursive (notmuch_database_t *notmuch,
 * scandir results, then it might be a directory (and if not,
 * then we'll stat and return immediately in the next level of
 * recursion). */
+#ifdef _DIRENT_HAVE_D_TYPE
if (entry-d_type != DT_DIR 
entry-d_type != DT_LNK 
entry-d_type != DT_UNKNOWN)
+#else
+   if (stat(entry-d_name, statbuf) == -1)
+   continue;
+   if (!(statbuf.st_mode  S_IFDIR) 
+   !(statbuf.st_mode  S_IFLNK))
+#endif
{
continue;
}
@@ -427,7 +447,11 @@ add_files_recursive (notmuch_database_t *notmuch,
 *
 * In either case, a stat does the trick.
 */
+#ifdef _DIRENT_HAVE_D_TYPE
if (entry-d_type == DT_LNK || entry-d_type == DT_UNKNOWN) {
+#else
+   if (stat(entry-d_name, statbuf) == -1 || statbuf.st_mode  S_IFLNK) {
+#endif
int err;
 
next = talloc_asprintf (notmuch, %s/%s, path, entry-d_name);
@@ -443,7 +467,11 @@ add_files_recursive (notmuch_database_t *notmuch,
 
if (! S_ISREG (st.st_mode))
continue;
+#ifdef _DIRENT_HAVE_D_TYPE
} else if (entry-d_type != DT_REG) {
+#else
+   } else if (statbuf.st_mode  S_IFREG) {
+#endif
continue;
}
 
-- 
1.7.3.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH 4/4] Explicitly type void* pointers

[Vladimir Marek]

Hi,

 Hi, does notmuch not compile without this? IIRC talloc_steal is a macro
 that's supposed to provide type safety (at least with GCC), and I'd be
 hesitant about adding the casts. Please look in your talloc.h.

It does not compile. It might be that I'm using Sun/Oracle CC instead of
gcc. The error looks like this:

lib/database.cc, line 1368: Error: Cannot assign void* to const char*.


When looking into talloc documentation, the definition seems to be:

void* talloc_steal ( const void * new_ctx, const void * ptr )

http://talloc.samba.org/talloc/doc/html/group__talloc.html#gaccc66139273e727183fb5bdda11ef82c


When looking into talloc.h, it says:

/* try to make talloc_set_destructor() and talloc_steal() type safe,
   if we have a recent gcc */

So, maybe the way to satisfy everyone would be:

notmuch_message_t *tmp_message = message;
talloc_steal(notmuch, tmp_message);
return(tmp_message);

Or alternatively,

#if (__GNUC__ = 3)
   return talloc_steal (notmuch, message);
#else
   return (notmuch_message_t*) talloc_steal (notmuch, message);
#fi


Of course I'm happy either way :)

Thank you
-- 
Vlad
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch