Use of strcasestr

[Aaron Ecay]

--- 2010ko Apirilak 12an, Tomas Carnecky-ek idatzi zuen:

>
> Adding that function to compat/ probably is the way to go, but the whole
> compat mechanism doesn't work here. It's like if compat/Makefile.local was
> not included in the top-level makefile, notmuch_compat_srcs is empty
> there. Any ideas how to debug that?

If you are referring to the bug I think you are, I sent a patch to the list to
fix it yesterday.

id:1271029494-89014-2-git-send-email-aaronecay at gmail.com

Aaron

[PATCH] Add compat version of strcasestr

[Dirk Hohndel]

Funny that we both submitted patches...

cworth can decide which one he likes better :-)

/D

On Tue, 13 Apr 2010 05:45:51 +0200, Tomas Carnecky  wrote:
> strcasestr is not part of any standard (unlike for example strcasecmp) and 
> thus
> not available on all platforms (in my case Solaris).
> 
> ---
>  compat/Makefile.local|4 
>  compat/compat.h  |4 
>  compat/have_strcasestr.c |8 
>  compat/strcasestr.c  |   15 +++
>  configure|   19 +--
>  5 files changed, 48 insertions(+), 2 deletions(-)
>  create mode 100644 compat/have_strcasestr.c
>  create mode 100644 compat/strcasestr.c
> 
> diff --git a/compat/Makefile.local b/compat/Makefile.local
> index 81e6c70..328eca2 100644
> --- a/compat/Makefile.local
> +++ b/compat/Makefile.local
> @@ -8,3 +8,7 @@ notmuch_compat_srcs =
>  ifneq ($(HAVE_GETLINE),1)
>  notmuch_compat_srcs += $(dir)/getline.c $(dir)/getdelim.c
>  endif
> +
> +ifneq ($(HAVE_STRCASESTR),1)
> +notmuch_compat_srcs += $(dir)/strcasestr.c
> +endif
> diff --git a/compat/compat.h b/compat/compat.h
> index d639e0f..be70bd8 100644
> --- a/compat/compat.h
> +++ b/compat/compat.h
> @@ -38,4 +38,8 @@ getdelim (char **lineptr, size_t *n, int delimiter, FILE 
> *fp);
>  
>  #endif /* !HAVE_GETLINE */
>  
> +#if !HAVE_STRCASESTR
> +char *strcasestr(char *a, char *b);
> +#endif /* !HAVE_STRCASESTR */
> +
>  #endif /* NOTMUCH_COMPAT_H */
> diff --git a/compat/have_strcasestr.c b/compat/have_strcasestr.c
> new file mode 100644
> index 000..36e760d
> --- /dev/null
> +++ b/compat/have_strcasestr.c
> @@ -0,0 +1,8 @@
> +
> +#include 
> +#include 
> +
> +int main()
> +{
> + return strcasestr("","");
> +}
> diff --git a/compat/strcasestr.c b/compat/strcasestr.c
> new file mode 100644
> index 000..a4188b9
> --- /dev/null
> +++ b/compat/strcasestr.c
> @@ -0,0 +1,15 @@
> +
> +#include 
> +#include 
> +
> +char *strcasestr(char *a, char *b)
> +{
> +  size_t l;
> +  char f[3];
> +  
> +  snprintf(f, sizeof(f), "%c%c", tolower(*b), toupper(*b));
> +  for (l = strcspn(a, f); l != strlen(a); l += strcspn(a + l + 1, f) + 1)
> +if (strncasecmp(a + l, b, strlen(b)) == 0)
> +  return a + l;
> +  return NULL;
> +}
> diff --git a/configure b/configure
> index 5af7852..add2da6 100755
> --- a/configure
> +++ b/configure
> @@ -310,6 +310,17 @@ else
>  fi
>  rm -f compat/have_getline
>  
> +printf "Checking for strcasestr... "
> +if ${CC} -o compat/have_strcasestr compat/have_strcasestr.c > /dev/null 2>&1
> +then
> +printf "Yes.\n"
> +have_strcasestr=1
> +else
> +printf "No (will use our own instead).\n"
> +have_strcasestr=0
> +fi
> +rm -f compat/have_strcasestr
> +
>  cat <  
>  All required packages were found. You may now run the following
> @@ -384,6 +395,10 @@ zsh_completion_dir = 
> \$(prefix)/share/zsh/functions/Completion/Unix
>  # build its own version)
>  HAVE_GETLINE = ${have_getline}
>  
> +# Whether the strcasestr function is available (if not, then notmuch will
> +# build its own version) 
> +HAVE_STRCASESTR = ${have_strcasestr} 
> +
>  # Flags needed to compile and link against Xapian
>  XAPIAN_CXXFLAGS = ${xapian_cxxflags}
>  XAPIAN_LDFLAGS = ${xapian_ldflags}
> @@ -405,9 +420,9 @@ VALGRIND_CFLAGS = ${valgrind_cflags}
>  # Combined flags for compiling and linking against all of the above
>  CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)  \\
>  \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND)   \\
> -\$(VALGRIND_CFLAGS)
> +\$(VALGRIND_CFLAGS) -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
>  CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
>\$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
> -  \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS)
> +  \$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) 
> -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
>  CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
>  EOF
> -- 
> 1.7.0.2
> 
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch

-- 
Dirk Hohndel
Intel Open Source Technology Center

[PATCH] Add simplistic reimplementation of strcasestr to compat library

[Dirk Hohndel]

While all systems that I have access to support strcasestr, it is
in fact not part of POSIX. So here's a fallback reimplementation
based on POSIX functions.

Signed-off-by: Dirk Hohndel 
---
 compat/Makefile.local|4 
 compat/have_strcasestr.c |   10 ++
 compat/strcasestr.c  |   41 +
 configure|   15 +++
 4 files changed, 70 insertions(+), 0 deletions(-)
 create mode 100644 compat/have_strcasestr.c
 create mode 100644 compat/strcasestr.c

diff --git a/compat/Makefile.local b/compat/Makefile.local
index 81e6c70..2a52a14 100644
--- a/compat/Makefile.local
+++ b/compat/Makefile.local
@@ -8,3 +8,7 @@ notmuch_compat_srcs =
 ifneq ($(HAVE_GETLINE),1)
 notmuch_compat_srcs += $(dir)/getline.c $(dir)/getdelim.c
 endif
+
+ifneq ($(HAVE_STRCASESTR),1)
+notmuch_compat_srcs += $(dir)/strcasestr.c 
+endif
\ No newline at end of file
diff --git a/compat/have_strcasestr.c b/compat/have_strcasestr.c
new file mode 100644
index 000..c0fb762
--- /dev/null
+++ b/compat/have_strcasestr.c
@@ -0,0 +1,10 @@
+#define _GNU_SOURCE
+#include 
+
+int main()
+{
+char *found;
+const char *haystack, *needle;
+
+found = strcasestr(haystack, needle);
+}
diff --git a/compat/strcasestr.c b/compat/strcasestr.c
new file mode 100644
index 000..50bc89d
--- /dev/null
+++ b/compat/strcasestr.c
@@ -0,0 +1,41 @@
+/*
+ * slow simplistic reimplementation of strcasestr for systems that
+ * don't include it in their library
+ *
+ * based on a GPL implementation in OpenTTD found under GPL v2
+
+   This program is free software; you can redistribute it and/or
+   modify it under the terms of the GNU General Public License as
+   published by the Free Software Foundation, version 2.
+
+   This program is distributed in the hope that it will be useful, but
+   WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+   02110-1301, USA.  */
+
+/* Imported into notmuch by Dirk Hohndel - original author unknown. */
+/* the semantic here actually puzzles me:
+   how can haystack be const char * - yet the return value is char *
+   after all, it points to a sub-string of haystack... */
+
+#include 
+
+char *strcasestr(const char *haystack, const char *needle)
+{
+   size_t hay_len = strlen(haystack);
+   size_t needle_len = strlen(needle);
+   while (hay_len >= needle_len) {
+   if (strncasecmp(haystack, needle, needle_len) == 0) 
+   return (char *) haystack;
+
+   haystack++;
+   hay_len--;
+   }
+
+   return NULL;
+}
diff --git a/configure b/configure
index 5af7852..023aa40 100755
--- a/configure
+++ b/configure
@@ -310,6 +310,17 @@ else
 fi
 rm -f compat/have_getline

+printf "Checking for strcasestr... "
+if ${CC} -o compat/have_strcasestr compat/have_strcasestr.c > /dev/null 2>&1
+then
+printf "Yes.\n"
+have_strcasestr=1
+else
+printf "No (will use our own instead).\n"
+have_strcasestr=0
+fi
+rm -f compat/have_strcasestr
+
 cat <

Use of strcasestr

[Dirk Hohndel]

On Tue, 13 Apr 2010 04:04:39 +0200, Tomas Carnecky  wrote:
> On 4/12/10 10:18 PM, Mikhail Gusarov wrote:
> >
> > Twas brillig at 15:58:10 12.04.2010 UTC+02 when tom at dbservice.com did 
> > gyre and gimble:
> >
> >   TC>  In 4fd9ea0 (guess From address from Received headers, 2010-04-06) 
> > you introduced
> >   TC>  strcasestr, which is not portable, see 82e47ec (notmuch reply: Use 
> > strstr
> >   TC>  instead of strcasestr for portability., 2010-02-04).
> >
> >   TC>  Is strcasestr really necessary there or can it be replaced with 
> > strstr?
> >
> > strcasecmp is POSIX.1-2001.
> 
> Indeed it is, but the code uses strcasestr and I couldn't find any 
> indication which standard that function is part of.
> 
> Adding that function to compat/ probably is the way to go, but the whole 
> compat mechanism doesn't work here. It's like if compat/Makefile.local 
> was not included in the top-level makefile, notmuch_compat_srcs is empty 
> there. Any ideas how to debug that?

While I don't have access to a system that doesn't provide strcasest
right now... I'll submit a patch in a moment that should add a
replacement function to compat.

Please check and make sure this works...

Thanks

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

please eat my data!

[Servilio Afre Puentes]

On 12 April 2010 13:47, Dirk Hohndel  wrote:
> On Mon, 12 Apr 2010 10:14:05 -0700, Stewart Smith  flamingspork.com> wrote:
>> On Mon, 12 Apr 2010 17:24:35 +0200, "Sebastian Spaeth" > SSpaeth.de> wrote:
>> > What I find intersting is that we have a 2x speedup and a 10x speedup
>> > for different queries. Olly was saying on IRC that both *should* really be
>> > behaving in much the same manner.
>>
>> Remember that on ext3 (and pretty sure ext4) fsync is the same as
>> sync(). So performance depends on how much dirty data you have in your cache.
>>
>> libeatmydata also gets rid of msync(), O_SYNC etc as well.
>
> Which is why so many of us have started to use BTRFS...

How stable is it now? What kernel version and distro are you using?

Servilio

please eat my data!

[Sebastian Spaeth]

On 2010-04-12, Jameson Rollins wrote:
> On Mon, 12 Apr 2010 15:33:35 +0200, "Sebastian Spaeth" > Wow, this is really 
> interesting, Sebastian.  For those of us not in the
> know, can you explain what libeatmydata is and how it's used?

Hehe, I just got the pointer to it on IRC myself:

http://www.flamingspork.com/projects/libeatmydata/

You download and untar the thing, and "make" it, which produces
libeatmydata.so. Running a binary foo with LD_PRELOAD=./libeatmydata.so foo
will then effectively make all fsyncs a Noop. Not something you want on
your production systems, but great to test how much of a penality those
fsyncs really are.

What I find intersting is that we have a 2x speedup and a 10x speedup
for different queries. Olly was saying on IRC that both *should* really be
behaving in much the same manner.

Sebastian

[PATCH] notmuch new --new-tags=tags...

[Michal Sojka]

On Mon, 12 Apr 2010, Jameson Rollins wrote:
> On Mon, 12 Apr 2010 10:00:37 +0200, "Sebastian Spaeth"  SSpaeth.de> wrote:
> > On 2010-04-10, Anthony Towns wrote:
> > > The attached patch makes "notmuch new --new-tags=unread,new" set the
> > > "unread" and "new" tags on any new mail it finds rather than "unread"
> > > and "inbox". Or whatever other tags you happen to specify.
> > 
> > Thanks for the patch. I can't comment on the code quality, but rather
> > than having to specify the set of new tags on the command line every
> > time, I think it would make more sense to put them in the notmuch config
> > file as this patch does:
> > id:1268432006-24333-2-git-send-email-bgamari.foss at gmail.com
> 
> I was thinking about this, and it seems to me that we really need is a
> way to just specify which tags should be applied to new messages based
> on search terms.  It's becoming pretty clear that most people are doing
> some sort of post-notmuch-new tag processing to modify the tags of new
> messages to suite their needs.  Why not just integrate this directly
> into the notmuch-new processing itself?  It seems like if this was
> integrated into notmuch-new directly, the entire processing of new
> messages could be sped up considerably, so that one wouldn't have to
> call multiple notmuch-new processes in succession.

In my mailstore abstraction patches, I suppose that tags for new
messages are derived in some way from mailstore. The idea is, that once
we have a mail store which can store tags (e.g. git-based mails store),
the tags will be assigned on delivery by tools like procmail and notmuch
will only index that tags for faster searching. The problem is the it
will take some time to implement all of this and we want this
functionality now.

> 
> I'm not sure exactly what the best way to handle it would be, but I can
> imagine something like this:
> 
> [new-tags]
> +sent -new -- from:jrollins at finestructure.net
> +drafts -new -- folder:draft
> +notmuch -- from:notmuch at notmuchmail.org
> +unread +inbox -new -- tag:new
> 
> These are all just commands for "notmuch tag" that would be run on all
> the new messages as they're processed.  Each new message would be given
> "new" tag by default, and then the new tag commands would be run.  So it
> would be the equivalent of running the following commands:
> 
> notmuch new --new-tags=new
> notmuch tag +sent -new -- from:jrollins at finestructure.net
> notmuch tag +drafts -new -- folder:draft
> notmuch tag +notmuch -- from:notmuch at notmuchmail.org
> notmuch tag +unread +inbox -- tag:new

The problem I see with this approach is, that all notmuch searches are
build around Xapian. A simple implementation of the above would mean
that whenever you add a new message to the database, you would need to
execute a Xapian query to see if the new message matches your rule or
not. This would be, of course, very expensive.

If we do not want to use xapian for searching, we would have to
implement all the query parsing. building and whatever in notmuch and I
do not think it belongs there.

I do not know much about Xapian internals, so there might be a way of
doing this without reimplementing everything, so correct me if I'm
wrong.

-Michal

please eat my data!

[ra...@free.fr]

- "Jameson Rollins"  a ?crit :

> On Mon, 12 Apr 2010 15:33:35 +0200, "Sebastian Spaeth"
>  wrote:
> > fsync is really killing xapian (and notmuch). What suffers, are the
> > boolean prefixes (tag, id, and thread). Using libeatmydata (which
> > disables fsync) shows a 10x speedup for tagging. The speedup is
> only
> > factor 2 for e.g. from: searches. This is ext4 on recent stock
> > Ubuntu. Given that search by tag and thread are performed really
> often
> > (each time I advance a thread, for example), that really hurts.
> 
> Wow, this is really interesting, Sebastian.  For those of us not in
> the
> know, can you explain what libeatmydata is and how it's used?  It
> sounds
> like something I would *not* want to use!  So you didn't have to
> recompile here, and only had to set LD_PRELOAD=./libeatmydata.so?  Is
> there any drawback to what you're doing here?
> 
> jamie.
> 

It seems like it is a small library that implements fsync as no-op. Using 
LD_PRELOAD 
allows to overloads the libc's fsync definition by libeatmydata's one. Making 
writes faster,
but no longer crash-safe.

Matthieu

please eat my data!

[Dirk Hohndel]

On Mon, 12 Apr 2010 19:10:25 -0400, Servilio Afre Puentes  wrote:
> On 12 April 2010 13:47, Dirk Hohndel  wrote:
> > On Mon, 12 Apr 2010 10:14:05 -0700, Stewart Smith  > flamingspork.com> wrote:
> >> On Mon, 12 Apr 2010 17:24:35 +0200, "Sebastian Spaeth"  >> SSpaeth.de> wrote:
> >> > What I find intersting is that we have a 2x speedup and a 10x speedup
> >> > for different queries. Olly was saying on IRC that both *should* really 
> >> > be
> >> > behaving in much the same manner.
> >>
> >> Remember that on ext3 (and pretty sure ext4) fsync is the same as
> >> sync(). So performance depends on how much dirty data you have in your 
> >> cache.
> >>
> >> libeatmydata also gets rid of msync(), O_SYNC etc as well.
> >
> > Which is why so many of us have started to use BTRFS...
> 
> How stable is it now? What kernel version and distro are you using?

Several. Fedora 12 with 2.6.34-rc3. Moblin-2.1 (derivative) with 2.6.33.
Debian sid with 2.6.33

I've been using it for most everything I do since some point in the
2.6.32 rcs.

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

Use of strcasestr

[Tomas Carnecky]

In 4fd9ea0 (guess From address from Received headers, 2010-04-06) you 
introduced strcasestr, which is not portable, see 82e47ec (notmuch 
reply: Use strstr instead of strcasestr for portability., 2010-02-04).

Is strcasestr really necessary there or can it be replaced with strstr?

tom

please eat my data!

[Sebastian Spaeth]

fsync is really killing xapian (and notmuch). What suffers, are the
boolean prefixes (tag, id, and thread). Using libeatmydata (which
disables fsync) shows a 10x speedup for tagging. The speedup is only
factor 2 for e.g. from: searches. This is ext4 on recent stock
Ubuntu. Given that search by tag and thread are performed really often
(each time I advance a thread, for example), that really hurts.

With a warm file cache and a thread containing 11 messages:

---
time notmuch tag +test -- thread:0f4e
real0m0.677s
user0m0.030s
sys 0m0.020s
---
time LD_PRELOAD=./libeatmydata.so notmuch tag +test -- thread:0f4e

real0m0.040s
user0m0.020s
sys 0m0.020s
---

However tagging ~850 messages based on a from search is "ONLY" factor 2:
--
time notmuch tag +test -- from:"sebastian at sspaeth.de"

real0m2.355s
user0m1.240s
sys 0m0.040s
---
time LD_PRELOAD=./libeatmydata.so notmuch tag +test -- from:"sebastian at 
sspaeth.de"

real0m1.286s
user0m1.230s
sys 0m0.010s
---

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
> also sprach Michal Sojka  [2010.04.12.1347 +0200]:
> > > Wouldn't it be better to postpone synchronisation of the tags
> > > until after emacs is done with the message?
> > 
> > Theoretically, it would be possible, but if, for some reason, the
> > synchronization step would not happen, then the tags in the
> > database and in the mailstore will be inconsistent and next run of
> > notmuch new would reset the tags according to the state in
> > mailstore.
> 
> Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
> transactional anyway. There are many other ways in which the
> database and store can get out of sync. And you are about to add
> another redundancy.
> 
> > The current implementation takes tags in mailstore as
> > authoritative and ensures that tags in mailstore are always
> > updated before tags in the database.
> 
> So why store them in the database at all?

Because we want to use them in searches.

> > > I understand this might be hard to make work with mailstore
> > > abstraction. Wouldn't it make more sense to have emacs call
> > > 'notmuch cat', which returns the entire message, removes the
> > > unread tag, changes the filename, and updates the database?
> > 
> > I do not like the fact that cat would do two things - cat and tag.
> > And then, 'unread' tag is not the only one which can be changed.
> 
> I wouldn't want an unread tag in the first place, especially not
> with Maildir semantics. In this case, what should really happen is:

The bellow sounds reasonable. I only have a few notes.
> 
>   1. cat feeds a message to client

I'd not use cat here, I'd stay with show which transfers only text/*
parts. Recently I was surprised how fast notmuch is when client is
running locally and notmuch is invoked remotely over ssh (even over slow
connection).

>   2. client instructs notmuch to update tags
>  - some tags require changes in the database
>  - others require filename changes, which must be completed in
>unison with a database update so the new filename is stored.
>   3. user asks to see attachment, which the client can fulfill using
>  either a cached copy from (1.) in a tempfile, or by simply
>  asking for the message again, via notmuch search.

The latter option sounds well for me, but my elisp skills are not
sufficient for implementing this. Maybe, dme will do it in his new
client. On the other side, I'm not sure whether it has sense search for
the message again (in step 3) if you are not using maildir mailstore.
One possibility would be, when using maildir mailstore, to cheat the
client and use message-id instead of filename. Then, the client would
use message-id in cat command and implementation for open_file() in
maildir mailstore will search for the real file name.

What do you think?

> > > The message returned by cat would be stored in a temporary file
> > > for use by the client (emacs). And if the message was needed
> > > again, you could just search for it again.
> > > 
> > > I dislike the idea of heuristically probing a Maildir for files.
> > 
> > Well, I do not plan to use wired heuristics. At the end there will
> > be readdir() to traverse the cur/ directory to find the file with
> > the same part before flags. Since the S flag will probably be the
> > most frequent change, I may add one single test for added S flag
> > before trying more expensive readdir().
> 
> What is the point of storing these tags in the Maildir anyway? 

My point is to synchronize the status of reading of my emails with IMAP
server (through offlineimap). I use notmuch as my primary emails client,
but my mobile phone checks for new emails on the IMAP server and from
time to time I need to use other IMAP clients (on other computers).

I have offlineimap running in background so that whenever I read a
message in notmuch, after some short time, this information is
propagated back to IMAP server and when I go home from work, my mobile
phone shows zero unread mails.

> If you want to make this information (e.g. new, seen, unread)
> available to MUAs accessing Maildir directly, keep in mind that the
> database and mailstore will very quickly grow inconsistent until the
> next notmuch-new run, e.g. as messages are moved, or flags ('F') are
> added in a way that the notmuch database is not updated.

It depends on how you define quickly and if you run notmuch new also
very quickly, there is almost no inconsistency :))

-Michal

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka  [2010.04.12.1347 +0200]:
> > Wouldn't it be better to postpone synchronisation of the tags
> > until after emacs is done with the message?
> 
> Theoretically, it would be possible, but if, for some reason, the
> synchronization step would not happen, then the tags in the
> database and in the mailstore will be inconsistent and next run of
> notmuch new would reset the tags according to the state in
> mailstore.

Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
transactional anyway. There are many other ways in which the
database and store can get out of sync. And you are about to add
another redundancy.

> The current implementation takes tags in mailstore as
> authoritative and ensures that tags in mailstore are always
> updated before tags in the database.

So why store them in the database at all?

> > I understand this might be hard to make work with mailstore
> > abstraction. Wouldn't it make more sense to have emacs call
> > 'notmuch cat', which returns the entire message, removes the
> > unread tag, changes the filename, and updates the database?
> 
> I do not like the fact that cat would do two things - cat and tag.
> And then, 'unread' tag is not the only one which can be changed.

I wouldn't want an unread tag in the first place, especially not
with Maildir semantics. In this case, what should really happen is:

  1. cat feeds a message to client
  2. client instructs notmuch to update tags
 - some tags require changes in the database
 - others require filename changes, which must be completed in
   unison with a database update so the new filename is stored.
  3. user asks to see attachment, which the client can fulfill using
 either a cached copy from (1.) in a tempfile, or by simply
 asking for the message again, via notmuch search.

> > The message returned by cat would be stored in a temporary file
> > for use by the client (emacs). And if the message was needed
> > again, you could just search for it again.
> > 
> > I dislike the idea of heuristically probing a Maildir for files.
> 
> Well, I do not plan to use wired heuristics. At the end there will
> be readdir() to traverse the cur/ directory to find the file with
> the same part before flags. Since the S flag will probably be the
> most frequent change, I may add one single test for added S flag
> before trying more expensive readdir().

What is the point of storing these tags in the Maildir anyway? If
you want to make this information (e.g. new, seen, unread) available
to MUAs accessing Maildir directly, keep in mind that the database
and mailstore will very quickly grow inconsistent until the next
notmuch-new run, e.g. as messages are moved, or flags ('F') are
added in a way that the notmuch database is not updated.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"friendships last when each friend thinks he has
 a slight superiority over the other."
   -- honor? de balzac

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100412/2656e7a1/attachment.pgp>

Plans for the 0.2 release (this week)

[Mike Kelly]

On Thu, Apr 08, 2010 at 10:34:59AM -0400, Mike Kelly wrote:
> On Wed, Apr 07, 2010 at 11:23:14PM +0100, James Westby wrote:
> > On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth  wrote:
> > >   * Anything else that people want, (especially things that already
> > > exist and that you're already using). Support for maildir flags on
> > > import would be great here. I'm still waiting to see a complete
> > > solution I think.
> > 
> > id:1268515677-12692-1-git-send-email-jw+debian at jameswestby.net
> 
> I haven't tested yet, but this is definitely a feature I want/need (due
> to the... unique behavior of a certain bug tracker I use at work).

I've now tested this patch, and it appears to work properly. Thanks!

-- 
Mike Kelly

[PATCH] notmuch new --new-tags=tags...

[Tomas Carnecky]

On 4/12/10 1:59 PM, Jameson Rollins wrote:
> On Mon, 12 Apr 2010 10:00:37 +0200, "Sebastian Spaeth" SSpaeth.de>  wrote:
>> On 2010-04-10, Anthony Towns wrote:
>>> The attached patch makes "notmuch new --new-tags=unread,new" set the
>>> "unread" and "new" tags on any new mail it finds rather than "unread"
>>> and "inbox". Or whatever other tags you happen to specify.
>>
>> Thanks for the patch. I can't comment on the code quality, but rather
>> than having to specify the set of new tags on the command line every
>> time, I think it would make more sense to put them in the notmuch config
>> file as this patch does:
>> id:1268432006-24333-2-git-send-email-bgamari.foss at gmail.com
>
> I was thinking about this, and it seems to me that we really need is a
> way to just specify which tags should be applied to new messages based
> on search terms.  It's becoming pretty clear that most people are doing
> some sort of post-notmuch-new tag processing to modify the tags of new
> messages to suite their needs.  Why not just integrate this directly
> into the notmuch-new processing itself?  It seems like if this was
> integrated into notmuch-new directly, the entire processing of new
> messages could be sped up considerably, so that one wouldn't have to
> call multiple notmuch-new processes in succession.
>
> I'm not sure exactly what the best way to handle it would be, but I can
> imagine something like this:
>
> [new-tags]
> +sent -new -- from:jrollins at finestructure.net
> +drafts -new -- folder:draft
> +notmuch -- from:notmuch at notmuchmail.org
> +unread +inbox -new -- tag:new
>
> These are all just commands for "notmuch tag" that would be run on all
> the new messages as they're processed.  Each new message would be given
> "new" tag by default, and then the new tag commands would be run.  So it
> would be the equivalent of running the following commands:
>
> notmuch new --new-tags=new
> notmuch tag +sent -new -- from:jrollins at finestructure.net
> notmuch tag +drafts -new -- folder:draft
> notmuch tag +notmuch -- from:notmuch at notmuchmail.org
> notmuch tag +unread +inbox -- tag:new
>
> This would make things much easier for everyone who is doing post-new
> tag processing, which I think is probably most people.  And I'm sure it
> could be made much more efficient (if coded properly) than running all
> these notmuch commands in succession, especially for people who have a
> lot of post-new tag processing to do.  Keeping the syntax identical to
> the notmuch-tag command syntax would keep things simple as well.
>
> Do people who do a lot of post-notmuch-new tag processing think
> something like this would suite their needs?

I have a patch which adds support for hooks which are run when tags are 
added, removed or new messages added to notmuch. But perhaps the 
fork/exec overhead of running the hooks would slow the processing down 
too much.
See http://caurea.org/2009/12/22/a-different-approach-to-email-tagging/, 
though that didn't work out quite how I expected. Classifying spam/ham 
is easy (that's what dspam was written for), but patch/not-patch 
resulted in a lot false-positives, especially when people quote emails 
which included patches. Same with the 'notmuch' and 'xorg' tags: dspam 
had trouble figuring out to which mailing list Carl sent the email (he 
sends emails to both lists).

tom

[PATCH] notmuch new --new-tags=tags...

[Jameson Rollins]

On Tue, 13 Apr 2010 03:15:28 +1000, Anthony Towns  wrote:
> If you say "they're just notmuch tag commands applied at new time" you
> expect to have the same search behaviour as Xapian...

True, but that's not exactly what I was saying.  I was just saying to
use the same syntax.  But I expect that things might have to work
differently under the hood.  But it might be too much work to come up
with something to parse the search syntax and apply the tag accordingly.

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100412/cefeaffe/attachment.pgp>

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
> also sprach Michal Sojka  [2010.04.08.1713 +0200]:
> >I'm working on the solution - if the mailstore cannot open the
> >message with the name passed, it tries different names with
> >different maildir flags.
> 
> Wouldn't it be better to postpone synchronisation of the tags until
> after emacs is done with the message?

Theoretically, it would be possible, but if, for some reason, the
synchronization step would not happen, then the tags in the database and
in the mailstore will be inconsistent and next run of notmuch new would
reset the tags according to the state in mailstore.

The current implementation takes tags in mailstore as authoritative and
ensures that tags in mailstore are always updated before tags in the
database.

> I understand this might be hard to make work with mailstore
> abstraction. Wouldn't it make more sense to have emacs call 'notmuch
> cat', which returns the entire message, removes the unread tag,
> changes the filename, and updates the database?

I do not like the fact that cat would do two things - cat and tag. And
then, 'unread' tag is not the only one which can be changed.

> The message returned by cat would be stored in a temporary file for
> use by the client (emacs). And if the message was needed again, you
> could just search for it again.
> 
> I dislike the idea of heuristically probing a Maildir for files.

Well, I do not plan to use wired heuristics. At the end there will be
readdir() to traverse the cur/ directory to find the file with the same
part before flags. Since the S flag will probably be the most frequent
change, I may add one single test for added S flag before trying more
expensive readdir().

-Michal

[PATCH] notmuch new --new-tags=tags...

[Jameson Rollins]

On Mon, 12 Apr 2010 17:11:24 +0200, Michal Sojka  wrote:
> In my mailstore abstraction patches, I suppose that tags for new
> messages are derived in some way from mailstore. The idea is, that once
> we have a mail store which can store tags (e.g. git-based mails store),
> the tags will be assigned on delivery by tools like procmail and notmuch
> will only index that tags for faster searching. The problem is the it
> will take some time to implement all of this and we want this
> functionality now.

This is a big effort, and not one that all notmuch users are really
interested in, so I would like to not see it specifically guide notmuch
development.  There are lots of features that would be nice to have
soon.

In any event, I wasn't suggesting we drop everything to do this.  I was
just curious what people thought of this approach.

> The problem I see with this approach is, that all notmuch searches are
> build around Xapian. A simple implementation of the above would mean
> that whenever you add a new message to the database, you would need to
> execute a Xapian query to see if the new message matches your rule or
> not. This would be, of course, very expensive.

This does sound like a potential issue.  I definitely don't understand
how new messages are added to the database.  I was mostly suggesting a
syntax for adding tag as new messages are added, though, not that an
actual xapian search term.  I don't know if they can be decoupled,
though.

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100412/01f92983/attachment-0001.pgp>

[notmuch] Bulk message tagging

[Mark Anderson]

On Sat, 10 Apr 2010 08:56:48 -0500, Xavier Maillard  wrote:
> Hi,
> 
> On Tue, 6 Apr 2010 13:51:01 -0600, Mark Anderson  
> wrote:
> > 
> > I think that '*' is definitely an awesome command, but I wonder if we
> > shouldn't have another command for the notmuch-search buffer which means
> > 'tag all the threads that I can see in this buffer'.
> 
> This is exactly what my initial post asked for. '*' is not
> totally satisfying for me due to the "limitations" you
> exposed. Though It is a good and acceptable workaround for me.
> As said, I just have to pay attention to redo my search query
> before pressing the '*' key.

Another problem I have is that I often _don't_ want to refresh my
search.  Some of my mail processing, while not visible in the search
window, since we don't have a way to refresh tags yet, will remove some of
the current search results from matching the search query.

For example, I like to have notmuch folder definitions with "tag:unread"
in them.  For those search views, once I've looked at any of the mails,
they no longer match the query.  Sometimes I want to refresh the search
so that those mails are no longer visible, sometimed I want to apply an
action on all the visible messages which I've just processed.

When I visit a folder view, what intent do I have?  Am I returning to a
'moment-ago' processing view that was interrupted?  Or am I wanting to
do the search again on the current contents of the database?  I could
easily see mental models that match either way.

I think I read that Carl plans to update the tags in a search view at
some point, without removing threads automatically.

Perhaps there ought to be a way to colorize threads which are displayed
but no longer match the search criterion?

-Mark

[PATCH] emacs: Add notmuch-hello.el, a friendly frontend to notmuch

[David Edmondson]

commit e55dc251b9e8001fe16873fadac565563e606d69
Author: David Edmondson 
Date:   Mon Apr 12 11:12:23 2010 +0100

emacs: Add notmuch-hello.el, a friendly frontend to notmuch

New  emacs/notmuch-hello.el
diff --git a/emacs/notmuch-hello.el b/emacs/notmuch-hello.el
new file mode 100644
index 000..279c424
--- /dev/null
+++ b/emacs/notmuch-hello.el
@@ -0,0 +1,122 @@
+;; notmuch-hello.el --- welcome to notmuch, a frontend
+;;
+;; Copyright ? David Edmondson
+;;
+;; This file is part of Notmuch.
+;;
+;; Notmuch is free software: you can redistribute it and/or modify it
+;; under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+;;
+;; Notmuch is distributed in the hope that it will be useful, but
+;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+;; General Public License for more details.
+;;
+;; You should have received a copy of the GNU General Public License
+;; along with Notmuch.  If not, see .
+;;
+;; Authors: David Edmondson 
+
+(require 'widget)
+
+(require 'notmuch-lib)
+
+(defvar notmuch-hello-searches-to-save 10)
+(defvar notmuch-hello-search-width 60)
+
+(defvar notmuch-hello-recent-searches nil)
+
+(defun notmuch-hello-save-search (search)
+  (if (not (memq search notmuch-hello-recent-searches))
+  (push search notmuch-hello-recent-searches))
+  (if (> (length notmuch-hello-recent-searches)
+notmuch-hello-searches-to-save)
+  (setq notmuch-hello-recent-searches (butlast 
notmuch-hello-recent-searches
+
+(defun notmuch-hello-trim (search)
+  "Trim whitespace."
+  (if (string-match "^[[:space:]]*\\(.*[^[:space:]]\\)[[:space:]]*$" search)
+  (match-string 1 search)
+search))
+
+(defun notmuch-hello-search (search)
+  (let ((search (notmuch-hello-trim search)))
+(notmuch-hello-save-search search)
+(notmuch-search search)))
+
+(defun notmuch-hello ()
+  (interactive)
+
+  (switch-to-buffer "*notmuch-hello*")
+  (kill-all-local-variables)
+  (let ((inhibit-read-only t))
+(erase-buffer))
+
+  (let ((all (overlay-lists)))
+;; Delete all the overlays.
+(mapcar 'delete-overlay (car all))
+(mapcar 'delete-overlay (cdr all)))
+
+  (widget-insert "\nWelcome to notmuch.\n\n")
+  (let ((start (point)))
+(widget-insert "Search: ")
+(widget-create 'editable-field
+  :size notmuch-hello-search-width
+  :action (lambda (widget  ignore)
+(let ((search (widget-value widget)))
+  (notmuch-hello-search search
+(widget-insert "\n")
+(indent-rigidly start (point) 4))
+
+  (when notmuch-hello-recent-searches
+(widget-insert "\nRecent searches:\n\n")
+(let ((start (point)))
+  (mapcar '(lambda (search)
+(widget-create 'editable-field
+   :size notmuch-hello-search-width
+   :action (lambda (widget  ignore)
+ (let ((search (widget-value widget)))
+   (notmuch-hello-search search)))
+   search)
+(widget-insert "\n"))
+ notmuch-hello-recent-searches)
+  (indent-rigidly start (point) 4)))
+
+  (widget-insert "\nFolders:\n\n")
+  (let ((start (point)))
+(mapcar '(lambda (folder)
+  (let ((w (widget-create 'push-button
+  :notify (lambda (widget  ignore)
+(notmuch-search (widget-get 
widget :notmuch-search-terms)))
+  "open")))
+(widget-put w :notmuch-search-terms (cdr folder)))
+  (widget-insert (format " %6s %s\n" (notmuch-folder-count (cdr 
folder)) (car folder
+   notmuch-folders)
+(indent-rigidly start (point) 4))
+
+  (widget-insert "\nAll tags:\n\n")
+  (let ((start (point)))
+(mapcar '(lambda (tag)
+  (let ((w (widget-create 'push-button
+  :notify (lambda (widget  ignore)
+(notmuch-search (widget-get 
widget :notmuch-search-terms)))
+  "open")))
+(widget-put w :notmuch-search-terms (concat "tag:" tag)))
+  (widget-insert (format " %6s %s\n" (notmuch-folder-count
+  (concat "tag:" tag))
+ tag)))
+   (process-lines notmuch-command "search-tags"))
+(indent-rigidly start (point) 4))
+
+  (use-local-map widget-keymap)
+  (local-set-key "=" 'notmuch-hello)
+  (local-set-key "q" '(lambda () (interactive) (kill-buffer (current-buffer
+
+  (widget-setup)
+
+  ;; Always 

please eat my data!

[Dirk Hohndel]

On Mon, 12 Apr 2010 10:14:05 -0700, Stewart Smith  
wrote:
> On Mon, 12 Apr 2010 17:24:35 +0200, "Sebastian Spaeth"  SSpaeth.de> wrote:
> > What I find intersting is that we have a 2x speedup and a 10x speedup
> > for different queries. Olly was saying on IRC that both *should* really be
> > behaving in much the same manner.
> 
> Remember that on ext3 (and pretty sure ext4) fsync is the same as
> sync(). So performance depends on how much dirty data you have in your cache.
> 
> libeatmydata also gets rid of msync(), O_SYNC etc as well.

Which is why so many of us have started to use BTRFS...

Much smaller performance degradation when doing frequent fsync's

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

[PATCH] Fixing possible dereferencing of NULL pointer

[Dirk Hohndel]

I must have misunderstood the cases in which this function can be called
It seemed odd to try to manage authors when author==NULL, but that's
what we appear to be doing; so now we check that autho != NULL and bail
otherwise.

Signed-off-by: Dirk Hohndel 
---
 lib/thread.cc |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/lib/thread.cc b/lib/thread.cc
index c3c83a3..93a7264 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -88,7 +88,7 @@ _thread_move_matched_author (notmuch_thread_t *thread,
 char *currentauthor;
 int idx,nmstart,author_len,authors_len;

-if (thread->authors == NULL)
+if (thread->authors == NULL || author == NULL)
return;
 if (thread->nonmatched_authors == NULL)
thread->nonmatched_authors = thread->authors;
-- 
1.6.6.1


-- 
Dirk Hohndel
Intel Open Source Technology Center

sup-like label listings (elisp)

[Sebastian Spaeth]

On Mon, 12 Apr 2010 09:14:27 +0100, David Edmondson  wrote:
> How about this approach:

ooh! me likey!

Thanks, very nice to get an overview over all my tags.

[PATCH] RFC: User-Agent header

[Sebastian Spaeth]

OK, final post from me on this issue. This is what I am using in my
.emacs now. It does not clutter up the message compose window with a
User-Agent header as it is hidden by message mode. It is also not shown
by default in the notmuch-show modes but viewing the full headers
confirms that it is indeed being send. It works for "c-x m" and "m" in
notmuch-show mode, AND it also work for "r"eplies in notmuch-show mode.

I put this in the message mode hook because we will then not call the
notmuch binary each time we start up emacs, but it will be dynamically
be created when message mode is invoked. Not sure if others have a
cleverer solution here.

;; set the User-Agent string whenever we invoke message mode
(add-hook 'message-mode-hook '(lambda()
;; check if User-Agent is a required header and set it if not
(if (not (memq 'User-Agent message-required-mail-headers))
(setq message-required-mail-headers 
  (append message-required-mail-headers '(User-Agent
;; hide the User-Agent header if not already hidden
(if (not (memq '"^User-Agent:" message-hidden-headers))
(setq message-hidden-headers 
  (append message-hidden-headers '("^User-Agent:"
;; create user agent string
(let ((notmuch-user-agent (concat 
   (substring (shell-command-to-string (concat 
notmuch-command " --version")) 0 -1)
   " (Emacs " emacs-version "/"
system-configuration ")")))
  (setq message-newsreader notmuch-user-agent))
))

Sebastian

please eat my data!

[Jameson Rollins]

On Mon, 12 Apr 2010 15:33:35 +0200, "Sebastian Spaeth"  wrote:
> fsync is really killing xapian (and notmuch). What suffers, are the
> boolean prefixes (tag, id, and thread). Using libeatmydata (which
> disables fsync) shows a 10x speedup for tagging. The speedup is only
> factor 2 for e.g. from: searches. This is ext4 on recent stock
> Ubuntu. Given that search by tag and thread are performed really often
> (each time I advance a thread, for example), that really hurts.

Wow, this is really interesting, Sebastian.  For those of us not in the
know, can you explain what libeatmydata is and how it's used?  It sounds
like something I would *not* want to use!  So you didn't have to
recompile here, and only had to set LD_PRELOAD=./libeatmydata.so?  Is
there any drawback to what you're doing here?

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100412/74749a1d/attachment.pgp>

[PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka  [2010.04.08.1713 +0200]:
>I'm working on the solution - if the mailstore cannot open the
>message with the name passed, it tries different names with
>different maildir flags.

Wouldn't it be better to postpone synchronisation of the tags until
after emacs is done with the message?

I understand this might be hard to make work with mailstore
abstraction. Wouldn't it make more sense to have emacs call 'notmuch
cat', which returns the entire message, removes the unread tag,
changes the filename, and updates the database?

The message returned by cat would be stored in a temporary file for
use by the client (emacs). And if the message was needed again, you
could just search for it again.

I dislike the idea of heuristically probing a Maildir for files.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/

"i don't think so," said rene descartes. just then, he vanished.

spamtraps: madduck.bogus at madduck.net
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature (see http://martin-krafft.net/gpg/)
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100412/4df774f5/attachment.pgp>

please eat my data!

[Stewart Smith]

On Mon, 12 Apr 2010 17:24:35 +0200, "Sebastian Spaeth"  wrote:
> What I find intersting is that we have a 2x speedup and a 10x speedup
> for different queries. Olly was saying on IRC that both *should* really be
> behaving in much the same manner.

Remember that on ext3 (and pretty sure ext4) fsync is the same as
sync(). So performance depends on how much dirty data you have in your cache.

libeatmydata also gets rid of msync(), O_SYNC etc as well.

-- 
Stewart Smith

[PATCH] notmuch new --new-tags=tags...

[Sebastian Spaeth]

On 2010-04-10, Anthony Towns wrote:
> Hi *,
> 
> The attached patch makes "notmuch new --new-tags=unread,new" set the
> "unread" and "new" tags on any new mail it finds rather than "unread"
> and "inbox". Or whatever other tags you happen to specify.

Thanks for the patch. I can't comment on the code quality, but rather
than having to specify the set of new tags on the command line every
time, I think it would make more sense to put them in the notmuch config
file as this patch does:
id:1268432006-24333-2-git-send-email-bgamari.foss at gmail.com

Looking forward to configurable notmuch tags.

Sebastian

sup-like label listings (elisp)

[David Edmondson]

Oops. That one wasn't complete. Try:

commit 0c55967141e7685b0ba23b45a74c1e48a5964f6c
Author: David Edmondson 
Date:   Mon Apr 12 09:24:44 2010 +0100

emacs: More flexible folder mode construction

Allow callers to `notmuch-folder' to optionally specify the alist of
folders to be shown and a title for the buffer.

Add `notmuch-folder-all-tags' and `notmuch-folder-all-tags-unread'
based on the above.

Modified emacs/notmuch.el
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 6d44249..b6a5e5f 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -888,16 +888,43 @@ Currently available key bindings:
 (if search
(notmuch-search (cdr search) notmuch-search-oldest-first

+(defun notmuch-folder-all-tags-unread ()
+  "Show the notmuch folder view for messages tagged `unread' for
+all tags."
+  (interactive)
+  (notmuch-folder-all-tags "tag:unread"))
+
+(defun notmuch-folder-all-tags ( search-restriction)
+  "Show the notmuch folder view for all tags. The optional
+parameter `search-restriction' allows the tag based search to be
+refined."
+  (interactive)
+  (notmuch-folder
+   (mapcar '(lambda (tag)
+ (cons tag (concat "tag:" tag
+   (if search-restriction
+   (concat " AND ( " search-restriction " )")
+ ""
+  (process-lines notmuch-command "search-tags"))
+   search-restriction))
+
 ;;;###autoload
-(defun notmuch-folder ()
+(defun notmuch-folder ( folders title)
   "Show the notmuch folder view and update the displayed counts."
   (interactive)
-  (let ((buffer (get-buffer-create "*notmuch-folders*")))
+  (let ((buffer (get-buffer-create
+(concat "*notmuch-folders"
+(if title (concat "-" title) "")
+"*")))
+   (folders (or folders notmuch-folders)))
 (switch-to-buffer buffer)
 (let ((inhibit-read-only t)
  (n (line-number-at-pos)))
   (erase-buffer)
   (notmuch-folder-mode)
+  ;; Must come after `notmuch-folder-mode', as that kills all
+  ;; local variables.
+  (set (make-local-variable 'notmuch-folders) folders)
   (notmuch-folder-add notmuch-folders)
   (goto-char (point-min))
   (goto-line n


dme.
-- 
David Edmondson, http://dme.org

sup-like label listings (elisp)

[David Edmondson]

How about this approach:

commit 0f591b5ac149179540327f1d39b593c043ec
Author: David Edmondson 
Date:   Mon Apr 12 09:13:15 2010 +0100

emacs: More flexible folder mode construction

Allow callers to `notmuch-folder' to optionally specify the alist of
folders to be shown and a title for the buffer.

Add `notmuch-folder-all-tags' and `notmuch-folder-all-tags-unread'
based on the above.

Modified emacs/notmuch.el
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 6d44249..7b867c6 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -888,17 +888,41 @@ Currently available key bindings:
 (if search
(notmuch-search (cdr search) notmuch-search-oldest-first

+(defun notmuch-folder-all-tags-unread ()
+  "Show the notmuch folder view for messages tagged `unread' for
+all tags."
+  (interactive)
+  (notmuch-folder-all-tags "tag:unread"))
+
+(defun notmuch-folder-all-tags ( search-restriction)
+  "Show the notmuch folder view for all tags. The optional
+parameter `search-restriction' allows the tag based search to be
+refined."
+  (interactive)
+  (notmuch-folder
+   (mapcar '(lambda (tag)
+ (cons tag (concat "tag:" tag
+   (if search-restriction
+   (concat " AND ( " search-restriction " )")
+ ""
+  (process-lines notmuch-command "search-tags"))
+   search-restriction))
+
 ;;;###autoload
-(defun notmuch-folder ()
+(defun notmuch-folder ( folders title)
   "Show the notmuch folder view and update the displayed counts."
   (interactive)
-  (let ((buffer (get-buffer-create "*notmuch-folders*")))
+  (let ((buffer (get-buffer-create
+(concat "*notmuch-folders"
+(if title (concat "-" title) "")
+"*")))
+   (folders (or folders notmuch-folders)))
 (switch-to-buffer buffer)
 (let ((inhibit-read-only t)
  (n (line-number-at-pos)))
   (erase-buffer)
   (notmuch-folder-mode)
-  (notmuch-folder-add notmuch-folders)
+  (notmuch-folder-add folders)
   (goto-char (point-min))
   (goto-line n



dme.
-- 
David Edmondson, http://dme.org

Use of strcasestr

[Dirk Hohndel]

On Mon, 12 Apr 2010 15:58:10 +0200, Tomas Carnecky  wrote:
> In 4fd9ea0 (guess From address from Received headers, 2010-04-06) you 
> introduced strcasestr, which is not portable, see 82e47ec (notmuch 
> reply: Use strstr instead of strcasestr for portability., 2010-02-04).
> 
> Is strcasestr really necessary there or can it be replaced with strstr?

Well, I'm trying to make sure that rAndOm case in email addresses
doesn't throw of the matching - it's quite common for people to write
email addresses as First.Last at Company.Com - the RFCs that I've seen
(can't double check, on a plane right now) seem to indicate that you
can't rely on the headers being lowercase - so even preprocessing the
configured email addresses might not be enough.
I guess we could selectively force some of the headers to be stored as
lower case - Received, X-Original-To, Envelope-To in my case - but that
seems wrong somehow.

Open to better ideas...

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center

[PATCH] notmuch new --new-tags=tags...

[Jameson Rollins]

On Mon, 12 Apr 2010 10:00:37 +0200, "Sebastian Spaeth"  wrote:
> On 2010-04-10, Anthony Towns wrote:
> > The attached patch makes "notmuch new --new-tags=unread,new" set the
> > "unread" and "new" tags on any new mail it finds rather than "unread"
> > and "inbox". Or whatever other tags you happen to specify.
> 
> Thanks for the patch. I can't comment on the code quality, but rather
> than having to specify the set of new tags on the command line every
> time, I think it would make more sense to put them in the notmuch config
> file as this patch does:
> id:1268432006-24333-2-git-send-email-bgamari.foss at gmail.com

I was thinking about this, and it seems to me that we really need is a
way to just specify which tags should be applied to new messages based
on search terms.  It's becoming pretty clear that most people are doing
some sort of post-notmuch-new tag processing to modify the tags of new
messages to suite their needs.  Why not just integrate this directly
into the notmuch-new processing itself?  It seems like if this was
integrated into notmuch-new directly, the entire processing of new
messages could be sped up considerably, so that one wouldn't have to
call multiple notmuch-new processes in succession.

I'm not sure exactly what the best way to handle it would be, but I can
imagine something like this:

[new-tags]
+sent -new -- from:jrollins at finestructure.net
+drafts -new -- folder:draft
+notmuch -- from:notmuch at notmuchmail.org
+unread +inbox -new -- tag:new

These are all just commands for "notmuch tag" that would be run on all
the new messages as they're processed.  Each new message would be given
"new" tag by default, and then the new tag commands would be run.  So it
would be the equivalent of running the following commands:

notmuch new --new-tags=new
notmuch tag +sent -new -- from:jrollins at finestructure.net
notmuch tag +drafts -new -- folder:draft
notmuch tag +notmuch -- from:notmuch at notmuchmail.org
notmuch tag +unread +inbox -- tag:new

This would make things much easier for everyone who is doing post-new
tag processing, which I think is probably most people.  And I'm sure it
could be made much more efficient (if coded properly) than running all
these notmuch commands in succession, especially for people who have a
lot of post-new tag processing to do.  Keeping the syntax identical to
the notmuch-tag command syntax would keep things simple as well.

Do people who do a lot of post-notmuch-new tag processing think
something like this would suite their needs?

jamie.
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20100412/682fa27e/attachment.pgp>

[PATCH] notmuch new --new-tags=tags...

[Scott Robinson]

Excerpts from Tomas Carnecky's message of Mon Apr 12 06:25:23 -0600 2010:
> I have a patch which adds support for hooks which are run when tags are 
> added, removed or new messages added to notmuch. But perhaps the 
> fork/exec overhead of running the hooks would slow the processing down 
> too much.

Rather than a fork/exec for every message, why not have a persistent
subprocess that receives message IDs (+ filenames?) and can then do
arbitrary work and return back the proper tags?

RFC: User-Agent header

[David Edmondson]

On Mon, 12 Apr 2010 00:12:15 +0200, Sebastian Spaeth  
wrote:
> After some research, this is what I found/propose:
> ...

Please make the insertion of User-Agent optional. Some might not want
it.

dme.
-- 
David Edmondson, http://dme.org

RFC: User-Agent header

[Sebastian Spaeth]

After some research, this is what I found/propose:

With some simple elisp am I using this User-Agent header now:

  User-Agent: notmuch version 0.1 (Emacs 23.1.1/i486-pc-linux-gnu)

This needs to be done:

1) Add "User-Agent" to the variable "message-required-headers" (it is
(optional . User-Agent) by default in message-mode.

2) Message mode will then insert a User-Agent header and fill it with
the (local) variable "message-newsreader". This is how I set my
message-newsreader now (leading to above string): 

(setq message-newsreader (concat (substring (shell-command-to-string (concat 
notmuch-command " --version")) 0 -1) " (Emacs " emacs-version "/" 
system-configuration ")"))

This works fine for composing mails with both "m" (from within notmuch)
and "c-x m" from outside notmuch. It won't work with "r" as message mode
does not seem to be involved in setting up the headers at all then. (So
notmuch-reply.c might just want to also set up the User-Agent header by
default).

sebastian

Re: sup-like label listings (elisp)

[David Edmondson]

Oops. That one wasn't complete. Try:

commit 0c55967141e7685b0ba23b45a74c1e48a5964f6c
Author: David Edmondson d...@dme.org
Date:   Mon Apr 12 09:24:44 2010 +0100

emacs: More flexible folder mode construction

Allow callers to `notmuch-folder' to optionally specify the alist of
folders to be shown and a title for the buffer.

Add `notmuch-folder-all-tags' and `notmuch-folder-all-tags-unread'
based on the above.

Modified emacs/notmuch.el
diff --git a/emacs/notmuch.el b/emacs/notmuch.el
index 6d44249..b6a5e5f 100644
--- a/emacs/notmuch.el
+++ b/emacs/notmuch.el
@@ -888,16 +888,43 @@ Currently available key bindings:
 (if search
(notmuch-search (cdr search) notmuch-search-oldest-first
 
+(defun notmuch-folder-all-tags-unread ()
+  Show the notmuch folder view for messages tagged `unread' for
+all tags.
+  (interactive)
+  (notmuch-folder-all-tags tag:unread))
+
+(defun notmuch-folder-all-tags (optional search-restriction)
+  Show the notmuch folder view for all tags. The optional
+parameter `search-restriction' allows the tag based search to be
+refined.
+  (interactive)
+  (notmuch-folder
+   (mapcar '(lambda (tag)
+ (cons tag (concat tag: tag
+   (if search-restriction
+   (concat  AND (  search-restriction  ))
+ 
+  (process-lines notmuch-command search-tags))
+   search-restriction))
+
 ;;;###autoload
-(defun notmuch-folder ()
+(defun notmuch-folder (optional folders title)
   Show the notmuch folder view and update the displayed counts.
   (interactive)
-  (let ((buffer (get-buffer-create *notmuch-folders*)))
+  (let ((buffer (get-buffer-create
+(concat *notmuch-folders
+(if title (concat - title) )
+*)))
+   (folders (or folders notmuch-folders)))
 (switch-to-buffer buffer)
 (let ((inhibit-read-only t)
  (n (line-number-at-pos)))
   (erase-buffer)
   (notmuch-folder-mode)
+  ;; Must come after `notmuch-folder-mode', as that kills all
+  ;; local variables.
+  (set (make-local-variable 'notmuch-folders) folders)
   (notmuch-folder-add notmuch-folders)
   (goto-char (point-min))
   (goto-line n


dme.
-- 
David Edmondson, http://dme.org
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] RFC: User-Agent header

[Sebastian Spaeth]

OK, final post from me on this issue. This is what I am using in my
.emacs now. It does not clutter up the message compose window with a
User-Agent header as it is hidden by message mode. It is also not shown
by default in the notmuch-show modes but viewing the full headers
confirms that it is indeed being send. It works for c-x m and m in
notmuch-show mode, AND it also work for replies in notmuch-show mode.

I put this in the message mode hook because we will then not call the
notmuch binary each time we start up emacs, but it will be dynamically
be created when message mode is invoked. Not sure if others have a
cleverer solution here.

;; set the User-Agent string whenever we invoke message mode
(add-hook 'message-mode-hook '(lambda()
;; check if User-Agent is a required header and set it if not
(if (not (memq 'User-Agent message-required-mail-headers))
(setq message-required-mail-headers 
  (append message-required-mail-headers '(User-Agent
;; hide the User-Agent header if not already hidden
(if (not (memq '^User-Agent: message-hidden-headers))
(setq message-hidden-headers 
  (append message-hidden-headers '(^User-Agent:
;; create user agent string
(let ((notmuch-user-agent (concat 
   (substring (shell-command-to-string (concat 
notmuch-command  --version)) 0 -1)
(Emacs  emacs-version /
system-configuration 
  (setq message-newsreader notmuch-user-agent))
))

Sebastian
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: sup-like label listings (elisp)

[Sebastian Spaeth]

On Mon, 12 Apr 2010 09:14:27 +0100, David Edmondson d...@dme.org wrote:
 How about this approach:

ooh! me likey!

Thanks, very nice to get an overview over all my tags.

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[martin f krafft]

also sprach Michal Sojka sojk...@fel.cvut.cz [2010.04.08.1713 +0200]:
I'm working on the solution - if the mailstore cannot open the
message with the name passed, it tries different names with
different maildir flags.

Wouldn't it be better to postpone synchronisation of the tags until
after emacs is done with the message?

I understand this might be hard to make work with mailstore
abstraction. Wouldn't it make more sense to have emacs call 'notmuch
cat', which returns the entire message, removes the unread tag,
changes the filename, and updates the database?

The message returned by cat would be stored in a temporary file for
use by the client (emacs). And if the message was needed again, you
could just search for it again.

I dislike the idea of heuristically probing a Maildir for files.

-- 
martin | http://madduck.net/ | http://two.sentenc.es/
 
i don't think so, said rene descartes. just then, he vanished.
 
spamtraps: madduck.bo...@madduck.net


digital_signature_gpg.asc
Description: Digital signature (see http://martin-krafft.net/gpg/)
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
 also sprach Michal Sojka sojk...@fel.cvut.cz [2010.04.08.1713 +0200]:
 I'm working on the solution - if the mailstore cannot open the
 message with the name passed, it tries different names with
 different maildir flags.
 
 Wouldn't it be better to postpone synchronisation of the tags until
 after emacs is done with the message?

Theoretically, it would be possible, but if, for some reason, the
synchronization step would not happen, then the tags in the database and
in the mailstore will be inconsistent and next run of notmuch new would
reset the tags according to the state in mailstore.

The current implementation takes tags in mailstore as authoritative and
ensures that tags in mailstore are always updated before tags in the
database.

 I understand this might be hard to make work with mailstore
 abstraction. Wouldn't it make more sense to have emacs call 'notmuch
 cat', which returns the entire message, removes the unread tag,
 changes the filename, and updates the database?

I do not like the fact that cat would do two things - cat and tag. And
then, 'unread' tag is not the only one which can be changed.
 
 The message returned by cat would be stored in a temporary file for
 use by the client (emacs). And if the message was needed again, you
 could just search for it again.
 
 I dislike the idea of heuristically probing a Maildir for files.

Well, I do not plan to use wired heuristics. At the end there will be
readdir() to traverse the cur/ directory to find the file with the same
part before flags. Since the S flag will probably be the most frequent
change, I may add one single test for added S flag before trying more
expensive readdir().

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] notmuch new --new-tags=tags...

[Jameson Rollins]

On Mon, 12 Apr 2010 10:00:37 +0200, Sebastian Spaeth sebast...@sspaeth.de 
wrote:
 On 2010-04-10, Anthony Towns wrote:
  The attached patch makes notmuch new --new-tags=unread,new set the
  unread and new tags on any new mail it finds rather than unread
  and inbox. Or whatever other tags you happen to specify.
 
 Thanks for the patch. I can't comment on the code quality, but rather
 than having to specify the set of new tags on the command line every
 time, I think it would make more sense to put them in the notmuch config
 file as this patch does:
 id:1268432006-24333-2-git-send-email-bgamari.f...@gmail.com

I was thinking about this, and it seems to me that we really need is a
way to just specify which tags should be applied to new messages based
on search terms.  It's becoming pretty clear that most people are doing
some sort of post-notmuch-new tag processing to modify the tags of new
messages to suite their needs.  Why not just integrate this directly
into the notmuch-new processing itself?  It seems like if this was
integrated into notmuch-new directly, the entire processing of new
messages could be sped up considerably, so that one wouldn't have to
call multiple notmuch-new processes in succession.

I'm not sure exactly what the best way to handle it would be, but I can
imagine something like this:

[new-tags]
+sent -new -- from:jroll...@finestructure.net
+drafts -new -- folder:draft
+notmuch -- from:notmuch@notmuchmail.org
+unread +inbox -new -- tag:new

These are all just commands for notmuch tag that would be run on all
the new messages as they're processed.  Each new message would be given
new tag by default, and then the new tag commands would be run.  So it
would be the equivalent of running the following commands:

notmuch new --new-tags=new
notmuch tag +sent -new -- from:jroll...@finestructure.net
notmuch tag +drafts -new -- folder:draft
notmuch tag +notmuch -- from:notmuch@notmuchmail.org
notmuch tag +unread +inbox -- tag:new

This would make things much easier for everyone who is doing post-new
tag processing, which I think is probably most people.  And I'm sure it
could be made much more efficient (if coded properly) than running all
these notmuch commands in succession, especially for people who have a
lot of post-new tag processing to do.  Keeping the syntax identical to
the notmuch-tag command syntax would keep things simple as well.

Do people who do a lot of post-notmuch-new tag processing think
something like this would suite their needs?

jamie.


pgpe08ISwnk6p.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] notmuch new --new-tags=tags...

[Tomas Carnecky]

On 4/12/10 1:59 PM, Jameson Rollins wrote:

On Mon, 12 Apr 2010 10:00:37 +0200, Sebastian Spaethsebast...@sspaeth.de  
wrote:

On 2010-04-10, Anthony Towns wrote:

The attached patch makes notmuch new --new-tags=unread,new set the
unread and new tags on any new mail it finds rather than unread
and inbox. Or whatever other tags you happen to specify.


Thanks for the patch. I can't comment on the code quality, but rather
than having to specify the set of new tags on the command line every
time, I think it would make more sense to put them in the notmuch config
file as this patch does:
id:1268432006-24333-2-git-send-email-bgamari.f...@gmail.com


I was thinking about this, and it seems to me that we really need is a
way to just specify which tags should be applied to new messages based
on search terms.  It's becoming pretty clear that most people are doing
some sort of post-notmuch-new tag processing to modify the tags of new
messages to suite their needs.  Why not just integrate this directly
into the notmuch-new processing itself?  It seems like if this was
integrated into notmuch-new directly, the entire processing of new
messages could be sped up considerably, so that one wouldn't have to
call multiple notmuch-new processes in succession.

I'm not sure exactly what the best way to handle it would be, but I can
imagine something like this:

[new-tags]
+sent -new -- from:jroll...@finestructure.net
+drafts -new -- folder:draft
+notmuch -- from:notmuch@notmuchmail.org
+unread +inbox -new -- tag:new

These are all just commands for notmuch tag that would be run on all
the new messages as they're processed.  Each new message would be given
new tag by default, and then the new tag commands would be run.  So it
would be the equivalent of running the following commands:

notmuch new --new-tags=new
notmuch tag +sent -new -- from:jroll...@finestructure.net
notmuch tag +drafts -new -- folder:draft
notmuch tag +notmuch -- from:notmuch@notmuchmail.org
notmuch tag +unread +inbox -- tag:new

This would make things much easier for everyone who is doing post-new
tag processing, which I think is probably most people.  And I'm sure it
could be made much more efficient (if coded properly) than running all
these notmuch commands in succession, especially for people who have a
lot of post-new tag processing to do.  Keeping the syntax identical to
the notmuch-tag command syntax would keep things simple as well.

Do people who do a lot of post-notmuch-new tag processing think
something like this would suite their needs?


I have a patch which adds support for hooks which are run when tags are 
added, removed or new messages added to notmuch. But perhaps the 
fork/exec overhead of running the hooks would slow the processing down 
too much.
See http://caurea.org/2009/12/22/a-different-approach-to-email-tagging/, 
though that didn't work out quite how I expected. Classifying spam/ham 
is easy (that's what dspam was written for), but patch/not-patch 
resulted in a lot false-positives, especially when people quote emails 
which included patches. Same with the 'notmuch' and 'xorg' tags: dspam 
had trouble figuring out to which mailing list Carl sent the email (he 
sends emails to both lists).


tom


___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Mailstore abstraction v4 - part 2 (maildir synchronization)

[Michal Sojka]

On Mon, 12 Apr 2010, martin f krafft wrote:
 also sprach Michal Sojka sojk...@fel.cvut.cz [2010.04.12.1347 +0200]:
   Wouldn't it be better to postpone synchronisation of the tags
   until after emacs is done with the message?
  
  Theoretically, it would be possible, but if, for some reason, the
  synchronization step would not happen, then the tags in the
  database and in the mailstore will be inconsistent and next run of
  notmuch new would reset the tags according to the state in
  mailstore.
 
 Well, sure. But then again, notmuch (nor IMAP or Maildir) isn't
 transactional anyway. There are many other ways in which the
 database and store can get out of sync. And you are about to add
 another redundancy.
 
  The current implementation takes tags in mailstore as
  authoritative and ensures that tags in mailstore are always
  updated before tags in the database.
 
 So why store them in the database at all?

Because we want to use them in searches.

   I understand this might be hard to make work with mailstore
   abstraction. Wouldn't it make more sense to have emacs call
   'notmuch cat', which returns the entire message, removes the
   unread tag, changes the filename, and updates the database?
  
  I do not like the fact that cat would do two things - cat and tag.
  And then, 'unread' tag is not the only one which can be changed.
 
 I wouldn't want an unread tag in the first place, especially not
 with Maildir semantics. In this case, what should really happen is:

The bellow sounds reasonable. I only have a few notes.
 
   1. cat feeds a message to client

I'd not use cat here, I'd stay with show which transfers only text/*
parts. Recently I was surprised how fast notmuch is when client is
running locally and notmuch is invoked remotely over ssh (even over slow
connection).

   2. client instructs notmuch to update tags
  - some tags require changes in the database
  - others require filename changes, which must be completed in
unison with a database update so the new filename is stored.
   3. user asks to see attachment, which the client can fulfill using
  either a cached copy from (1.) in a tempfile, or by simply
  asking for the message again, via notmuch search.

The latter option sounds well for me, but my elisp skills are not
sufficient for implementing this. Maybe, dme will do it in his new
client. On the other side, I'm not sure whether it has sense search for
the message again (in step 3) if you are not using maildir mailstore.
One possibility would be, when using maildir mailstore, to cheat the
client and use message-id instead of filename. Then, the client would
use message-id in cat command and implementation for open_file() in
maildir mailstore will search for the real file name.

What do you think?

   The message returned by cat would be stored in a temporary file
   for use by the client (emacs). And if the message was needed
   again, you could just search for it again.
   
   I dislike the idea of heuristically probing a Maildir for files.
  
  Well, I do not plan to use wired heuristics. At the end there will
  be readdir() to traverse the cur/ directory to find the file with
  the same part before flags. Since the S flag will probably be the
  most frequent change, I may add one single test for added S flag
  before trying more expensive readdir().
 
 What is the point of storing these tags in the Maildir anyway? 

My point is to synchronize the status of reading of my emails with IMAP
server (through offlineimap). I use notmuch as my primary emails client,
but my mobile phone checks for new emails on the IMAP server and from
time to time I need to use other IMAP clients (on other computers).

I have offlineimap running in background so that whenever I read a
message in notmuch, after some short time, this information is
propagated back to IMAP server and when I go home from work, my mobile
phone shows zero unread mails.

 If you want to make this information (e.g. new, seen, unread)
 available to MUAs accessing Maildir directly, keep in mind that the
 database and mailstore will very quickly grow inconsistent until the
 next notmuch-new run, e.g. as messages are moved, or flags ('F') are
 added in a way that the notmuch database is not updated.

It depends on how you define quickly and if you run notmuch new also
very quickly, there is almost no inconsistency :))

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

please eat my data!

[Sebastian Spaeth]

fsync is really killing xapian (and notmuch). What suffers, are the
boolean prefixes (tag, id, and thread). Using libeatmydata (which
disables fsync) shows a 10x speedup for tagging. The speedup is only
factor 2 for e.g. from: searches. This is ext4 on recent stock
Ubuntu. Given that search by tag and thread are performed really often
(each time I advance a thread, for example), that really hurts.

With a warm file cache and a thread containing 11 messages:

---
time notmuch tag +test -- thread:0f4e
real0m0.677s
user0m0.030s
sys 0m0.020s
---
time LD_PRELOAD=./libeatmydata.so notmuch tag +test -- thread:0f4e

real0m0.040s
user0m0.020s
sys 0m0.020s
---

However tagging ~850 messages based on a from search is ONLY factor 2:
--
time notmuch tag +test -- from:sebast...@sspaeth.de

real0m2.355s
user0m1.240s
sys 0m0.040s
---
time LD_PRELOAD=./libeatmydata.so notmuch tag +test -- 
from:sebast...@sspaeth.de

real0m1.286s
user0m1.230s
sys 0m0.010s
---
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] notmuch new --new-tags=tags...

[Scott Robinson]

Excerpts from Tomas Carnecky's message of Mon Apr 12 06:25:23 -0600 2010:
 I have a patch which adds support for hooks which are run when tags are 
 added, removed or new messages added to notmuch. But perhaps the 
 fork/exec overhead of running the hooks would slow the processing down 
 too much.

Rather than a fork/exec for every message, why not have a persistent
subprocess that receives message IDs (+ filenames?) and can then do
arbitrary work and return back the proper tags?
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Use of strcasestr

[Tomas Carnecky]

In 4fd9ea0 (guess From address from Received headers, 2010-04-06) you 
introduced strcasestr, which is not portable, see 82e47ec (notmuch 
reply: Use strstr instead of strcasestr for portability., 2010-02-04).


Is strcasestr really necessary there or can it be replaced with strstr?

tom

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: please eat my data!

[Jameson Rollins]

On Mon, 12 Apr 2010 15:33:35 +0200, Sebastian Spaeth sebast...@sspaeth.de 
wrote:
 fsync is really killing xapian (and notmuch). What suffers, are the
 boolean prefixes (tag, id, and thread). Using libeatmydata (which
 disables fsync) shows a 10x speedup for tagging. The speedup is only
 factor 2 for e.g. from: searches. This is ext4 on recent stock
 Ubuntu. Given that search by tag and thread are performed really often
 (each time I advance a thread, for example), that really hurts.

Wow, this is really interesting, Sebastian.  For those of us not in the
know, can you explain what libeatmydata is and how it's used?  It sounds
like something I would *not* want to use!  So you didn't have to
recompile here, and only had to set LD_PRELOAD=./libeatmydata.so?  Is
there any drawback to what you're doing here?

jamie.


pgpWPNrbFpSYs.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: please eat my data!

[racin]

- Jameson Rollins jroll...@finestructure.net a écrit :

 On Mon, 12 Apr 2010 15:33:35 +0200, Sebastian Spaeth
 sebast...@sspaeth.de wrote:
  fsync is really killing xapian (and notmuch). What suffers, are the
  boolean prefixes (tag, id, and thread). Using libeatmydata (which
  disables fsync) shows a 10x speedup for tagging. The speedup is
 only
  factor 2 for e.g. from: searches. This is ext4 on recent stock
  Ubuntu. Given that search by tag and thread are performed really
 often
  (each time I advance a thread, for example), that really hurts.
 
 Wow, this is really interesting, Sebastian.  For those of us not in
 the
 know, can you explain what libeatmydata is and how it's used?  It
 sounds
 like something I would *not* want to use!  So you didn't have to
 recompile here, and only had to set LD_PRELOAD=./libeatmydata.so?  Is
 there any drawback to what you're doing here?
 
 jamie.
 

It seems like it is a small library that implements fsync as no-op. Using 
LD_PRELOAD 
allows to overloads the libc's fsync definition by libeatmydata's one. Making 
writes faster,
but no longer crash-safe.

Matthieu
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] notmuch new --new-tags=tags...

[Michal Sojka]

On Mon, 12 Apr 2010, Jameson Rollins wrote:
 On Mon, 12 Apr 2010 10:00:37 +0200, Sebastian Spaeth sebast...@sspaeth.de 
 wrote:
  On 2010-04-10, Anthony Towns wrote:
   The attached patch makes notmuch new --new-tags=unread,new set the
   unread and new tags on any new mail it finds rather than unread
   and inbox. Or whatever other tags you happen to specify.
  
  Thanks for the patch. I can't comment on the code quality, but rather
  than having to specify the set of new tags on the command line every
  time, I think it would make more sense to put them in the notmuch config
  file as this patch does:
  id:1268432006-24333-2-git-send-email-bgamari.f...@gmail.com
 
 I was thinking about this, and it seems to me that we really need is a
 way to just specify which tags should be applied to new messages based
 on search terms.  It's becoming pretty clear that most people are doing
 some sort of post-notmuch-new tag processing to modify the tags of new
 messages to suite their needs.  Why not just integrate this directly
 into the notmuch-new processing itself?  It seems like if this was
 integrated into notmuch-new directly, the entire processing of new
 messages could be sped up considerably, so that one wouldn't have to
 call multiple notmuch-new processes in succession.

In my mailstore abstraction patches, I suppose that tags for new
messages are derived in some way from mailstore. The idea is, that once
we have a mail store which can store tags (e.g. git-based mails store),
the tags will be assigned on delivery by tools like procmail and notmuch
will only index that tags for faster searching. The problem is the it
will take some time to implement all of this and we want this
functionality now.

 
 I'm not sure exactly what the best way to handle it would be, but I can
 imagine something like this:
 
 [new-tags]
 +sent -new -- from:jroll...@finestructure.net
 +drafts -new -- folder:draft
 +notmuch -- from:notmuch@notmuchmail.org
 +unread +inbox -new -- tag:new
 
 These are all just commands for notmuch tag that would be run on all
 the new messages as they're processed.  Each new message would be given
 new tag by default, and then the new tag commands would be run.  So it
 would be the equivalent of running the following commands:
 
 notmuch new --new-tags=new
 notmuch tag +sent -new -- from:jroll...@finestructure.net
 notmuch tag +drafts -new -- folder:draft
 notmuch tag +notmuch -- from:notmuch@notmuchmail.org
 notmuch tag +unread +inbox -- tag:new

The problem I see with this approach is, that all notmuch searches are
build around Xapian. A simple implementation of the above would mean
that whenever you add a new message to the database, you would need to
execute a Xapian query to see if the new message matches your rule or
not. This would be, of course, very expensive.

If we do not want to use xapian for searching, we would have to
implement all the query parsing. building and whatever in notmuch and I
do not think it belongs there.

I do not know much about Xapian internals, so there might be a way of
doing this without reimplementing everything, so correct me if I'm
wrong.

-Michal
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: please eat my data!

[Sebastian Spaeth]

On 2010-04-12, Jameson Rollins wrote:
 On Mon, 12 Apr 2010 15:33:35 +0200, Sebastian Spaeth  Wow, this is really 
 interesting, Sebastian.  For those of us not in the
 know, can you explain what libeatmydata is and how it's used?

Hehe, I just got the pointer to it on IRC myself:

http://www.flamingspork.com/projects/libeatmydata/

You download and untar the thing, and make it, which produces
libeatmydata.so. Running a binary foo with LD_PRELOAD=./libeatmydata.so foo
will then effectively make all fsyncs a Noop. Not something you want on
your production systems, but great to test how much of a penality those
fsyncs really are.

What I find intersting is that we have a 2x speedup and a 10x speedup
for different queries. Olly was saying on IRC that both *should* really be
behaving in much the same manner.

Sebastian
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] notmuch new --new-tags=tags...

[Jameson Rollins]

On Mon, 12 Apr 2010 17:11:24 +0200, Michal Sojka sojk...@fel.cvut.cz wrote:
 In my mailstore abstraction patches, I suppose that tags for new
 messages are derived in some way from mailstore. The idea is, that once
 we have a mail store which can store tags (e.g. git-based mails store),
 the tags will be assigned on delivery by tools like procmail and notmuch
 will only index that tags for faster searching. The problem is the it
 will take some time to implement all of this and we want this
 functionality now.

This is a big effort, and not one that all notmuch users are really
interested in, so I would like to not see it specifically guide notmuch
development.  There are lots of features that would be nice to have
soon.

In any event, I wasn't suggesting we drop everything to do this.  I was
just curious what people thought of this approach.

 The problem I see with this approach is, that all notmuch searches are
 build around Xapian. A simple implementation of the above would mean
 that whenever you add a new message to the database, you would need to
 execute a Xapian query to see if the new message matches your rule or
 not. This would be, of course, very expensive.

This does sound like a potential issue.  I definitely don't understand
how new messages are added to the database.  I was mostly suggesting a
syntax for adding tag as new messages are added, though, not that an
actual xapian search term.  I don't know if they can be decoupled,
though.

jamie.


pgp1t9TIHZk4T.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: please eat my data!

[Stewart Smith]

On Mon, 12 Apr 2010 17:24:35 +0200, Sebastian Spaeth sebast...@sspaeth.de 
wrote:
 What I find intersting is that we have a 2x speedup and a 10x speedup
 for different queries. Olly was saying on IRC that both *should* really be
 behaving in much the same manner.

Remember that on ext3 (and pretty sure ext4) fsync is the same as
sync(). So performance depends on how much dirty data you have in your cache.

libeatmydata also gets rid of msync(), O_SYNC etc as well.

-- 
Stewart Smith
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] notmuch new --new-tags=tags...

[Anthony Towns]

On Tue, Apr 13, 2010 at 01:55, Jameson Rollins
jroll...@finestructure.net wrote:
 On Mon, 12 Apr 2010 17:11:24 +0200, Michal Sojka sojk...@fel.cvut.cz wrote:
 The problem I see with this approach is, that all notmuch searches are
 build around Xapian. ...
 This does sound like a potential issue.  I definitely don't understand
 how new messages are added to the database.  I was mostly suggesting a
 syntax for adding tag as new messages are added, though, not that an
 actual xapian search term.  I don't know if they can be decoupled,
 though.

If you say they're just notmuch tag commands applied at new time you
expect to have the same search behaviour as Xapian...

Maybe you could do that by temporarily inserting the mail into an
inmemory Xapian database, since you're only checking to see if that
particular one matches.

On the other hand, maybe having it be a separate syntax would be good
-- then you could justify using information notmuch doesn't usually
have -- like file/path names, Received or Delivered-To headers, and so
on.

On the gripping hand, maybe notmuch tag should simply be fast enough
that running a bunch of them after notmuch new isn't an issue.

Cheers,
aj

-- 
Anthony Towns a...@erisian.com.au
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] Fixing possible dereferencing of NULL pointer

[Dirk Hohndel]

I must have misunderstood the cases in which this function can be called
It seemed odd to try to manage authors when author==NULL, but that's
what we appear to be doing; so now we check that autho != NULL and bail
otherwise.

Signed-off-by: Dirk Hohndel hohn...@infradead.org
---
 lib/thread.cc |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/lib/thread.cc b/lib/thread.cc
index c3c83a3..93a7264 100644
--- a/lib/thread.cc
+++ b/lib/thread.cc
@@ -88,7 +88,7 @@ _thread_move_matched_author (notmuch_thread_t *thread,
 char *currentauthor;
 int idx,nmstart,author_len,authors_len;
 
-if (thread-authors == NULL)
+if (thread-authors == NULL || author == NULL)
return;
 if (thread-nonmatched_authors == NULL)
thread-nonmatched_authors = thread-authors;
-- 
1.6.6.1


-- 
Dirk Hohndel
Intel Open Source Technology Center
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [notmuch] Bulk message tagging

[Mark Anderson]

On Sat, 10 Apr 2010 08:56:48 -0500, Xavier Maillard x...@gnu.org wrote:
 Hi,
 
 On Tue, 6 Apr 2010 13:51:01 -0600, Mark Anderson markr.ander...@amd.com 
 wrote:
  
  I think that '*' is definitely an awesome command, but I wonder if we
  shouldn't have another command for the notmuch-search buffer which means
  'tag all the threads that I can see in this buffer'.
 
 This is exactly what my initial post asked for. '*' is not
 totally satisfying for me due to the limitations you
 exposed. Though It is a good and acceptable workaround for me.
 As said, I just have to pay attention to redo my search query
 before pressing the '*' key.

Another problem I have is that I often _don't_ want to refresh my
search.  Some of my mail processing, while not visible in the search
window, since we don't have a way to refresh tags yet, will remove some of
the current search results from matching the search query.

For example, I like to have notmuch folder definitions with tag:unread
in them.  For those search views, once I've looked at any of the mails,
they no longer match the query.  Sometimes I want to refresh the search
so that those mails are no longer visible, sometimed I want to apply an
action on all the visible messages which I've just processed.

When I visit a folder view, what intent do I have?  Am I returning to a
'moment-ago' processing view that was interrupted?  Or am I wanting to
do the search again on the current contents of the database?  I could
easily see mental models that match either way.

I think I read that Carl plans to update the tags in a search view at
some point, without removing threads automatically.

Perhaps there ought to be a way to colorize threads which are displayed
but no longer match the search criterion?

-Mark

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] notmuch new --new-tags=tags...

[Jameson Rollins]

On Tue, 13 Apr 2010 03:15:28 +1000, Anthony Towns a...@erisian.com.au wrote:
 If you say they're just notmuch tag commands applied at new time you
 expect to have the same search behaviour as Xapian...

True, but that's not exactly what I was saying.  I was just saying to
use the same syntax.  But I expect that things might have to work
differently under the hood.  But it might be too much work to come up
with something to parse the search syntax and apply the tag accordingly.

jamie.


pgpvRM43INvbD.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Plans for the 0.2 release (this week)

[Mike Kelly]

On Thu, Apr 08, 2010 at 10:34:59AM -0400, Mike Kelly wrote:
 On Wed, Apr 07, 2010 at 11:23:14PM +0100, James Westby wrote:
  On Wed, 07 Apr 2010 15:12:44 -0700, Carl Worth cwo...@cworth.org wrote:
 * Anything else that people want, (especially things that already
   exist and that you're already using). Support for maildir flags on
   import would be great here. I'm still waiting to see a complete
   solution I think.
  
  id:1268515677-12692-1-git-send-email-jw+deb...@jameswestby.net
 
 I haven't tested yet, but this is definitely a feature I want/need (due
 to the... unique behavior of a certain bug tracker I use at work).

I've now tested this patch, and it appears to work properly. Thanks!

-- 
Mike Kelly
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Use of strcasestr

[Dirk Hohndel]

On Mon, 12 Apr 2010 15:58:10 +0200, Tomas Carnecky t...@dbservice.com wrote:
 In 4fd9ea0 (guess From address from Received headers, 2010-04-06) you 
 introduced strcasestr, which is not portable, see 82e47ec (notmuch 
 reply: Use strstr instead of strcasestr for portability., 2010-02-04).
 
 Is strcasestr really necessary there or can it be replaced with strstr?

Well, I'm trying to make sure that rAndOm case in email addresses
doesn't throw of the matching - it's quite common for people to write
email addresses as first.l...@company.com - the RFCs that I've seen
(can't double check, on a plane right now) seem to indicate that you
can't rely on the headers being lowercase - so even preprocessing the
configured email addresses might not be enough.
I guess we could selectively force some of the headers to be stored as
lower case - Received, X-Original-To, Envelope-To in my case - but that
seems wrong somehow.

Open to better ideas...

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Use of strcasestr

[Mikhail Gusarov]

Twas brillig at 15:58:10 12.04.2010 UTC+02 when t...@dbservice.com did gyre and 
gimble:

 TC In 4fd9ea0 (guess From address from Received headers, 2010-04-06) you 
introduced
 TC strcasestr, which is not portable, see 82e47ec (notmuch reply: Use strstr
 TC instead of strcasestr for portability., 2010-02-04).

 TC Is strcasestr really necessary there or can it be replaced with strstr?

strcasecmp is POSIX.1-2001.

If you know any OS which does not have it, add a new file to compat/ subdir.

-- 
  http://fossarchy.blogspot.com/


pgp3d80eMLC7B.pgp
Description: PGP signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: please eat my data!

[Servilio Afre Puentes]

On 12 April 2010 13:47, Dirk Hohndel hohn...@infradead.org wrote:
 On Mon, 12 Apr 2010 10:14:05 -0700, Stewart Smith stew...@flamingspork.com 
 wrote:
 On Mon, 12 Apr 2010 17:24:35 +0200, Sebastian Spaeth 
 sebast...@sspaeth.de wrote:
  What I find intersting is that we have a 2x speedup and a 10x speedup
  for different queries. Olly was saying on IRC that both *should* really be
  behaving in much the same manner.

 Remember that on ext3 (and pretty sure ext4) fsync is the same as
 sync(). So performance depends on how much dirty data you have in your cache.

 libeatmydata also gets rid of msync(), O_SYNC etc as well.

 Which is why so many of us have started to use BTRFS...

How stable is it now? What kernel version and distro are you using?

Servilio
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: please eat my data!

[Dirk Hohndel]

On Mon, 12 Apr 2010 19:10:25 -0400, Servilio Afre Puentes servi...@gmail.com 
wrote:
 On 12 April 2010 13:47, Dirk Hohndel hohn...@infradead.org wrote:
  On Mon, 12 Apr 2010 10:14:05 -0700, Stewart Smith 
  stew...@flamingspork.com wrote:
  On Mon, 12 Apr 2010 17:24:35 +0200, Sebastian Spaeth 
  sebast...@sspaeth.de wrote:
   What I find intersting is that we have a 2x speedup and a 10x speedup
   for different queries. Olly was saying on IRC that both *should* really 
   be
   behaving in much the same manner.
 
  Remember that on ext3 (and pretty sure ext4) fsync is the same as
  sync(). So performance depends on how much dirty data you have in your 
  cache.
 
  libeatmydata also gets rid of msync(), O_SYNC etc as well.
 
  Which is why so many of us have started to use BTRFS...
 
 How stable is it now? What kernel version and distro are you using?

Several. Fedora 12 with 2.6.34-rc3. Moblin-2.1 (derivative) with 2.6.33.
Debian sid with 2.6.33

I've been using it for most everything I do since some point in the
2.6.32 rcs.

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Use of strcasestr

[Tomas Carnecky]

On 4/12/10 10:18 PM, Mikhail Gusarov wrote:


Twas brillig at 15:58:10 12.04.2010 UTC+02 when t...@dbservice.com did gyre and 
gimble:

  TC  In 4fd9ea0 (guess From address from Received headers, 2010-04-06) you 
introduced
  TC  strcasestr, which is not portable, see 82e47ec (notmuch reply: Use strstr
  TC  instead of strcasestr for portability., 2010-02-04).

  TC  Is strcasestr really necessary there or can it be replaced with strstr?

strcasecmp is POSIX.1-2001.


Indeed it is, but the code uses strcasestr and I couldn't find any 
indication which standard that function is part of.


Adding that function to compat/ probably is the way to go, but the whole 
compat mechanism doesn't work here. It's like if compat/Makefile.local 
was not included in the top-level makefile, notmuch_compat_srcs is empty 
there. Any ideas how to debug that?


tom

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Use of strcasestr

[Aaron Ecay]

--- 2010ko Apirilak 12an, Tomas Carnecky-ek idatzi zuen:


 Adding that function to compat/ probably is the way to go, but the whole
 compat mechanism doesn't work here. It's like if compat/Makefile.local was
 not included in the top-level makefile, notmuch_compat_srcs is empty
 there. Any ideas how to debug that?

If you are referring to the bug I think you are, I sent a patch to the list to
fix it yesterday.

id:1271029494-89014-2-git-send-email-aarone...@gmail.com

Aaron
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] Add compat version of strcasestr

[Tomas Carnecky]

strcasestr is not part of any standard (unlike for example strcasecmp) and thus
not available on all platforms (in my case Solaris).

---
 compat/Makefile.local|4 
 compat/compat.h  |4 
 compat/have_strcasestr.c |8 
 compat/strcasestr.c  |   15 +++
 configure|   19 +--
 5 files changed, 48 insertions(+), 2 deletions(-)
 create mode 100644 compat/have_strcasestr.c
 create mode 100644 compat/strcasestr.c

diff --git a/compat/Makefile.local b/compat/Makefile.local
index 81e6c70..328eca2 100644
--- a/compat/Makefile.local
+++ b/compat/Makefile.local
@@ -8,3 +8,7 @@ notmuch_compat_srcs =
 ifneq ($(HAVE_GETLINE),1)
 notmuch_compat_srcs += $(dir)/getline.c $(dir)/getdelim.c
 endif
+
+ifneq ($(HAVE_STRCASESTR),1)
+notmuch_compat_srcs += $(dir)/strcasestr.c
+endif
diff --git a/compat/compat.h b/compat/compat.h
index d639e0f..be70bd8 100644
--- a/compat/compat.h
+++ b/compat/compat.h
@@ -38,4 +38,8 @@ getdelim (char **lineptr, size_t *n, int delimiter, FILE *fp);
 
 #endif /* !HAVE_GETLINE */
 
+#if !HAVE_STRCASESTR
+char *strcasestr(char *a, char *b);
+#endif /* !HAVE_STRCASESTR */
+
 #endif /* NOTMUCH_COMPAT_H */
diff --git a/compat/have_strcasestr.c b/compat/have_strcasestr.c
new file mode 100644
index 000..36e760d
--- /dev/null
+++ b/compat/have_strcasestr.c
@@ -0,0 +1,8 @@
+
+#include string.h
+#include strings.h
+
+int main()
+{
+   return strcasestr(,);
+}
diff --git a/compat/strcasestr.c b/compat/strcasestr.c
new file mode 100644
index 000..a4188b9
--- /dev/null
+++ b/compat/strcasestr.c
@@ -0,0 +1,15 @@
+
+#include string.h
+#include strings.h
+
+char *strcasestr(char *a, char *b)
+{
+  size_t l;
+  char f[3];
+  
+  snprintf(f, sizeof(f), %c%c, tolower(*b), toupper(*b));
+  for (l = strcspn(a, f); l != strlen(a); l += strcspn(a + l + 1, f) + 1)
+if (strncasecmp(a + l, b, strlen(b)) == 0)
+  return a + l;
+  return NULL;
+}  
diff --git a/configure b/configure
index 5af7852..add2da6 100755
--- a/configure
+++ b/configure
@@ -310,6 +310,17 @@ else
 fi
 rm -f compat/have_getline
 
+printf Checking for strcasestr... 
+if ${CC} -o compat/have_strcasestr compat/have_strcasestr.c  /dev/null 21
+then
+printf Yes.\n
+have_strcasestr=1
+else
+printf No (will use our own instead).\n
+have_strcasestr=0
+fi
+rm -f compat/have_strcasestr
+
 cat EOF
 
 All required packages were found. You may now run the following
@@ -384,6 +395,10 @@ zsh_completion_dir = 
\$(prefix)/share/zsh/functions/Completion/Unix
 # build its own version)
 HAVE_GETLINE = ${have_getline}
 
+# Whether the strcasestr function is available (if not, then notmuch will
+# build its own version) 
+HAVE_STRCASESTR = ${have_strcasestr} 
+
 # Flags needed to compile and link against Xapian
 XAPIAN_CXXFLAGS = ${xapian_cxxflags}
 XAPIAN_LDFLAGS = ${xapian_ldflags}
@@ -405,9 +420,9 @@ VALGRIND_CFLAGS = ${valgrind_cflags}
 # Combined flags for compiling and linking against all of the above
 CONFIGURE_CFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)  \\
   \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND)   \\
-  \$(VALGRIND_CFLAGS)
+  \$(VALGRIND_CFLAGS) -DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
 CONFIGURE_CXXFLAGS = -DHAVE_GETLINE=\$(HAVE_GETLINE) \$(GMIME_CFLAGS)\\
 \$(TALLOC_CFLAGS) -DHAVE_VALGRIND=\$(HAVE_VALGRIND) \\
-\$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS)
+\$(VALGRIND_CFLAGS) \$(XAPIAN_CXXFLAGS) 
-DHAVE_STRCASESTR=\$(HAVE_STRCASESTR)
 CONFIGURE_LDFLAGS =  \$(GMIME_LDFLAGS) \$(TALLOC_LDFLAGS) \$(XAPIAN_LDFLAGS)
 EOF
-- 
1.7.0.2

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: Use of strcasestr

[Dirk Hohndel]

On Tue, 13 Apr 2010 04:04:39 +0200, Tomas Carnecky t...@dbservice.com wrote:
 On 4/12/10 10:18 PM, Mikhail Gusarov wrote:
 
  Twas brillig at 15:58:10 12.04.2010 UTC+02 when t...@dbservice.com did gyre 
  and gimble:
 
TC  In 4fd9ea0 (guess From address from Received headers, 2010-04-06) 
  you introduced
TC  strcasestr, which is not portable, see 82e47ec (notmuch reply: Use 
  strstr
TC  instead of strcasestr for portability., 2010-02-04).
 
TC  Is strcasestr really necessary there or can it be replaced with 
  strstr?
 
  strcasecmp is POSIX.1-2001.
 
 Indeed it is, but the code uses strcasestr and I couldn't find any 
 indication which standard that function is part of.
 
 Adding that function to compat/ probably is the way to go, but the whole 
 compat mechanism doesn't work here. It's like if compat/Makefile.local 
 was not included in the top-level makefile, notmuch_compat_srcs is empty 
 there. Any ideas how to debug that?

While I don't have access to a system that doesn't provide strcasest
right now... I'll submit a patch in a moment that should add a
replacement function to compat.

Please check and make sure this works...

Thanks

/D

-- 
Dirk Hohndel
Intel Open Source Technology Center
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

[PATCH] Add simplistic reimplementation of strcasestr to compat library

[Dirk Hohndel]

While all systems that I have access to support strcasestr, it is
in fact not part of POSIX. So here's a fallback reimplementation
based on POSIX functions.

Signed-off-by: Dirk Hohndel hohn...@infradead.org
---
 compat/Makefile.local|4 
 compat/have_strcasestr.c |   10 ++
 compat/strcasestr.c  |   41 +
 configure|   15 +++
 4 files changed, 70 insertions(+), 0 deletions(-)
 create mode 100644 compat/have_strcasestr.c
 create mode 100644 compat/strcasestr.c

diff --git a/compat/Makefile.local b/compat/Makefile.local
index 81e6c70..2a52a14 100644
--- a/compat/Makefile.local
+++ b/compat/Makefile.local
@@ -8,3 +8,7 @@ notmuch_compat_srcs =
 ifneq ($(HAVE_GETLINE),1)
 notmuch_compat_srcs += $(dir)/getline.c $(dir)/getdelim.c
 endif
+
+ifneq ($(HAVE_STRCASESTR),1)
+notmuch_compat_srcs += $(dir)/strcasestr.c 
+endif
\ No newline at end of file
diff --git a/compat/have_strcasestr.c b/compat/have_strcasestr.c
new file mode 100644
index 000..c0fb762
--- /dev/null
+++ b/compat/have_strcasestr.c
@@ -0,0 +1,10 @@
+#define _GNU_SOURCE
+#include strings.h
+
+int main()
+{
+char *found;
+const char *haystack, *needle;
+
+found = strcasestr(haystack, needle);
+}
diff --git a/compat/strcasestr.c b/compat/strcasestr.c
new file mode 100644
index 000..50bc89d
--- /dev/null
+++ b/compat/strcasestr.c
@@ -0,0 +1,41 @@
+/*
+ * slow simplistic reimplementation of strcasestr for systems that
+ * don't include it in their library
+ *
+ * based on a GPL implementation in OpenTTD found under GPL v2
+
+   This program is free software; you can redistribute it and/or
+   modify it under the terms of the GNU General Public License as
+   published by the Free Software Foundation, version 2.
+
+   This program is distributed in the hope that it will be useful, but
+   WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program; if not, write to the Free Software
+   Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+   02110-1301, USA.  */
+
+/* Imported into notmuch by Dirk Hohndel - original author unknown. */
+/* the semantic here actually puzzles me:
+   how can haystack be const char * - yet the return value is char *
+   after all, it points to a sub-string of haystack... */
+
+#include string.h
+
+char *strcasestr(const char *haystack, const char *needle)
+{
+   size_t hay_len = strlen(haystack);
+   size_t needle_len = strlen(needle);
+   while (hay_len = needle_len) {
+   if (strncasecmp(haystack, needle, needle_len) == 0) 
+   return (char *) haystack;
+
+   haystack++;
+   hay_len--;
+   }
+
+   return NULL;
+}
diff --git a/configure b/configure
index 5af7852..023aa40 100755
--- a/configure
+++ b/configure
@@ -310,6 +310,17 @@ else
 fi
 rm -f compat/have_getline
 
+printf Checking for strcasestr... 
+if ${CC} -o compat/have_strcasestr compat/have_strcasestr.c  /dev/null 21
+then
+printf Yes.\n
+have_strcasestr=1
+else
+printf No (will use our own instead).\n
+have_strcasestr=0
+fi
+rm -f compat/have_strcasestr
+
 cat EOF
 
 All required packages were found. You may now run the following
@@ -384,6 +395,10 @@ zsh_completion_dir = 
\$(prefix)/share/zsh/functions/Completion/Unix
 # build its own version)
 HAVE_GETLINE = ${have_getline}
 
+# Whether the strcasestr function is available (if not, then notmuch will
+# build its own version)
+HAVE_STRCASESTR = ${have_strcasestr}
+
 # Flags needed to compile and link against Xapian
 XAPIAN_CXXFLAGS = ${xapian_cxxflags}
 XAPIAN_LDFLAGS = ${xapian_ldflags}
-- 
1.6.6.1


-- 
Dirk Hohndel
Intel Open Source Technology Center
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Add simplistic reimplementation of strcasestr to compat library

[Tomas Carnecky]

On 4/13/10 6:10 AM, Dirk Hohndel wrote:


While all systems that I have access to support strcasestr, it is
in fact not part of POSIX. So here's a fallback reimplementation
based on POSIX functions.


Your patch is missing the part where it adds -DHAVE_STRCASESTR=.. to 
CONFIGURE_C{XX,}FLAGS.


And I still have to figure out the licencing of my patch (need to wait 
for the original author to respond). So maybe your patch, with the 
missing parts added, is better.


tom

___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch

Re: [PATCH] Add simplistic reimplementation of strcasestr to compat library

[Anthony Towns]

On Tue, Apr 13, 2010 at 14:10, Dirk Hohndel hohn...@infradead.org wrote:
 +/* the semantic here actually puzzles me:
 +   how can haystack be const char * - yet the return value is char *
 +   after all, it points to a sub-string of haystack... */

Dunno if this is a question from the original source, but the answer
if anyone's interested is probably because C doesn't have templates --
you'd ideally like to have it treated as:

char *strcasestr(char *haystack, const char *needle);

for when you're doing a search and replace on the needle (say), and:

const char *strcasestr(const char *haystack, const char *needle);

for when you're doing a search for the needle in something you can't
modify. But C isn't clever enough to let you say that with just one
function (and no fancy #defines), so you have to drop some of the
typechecking with the (char*) cast on the return value if you want to
handle both use cases, without the compiler complaining about
const-non-const conversions in otherwise correct code in one case or
the other.

Cheers,
aj

-- 
Anthony Towns a...@erisian.com.au
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch