[notmuch] 25 minutes load time with emacs -f notmuch

2009-11-21 Thread Stefan Schmidt
Hello.

Disclaimer: I'm using vim, in combination with mutt for email, for years, but
never dealt with emacs. Please have this in mind and spot any emacs user errors
in this report. :)

I have first seen notmuch several weeks ago as it seems a silent project. Being
more then happy now that it envolves quickly and a real developer community
builds around it.

But now to my problem. Getting m mail indexed was easy enough:

stefan at excalibur:~$ du -chs not-much-mail/
1.5Gnot-much-mail/
1.5Gtotal
stefan at excalibur:~$ time notmuch new
Found 103677 total files.
Processed 103677 total files in 42m 30s (40 files/sec.).
Added 100899 new messages to the database (not much, really).

Tip: If you have any sub-directories that are archives (that is,
they will never receive new mail), marking these directories as
read-only (chmod u-w /path/to/dir) will make "notmuch new"
much more efficient (it won't even look in those directories).

real43m0.943s
user22m46.513s
sys 0m39.418s


I put (require  'notmuch) in my ~/.emacs ans start emacs with the -f notmuch
option to enter the notmuch mode. What happends then is that a notmuch process
gets started and emacs waits for the return.

23649 pts/1SN+0:00  |   \_ emacs -f notmuch
23651 ?RNs0:03  |   \_ /usr/local/bin/notmuch search
--sort=oldest-first tag:inbox

Sadly that takes around 25 minutes here on an Intel Core2Duo notbeook (Thinkpad
X200s). I tried this several times now. CPU load was low (~10%) during this time
so it is mostly IO bound.

I checked that I don't have any big files like mutt header caches left and all
my mail is stored in maildir format diretcly from offlineimap. I'm more then
happy to test any patches on this issue or do some debugging myself if I get
some hints where to look.

regards
Stefan Schmidt


[notmuch] 25 minutes load time with emacs -f notmuch

2009-11-21 Thread Stefan Schmidt
Hello.

On Sat, 2009-11-21 at 08:12, Bdale Garbee wrote:
> On Sat, 2009-11-21 at 15:51 +0100, Stefan Schmidt wrote:
> 
> > Sadly that takes around 25 minutes here on an Intel Core2Duo notbeook 
> > (Thinkpad
> > X200s). I tried this several times now. CPU load was low (~10%) during this 
> > time
> > so it is mostly IO bound.
> 
> I see the same behavior on my notebook.  
> 
> I gather from talking to keithp that things like the 'state of already
> being read' aren't being picked up from the file names in the local
> Maildir yet.  Thus I suspect it's a fairly unusual / worst case scenario
> trying to start up with 178k (in my case) supposedly-unread messages
> tagged inbox.

Using the read flag during notmuch new would indeed be nice. But some further
testing brings some doubts that it is an overload due to to many unread
messages.

I executed "/usr/local/bin/notmuch search --sort=oldest-first tag:inbox" by hand
and from the 21 minutes it took it stayed around 20 in a state where no new
message where printed and then sudenly all the rest comes up.

In my case only 80 messages were printed before the gap. All of them had a wrong
year in the timestamp. 1900 and 1970. Maybe notmuch just comes into a bad state
with this dates?

Bdale, can you confirm this for your case?

I will remove these mails and re-generate the notmuch index to test this out
after dinner later today.

regards
Stefan Schmidt


[notmuch] notmuch vim support

2009-11-21 Thread Stefan Schmidt
Hello.

On Sat, 2009-11-21 at 13:39, Bart Trojanowski wrote:
> 
> I've spent a few days on porting notmuch.el to vim.  It's not as feature
> rich as notmuch.el yet, but it's coming along nicely.  The results of my
> work are here:

Great stuff! I hoped for someone stepping up early to provide alternative
interfaces.

> I'd love for this to be included in the official project, but I am not
> sure if you'd prefer to merge at this point or wait will I am done.

IMHO it would be best to do the development inside the main tree. You could use
branches there too and it would be easier to find and test. And as notmuch is
still under heavy development its not a problem that the vim interface is as
well.

regards
Stefan Schmidt


[notmuch] 25 minutes load time with emacs -f notmuch

2009-11-21 Thread Stefan Schmidt
Hello.

On Sat, 2009-11-21 at 18:16, Carl Worth wrote:
> On Sat, 21 Nov 2009 08:12:52 -0700, Bdale Garbee  wrote:
> > I haven't figured out how to quickly tag everything as already read or
> > archived or whatever .. can someone who knows more about what's going on
> > confirm my hypothesis and if so, suggest the best approach to getting to
> > a happier state?
> 
> See my message up-thread. The only reasonable ways all really do involve
> at least a little bit of C-code hacking to either prevent those tags
> from getting put there by "notmuch new" or to make it easier to get them
> off afterwards.

Let's see if I come up with something here.

> And I can't help but apologize. I've known about all these issues, and
> wouldn't have invited people to try things out in the current state. But
> it was nice of Keith to share this with everyone. And it's nice of all
> you to come take a look at things.

Getting it out now was a good move. It had enough code to actually do omething
usefull and many people waited for something like this. The increasing number of
contributors in such a short time shows it very well. :)

regards
Stefan Schmidt


[notmuch] 25 minutes load time with emacs -f notmuch

2009-11-21 Thread Stefan Schmidt
Hello.

On Sat, 2009-11-21 at 18:26, Carl Worth wrote:
> On Sat, 21 Nov 2009 16:36:55 +0100, Stefan Schmidt  datenfreihafen.org> wrote:
> 
> > In my case only 80 messages were printed before the gap. All of them had a 
> > wrong
> > year in the timestamp. 1900 and 1970. Maybe notmuch just comes into a bad 
> > state
> > with this dates?
> 
> I don't think the bogus dates are throwing anything off. It's more
> likely that you just have a number of messages with no Date header on
> them at all. And for such messages, notmuch just chooses a time_t value
> of 0 so you'll see whatever that 0 maps to on your system---a date of
> 1970 there is not surprising. :-)

Yeah, I figured that removing the offending messages and re-run it brought
nothing. Time to look at the source. :)

regards
Stefan Schmidt


[notmuch] 25 minutes load time with emacs -f notmuch

2009-11-21 Thread Stefan Schmidt
Hello.

On Sat, 2009-11-21 at 18:07, Carl Worth wrote:
> On Sat, 21 Nov 2009 15:51:11 +0100, Stefan Schmidt  datenfreihafen.org> wrote:
> > Disclaimer: I'm using vim, in combination with mutt for email, for years, 
> > but
> > never dealt with emacs. Please have this in mind and spot any emacs user 
> > errors
> > in this report. :)
> 
> Hi Stefan, welcome to Notmuch! And don't worry, we don't discriminate
> (too much) against non-emacs users around here.

:)

> > I have first seen notmuch several weeks ago as it seems a silent project. 
> > Being
> > more then happy now that it envolves quickly and a real developer community
> > builds around it.
> 
> Yes. Notmuch was a silent project since it was just something that I was
> doing for myself. I was always writing it as free software, and even had
> a public git repository available, but hadn't advertised it at all yet.

Yup, I had the repo on my disk a week before Keith blogged about it. Just nice
that it was going crazy that fast and people start using it and contributing to
it.

> > But now to my problem. Getting m mail indexed was easy enough:
> > 
> > stefan at excalibur:~$ du -chs not-much-mail/
> > 1.5Gnot-much-mail/
> > 1.5Gtotal
> > stefan at excalibur:~$ time notmuch new
> > Found 103677 total files.
> > Processed 103677 total files in 42m 30s (40 files/sec.).
> > Added 100899 new messages to the database (not much, really).
> 
> Good. I'm glad that went fairly smoothly for you.
> 
> Though, frankly, I think we need to fix "notmuch new" to do much better
> than 40 files/sec.

As a sidenote. That one is on a notebook with a slow 5400 disk and crypt + lvm +
ext3 on top. Perhaps I should put some money back for an X25 SSD. ;)

> > I put (require  'notmuch) in my ~/.emacs ans start emacs with the -f notmuch
> > option to enter the notmuch mode.
> 
> I'm glad you've figured that much out. I feel bad that that's not even
> in the documentation anywhere yet.

I have to admit it took me some time. Something like below should help?

> > What happends then is that a notmuch process gets started and emacs
> > waits for the return.
> 
> OK. This is a known shortcoming. As Bdale supposes, this problem is from
> notmuch trying to load and construct every thread in your
> database. There are actually several different bugs/missing features
> here that should be addressed:
> 
>   * "notmuch new" should look at the R flag in maildir files to
> determine that they are read and do not need to be marked as "inbox"
> and "unread"

I think that's what I will try to get working here. Sounds the nearest solution
to my problem. That in combination with the just merged tags-based-on-folders
patch should make me a lot happier. :)


>From 8f95e039e98addd0f4be7c31e41e534f1b519a5d Mon Sep 17 00:00:00 2001
From: Stefan Schmidt <ste...@datenfreihafen.org>
Date: Sat, 21 Nov 2009 20:31:55 +0100
Subject: [PATCH] INSTALL: emacs install dokumentation.

Write down the steps needed to install and actuall use notmuch in emacs. Should
help emacs newbies.

Signed-off-by: Stefan Schmidt 
---
 INSTALL |8 
 1 files changed, 8 insertions(+), 0 deletions(-)

diff --git a/INSTALL b/INSTALL
index de268b6..64b8e36 100644
--- a/INSTALL
+++ b/INSTALL
@@ -14,6 +14,14 @@ Notmuch are satisfied. If they are not, the configure script 
will
 notice that and provide instructions on where to obtain the necessary
 dependencies.

+notmuch.el installation
+---
+Installing the notmuch.el emacs lisp function systemwide:
+
+   sudo make install-emacs
+
+Each user needs to add (require 'notmuch) in his ~/.emacs to activate it.
+
 Dependencies
 
 Notmuch depends on three libraries: Xapian, GMime 2.4, and Talloc
-- 
1.6.5.3

regards
Stefan Schmidt


[notmuch] notmuch vim support

2009-11-21 Thread Stefan Schmidt
Hello.

[Resend, first one missed the list]

On Sat, 2009-11-21 at 14:14, Bart Trojanowski wrote:
> * Stefan Schmidt  [091121 14:06]:
> > IMHO it would be best to do the development inside the main tree. You could 
> > use
> > branches there too and it would be easier to find and test. And as notmuch 
> > is
> > still under heavy development its not a problem that the vim interface is as
> > well.
> 
> Perfect.  How would you like to proceed with the initial integration?
> Pulling my tree, or me squashing all the changes into a single patch?
> Also, are you OK with where the files live (ie under a vim directory).

Just started to use it. Its Carls trun to decide such things. :)

regards
Stefan Schmidt


[notmuch] [PATCH] notmuch-config: Fix memleaks.

2009-11-21 Thread Stefan Schmidt
>From 4bb598e6e0fa32d5184be54cb5c66fe364eba374 Mon Sep 17 00:00:00 2001
From: Holger Freyther <ze...@selfish.org>
Date: Sat, 21 Nov 2009 22:45:23 +0100
Subject: [PATCH] notmuch-config: Fix memleaks.

While talloc is great we need to free the g_error by hand.

Tested-by: Stefan Schmidt 
Signed-off-by: Holger Freyther 
---
 notmuch-config.c |3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)

diff --git a/notmuch-config.c b/notmuch-config.c
index 7252a19..aaa0372 100644
--- a/notmuch-config.c
+++ b/notmuch-config.c
@@ -207,9 +207,11 @@ notmuch_config_open (void *ctx,
fprintf (stderr, "Error reading configuration file %s: %s\n",
 config->filename, error->message);
talloc_free (config);
+   g_error_free (error);
return NULL;
}

+   g_error_free (error);
is_new = 1;
 }

@@ -308,6 +310,7 @@ notmuch_config_save (notmuch_config_t *config)
 if (! g_file_set_contents (config->filename, data, length, )) {
fprintf (stderr, "Error saving configuration to %s: %s\n",
 config->filename, error->message);
+   g_error_free (error);
return 1;
 }

-- 
1.6.5.3



[notmuch] [PATCH 2/2] notmuch-new: Tag mails not as unread when the seen flag in the maildir is set.

2009-11-22 Thread Stefan Schmidt
With the new notmuch_message_get_flags() function we can get the information if
a message was already flagged as seen in maildir by another mailer or tool. This
gives a more realistic picture instead of flagging all as unread.

Signed-off-by: Stefan Schmidt 
---
 Makefile  |2 +-
 notmuch-new.c |   16 +++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 3fedcf1..dfcfc70 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,6 @@
 # Default FLAGS, (can be overridden by user such as "make CFLAGS=-O2")
 WARN_FLAGS=-Wall -Wextra -Wmissing-declarations -Wwrite-strings -Wswitch-enum
-CFLAGS=-O2
+CFLAGS=-O0 -ggdb3

 # Additional flags that we will append to whatever the user set.
 # These aren't intended for the user to manipulate.
diff --git a/notmuch-new.c b/notmuch-new.c
index bc35b4e..ef4429d 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -41,8 +41,22 @@ handle_sigint (unused (int sig))
 static void
 tag_inbox_and_unread (notmuch_message_t *message)
 {
-notmuch_message_add_tag (message, "inbox");
+char *buf;
+int i;
+
+buf = notmuch_message_get_flags (message);
+for (i = 0; i < strlen (buf); i++) {
+/* If the S flag is set the message can be tagged as read */
+if (buf[i] == 'S') {
+notmuch_message_add_tag (message, "read");
+goto inbox;
+}
+}
+
 notmuch_message_add_tag (message, "unread");
+
+inbox:
+notmuch_message_add_tag (message, "inbox");
 }

 static void
-- 
1.6.5.3



[notmuch] [PATCH 1/2] lib/message: Add function to get maildir flags.

2009-11-22 Thread Stefan Schmidt
Hello.

On Sat, 2009-11-21 at 16:28, Keith Packard wrote:
> On Sun, 22 Nov 2009 01:11:00 +0100, Stefan Schmidt  datenfreihafen.org> wrote:
> 
> > +const char *
> > +notmuch_message_get_flags (notmuch_message_t *message)
> 
> This function should interpret the flags that it finds and return a
> suitable set of notmuch tags. I'd suggest that 'unread' messages get
> both 'unread' and 'inbox' tags, as Maildir doesn't distinguish between
> 'don't show this to be by default again please' and 'I've read this
> message'. It seems best to hide the maildir-specific details inside the
> library instead of exposing them.

Thanks for the review. On a second thought the interface was really a bit ugly.
:)

I'm just back to my box and going through the outstanding mails shows me that
Michiel Buddingh has a more complete patch on the
convert-maildir-flags-into-tags issue which Carl has tagged for review. Will
wait what comes out of it and if anything is left for me to. :)

regards
Stefan Schmidt
-- next part --
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 250 bytes
Desc: Digital signature
URL: 
<http://notmuchmail.org/pipermail/notmuch/attachments/20091122/3c68c8a9/attachment.pgp>


[notmuch] [PATCH 1/2] lib/message: Add function to get maildir flags.

2009-11-22 Thread Stefan Schmidt
Hello.

On Sun, 2009-11-22 at 19:55, Michiel Buddingh' wrote:
> Stefan Schmidt  wrote:
> > > This function should interpret the flags that it finds and return a
> > > suitable set of notmuch tags. I'd suggest that 'unread' messages get
> > > both 'unread' and 'inbox' tags, as Maildir doesn't distinguish between
> > > 'don't show this to be by default again please' and 'I've read this
> > > message'. It seems best to hide the maildir-specific details inside the
> > > library instead of exposing them.
> >
> > Thanks for the review. On a second thought the interface was really a bit 
> > ugly.
> > :)
> >
> > I'm just back to my box and going through the outstanding mails shows me 
> > that
> > Michiel Buddingh has a more complete patch on the
> > convert-maildir-flags-into-tags issue which Carl has tagged for review. Will
> > wait what comes out of it and if anything is left for me to. :)
> 
> Apologies.  In my haste to cover up my appalling and incorrect first patch, I
> neglected to review the archives to see if someone had already done this. 
> Sorry
> for stealing your thunder.

No need to be sorry. I'm interestecd in having this itch scratched, but don't
care who is doing it. :)

Just go ahead and get it in after Carl's review.

regards
Stefan Schmidt


[notmuch] [PATCH] Makefile: Enable backslash escapes for echo.

2009-11-26 Thread Stefan Schmidt
This fixes a visual glitch during a silent compile.
Before:
Use "make V=1" to see the verbose compile lines.\n  CC  debugger.o
  CCgmime-filter-reply.o

After:
Use "make V=1" to see the verbose compile lines.
  CCdebugger.o
  CCgmime-filter-reply.o

Signed-off-by: Stefan Schmidt 
---
 Makefile |2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/Makefile b/Makefile
index 2cd1b1b..2d19a6e 100644
--- a/Makefile
+++ b/Makefile
@@ -41,7 +41,7 @@ include Makefile.config
 # user how to enable verbose compiles.
 ifeq ($(V),)
 quiet_DOC := "Use \"$(MAKE) V=1\" to see the verbose compile lines.\n"
-quiet = @echo $(quiet_DOC)$(eval quiet_DOC:=)"  $1 $@"; $($1)
+quiet = @echo -e $(quiet_DOC)$(eval quiet_DOC:=)"  $1  $@"; $($1)
 endif
 # The user has explicitly enabled quiet compilation.
 ifeq ($(V),0)
-- 
1.6.5.3