[PATCH] Add configurable changed tag to messages that have been changed on disk
Excerpts from David Mazieres's message of 2014-04-06 22:19:19 +0200: > Gaute Hope writes: > > > When one of the source files for a message is changed on disk, renamed, > > deleted or a new source file is added. A configurable changed tag is > > is added. The tag can be configured under the option 'changed_tags' in > > the [new] section, the default is none. Tests have been updated to > > accept the new config option. > > > > notmuch-setup now asks for a changed tag after the new tags question. > > > > This could be useful for for example 'afew' to detect remote changes in > > IMAP folders and update the FolderNameFilter to also add tags or remove > > tags when a _existing_ message has been added to or removed from a > > maildir. > > I think this is the wrong way to achieve such functionality, because > then the change tag A) is expensive to remove, B) is easy to misuse > (remember to call fsync everywhere before deleting the change tag), and > C) can be used by only one application. > > A better approach would be to add a new "modtime" xapian value that is > updated whenever the tags or any other terms (such as XFDIRENTRY) are > added to or deleted from a docid. If it's a Xapian value, rather than a > term, then modtime will be queriable just like date, allowing multiple > applications to query all docids modified since the last time they ran. > > [... snip] This could also solve it, and probably have more uses. I don't quite see how the opposite problem (for my use case) can be solved by this without using a 'localchange' tag. This is to sync tag to maildir sync, when a new tag has been added (by e.g. a user interaction in a client) it needs to be copied to the maildir, if it is not done in the same go a different application won't know whether the change was local or remote. How did you solve this? I would suggest using a Xapian- or Index-time which gets a tick everytime a modification is made to the index. Atomic operations could operate on the same time in case this distinction turns out to be useful. Perhaps something like this already exists in Xapian? This way clock skew, clock resolution (lots of operations happening in the same second, msec or nanosec) problems won't be an issue. The crux will be to make sure all write-operations trigger a tick on the indextime. Regards, Gaute
Synchronization success stories?
I'm thrilled by using notmuch to manage my mail. Low-latency search is very important to me. But I use computers in a couple of places---several of which are laptops. Has anyone stories to share of successful multi-computer notmuch sync, for a corpus of a quarter-million messages or so? I've tried offlineimap---it (and my Exchange sever) get grouchy with mailboxes of that size. I tried keeping ~/Maildir/ in Google Drive; it took weeks to do the initial sync and I gave up. I'm trying bittorrent-sync now, with no obivous failures. -Brian -- Brian Sniffen Information Security Akamai Technologies
[Patch v7 5/6] notmuch-new: backup tags before database upgrade
On Sat, 05 Apr 2014, David Bremner wrote:
> All we do here is calculate the backup filename, and call the existing
> dump routine.
>
> Also take the opportity to add a message about being safe to
opportunity
> interrupt.
> ---
> notmuch-new.c| 29 -
> test/T530-upgrade.sh | 4 +++-
> 2 files changed, 31 insertions(+), 2 deletions(-)
>
> diff --git a/notmuch-new.c b/notmuch-new.c
> index 82acf69..d269c7c 100644
> --- a/notmuch-new.c
> +++ b/notmuch-new.c
> @@ -989,8 +989,35 @@ notmuch_new_command (notmuch_config_t *config, int argc,
> char *argv[])
> return EXIT_FAILURE;
>
> if (notmuch_database_needs_upgrade (notmuch)) {
> - if (add_files_state.verbosity >= VERBOSITY_NORMAL)
> + time_t now = time (NULL);
> + struct tm *gm_time = gmtime ();
> +
> + /* since dump files are written atomically, the amount of
> + * harm from overwriting one within a second seems
> + * relatively small. */
> +
> + const char *backup_name =
> + talloc_asprintf (notmuch,
> "%s/dump-%04d%02d%02dT%02d%02d%02d.gz",
> + dot_notmuch_path,
> + gm_time->tm_year + 1900,
> + gm_time->tm_mon + 1,
> + gm_time->tm_mday,
> + gm_time->tm_hour,
> + gm_time->tm_min,
> + gm_time->tm_sec);
> +
> + if (add_files_state.verbosity >= VERBOSITY_NORMAL) {
> printf ("Welcome to a new version of notmuch! Your database
> will now be upgraded.\n");
> + printf ("This process is safe to interrupt.\n");
> + printf ("Backing up tags to %s...\n", backup_name);
> + }
> +
> + if (notmuch_database_dump (notmuch, backup_name, "",
> +DUMP_FORMAT_BATCH_TAG, TRUE)) {
> + fprintf (stderr, "Backup failed. Aborting upgrade.");
> + return EXIT_FAILURE;
> + }
> +
> gettimeofday (_files_state.tv_start, NULL);
> notmuch_database_upgrade (notmuch,
> add_files_state.verbosity >=
> VERBOSITY_NORMAL ? upgrade_print_progress : NULL,
> diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
> index 67bbf31..d46e3d1 100755
> --- a/test/T530-upgrade.sh
> +++ b/test/T530-upgrade.sh
> @@ -26,9 +26,11 @@ output=$(notmuch search path:foo)
> test_expect_equal "$output" ""
>
> test_begin_subtest "database upgrade from format version 1"
> -output=$(notmuch new)
> +output=$(notmuch new | sed -e 's/^Backing up tags to .*$/Backing up tags to
> FILENAME/')
> test_expect_equal "$output" "\
> Welcome to a new version of notmuch! Your database will now be upgraded.
> +This process is safe to interrupt.
> +Backing up tags to FILENAME
> Your notmuch database has now been upgraded to database format version 2.
> No new mail."
>
> --
> 1.9.0
>
> ___
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
[Patch v7 4/6] restore: transparently support gzipped input
On Sat, 05 Apr 2014, David Bremner wrote:
> We rely completely on zlib to do the right thing in detecting gzipped
> input. Since our dump format is chosen to be 7 bit ascii, this should
> be fine.
> ---
> doc/man1/notmuch-restore.rst | 8
> notmuch-restore.c| 93
> +---
> test/T240-dump-restore.sh| 14 +++
> 3 files changed, 92 insertions(+), 23 deletions(-)
>
> diff --git a/doc/man1/notmuch-restore.rst b/doc/man1/notmuch-restore.rst
> index d6cf19a..936b138 100644
> --- a/doc/man1/notmuch-restore.rst
> +++ b/doc/man1/notmuch-restore.rst
> @@ -50,6 +50,14 @@ Supported options for **restore** include
> format, this heuristic, based the fact that batch-tag format
> contains no parentheses, should be accurate.
>
> +GZIPPED INPUT
> +=
> +
> +\ **notmuch restore** will detect if the input is compressed in
> +**gzip(1)** format and automatically decompress it while reading. This
> +detection does not depend on file naming and in particular works for
> +standard input.
> +
> SEE ALSO
>
>
> diff --git a/notmuch-restore.c b/notmuch-restore.c
> index c54d513..7abee0a 100644
> --- a/notmuch-restore.c
> +++ b/notmuch-restore.c
> @@ -22,6 +22,7 @@
> #include "hex-escape.h"
> #include "tag-util.h"
> #include "string-util.h"
> +#include "zlib-extra.h"
>
> static regex_t regex;
>
> @@ -128,10 +129,10 @@ notmuch_restore_command (notmuch_config_t *config, int
> argc, char *argv[])
> tag_op_list_t *tag_ops;
>
> char *input_file_name = NULL;
> -FILE *input = stdin;
> +const char *name_for_error = NULL;
> +gzFile input = NULL;
> char *line = NULL;
> void *line_ctx = NULL;
> -size_t line_size;
> ssize_t line_len;
>
> int ret = 0;
> @@ -157,39 +158,69 @@ notmuch_restore_command (notmuch_config_t *config, int
> argc, char *argv[])
> };
>
> opt_index = parse_arguments (argc, argv, options, 1);
> -if (opt_index < 0)
> - return EXIT_FAILURE;
> +if (opt_index < 0) {
> + ret = EXIT_FAILURE;
> + goto DONE;
> +}
> +
> +name_for_error = input_file_name ? input_file_name : "stdin";
>
> if (! accumulate)
> flags |= TAG_FLAG_REMOVE_ALL;
>
> -if (input_file_name) {
> - input = fopen (input_file_name, "r");
> - if (input == NULL) {
> - fprintf (stderr, "Error opening %s for reading: %s\n",
> - input_file_name, strerror (errno));
> - return EXIT_FAILURE;
> +errno = 0;
> +if (input_file_name)
> + input = gzopen (input_file_name, "r");
> +else {
> + int infd = dup (STDIN_FILENO);
> + if (infd < 0) {
> + fprintf (stderr, "Error duping stdin: %s\n",
> + strerror (errno));
> + ret = EXIT_FAILURE;
> + goto DONE;
> }
> + input = gzdopen (infd, "r");
> + if (! input)
> + close (infd);
> +}
> +
> +if (input == NULL) {
> + fprintf (stderr, "Error opening %s for (gzip) reading: %s\n",
> + name_for_error, strerror (errno));
> + ret = EXIT_FAILURE;
> + goto DONE;
> }
>
> if (opt_index < argc) {
> fprintf (stderr, "Unused positional parameter: %s\n", argv[opt_index]);
> - return EXIT_FAILURE;
> + ret = EXIT_FAILURE;
> + goto DONE;
> }
>
> tag_ops = tag_op_list_create (config);
> if (tag_ops == NULL) {
> fprintf (stderr, "Out of memory.\n");
> - return EXIT_FAILURE;
> + ret = EXIT_FAILURE;
> + goto DONE;
> }
>
> do {
> - line_len = getline (, _size, input);
> + util_status_t status;
> +
> + status = gz_getline (line_ctx, , _len, input);
>
> /* empty input file not considered an error */
> - if (line_len < 0)
> - return EXIT_SUCCESS;
> + if (status == UTIL_EOF) {
> + ret = EXIT_SUCCESS;
> + goto DONE;
> + }
>
> + if (status) {
> + fprintf (stderr, "Error reading (gzipped) input: %s\n",
> + gz_error_string(status, input));
> + ret = EXIT_FAILURE;
> + goto DONE;
> + }
> } while ((line_len == 0) ||
>(line[0] == '#') ||
>/* the cast is safe because we checked about for line_len < 0 */
> @@ -254,21 +285,37 @@ notmuch_restore_command (notmuch_config_t *config, int
> argc, char *argv[])
> if (ret)
> break;
>
> -} while ((line_len = getline (, _size, input)) != -1);
> +} while (! (ret = gz_getline (line_ctx, , _len, input)));
> +
>
> -if (line_ctx != NULL)
> - talloc_free (line_ctx);
> +/* EOF is normal loop termination condition, UTIL_SUCCESS is
> + * impossible here */
> +if (ret == UTIL_EOF) {
> + ret = UTIL_SUCCESS;
> +} else {
> + fprintf (stderr, "Error reading (gzipped) input: %s\n",
> + gz_error_string (ret, input));
ret = EXIT_FAILURE; here?
> +}
> +
> +/* currently
[PATCH] Add configurable changed tag to messages that have been changed on disk
Gaute Hope writes: >> A better approach would be to add a new "modtime" xapian value that is >> updated whenever the tags or any other terms (such as XFDIRENTRY) are >> added to or deleted from a docid. If it's a Xapian value, rather than a >> term, then modtime will be queriable just like date, allowing multiple >> applications to query all docids modified since the last time they ran. >> >> [... snip] > > This could also solve it, and probably have more uses. I don't quite see > how the opposite problem (for my use case) can be solved by this without > using a 'localchange' tag. This is to sync tag to maildir sync, when a > new tag has been added (by e.g. a user interaction in a client) it needs > to be copied to the maildir, if it is not done in the same go a > different application won't know whether the change was local or remote. > How did you solve this? Why don't you just set maildir.synchronize_flags=true? When I synchronize mail across machines, I start by concurrently running "notmuch new" on both the local and remote machines, which picks up all the changed maildir flags. Then I synchronize the mail and the tags between the two maildirs. If maildir.synchronize=true, then atomically with setting the new tags I call notmuch_message_tags_to_maildir_flags() to sync the new tags to the maildir. The maildir flags question seems kind of independent of what we are talking about, which is just having an incremental way of examining the database. Right now, I have to scan everything to find tags that have changed since the last synchronization event. If I had modtime (or really it should be called "ctime", like inode change time), then I could look at only the few messages that changed, and it would probably shave 250msec off polling new mail for a 100,000-message maildir. Note you can't use the file system ctime/mtime because the file system may have changed since the last time you ran notmuch new. > I would suggest using a Xapian- or Index-time which gets a tick > everytime a modification is made to the index. Exactly. It could be a tick, or just the current time of day if your clock does not go backwards. (I'd be willing to do a full scan if the clock ever goes backwards.) The advantage of time is that you don't have to synchronously update some counter. > Atomic operations could operate on the same time in case this > distinction turns out to be useful. Perhaps something like this > already exists in Xapian? I don't think it's important for atomic operations to have the same timestamp. All that's important is that you be able to diff the database between the last time you scanned it. > This way clock skew, clock resolution (lots of operations happening in > the same second, msec or nanosec) problems won't be an issue. The crux > will be to make sure all write-operations trigger a tick on the > indextime. Clock skew is not really an issue. It takes years to amass hundreds of thousands of email messages. So adding 5 minutes of slop is not a big deal--you'll just scan a few messages needlessly. Making sure the write-operations update the time should be easy. Most or all of the changes are probably funneled through _notmuch_message_sync. Worst case, there are only 9 places in the source code that make use of a Xapian:WritableDatabase, so I'm pretty confident total changes wouldn't be much more than 50 lines of code. I would do it myself if there were any kind of indication that such a change could be upstreamed. I brought this up in January, 2011, and didn't get a huge amount of interest in the ctime idea. But I was also a lot less focused on what I needed. Now that I have a working distributed setup and am actually using notmuch for my mail, I have a much better understanding of what is needed. David
Re: [Patch v7 4/6] restore: transparently support gzipped input
On Sat, 05 Apr 2014, David Bremner da...@tethera.net wrote:
We rely completely on zlib to do the right thing in detecting gzipped
input. Since our dump format is chosen to be 7 bit ascii, this should
be fine.
---
doc/man1/notmuch-restore.rst | 8
notmuch-restore.c| 93
+---
test/T240-dump-restore.sh| 14 +++
3 files changed, 92 insertions(+), 23 deletions(-)
diff --git a/doc/man1/notmuch-restore.rst b/doc/man1/notmuch-restore.rst
index d6cf19a..936b138 100644
--- a/doc/man1/notmuch-restore.rst
+++ b/doc/man1/notmuch-restore.rst
@@ -50,6 +50,14 @@ Supported options for **restore** include
format, this heuristic, based the fact that batch-tag format
contains no parentheses, should be accurate.
+GZIPPED INPUT
+=
+
+\ **notmuch restore** will detect if the input is compressed in
+**gzip(1)** format and automatically decompress it while reading. This
+detection does not depend on file naming and in particular works for
+standard input.
+
SEE ALSO
diff --git a/notmuch-restore.c b/notmuch-restore.c
index c54d513..7abee0a 100644
--- a/notmuch-restore.c
+++ b/notmuch-restore.c
@@ -22,6 +22,7 @@
#include hex-escape.h
#include tag-util.h
#include string-util.h
+#include zlib-extra.h
static regex_t regex;
@@ -128,10 +129,10 @@ notmuch_restore_command (notmuch_config_t *config, int
argc, char *argv[])
tag_op_list_t *tag_ops;
char *input_file_name = NULL;
-FILE *input = stdin;
+const char *name_for_error = NULL;
+gzFile input = NULL;
char *line = NULL;
void *line_ctx = NULL;
-size_t line_size;
ssize_t line_len;
int ret = 0;
@@ -157,39 +158,69 @@ notmuch_restore_command (notmuch_config_t *config, int
argc, char *argv[])
};
opt_index = parse_arguments (argc, argv, options, 1);
-if (opt_index 0)
- return EXIT_FAILURE;
+if (opt_index 0) {
+ ret = EXIT_FAILURE;
+ goto DONE;
+}
+
+name_for_error = input_file_name ? input_file_name : stdin;
if (! accumulate)
flags |= TAG_FLAG_REMOVE_ALL;
-if (input_file_name) {
- input = fopen (input_file_name, r);
- if (input == NULL) {
- fprintf (stderr, Error opening %s for reading: %s\n,
- input_file_name, strerror (errno));
- return EXIT_FAILURE;
+errno = 0;
+if (input_file_name)
+ input = gzopen (input_file_name, r);
+else {
+ int infd = dup (STDIN_FILENO);
+ if (infd 0) {
+ fprintf (stderr, Error duping stdin: %s\n,
+ strerror (errno));
+ ret = EXIT_FAILURE;
+ goto DONE;
}
+ input = gzdopen (infd, r);
+ if (! input)
+ close (infd);
+}
+
+if (input == NULL) {
+ fprintf (stderr, Error opening %s for (gzip) reading: %s\n,
+ name_for_error, strerror (errno));
+ ret = EXIT_FAILURE;
+ goto DONE;
}
if (opt_index argc) {
fprintf (stderr, Unused positional parameter: %s\n, argv[opt_index]);
- return EXIT_FAILURE;
+ ret = EXIT_FAILURE;
+ goto DONE;
}
tag_ops = tag_op_list_create (config);
if (tag_ops == NULL) {
fprintf (stderr, Out of memory.\n);
- return EXIT_FAILURE;
+ ret = EXIT_FAILURE;
+ goto DONE;
}
do {
- line_len = getline (line, line_size, input);
+ util_status_t status;
+
+ status = gz_getline (line_ctx, line, line_len, input);
/* empty input file not considered an error */
- if (line_len 0)
- return EXIT_SUCCESS;
+ if (status == UTIL_EOF) {
+ ret = EXIT_SUCCESS;
+ goto DONE;
+ }
+ if (status) {
+ fprintf (stderr, Error reading (gzipped) input: %s\n,
+ gz_error_string(status, input));
+ ret = EXIT_FAILURE;
+ goto DONE;
+ }
} while ((line_len == 0) ||
(line[0] == '#') ||
/* the cast is safe because we checked about for line_len 0 */
@@ -254,21 +285,37 @@ notmuch_restore_command (notmuch_config_t *config, int
argc, char *argv[])
if (ret)
break;
-} while ((line_len = getline (line, line_size, input)) != -1);
+} while (! (ret = gz_getline (line_ctx, line, line_len, input)));
+
-if (line_ctx != NULL)
- talloc_free (line_ctx);
+/* EOF is normal loop termination condition, UTIL_SUCCESS is
+ * impossible here */
+if (ret == UTIL_EOF) {
+ ret = UTIL_SUCCESS;
+} else {
+ fprintf (stderr, Error reading (gzipped) input: %s\n,
+ gz_error_string (ret, input));
ret = EXIT_FAILURE; here?
+}
+
+/* currently this should not be after DONE: since we don't
+ * know if the xregcomp was reached
+ */
if (input_format == DUMP_FORMAT_SUP)
Re: [Patch v7 5/6] notmuch-new: backup tags before database upgrade
On Sat, 05 Apr 2014, David Bremner da...@tethera.net wrote:
All we do here is calculate the backup filename, and call the existing
dump routine.
Also take the opportity to add a message about being safe to
opportunity
interrupt.
---
notmuch-new.c| 29 -
test/T530-upgrade.sh | 4 +++-
2 files changed, 31 insertions(+), 2 deletions(-)
diff --git a/notmuch-new.c b/notmuch-new.c
index 82acf69..d269c7c 100644
--- a/notmuch-new.c
+++ b/notmuch-new.c
@@ -989,8 +989,35 @@ notmuch_new_command (notmuch_config_t *config, int argc,
char *argv[])
return EXIT_FAILURE;
if (notmuch_database_needs_upgrade (notmuch)) {
- if (add_files_state.verbosity = VERBOSITY_NORMAL)
+ time_t now = time (NULL);
+ struct tm *gm_time = gmtime (now);
+
+ /* since dump files are written atomically, the amount of
+ * harm from overwriting one within a second seems
+ * relatively small. */
+
+ const char *backup_name =
+ talloc_asprintf (notmuch,
%s/dump-%04d%02d%02dT%02d%02d%02d.gz,
+ dot_notmuch_path,
+ gm_time-tm_year + 1900,
+ gm_time-tm_mon + 1,
+ gm_time-tm_mday,
+ gm_time-tm_hour,
+ gm_time-tm_min,
+ gm_time-tm_sec);
+
+ if (add_files_state.verbosity = VERBOSITY_NORMAL) {
printf (Welcome to a new version of notmuch! Your database
will now be upgraded.\n);
+ printf (This process is safe to interrupt.\n);
+ printf (Backing up tags to %s...\n, backup_name);
+ }
+
+ if (notmuch_database_dump (notmuch, backup_name, ,
+DUMP_FORMAT_BATCH_TAG, TRUE)) {
+ fprintf (stderr, Backup failed. Aborting upgrade.);
+ return EXIT_FAILURE;
+ }
+
gettimeofday (add_files_state.tv_start, NULL);
notmuch_database_upgrade (notmuch,
add_files_state.verbosity =
VERBOSITY_NORMAL ? upgrade_print_progress : NULL,
diff --git a/test/T530-upgrade.sh b/test/T530-upgrade.sh
index 67bbf31..d46e3d1 100755
--- a/test/T530-upgrade.sh
+++ b/test/T530-upgrade.sh
@@ -26,9 +26,11 @@ output=$(notmuch search path:foo)
test_expect_equal $output
test_begin_subtest database upgrade from format version 1
-output=$(notmuch new)
+output=$(notmuch new | sed -e 's/^Backing up tags to .*$/Backing up tags to
FILENAME/')
test_expect_equal $output \
Welcome to a new version of notmuch! Your database will now be upgraded.
+This process is safe to interrupt.
+Backing up tags to FILENAME
Your notmuch database has now been upgraded to database format version 2.
No new mail.
--
1.9.0
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: v7 gzip dump restore patches
On Sat, 05 Apr 2014, David Bremner da...@tethera.net wrote: Supercedes id:1396554083-3892-2-git-send-email-da...@tethera.net - adds new analogues of strerror - util_error_string - gz_error_string LGTM other than the two nits I pointed out. I'd be happy to have the fixes for those simply amended in when you apply. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
Gaute Hope e...@gaute.vetsj.com writes: A better approach would be to add a new modtime xapian value that is updated whenever the tags or any other terms (such as XFDIRENTRY) are added to or deleted from a docid. If it's a Xapian value, rather than a term, then modtime will be queriable just like date, allowing multiple applications to query all docids modified since the last time they ran. [... snip] This could also solve it, and probably have more uses. I don't quite see how the opposite problem (for my use case) can be solved by this without using a 'localchange' tag. This is to sync tag to maildir sync, when a new tag has been added (by e.g. a user interaction in a client) it needs to be copied to the maildir, if it is not done in the same go a different application won't know whether the change was local or remote. How did you solve this? Why don't you just set maildir.synchronize_flags=true? When I synchronize mail across machines, I start by concurrently running notmuch new on both the local and remote machines, which picks up all the changed maildir flags. Then I synchronize the mail and the tags between the two maildirs. If maildir.synchronize=true, then atomically with setting the new tags I call notmuch_message_tags_to_maildir_flags() to sync the new tags to the maildir. The maildir flags question seems kind of independent of what we are talking about, which is just having an incremental way of examining the database. Right now, I have to scan everything to find tags that have changed since the last synchronization event. If I had modtime (or really it should be called ctime, like inode change time), then I could look at only the few messages that changed, and it would probably shave 250msec off polling new mail for a 100,000-message maildir. Note you can't use the file system ctime/mtime because the file system may have changed since the last time you ran notmuch new. I would suggest using a Xapian- or Index-time which gets a tick everytime a modification is made to the index. Exactly. It could be a tick, or just the current time of day if your clock does not go backwards. (I'd be willing to do a full scan if the clock ever goes backwards.) The advantage of time is that you don't have to synchronously update some counter. Atomic operations could operate on the same time in case this distinction turns out to be useful. Perhaps something like this already exists in Xapian? I don't think it's important for atomic operations to have the same timestamp. All that's important is that you be able to diff the database between the last time you scanned it. This way clock skew, clock resolution (lots of operations happening in the same second, msec or nanosec) problems won't be an issue. The crux will be to make sure all write-operations trigger a tick on the indextime. Clock skew is not really an issue. It takes years to amass hundreds of thousands of email messages. So adding 5 minutes of slop is not a big deal--you'll just scan a few messages needlessly. Making sure the write-operations update the time should be easy. Most or all of the changes are probably funneled through _notmuch_message_sync. Worst case, there are only 9 places in the source code that make use of a Xapian:WritableDatabase, so I'm pretty confident total changes wouldn't be much more than 50 lines of code. I would do it myself if there were any kind of indication that such a change could be upstreamed. I brought this up in January, 2011, and didn't get a huge amount of interest in the ctime idea. But I was also a lot less focused on what I needed. Now that I have a working distributed setup and am actually using notmuch for my mail, I have a much better understanding of what is needed. David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
Excerpts from dm-list-email-notmuch's message of 2014-04-10 17:31:04 +0200: Gaute Hope e...@gaute.vetsj.com writes: A better approach would be to add a new modtime xapian value that is updated whenever the tags or any other terms (such as XFDIRENTRY) are added to or deleted from a docid. If it's a Xapian value, rather than a term, then modtime will be queriable just like date, allowing multiple applications to query all docids modified since the last time they ran. [... snip] This could also solve it, and probably have more uses. I don't quite see how the opposite problem (for my use case) can be solved by this without using a 'localchange' tag. This is to sync tag to maildir sync, when a new tag has been added (by e.g. a user interaction in a client) it needs to be copied to the maildir, if it is not done in the same go a different application won't know whether the change was local or remote. How did you solve this? Why don't you just set maildir.synchronize_flags=true? When I synchronize mail across machines, I start by concurrently running notmuch new on both the local and remote machines, which picks up all the changed maildir flags. Then I synchronize the mail and the tags between the two maildirs. If maildir.synchronize=true, then atomically with setting the new tags I call notmuch_message_tags_to_maildir_flags() to sync the new tags to the maildir. I am talking about syncing tags to a maildir _folder_, not flags. It could be implemented as maildir.synchronize is now, but it would be a larger feature which could work in a lot of different ways. The maildir flags question seems kind of independent of what we are talking about, which is just having an incremental way of examining the database. Right now, I have to scan everything to find tags that have changed since the last synchronization event. If I had modtime (or really it should be called ctime, like inode change time), then I could look at only the few messages that changed, and it would probably shave 250msec off polling new mail for a 100,000-message maildir. Note you can't use the file system ctime/mtime because the file system may have changed since the last time you ran notmuch new. If you have a unreliable clock or use a badly configured system you could risk detecting changes in the case where application time stamp is set in the future, a mod time now. Then the app won't know there has been a change. The same could happen if the clock is in the past, and the modtime is set, the clock is updated and the app won't know there has been a change. The only way to know is to do a full scan of the entire db. This could be very expansive, and comparable to initial indexing, for some actions. You would not necessarily, or reliably, be able to detect this. With an internal tick this wouldn't be an issue. I would suggest using a Xapian- or Index-time which gets a tick everytime a modification is made to the index. Exactly. It could be a tick, or just the current time of day if your clock does not go backwards. (I'd be willing to do a full scan if the clock ever goes backwards.) The advantage of time is that you don't have to synchronously update some counter. Atomic operations could operate on the same time in case this distinction turns out to be useful. Perhaps something like this already exists in Xapian? I don't think it's important for atomic operations to have the same timestamp. All that's important is that you be able to diff the database between the last time you scanned it. Yeah, it is not necessary for anything I am planning on doing, but it would be a way for other apps to know that a set of changes were done at the same time. This way clock skew, clock resolution (lots of operations happening in the same second, msec or nanosec) problems won't be an issue. The crux will be to make sure all write-operations trigger a tick on the indextime. Clock skew is not really an issue. It takes years to amass hundreds of thousands of email messages. So adding 5 minutes of slop is not a big deal--you'll just scan a few messages needlessly. Yes, but you risk missing changes without knowing. That is an issue for my use case. Making sure the write-operations update the time should be easy. Most or all of the changes are probably funneled through _notmuch_message_sync. Worst case, there are only 9 places in the source code that make use of a Xapian:WritableDatabase, so I'm pretty confident total changes wouldn't be much more than 50 lines of code. Yes :) I would do it myself if there were any kind of indication that such a change could be upstreamed. I brought this up in January, 2011, and didn't get a huge amount of interest in the ctime idea. But I was also a lot less focused on what I needed. Now that I have a working distributed setup and am actually using notmuch for my mail, I have a much better understanding of what is
Re: [PATCH v4 2/3] emacs: add notmuch-version.el.tmpl and create notmuch-version.el from it
Tomi Ollila tomi.oll...@iki.fi writes: The notmuch cli program and emacs lisp versions may differ (especially in remote usage). It helps to resolve problems if we can determine the versions of notmuch cli and notmuch emacs mua separately. The build process now creates notmuch-version.el from template file by filling the version info to notmuch-emacs-version variable. --- Alternative to id:1395261431-24668-2-git-send-email-tomi.oll...@iki.fi only change being in notmuch-emacs-version docstring. pushed this series with the two alternate patches d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
