Re: [announce] Bower 0.4
> This is the command to save drafts: > > notmuch-deliver Drafts --tag=draft \ > --remove-tag=inbox --remove-tag=unread < message_file > > Does it work if you run it manually? Just make a dummy message_file > with Date:, From:, To:, Subject: headers and a body. The manual command works! So I think I understand my problem. The reason is that in put in my .config/bower/bower.conf: notmuch_deliver=/bin/true To solve the problem of "duplicated sent email" with Gmail: http://support.google.com/mail/bin/answer.py?hl=en&answer=78892 But I didn't realize that notmuch_deliver was also used for postponed messages. What is the good config? (I see that you're using gmail too :-) ...)? thanks, David ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[Patch v2 6/6] test: add broken roundtrip test
We demonstrate the current notmuch restore parser being confused by
message-id's and tags containing non alpha numeric characters
(particularly space and parentheses are problematic because they are
not escaped by notmuch dump).
We save the files as hex escaped on disk so that the output from the
failing test will not confuse the terminal emulator of people running
the test.
---
test/dump-restore |9 +
1 file changed, 9 insertions(+)
diff --git a/test/dump-restore b/test/dump-restore
index f25f7cf..6923952 100755
--- a/test/dump-restore
+++ b/test/dump-restore
@@ -78,4 +78,13 @@ test_begin_subtest "dump --output=outfile -- from:cworth"
notmuch dump --output=dump-outfile-dash-inbox.actual -- from:cworth
test_expect_equal_file dump-cworth.expected dump-outfile-dash-inbox.actual
+test_expect_success 'roundtripping random message-ids and tags' \
+'test_subtest_known_broken && set -o pipefail &&
+ ${TEST_DIRECTORY}/random-corpus --num-messages=10
--config-path=${NOTMUCH_CONFIG} &&
+ notmuch dump | ${TEST_DIRECTORY}/hex-xcode --direction=encode >
EXPECTED.$test_count &&
+ notmuch tag -random-corpus tag:random-corpus &&
+ ${TEST_DIRECTORY}/hex-xcode --direction=decode < EXPECTED.$test_count |
notmuch restore 2>/dev/null &&
+ notmuch dump | ${TEST_DIRECTORY}/hex-xcode --direction=encode >
OUTPUT.$test_count &&
+ test_cmp EXPECTED.$test_count OUTPUT.$test_count'
+
test_done
--
1.7.10.4
[Patch v2 5/6] test: add generator for random "stub" messages
Initial use case is testing dump and restore, so we only have
message-ids and tags.
The message ID's are nothing like RFC compliant, but it doesn't seem
any harder to roundtrip random UTF-8 strings than RFC-compliant ones.
Tags are UTF-8, even though notmuch is in principle more generous than
that.
---
test/.gitignore |1 +
test/Makefile.local |8 ++
test/basic |2 +-
test/random-corpus.c | 203 ++
4 files changed, 213 insertions(+), 1 deletion(-)
create mode 100644 test/random-corpus.c
diff --git a/test/.gitignore b/test/.gitignore
index be7ab5e..1eff7ce 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -4,4 +4,5 @@ smtp-dummy
symbol-test
arg-test
hex-xcode
+random-corpus
tmp.*
diff --git a/test/Makefile.local b/test/Makefile.local
index ab13814..501aa4a 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -16,6 +16,13 @@ $(dir)/arg-test: $(dir)/arg-test.o command-line-arguments.o
util/libutil.a
$(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o util/libutil.a
$(call quiet,CC) -I. $^ -o $@ -ltalloc
+random_corpus_deps = $(dir)/random-corpus.o $(dir)/database-test.o \
+ notmuch-config.o command-line-arguments.o \
+ lib/libnotmuch.a util/libutil.a
+
+$(dir)/random-corpus: $(random_corpus_deps)
+ $(call quiet,CC) $(CFLAGS_FINAL) $^ -o $@ $(CONFIGURE_LDFLAGS)
+
$(dir)/smtp-dummy: $(smtp_dummy_modules)
$(call quiet,CC) $^ -o $@
@@ -26,6 +33,7 @@ $(dir)/symbol-test: $(dir)/symbol-test.o
TEST_BINARIES=$(dir)/arg-test \
$(dir)/hex-xcode \
+ $(dir)/random-corpus \
$(dir)/smtp-dummy \
$(dir)/symbol-test
diff --git a/test/basic b/test/basic
index 3eebb37..d923d5a 100755
--- a/test/basic
+++ b/test/basic
@@ -54,7 +54,7 @@ test_begin_subtest 'Ensure that all available tests will be
run by notmuch-test'
eval $(sed -n -e '/^TESTS="$/,/^"$/p' $TEST_DIRECTORY/notmuch-test)
tests_in_suite=$(for i in $TESTS; do echo $i; done | sort)
available=$(find "$TEST_DIRECTORY" -maxdepth 1 -type f -executable -printf
'%f\n' | \
-sed -r -e
"/^(aggregate-results.sh|notmuch-test|smtp-dummy|test-verbose|symbol-test|arg-test|hex-escape)$/d"
| \
+sed -r -e
"/^(aggregate-results.sh|notmuch-test|smtp-dummy|test-verbose|symbol-test|arg-test|hex-escape|random-corpus)$/d"
| \
sort)
test_expect_equal "$tests_in_suite" "$available"
diff --git a/test/random-corpus.c b/test/random-corpus.c
new file mode 100644
index 000..a318267
--- /dev/null
+++ b/test/random-corpus.c
@@ -0,0 +1,203 @@
+/*
+ * Generate a random corpus of stub messages.
+ *
+ * Initial use case is testing dump and restore, so we only have
+ * message-ids and tags.
+ *
+ * Generated message-id's and tags are intentionally nasty.
+ *
+ * Copyright (c) 2012 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner
+ */
+
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "notmuch-client.h"
+#include "command-line-arguments.h"
+#include "database-test.h"
+
+/* Current largest UTF-32 value defined. Note that most of these will
+ * be printed as boxes in most fonts.
+ */
+
+#define GLYPH_MAX 0x10FFFE
+#define GLYPH_CLASSES 4
+
+static gunichar
+random_unichar ()
+{
+int start=1, stop=GLYPH_MAX;
+int class = random() % 2;
+
+/*
+ * Choose about half ascii as test characters, as ascii
+ * punctation and whitespace is the main cause of problems for
+ * the (old) restore parser
+*/
+switch (class) {
+case 0:
+ /* ascii */
+ start=0x01;
+ stop=0x7f;
+ break;
+case 1:
+ /* the rest of unicode */
+ start=0x80;
+ stop=GLYPH_MAX;
+}
+
+if (start == stop)
+ return start;
+else
+ return start + (random() % (stop - start + 1));
+}
+
+static char *
+random_utf8_string (void *ctx, size_t char_count)
+{
+
+gchar *buf = NULL;
+size_t buf_size = 0;
+
+size_t offset = 0;
+
+size_t i;
+
+buf = talloc_realloc (ctx, NULL, gchar, char_count);
+buf_size = char_count;
+
+for (i = 0; i < char_count; i++) {
+ gunichar randomchar;
+ size_t written;
+
+ /* 6 for one glyph, one for null */
+ if (buf_size - offset < 8) {
+ bu
[Patch v2 4/6] test: add database routines for testing.
Initially, provide a way to create "stub" messages in the notmuch
database without corresponding files. This is essentially cut and
paste from lib/database.cc. This is a seperate file since we don't
want to export these symbols from libnotmuch or bloat the library with
non-exported code.
---
test/database-test.c | 72 ++
test/database-test.h | 21 +++
2 files changed, 93 insertions(+)
create mode 100644 test/database-test.c
create mode 100644 test/database-test.h
diff --git a/test/database-test.c b/test/database-test.c
new file mode 100644
index 000..f0f1c8e
--- /dev/null
+++ b/test/database-test.c
@@ -0,0 +1,72 @@
+/*
+ * Database routines intended only for testing, not exported from
+ * library.
+ *
+ * Copyright (c) 2012 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner
+ */
+
+#include "notmuch-private.h"
+#include "database-test.h"
+
+notmuch_status_t
+notmuch_database_add_stub_message (notmuch_database_t *notmuch,
+ const char *message_id,
+ const char **tags)
+{
+const char **tag;
+notmuch_status_t ret;
+notmuch_private_status_t private_status;
+notmuch_message_t *message;
+
+ret = _notmuch_database_ensure_writable (notmuch);
+if (ret)
+ return ret;
+
+message = _notmuch_message_create_for_message_id (notmuch,
+ message_id,
+ &private_status);
+if (message == NULL) {
+ return COERCE_STATUS (private_status,
+ "Unexpected status value from
_notmuch_message_create_for_message_id");
+
+}
+
+if (private_status == NOTMUCH_PRIVATE_STATUS_NO_DOCUMENT_FOUND) {
+ _notmuch_message_add_term (message, "type", "mail");
+} else {
+ return NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID;
+}
+
+if (tags) {
+ ret = notmuch_message_freeze (message);
+ if (ret)
+ return ret;
+
+ for (tag = tags; *tag; tag++) {
+ ret = notmuch_message_add_tag (message, *tag);
+ if (ret)
+ return ret;
+ }
+}
+
+ret = notmuch_message_thaw (message);
+if (ret)
+ return ret;
+
+return NOTMUCH_STATUS_SUCCESS;
+}
diff --git a/test/database-test.h b/test/database-test.h
new file mode 100644
index 000..84f7988
--- /dev/null
+++ b/test/database-test.h
@@ -0,0 +1,21 @@
+#ifndef _DATABASE_TEST_H
+#define _DATABASE_TEST_H
+/* Add a new stub message to the given notmuch database.
+ *
+ * At least the following return values are possible:
+ *
+ * NOTMUCH_STATUS_SUCCESS: Message successfully added to database.
+ *
+ * NOTMUCH_STATUS_DUPLICATE_MESSAGE_ID: Message has the same message
+ * ID as another message already in the database.
+ *
+ * NOTMUCH_STATUS_READ_ONLY_DATABASE: Database was opened in read-only
+ * mode so no message can be added.
+ */
+
+notmuch_status_t
+notmuch_database_add_stub_message (notmuch_database_t *database,
+ const char *message_id,
+ const char **tag_list);
+
+#endif
--
1.7.10.4
[Patch v2 3/6] test/hex-escaping: new test for hex escaping routines
These are more like unit tests, to (try to) make sure the library
functionality is working before building more complicated things on
top of it.
---
test/hex-escaping | 26 ++
test/notmuch-test |1 +
2 files changed, 27 insertions(+)
create mode 100755 test/hex-escaping
diff --git a/test/hex-escaping b/test/hex-escaping
new file mode 100755
index 000..f34cc8c
--- /dev/null
+++ b/test/hex-escaping
@@ -0,0 +1,26 @@
+#!/usr/bin/env bash
+test_description="hex encoding and decoding"
+. ./test-lib.sh
+
+test_begin_subtest "round trip"
+find $TEST_DIRECTORY/corpus -type f -print | sort | xargs cat > EXPECTED
+$TEST_DIRECTORY/hex-xcode --direction=encode < EXPECTED |
$TEST_DIRECTORY/hex-xcode --direction=decode > OUTPUT
+test_expect_equal_file OUTPUT EXPECTED
+
+test_begin_subtest "punctuation"
+tag1='comic_swear=$&^%$^%\\//-+$^%$'
+tag_enc1=$($TEST_DIRECTORY/hex-xcode --direction=encode "$tag1")
+test_expect_equal "$tag_enc1"
"comic_swear=%24%26%5e%25%24%5e%25%5c%5c%2f%2f-+%24%5e%25%24"
+
+test_begin_subtest "round trip newlines"
+printf 'this\n tag\t has\n spaces\n' > EXPECTED.$test_count
+$TEST_DIRECTORY/hex-xcode --direction=encode < EXPECTED.$test_count |\
+ $TEST_DIRECTORY/hex-xcode --direction=decode > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+
+test_begin_subtest "round trip 8bit chars"
+echo '%c3%91%c3%a5%c3%b0%c3%a3%c3%a5%c3%a9-%c3%8f%c3%8a' > EXPECTED.$test_count
+$TEST_DIRECTORY/hex-xcode --direction=decode < EXPECTED.$test_count |\
+ $TEST_DIRECTORY/hex-xcode --direction=encode > OUTPUT.$test_count
+test_expect_equal_file EXPECTED.$test_count OUTPUT.$test_count
+test_done
diff --git a/test/notmuch-test b/test/notmuch-test
index cc732c3..43b5734 100755
--- a/test/notmuch-test
+++ b/test/notmuch-test
@@ -60,6 +60,7 @@ TESTS="
emacs-hello
emacs-show
missing-headers
+ hex-escaping
"
TESTS=${NOTMUCH_TESTS:=$TESTS}
--
1.7.10.4
[Patch v2 2/6] test/hex-xcode: new test binary
This program is used both as a test-bed/unit-tester for
../util/hex-escape.c, and also as a utility in future tests of dump
and restore.
---
test/.gitignore |1 +
test/Makefile.local | 10 -
test/basic |2 +-
test/hex-xcode.c| 103 +++
4 files changed, 114 insertions(+), 2 deletions(-)
create mode 100644 test/hex-xcode.c
diff --git a/test/.gitignore b/test/.gitignore
index e63c689..be7ab5e 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -3,4 +3,5 @@ corpus.mail
smtp-dummy
symbol-test
arg-test
+hex-xcode
tmp.*
diff --git a/test/Makefile.local b/test/Makefile.local
index c7f1435..ab13814 100644
--- a/test/Makefile.local
+++ b/test/Makefile.local
@@ -13,6 +13,9 @@ smtp_dummy_modules = $(smtp_dummy_srcs:.c=.o)
$(dir)/arg-test: $(dir)/arg-test.o command-line-arguments.o util/libutil.a
$(call quiet,CC) -I. $^ -o $@
+$(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o util/libutil.a
+ $(call quiet,CC) -I. $^ -o $@ -ltalloc
+
$(dir)/smtp-dummy: $(smtp_dummy_modules)
$(call quiet,CC) $^ -o $@
@@ -21,7 +24,12 @@ $(dir)/symbol-test: $(dir)/symbol-test.o
.PHONY: test check
-test-binaries: $(dir)/arg-test $(dir)/smtp-dummy $(dir)/symbol-test
+TEST_BINARIES=$(dir)/arg-test \
+ $(dir)/hex-xcode \
+ $(dir)/smtp-dummy \
+ $(dir)/symbol-test
+
+test-binaries: $(TEST_BINARIES)
test: all test-binaries
@${dir}/notmuch-test $(OPTIONS)
diff --git a/test/basic b/test/basic
index d6aed24..3eebb37 100755
--- a/test/basic
+++ b/test/basic
@@ -54,7 +54,7 @@ test_begin_subtest 'Ensure that all available tests will be
run by notmuch-test'
eval $(sed -n -e '/^TESTS="$/,/^"$/p' $TEST_DIRECTORY/notmuch-test)
tests_in_suite=$(for i in $TESTS; do echo $i; done | sort)
available=$(find "$TEST_DIRECTORY" -maxdepth 1 -type f -executable -printf
'%f\n' | \
-sed -r -e
"/^(aggregate-results.sh|notmuch-test|smtp-dummy|test-verbose|symbol-test|arg-test)$/d"
| \
+sed -r -e
"/^(aggregate-results.sh|notmuch-test|smtp-dummy|test-verbose|symbol-test|arg-test|hex-escape)$/d"
| \
sort)
test_expect_equal "$tests_in_suite" "$available"
diff --git a/test/hex-xcode.c b/test/hex-xcode.c
new file mode 100644
index 000..eec6541
--- /dev/null
+++ b/test/hex-xcode.c
@@ -0,0 +1,103 @@
+/* No, nothing to to with IDE from Apple Inc.
+ testbed for ../util/hex-escape.c.
+
+ usage:
+ hex-xcode [--direction=(encode|decode)] [--omit-newline] < file
+ hex-xcode [--direction=(encode|decode)] [--omit-newline] arg1 arg2 arg3 ...
+
+ */
+
+#include "notmuch-client.h"
+#include "hex-escape.h"
+#include
+
+
+enum direction {
+ENCODE,
+DECODE
+};
+
+static int
+xcode (void *ctx, enum direction dir, char *in, char **buf_p, size_t *size_p)
+{
+hex_status_t status;
+
+if (dir == ENCODE)
+ status = hex_encode (ctx, in, buf_p, size_p);
+else
+ status = hex_decode (ctx, in, buf_p, size_p);
+
+if (status == HEX_SUCCESS)
+ fputs (*buf_p, stdout);
+
+return status;
+}
+
+
+int
+main (int argc, char **argv)
+{
+
+
+enum direction dir = DECODE;
+int omit_newline = FALSE;
+
+notmuch_opt_desc_t options[] = {
+ { NOTMUCH_OPT_KEYWORD, &dir, "direction", 'd',
+ (notmuch_keyword_t []){ { "encode", ENCODE },
+ { "decode", DECODE },
+ { 0, 0 } } },
+ { NOTMUCH_OPT_BOOLEAN, &omit_newline, "omit-newline", 'n', 0 },
+ { 0, 0, 0, 0, 0 }
+};
+
+int opt_index = parse_arguments (argc, argv, options, 1);
+
+if (opt_index < 0)
+ exit (1);
+
+void *ctx = talloc_new (NULL);
+
+char *line = NULL;
+size_t line_size;
+ssize_t line_len;
+
+char *buffer = NULL;
+size_t buf_size = 0;
+
+notmuch_bool_t read_stdin = TRUE;
+
+for (; opt_index < argc; opt_index++) {
+
+ if (xcode (ctx, dir, argv[opt_index],
+ &buffer, &buf_size) != HEX_SUCCESS)
+ return 1;
+
+ if (!omit_newline)
+ putchar ('\n');
+
+ read_stdin = FALSE;
+}
+
+if (!read_stdin)
+ return 0;
+
+while ((line_len = getline (&line, &line_size, stdin)) != -1) {
+
+ chomp_newline (line);
+
+ if (xcode (ctx, dir, line, &buffer, &buf_size) != HEX_SUCCESS)
+ return 1;
+
+ if (!omit_newline)
+ putchar ('\n');
+
+}
+
+if (line)
+ free (line);
+
+talloc_free (ctx);
+
+return 0;
+}
--
1.7.10.4
[Patch v2 1/6] hex-escape: (en|de)code strings to/from restricted character set
The character set is chosen to be suitable for pathnames, and the same
as that used by contrib/nmbug
[With additions by Jani Nikula]
---
util/Makefile.local |2 +-
util/hex-escape.c | 168 +++
util/hex-escape.h | 41 +
3 files changed, 210 insertions(+), 1 deletion(-)
create mode 100644 util/hex-escape.c
create mode 100644 util/hex-escape.h
diff --git a/util/Makefile.local b/util/Makefile.local
index c7cae61..3ca623e 100644
--- a/util/Makefile.local
+++ b/util/Makefile.local
@@ -3,7 +3,7 @@
dir := util
extra_cflags += -I$(srcdir)/$(dir)
-libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c
+libutil_c_srcs := $(dir)/xutil.c $(dir)/error_util.c $(dir)/hex-escape.c
libutil_modules := $(libutil_c_srcs:.c=.o)
diff --git a/util/hex-escape.c b/util/hex-escape.c
new file mode 100644
index 000..d8905d0
--- /dev/null
+++ b/util/hex-escape.c
@@ -0,0 +1,168 @@
+/* hex-escape.c - Manage encoding and decoding of byte strings into path names
+ *
+ * Copyright (c) 2011 David Bremner
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 3 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see http://www.gnu.org/licenses/ .
+ *
+ * Author: David Bremner
+ */
+
+#include
+#include
+#include
+#include
+#include "error_util.h"
+#include "hex-escape.h"
+
+static const size_t default_buf_size = 1024;
+
+static const char *output_charset =
+"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-_@=.:,";
+
+static const char escape_char = '%';
+
+static int
+is_output (char c)
+{
+return (strchr (output_charset, c) != NULL);
+}
+
+static int
+maybe_realloc (void *ctx, size_t needed, char **out, size_t *out_size)
+{
+if (*out_size < needed) {
+
+ if (*out == NULL)
+ *out = talloc_size (ctx, needed);
+ else
+ *out = talloc_realloc (ctx, *out, char, needed);
+
+ if (*out == NULL)
+ return 0;
+
+ *out_size = needed;
+}
+return 1;
+}
+
+hex_status_t
+hex_encode (void *ctx, const char *in, char **out, size_t *out_size)
+{
+
+const unsigned char *p;
+char *q;
+
+size_t escape_count = 0;
+size_t len = 0;
+size_t needed;
+
+assert (ctx); assert (in); assert (out); assert (out_size);
+
+for (p = (unsigned char *) in; *p; p++) {
+ escape_count += (!is_output (*p));
+ len++;
+}
+
+needed = len + escape_count * 2 + 1;
+
+if (*out == NULL)
+ *out_size = 0;
+
+if (!maybe_realloc (ctx, needed, out, out_size))
+ return HEX_OUT_OF_MEMORY;
+
+q = *out;
+p = (unsigned char *) in;
+
+while (*p) {
+ if (is_output (*p)) {
+ *q++ = *p++;
+ } else {
+ sprintf (q, "%%%02x", *p++);
+ q += 3;
+ }
+}
+
+*q = '\0';
+return HEX_SUCCESS;
+}
+
+/* Hex decode 'in' to 'out'.
+ *
+ * This must succeed for in == out to support hex_decode_inplace().
+ */
+static hex_status_t
+hex_decode_internal (const char *in, unsigned char *out)
+{
+char buf[3];
+
+while (*in) {
+ if (*in == escape_char) {
+ char *endp;
+
+ /* This also handles unexpected end-of-string. */
+ if (!isxdigit ((unsigned char) in[1]) ||
+ !isxdigit ((unsigned char) in[2]))
+ return HEX_SYNTAX_ERROR;
+
+ buf[0] = in[1];
+ buf[1] = in[2];
+ buf[2] = '\0';
+
+ *out = strtoul (buf, &endp, 16);
+
+ if (endp != buf + 2)
+ return HEX_SYNTAX_ERROR;
+
+ in += 3;
+ out++;
+ } else {
+ *out++ = *in++;
+ }
+}
+
+*out = '\0';
+
+return HEX_SUCCESS;
+}
+
+hex_status_t
+hex_decode_inplace (char *s)
+{
+/* A decoded string is never longer than the encoded one, so it is
+ * safe to decode a string onto itself. */
+return hex_decode_internal (s, (unsigned char *) s);
+}
+
+hex_status_t
+hex_decode (void *ctx, const char *in, char **out, size_t * out_size)
+{
+const char *p;
+size_t escape_count = 0;
+size_t needed = 0;
+
+assert (ctx); assert (in); assert (out); assert (out_size);
+
+size_t len = strlen (in);
+
+for (p = in; *p; p++)
+ escape_count += (*p == escape_char);
+
+needed = len - escape_count * 2 + 1;
+
+if (!maybe_realloc (ctx, needed, out, out_size))
+ return HEX_OUT_OF_MEMORY;
+
+return hex_decode_internal (in, (unsigned char *) *out);
+}
diff
Random message corpus, v2
This obsoletes the series id:"1344190393-22497-1-git-send-email-david at tethera.net" [Patch v2 1/6] hex-escape: (en|de)code strings to/from restricted [Patch v2 2/6] test/hex-xcode: new test binary [Patch v2 3/6] test/hex-escaping: new test for hex escaping routines In order to avoid the problem of trashing people's terminals, the new version hex escapes the files being compared. This brings in another three patches from the previous dump-restore series, but we need them anyway for the eventual new dump-restore format. [Patch v2 4/6] test: add database routines for testing. No changes since v1. [Patch v2 5/6] test: add generator for random "stub" messages - Simplified to have only two character classes. - Fix bug with "stop - start" as opposed to "start - stop" See id:"87393x3jz1.fsf at qmul.ac.uk" [Patch v2 6/6] test: add broken roundtrip test - use hexdump to encode non-ascii octets. See id:"87zk6524f0.fsf at qmul.ac.uk" - Both Mark and Jamie remarked on the use of 'test_expect_success' Basically there are two styles of running tests, "git style" as written here, or "notmuch style" using "test_begin_subtest". The git style seems to be more robust against failure of intermediate commands: for example, if the hex-xcode binary is missing, the corresponding notmuch style test shows (wrongly) as fixed (see also id:"1317317811-29540-1-git-send-email-thomas at schwinge.name" for Thomas converting some notmuch style tests to git style). Obviously the notmuch style is more pleasant to edit. Maybe there is a way to make them more robust; something like "set -e". Note that some failures still do not abort the test, because of a missing "set -o pipefile". Setting this locally in a test seems definitely wrong, but maybe it should be set globally in test-lib.sh
Alternative (raw) message store (i.e. instead of maildir)
On Sat, Aug 11, 2012 at 11:50 PM, Jameson Graef Rollins wrote: > On Sat, Aug 11 2012, Ciprian Dorin Craciun > wrote: >> My problem with it is that it doesn't scale... And I don't mean >> this in a theoretical sense, I mean it in the concrete one: I have >> about 661k emails... And a single `notmuch sync` takes a few tens of >> seconds... > > Hey, Ciprian. That sounds really slow, which makes me wonder if there > are other things going on here. > I have 155k messages, but notmuch new > takes a fraction of a second for me. This initial indexing certainly > takes a long time (hours potentially), but additions after that should > be really fast. What version of notmuch are you using? What version of > xapian? Don't think there is anything wrong here... Its just drags with the file system... So just to give a complete info: * hardware: Core i5, 8GiB RAM (7.5GiB of which is the FS cache), SSD (about 175MiB raw disk access); * `notmuch --version`: 0.13 (built from sources on latest ArchLinux); * `notmuch count`: 701820; * `notmuch new` (after adding 5925 new emails, at touching others): Processed 7017 total files in 3m 19s (35 files/sec.). Added 6061 new messages to the database. Detected 1116 file renames. * actually the entire thing took almost 5 minutes, but the first two it didn't display anything just acesing the disk; * `notmuch new` (another go, but this time I've `time`-d it): No new mail. real0m40.546s user0m4.523s sys 0m17.506s * `notmuch new` (yet another go, no change): No new mail. real0m39.190s user0m4.229s sys 0m17.697s * just to `du` the maildir (there are also 40k other files in other maildirs not included in this count): 8.7G.. real0m22.229s user0m1.023s sys 0m7.890s * on `new` no hooks are run; * the file system in cause is JFS; As such I doubt the problem is with notmuch itself, and I guess it's the file system interaction... Now I know I have a really obscure corner case, and I'm positively amazed on how good notmuch handles this situation. I just wandered if I could have fixed my problem by moving to an embedded DB, thus skipping all that syscall overhead... Ciprian.
Re: [announce] Bower 0.4
On Sun, 12 Aug 2012 21:33:39 +0100, Patrick Totzke wrote: > Quoting David Froger (2012-08-12 16:53:43) > > > > 3. Would it be possible to have the feature 'kill a thread' like Sup > > > > have? (see > > > > http://sup.rubyforge.org/README.txt). > > > > > > I haven't needed such a feature so I don't have a clear idea how it > > > should work. There are some patches/ideas about muting threads using > > > notmuch generally around here. I guess the simplest thing to do is hide > > > a thread if it has a certain tag, e.g. "muted" as suggested by notmuch > > > TODO file, unless you search for it explicitly. > > I like this solution, I will use it (adding 'and not tag:mutted' at the > > end of my search). > > When I'm not interessed with a long discussion on a mailling list, I > > like to be able to 'kill' > > the thread so it does not reappear with every new mail. > > I'm not sure if this actually solves your problem, because > notmuch's search is message, not thread based: > If you search for threads that match "is:inbox and not is:muted", you will > see threads > that contain messages which are tagged with "inbox" and not with "muted". > This means if you just tag one (or all) msg in a thread "muted" and there is > a new response, > this thread will contain such a matching message again and show up in your > result list. Thanks for the clarification, it's very helpfull! > > To make this work, you need to set up your tagging script (that you run after > 'notmuch new') > so that it updates those new messages accordingly. If a msg belongs to a > thread that has > muted messages, also tag the new one muted. > I can pretty much recommend Justus' tagging script afew: > https://github.com/teythoon/afew > One of the filters that it supports out of the box is exactly the "kill > threads" function > you want. Thanks, this script looks very cool and I think I will replace mine with it (I should have visited (http://notmuchmail.org/initial_tagging/ ...). You're right, the filter I'm interessed is in in aftew/filters/KillThreadsFilter.py, and do this: for all email M that does not have tag 'killed': I = thread Id of the email M if there are emails that have thread id I and the 'killed' tag: apply 'killed' tag on email E Thanks again for all the help. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: Alternative (raw) message store (i.e. instead of maildir)
Vladimir Marek writes: > Hi, > > I have objections against maildir too, but I tried to tackle it from > different perspective. Store the maildir in zip file and use fuse-zip to > manage it. It works sort of but it has two major disadvantages: huh... this is fairly interesting one of the downsides of a million odd files for mail is that filesystem dump and restore takes a *LOT* longer than if it's just giant files on disk. Combined with afuse (fuse automounter) this could be a pretty elegant solution to the problem of storing archival Maildirs. One large archival maildir here went from 6.5GB (du -sh on XFS) to a 2.3GB ZIP archive that will never, ever change. Think about the performance difference between creating 560,000 files for backup/restore versus copying a single 2.3GB file. > - fuse zip stores all changes in memory until unmounted > - fuse zip (and libzip for that matter) creates new temporary file when >updating archive, which takes considerable time when the archive is >very big. This isn't much of a hastle if you have maildir per time period and archive off. Maybe if you sync flags it may be... > Of course this solution would have some disadvantages too, but for me > the advantages would win. At the moment I'm not sure if I want to > continue working on that. Maybe if there would be more interested guys I'm *really* tempted to investigate making this work for archived mail. Of course, the list of mounted file systems could get insane depending on granularity I guess... -- Stewart Smith pgpZcxW0PhtqJ.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[announce] Bower 0.4
On Sun, 12 Aug 2012 17:53:43 +0200, David Froger wrote: > > > 4. postponed message doesn't work for me. When I postpone a message, it > > > says > > > 'message postponed', but when pressing R from Index view, it says 'No > > > postponed > > > message'. Maybe I need to create a directory to store postponed messages? > > > > The postponed message should be in Maildir/Drafts. Can you see it with > > `notmuch search tag:draft'? > No, `notmuch search tag:draft` returns 0. This is the command to save drafts: notmuch-deliver Drafts --tag=draft \ --remove-tag=inbox --remove-tag=unread < message_file Does it work if you run it manually? Just make a dummy message_file with Date:, From:, To:, Subject: headers and a body. Peter
multipart/alternative bug
On 13 August 2012 16:50, hellekin wrote: > > Hello, > > as I mentioned on IRC a few days ago, there are some cases where: > > - a thread only displays the first message > - key bindings do not work at all (except q) > Does this in ~/.emacs help ? (defvar gnus-inhibit-images nil "*testing") (set-variable 'gnus-inhibit-images nil) Don't ask why. I have no clue!
multipart/alternative bug
Hello, as I mentioned on IRC a few days ago, there are some cases where: - a thread only displays the first message - key bindings do not work at all (except q) By trial and error, I could nail down the issue to multipart/alternative support. I had set notmuch to only show the text/plain part of an email before hitting the bug, and by setting "Notmuch Show All Multipart/Alternative Parts" to *off* in notmuch-show configuration, the bug disappeared. I'm sorry not to be able to dig further, but I hope this tip will help users to avoid the issue, and developers to find and fix the bug. Cheers, == hk
Re: Alternative (raw) message store (i.e. instead of maildir)
On Sat, Aug 11, 2012 at 11:50 PM, Jameson Graef Rollins wrote: > On Sat, Aug 11 2012, Ciprian Dorin Craciun wrote: >> My problem with it is that it doesn't scale... And I don't mean >> this in a theoretical sense, I mean it in the concrete one: I have >> about 661k emails... And a single `notmuch sync` takes a few tens of >> seconds... > > Hey, Ciprian. That sounds really slow, which makes me wonder if there > are other things going on here. > I have 155k messages, but notmuch new > takes a fraction of a second for me. This initial indexing certainly > takes a long time (hours potentially), but additions after that should > be really fast. What version of notmuch are you using? What version of > xapian? Don't think there is anything wrong here... Its just drags with the file system... So just to give a complete info: * hardware: Core i5, 8GiB RAM (7.5GiB of which is the FS cache), SSD (about 175MiB raw disk access); * `notmuch --version`: 0.13 (built from sources on latest ArchLinux); * `notmuch count`: 701820; * `notmuch new` (after adding 5925 new emails, at touching others): Processed 7017 total files in 3m 19s (35 files/sec.). Added 6061 new messages to the database. Detected 1116 file renames. * actually the entire thing took almost 5 minutes, but the first two it didn't display anything just acesing the disk; * `notmuch new` (another go, but this time I've `time`-d it): No new mail. real0m40.546s user0m4.523s sys 0m17.506s * `notmuch new` (yet another go, no change): No new mail. real0m39.190s user0m4.229s sys 0m17.697s * just to `du` the maildir (there are also 40k other files in other maildirs not included in this count): 8.7G.. real0m22.229s user0m1.023s sys 0m7.890s * on `new` no hooks are run; * the file system in cause is JFS; As such I doubt the problem is with notmuch itself, and I guess it's the file system interaction... Now I know I have a really obscure corner case, and I'm positively amazed on how good notmuch handles this situation. I just wandered if I could have fixed my problem by moving to an embedded DB, thus skipping all that syscall overhead... Ciprian. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] News for new 'previous' behavior
--- NEWS |8 1 file changed, 8 insertions(+) diff --git a/NEWS b/NEWS index 9916c91..7c7b510 100644 --- a/NEWS +++ b/NEWS @@ -41,6 +41,14 @@ Experimental support for multi-line search result formats `notmuch-search-result-format` to make individual search results span multiple lines. +Next/previous in search and show now move by boundaries + + All "next" and "previous" commands in the search and show modes now + move to the next/previous result or message boundary. This doesn't + change the behavior of "next", but "previous" commands will first + move to the beginning of the current result or message if point is + inside the result or message. + Search now uses the JSON format internally This should address problems with unusual characters in authors and -- 1.7.10
[PATCH 2/2] test: emacs: run list-processes after accept-process-output in emacs 23.1
Quoth Tomi Ollila on Aug 05 at 2:13 pm: > When running emacs tests using emacs 23.1.1 the tests block (until timeout) > when emacs function (notmuch-test-wait) is called. > > There is an emacs bug #2930 titled: > 23.0.92; `accept-process-output' and `sleep-for' do not run sentinel > > It seems this is present in emacs 23.1. > > Calling list-processes after accept-process-output seems work around > this problem; in case Emacs version is 23.1 a defadvice is activated > to do just that. Should this workaround perhaps go in notmuch-test-wait directly, instead of being implemented as advice? If we do want to keep it as advice, should it go in notmuch-lib.el along with the few other compatibility functions? > --- > > Thanks to Austin for the comments and IRC discussions on the matter. > > test/test-lib.el |8 > 1 files changed, 8 insertions(+), 0 deletions(-) > > diff --git a/test/test-lib.el b/test/test-lib.el > index 52d9936..4330352 100644 > --- a/test/test-lib.el > +++ b/test/test-lib.el > @@ -35,6 +35,16 @@ > "Disable yes-or-no-p before executing kill-emacs" > (defun yes-or-no-p (prompt) t))) > > +;; Emacs bug #2930: > +;; 23.0.92; `accept-process-output' and `sleep-for' do not run sentinels > +;; seems to be present in Emacs 23.1. > +;; Running `list-processes' after `accept-process-output' seems to work > +;; around this problem. > +(if (and (= emacs-major-version 23) (= emacs-minor-version 1)) > + (defadvice accept-process-output (after run-list-processes activate) > +"run list-processes after executing accept-process-output" > +(list-processes))) > + > (defun notmuch-test-wait () >"Wait for process completion." >(while (get-buffer-process (current-buffer))
Re: multipart/alternative bug
On 13 August 2012 16:50, hellekin wrote: > > Hello, > > as I mentioned on IRC a few days ago, there are some cases where: > > - a thread only displays the first message > - key bindings do not work at all (except q) > Does this in ~/.emacs help ? (defvar gnus-inhibit-images nil "*testing") (set-variable 'gnus-inhibit-images nil) Don't ask why. I have no clue! ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: multipart/alternative bug
On Mon, Aug 13 2012, hellekin wrote: > as I mentioned on IRC a few days ago, there are some cases where: > > - a thread only displays the first message > - key bindings do not work at all (except q) > > By trial and error, I could nail down the issue to multipart/alternative > support. I had set notmuch to only show the text/plain part of an email > before hitting the bug, and by setting "Notmuch Show All > Multipart/Alternative Parts" to *off* in notmuch-show configuration, the > bug disappeared. > > I'm sorry not to be able to dig further, but I hope this tip will help > users to avoid the issue, and developers to find and fix the bug. Hi, hk. I'm going to go out on a limb and guess that this is actually probably *not* an issue with multipart/alternative but instead with whatever is in that alternative part. If you set notmuch-show-all-multipart/alternative to off you're only seeing the text/plain part. If there's something crazy in the alternative part, something that's not getting decoded correctly maybe, then that might screw up the buffer. Can you use the command line UI to determine what else is in the offending message? Is it just a text/html part, or is there something else more exotic? A lot of popular but broken MUAs stuff non-alternative stuff into alternative parts. You can grab the message id (id:XXX) of the offending message with 'c i' in the emacs UI, and the look at it on the command line with: notmuch show id:XXX hth. jamie. ps. In general the command line UI is a great resource for figuring out what might be causing problems in the emacs UI. pgpI7enuh1CXt.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
multipart/alternative bug
On Mon, Aug 13 2012, hellekin wrote: > as I mentioned on IRC a few days ago, there are some cases where: > > - a thread only displays the first message > - key bindings do not work at all (except q) > > By trial and error, I could nail down the issue to multipart/alternative > support. I had set notmuch to only show the text/plain part of an email > before hitting the bug, and by setting "Notmuch Show All > Multipart/Alternative Parts" to *off* in notmuch-show configuration, the > bug disappeared. > > I'm sorry not to be able to dig further, but I hope this tip will help > users to avoid the issue, and developers to find and fix the bug. Hi, hk. I'm going to go out on a limb and guess that this is actually probably *not* an issue with multipart/alternative but instead with whatever is in that alternative part. If you set notmuch-show-all-multipart/alternative to off you're only seeing the text/plain part. If there's something crazy in the alternative part, something that's not getting decoded correctly maybe, then that might screw up the buffer. Can you use the command line UI to determine what else is in the offending message? Is it just a text/html part, or is there something else more exotic? A lot of popular but broken MUAs stuff non-alternative stuff into alternative parts. You can grab the message id (id:XXX) of the offending message with 'c i' in the emacs UI, and the look at it on the command line with: notmuch show id:XXX hth. jamie. ps. In general the command line UI is a great resource for figuring out what might be causing problems in the emacs UI. -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 835 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20120813/38099c59/attachment.pgp>
multipart/alternative bug
Hello, as I mentioned on IRC a few days ago, there are some cases where: - a thread only displays the first message - key bindings do not work at all (except q) By trial and error, I could nail down the issue to multipart/alternative support. I had set notmuch to only show the text/plain part of an email before hitting the bug, and by setting "Notmuch Show All Multipart/Alternative Parts" to *off* in notmuch-show configuration, the bug disappeared. I'm sorry not to be able to dig further, but I hope this tip will help users to avoid the issue, and developers to find and fix the bug. Cheers, == hk ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] News for new 'previous' behavior
--- NEWS |8 1 file changed, 8 insertions(+) diff --git a/NEWS b/NEWS index 9916c91..7c7b510 100644 --- a/NEWS +++ b/NEWS @@ -41,6 +41,14 @@ Experimental support for multi-line search result formats `notmuch-search-result-format` to make individual search results span multiple lines. +Next/previous in search and show now move by boundaries + + All "next" and "previous" commands in the search and show modes now + move to the next/previous result or message boundary. This doesn't + change the behavior of "next", but "previous" commands will first + move to the beginning of the current result or message if point is + inside the result or message. + Search now uses the JSON format internally This should address problems with unusual characters in authors and -- 1.7.10 ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 2/2] test: emacs: run list-processes after accept-process-output in emacs 23.1
Quoth Tomi Ollila on Aug 05 at 2:13 pm: > When running emacs tests using emacs 23.1.1 the tests block (until timeout) > when emacs function (notmuch-test-wait) is called. > > There is an emacs bug #2930 titled: > 23.0.92; `accept-process-output' and `sleep-for' do not run sentinel > > It seems this is present in emacs 23.1. > > Calling list-processes after accept-process-output seems work around > this problem; in case Emacs version is 23.1 a defadvice is activated > to do just that. Should this workaround perhaps go in notmuch-test-wait directly, instead of being implemented as advice? If we do want to keep it as advice, should it go in notmuch-lib.el along with the few other compatibility functions? > --- > > Thanks to Austin for the comments and IRC discussions on the matter. > > test/test-lib.el |8 > 1 files changed, 8 insertions(+), 0 deletions(-) > > diff --git a/test/test-lib.el b/test/test-lib.el > index 52d9936..4330352 100644 > --- a/test/test-lib.el > +++ b/test/test-lib.el > @@ -35,6 +35,16 @@ > "Disable yes-or-no-p before executing kill-emacs" > (defun yes-or-no-p (prompt) t))) > > +;; Emacs bug #2930: > +;; 23.0.92; `accept-process-output' and `sleep-for' do not run sentinels > +;; seems to be present in Emacs 23.1. > +;; Running `list-processes' after `accept-process-output' seems to work > +;; around this problem. > +(if (and (= emacs-major-version 23) (= emacs-minor-version 1)) > + (defadvice accept-process-output (after run-list-processes activate) > +"run list-processes after executing accept-process-output" > +(list-processes))) > + > (defun notmuch-test-wait () >"Wait for process completion." >(while (get-buffer-process (current-buffer)) ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [announce] Bower 0.4
On Sun, 12 Aug 2012 17:53:43 +0200, David Froger wrote: > > > 4. postponed message doesn't work for me. When I postpone a message, it > > > says > > > 'message postponed', but when pressing R from Index view, it says 'No > > > postponed > > > message'. Maybe I need to create a directory to store postponed messages? > > > > The postponed message should be in Maildir/Drafts. Can you see it with > > `notmuch search tag:draft'? > No, `notmuch search tag:draft` returns 0. This is the command to save drafts: notmuch-deliver Drafts --tag=draft \ --remove-tag=inbox --remove-tag=unread < message_file Does it work if you run it manually? Just make a dummy message_file with Date:, From:, To:, Subject: headers and a body. Peter ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[announce] Bower 0.4
On Sun, 12 Aug 2012 14:40:23 +0200, David Froger wrote: > > Hi, Hi David, > > Thanks Peter, nice to see this new release and these new functionnalies! > > I'm using bower every day, both at home and at work, and it's really usefull > for > me. > > By the way, I have some questions: > > 1. To receive emails, I use a bash script that basically do: > > while true > do > offlineimap > notmuch new > notmuch tag ... > sleep 60 > done > > Would it be possible to have a hook 'pollmails' that is triggered from the > Index > view by pressing a key (p or P), which can be configure in > ~/.config/bower/bower.conf > like this: > pollmails=a_command.sh Basically, run an arbitrary command when a key is pressed. It would be fairly easy to add, but I wonder if it would be easy to stop adding more :-) You could bind a shortcut key to your window manager or desktop environment that runs the command instead. > 2. I've notice that errors could happend if I modify tags from bower while my > bash script is also modifiny tags of the incoming emails. Do you see any > solution? The asynchronous tagging will retry after 5 seconds. The notmuch error message will show and mess up your screen (press ^L), which I would like to fix somehow. Ultimately, I think notmuch should handle this more gracefully. > 3. Would it be possible to have the feature 'kill a thread' like Sup have? > (see > http://sup.rubyforge.org/README.txt). I haven't needed such a feature so I don't have a clear idea how it should work. There are some patches/ideas about muting threads using notmuch generally around here. I guess the simplest thing to do is hide a thread if it has a certain tag, e.g. "muted" as suggested by notmuch TODO file, unless you search for it explicitly. > 4. postponed message doesn't work for me. When I postpone a message, it says > 'message postponed', but when pressing R from Index view, it says 'No > postponed > message'. Maybe I need to create a directory to store postponed messages? The postponed message should be in Maildir/Drafts. Can you see it with `notmuch search tag:draft'? > > 5. When attaching a file, would it be possible to have '~' expanded to the > $HOME directory? Sure. > 6. In the index view, having the possiblity to scroll right and left (so one > can > see the email tags in the line in too long)? I'm not sure that's a great solution. Maybe have a key that popups up an info box, which would have enough room to show the whole subject line, the list of authors and the tags? > 7. less important, having light colors for white console? Maybe :-) Peter
