Re: add status value to _notmuch_message_ensure_metadata
David Bremner writes on februar 23, 2017 1:58: Gaute Hope writes: Gaute Hope writes on februar 20, 2017 10:27: David Bremner writes on februar 18, 2017 15:45: In id:1487339566.mz8acpov1j.astr...@strange.none , Gaute provided a traceback of an uncaught Xapian::DatabaseModifiedError. The fix for this is simple, but somewhat intrusive. [...] I haven't tested against Gaute's test case (needs more boost than I have handy). Alright then, attached is a non-boost version that takes a notmuch db path (absolute) as the first argument (no warranty). With the patches above this crashes in a predictable / preventable way, because notmuch_message_get_tags returns NULL. It isn't clear to me yet what the best API choice is here: minimize difference with the old API by returning NULL to indicate errors, or switch completely to the pattern of e.g. notmuch_query_search_messages_st. I suppose we could do something along the same lines and add new _st versions of the problematic functions Hi, Ideally if the error could be caught in `notmuch_threads_valid` or `notmuch_threads_get_thread` I think that would be the clearest, _st versions would be nice. As I mentioned in id:1487582192.57s86yczcg.astr...@strange.none it seems that at later arbitrary iterations (without re-loading the threads object) the functions return valid data (even `notmuch_thread_get_tags` does). Can this data be trusted? I feel like this should all be invalid at this point. Regards, Gaute ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] lib: remove notmuch_query_{count,search}_{threads,messages}
These 4 functions were originally deprecated in notmuch 0.21, more
than a year ago.
---
lib/notmuch.h | 50 --
test/T560-lib-error.sh | 4 ++--
2 files changed, 2 insertions(+), 52 deletions(-)
diff --git a/lib/notmuch.h b/lib/notmuch.h
index 16da8be9..5c9f5137 100644
--- a/lib/notmuch.h
+++ b/lib/notmuch.h
@@ -862,19 +862,6 @@ notmuch_query_search_threads_st (notmuch_query_t *query,
notmuch_threads_t **out);
/**
- * Like notmuch_query_search_threads_st, but without a status return.
- *
- * If a Xapian exception occurs this function will return NULL.
- *
- * @deprecated Deprecated as of libnotmuch 4.3 (notmuch 0.21). Please
- * use notmuch_query_search_threads_st instead.
- *
- */
-NOTMUCH_DEPRECATED(4,3)
-notmuch_threads_t *
-notmuch_query_search_threads (notmuch_query_t *query);
-
-/**
* Execute a query for messages, returning a notmuch_messages_t object
* which can be used to iterate over the results. The returned
* messages object is owned by the query and as such, will only be
@@ -919,19 +906,6 @@ notmuch_status_t
notmuch_query_search_messages_st (notmuch_query_t *query,
notmuch_messages_t **out);
/**
- * Like notmuch_query_search_messages, but without a status return.
- *
- * If a Xapian exception occurs this function will return NULL.
- *
- * @deprecated Deprecated as of libnotmuch 4.3 (notmuch 0.21). Please use
- * notmuch_query_search_messages_st instead.
- *
- */
-NOTMUCH_DEPRECATED(4,3)
-notmuch_messages_t *
-notmuch_query_search_messages (notmuch_query_t *query);
-
-/**
* Destroy a notmuch_query_t along with any associated resources.
*
* This will in turn destroy any notmuch_threads_t and
@@ -1016,18 +990,6 @@ notmuch_status_t
notmuch_query_count_messages_st (notmuch_query_t *query, unsigned int *count);
/**
- * like notmuch_query_count_messages_st, but without a status return.
- *
- * May return 0 in the case of errors.
- *
- * @deprecated Deprecated since libnotmuch 4.3 (notmuch 0.21). Please
- * use notmuch_query_count_messages_st instead.
- */
-NOTMUCH_DEPRECATED(4,3)
-unsigned int
-notmuch_query_count_messages (notmuch_query_t *query);
-
-/**
* Return the number of threads matching a search.
*
* This function performs a search and returns the number of unique thread IDs
@@ -1053,18 +1015,6 @@ notmuch_status_t
notmuch_query_count_threads_st (notmuch_query_t *query, unsigned *count);
/**
- * like notmuch_query_count_threads, but without a status return.
- *
- * May return 0 in case of errors.
- *
- * @deprecated Deprecated as of libnotmuch 4.3 (notmuch 0.21). Please
- * use notmuch_query_count_threads_st instead.
- */
-NOTMUCH_DEPRECATED(4,3)
-unsigned int
-notmuch_query_count_threads (notmuch_query_t *query);
-
-/**
* Get the thread ID of 'thread'.
*
* The returned string belongs to 'thread' and as such, should not be
diff --git a/test/T560-lib-error.sh b/test/T560-lib-error.sh
index 087c6bd7..a59eb74d 100755
--- a/test/T560-lib-error.sh
+++ b/test/T560-lib-error.sh
@@ -302,9 +302,9 @@ backup_database
test_begin_subtest "Xapian exception counting messages"
cat c_head - c_tail <<'EOF' | test_C ${MAIL_DIR}
{
+ int count;
notmuch_query_t *query=notmuch_query_create (db,
"id:87ocn0qh6d@yoom.home.cworth.org");
- int count = notmuch_query_count_messages (query);
- stat = (count == 0);
+ stat = notmuch_query_count_messages_st (query, &count);
}
EOF
sed 's/^\(A Xapian exception [^:]*\):.*$/\1/' < OUTPUT > OUTPUT.clean
--
2.11.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: add status value to _notmuch_message_ensure_metadata
Gaute Hope writes: > Gaute Hope writes on februar 20, 2017 10:27: >> David Bremner writes on februar 18, 2017 15:45: >>> In id:1487339566.mz8acpov1j.astr...@strange.none , Gaute provided a >>> traceback of an uncaught Xapian::DatabaseModifiedError. The fix for >>> this is simple, but somewhat intrusive. >>> >>> [...] >>> >>> I haven't tested against Gaute's test case (needs more boost than I >>> have handy). > > Alright then, attached is a non-boost version that takes a notmuch db > path (absolute) as the first argument (no warranty). > With the patches above this crashes in a predictable / preventable way, because notmuch_message_get_tags returns NULL. It isn't clear to me yet what the best API choice is here: minimize difference with the old API by returning NULL to indicate errors, or switch completely to the pattern of e.g. notmuch_query_search_messages_st. I suppose we could do something along the same lines and add new _st versions of the problematic functions ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[RFC PATCH] cli/show: include content type parameters in formatted output
This is primarily to be able to handle "text/plain; format=flowed",
but I don't see much point in making this specific to
format=flowed. Just include all content type parameters in the
formatted output, like this:
"content-type" : "text/plain",
"content-type-params" : [
{
"format" : "flowed"
}
],
It might make sense to change the content-type key in schemata to
include the parameters in a more structured fashion instead of adding
a separate key like here, but this doesn't require a change in
consumers or schemata version bump.
This was just a quick proof of concept, and obviously lacks tests,
schemata updates, etc.
---
notmuch-show.c | 13 +
1 file changed, 13 insertions(+)
diff --git a/notmuch-show.c b/notmuch-show.c
index 22fa655ad20d..7100b2d0b578 100644
--- a/notmuch-show.c
+++ b/notmuch-show.c
@@ -601,6 +601,19 @@ format_part_sprinter (const void *ctx, sprinter_t *sp,
mime_node_t *node,
sp->map_key (sp, "content-type");
sp->string (sp, g_mime_content_type_to_string (content_type));
+const GMimeParam *params = g_mime_content_type_get_params (content_type);
+if (params) {
+ sp->map_key (sp, "content-type-params");
+ sp->begin_list (sp);
+ sp->begin_map (sp);
+ for (; params; params = params->next) {
+ sp->map_key (sp, params->name);
+ sp->string (sp, params->value);
+ }
+ sp->end (sp);
+ sp->end (sp);
+}
+
if (cid) {
sp->map_key (sp, "content-id");
sp->string (sp, cid);
--
2.11.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: bug report
Rafael Avila de Espindola writes: andy@andy-x200:~/work/talks/2017/icps$ notmuch new Processed 230 total files in 7s (29 files/sec.). Added 212 new messages to the database. Removed 85 messages. Detected 18 file renames. Error: A Xapian exception occurred flushing database: Value in posting list too large. > > I have seen similar "Xapian exception". I don't remember the exact > message. It happens while running "notmuch new" while reading email on > emacs. It's hard to be sure without more details, but it sounds like a locking problem, which could also be be fixed by upgrading to current xapian and notmuch. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: bug report
>>> andy@andy-x200:~/work/talks/2017/icps$ notmuch new >>> Processed 230 total files in 7s (29 files/sec.). >>> Added 212 new messages to the database. Removed 85 messages. Detected 18 >>> file renames. >>> Error: A Xapian exception occurred flushing database: Value in posting list >>> too large. I have seen similar "Xapian exception". I don't remember the exact message. It happens while running "notmuch new" while reading email on emacs. > The other generic advice is to run 'xapian-check' to check your database > for corruption. On Debian and derivatives (e.g. Ubuntu) this is in the > package 'xapian-tools'. It found no errors for me. Cheers, Rafael ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH] lib: fix g_hash_table related read-after-free bug
On Wed, Feb 22 2017, David Bremner wrote:
> The two g_hash_table functions (insert, add) have different behaviour
> with respect to existing keys. g_hash_table_insert frees the new key,
> while g_hash_table_add (which is really g_hash_table_replace in
> disguise) frees the existing key. With this change 'ref' is live until
> the end of the function (assuming single-threaded access to
> 'hash'). We can't guarantee it will continue to be live in the
> future (i.e. there may be a future key duplication) so we copy it with
> the allocation context passed to parse_references (in practice this is
> the notmuch_message_t object whose parents we are finding).
>
> Thanks to Tomi for the simpler approach to the problem based on
> reading the fine glib manual.
> ---
Great work! LGTM.
Tomi
PS: tried to look whethet there were talloc_ref() (the glib way)...
there is "refcounting" but a bit different (unsuitable) way...
>
> this at least passes the --medium memory test. I'll run the full one
> but it probably needs a day or so to complete.
>
> lib/database.cc | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/lib/database.cc b/lib/database.cc
> index f0bfe566..eddb780c 100644
> --- a/lib/database.cc
> +++ b/lib/database.cc
> @@ -652,7 +652,7 @@ parse_references (void *ctx,
> ref = _parse_message_id (ctx, refs, &refs);
>
> if (ref && strcmp (ref, message_id)) {
> - g_hash_table_insert (hash, ref, NULL);
> + g_hash_table_add (hash, ref);
> last_ref = ref;
> }
> }
> @@ -661,7 +661,7 @@ parse_references (void *ctx,
> * reference to the database. We should avoid making a message
> * its own parent, thus the above check.
> */
> -return last_ref;
> +return talloc_strdup(ctx, last_ref);
> }
>
> notmuch_status_t
> --
> 2.11.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
release plans: 0.23.6, 0.24
I'd like to release another point release, 0.23.6 on Monday Feb. 27th. This will be the gpgconf --create-socketdir patch, and all going well something like id:20170222103207.1-1-da...@tethera.net. I'll then do a feature freeze for 0.24 Monday March 6, and hopefully a release on March 13. This double release process saves me from having to backport the patches for Debian; others may just want to wait for 0.24. d signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
[PATCH] lib: fix g_hash_table related read-after-free bug
The two g_hash_table functions (insert, add) have different behaviour
with respect to existing keys. g_hash_table_insert frees the new key,
while g_hash_table_add (which is really g_hash_table_replace in
disguise) frees the existing key. With this change 'ref' is live until
the end of the function (assuming single-threaded access to
'hash'). We can't guarantee it will continue to be live in the
future (i.e. there may be a future key duplication) so we copy it with
the allocation context passed to parse_references (in practice this is
the notmuch_message_t object whose parents we are finding).
Thanks to Tomi for the simpler approach to the problem based on
reading the fine glib manual.
---
this at least passes the --medium memory test. I'll run the full one
but it probably needs a day or so to complete.
lib/database.cc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/database.cc b/lib/database.cc
index f0bfe566..eddb780c 100644
--- a/lib/database.cc
+++ b/lib/database.cc
@@ -652,7 +652,7 @@ parse_references (void *ctx,
ref = _parse_message_id (ctx, refs, &refs);
if (ref && strcmp (ref, message_id)) {
- g_hash_table_insert (hash, ref, NULL);
+ g_hash_table_add (hash, ref);
last_ref = ref;
}
}
@@ -661,7 +661,7 @@ parse_references (void *ctx,
* reference to the database. We should avoid making a message
* its own parent, thus the above check.
*/
-return last_ref;
+return talloc_strdup(ctx, last_ref);
}
notmuch_status_t
--
2.11.0
___
notmuch mailing list
notmuch@notmuchmail.org
https://notmuchmail.org/mailman/listinfo/notmuch
Re: bug report
David Bremner writes: > Andy Wills writes: > >> >> andy@andy-x200:~/work/talks/2017/icps$ notmuch new >> Processed 230 total files in 7s (29 files/sec.). >> Added 212 new messages to the database. Removed 85 messages. Detected 18 >> file renames. >> Error: A Xapian exception occurred flushing database: Value in posting list >> too large. >> >> ...and it seems that, since I've been getting this error, new emails >> are not reliably being found by, for example M-x notmuch in emacs. >> > > Hi Andy; > > What version of notmuch is that? From the error message it sounds > pre-2014. That particular code was rewritten in notmuch 0.19, so if my > guess is correct it might be fixed by newer notmuch. The other generic advice is to run 'xapian-check' to check your database for corruption. On Debian and derivatives (e.g. Ubuntu) this is in the package 'xapian-tools'. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
