On Mon, 27 Oct 2014, Michal Sojka <sojkam1 at fel.cvut.cz> wrote:
> This adds an algorithm to filter out duplicate addresses from address
> outputs (sender, receivers). The algorithm can be configured with
> --filter-by command line option.
>
> The code here is an extended version of a patch from Jani Nikula.
Hi
As this is getting into the more controversial bike shedding region I
wonder if it would be worth splitting this into 2 patches: the first
could do the default dedupe based on name/address and the second could
do add the filter-by options.
I think the default deduping is obviously worth doing but I am not sure
about the rest. In any case I think the default deduping could go in
pre-freeze but I would recommend the rest is left until after.
> ---
> completion/notmuch-completion.bash | 6 ++-
> completion/notmuch-completion.zsh | 3 +-
> doc/man1/notmuch-search.rst | 38 +++++++++++++++
> notmuch-search.c | 98
> +++++++++++++++++++++++++++++++++++---
> test/T090-search-output.sh | 87 +++++++++++++++++++++++++++++++++
> test/T095-search-filter-by.sh | 64 +++++++++++++++++++++++++
> 6 files changed, 288 insertions(+), 8 deletions(-)
> create mode 100755 test/T095-search-filter-by.sh
>
> diff --git a/completion/notmuch-completion.bash
> b/completion/notmuch-completion.bash
> index cfbd389..6b6d43a 100644
> --- a/completion/notmuch-completion.bash
> +++ b/completion/notmuch-completion.bash
> @@ -305,12 +305,16 @@ _notmuch_search()
> COMPREPLY=( $( compgen -W "true false flag all" -- "${cur}" ) )
> return
> ;;
> + --filter-by)
> + COMPREPLY=( $( compgen -W "nameaddr name addr addrfold
> nameaddrfold" -- "${cur}" ) )
> + return
> + ;;
> esac
>
> ! $split &&
> case "${cur}" in
> -*)
> - local options="--format= --output= --sort= --offset= --limit=
> --exclude= --duplicate="
> + local options="--format= --output= --sort= --offset= --limit=
> --exclude= --duplicate= --filter-by="
> compopt -o nospace
> COMPREPLY=( $(compgen -W "$options" -- ${cur}) )
> ;;
> diff --git a/completion/notmuch-completion.zsh
> b/completion/notmuch-completion.zsh
> index 3e52a00..3e535df 100644
> --- a/completion/notmuch-completion.zsh
> +++ b/completion/notmuch-completion.zsh
> @@ -53,7 +53,8 @@ _notmuch_search()
> '--max-threads=[display only the first x threads from the search
> results]:number of threads to show: ' \
> '--first=[omit the first x threads from the search results]:number of
> threads to omit: ' \
> '--sort=[sort results]:sorting:((newest-first\:"reverse chronological
> order" oldest-first\:"chronological order"))' \
> - '--output=[select what to output]:output:((summary threads messages
> files tags sender recipients))'
> + '--output=[select what to output]:output:((summary threads messages
> files tags sender recipients))' \
> + '--filter-by=[filter out duplicate
> addresses]:filter-by:((nameaddr\:"both name and address part" name\:"name
> part" addr\:"address part" addrfold\:"case-insensitive address part"
> nameaddrfold\:"name and case-insensitive address part"))'
> }
>
> _notmuch()
> diff --git a/doc/man1/notmuch-search.rst b/doc/man1/notmuch-search.rst
> index b6607c9..84af2da 100644
> --- a/doc/man1/notmuch-search.rst
> +++ b/doc/man1/notmuch-search.rst
> @@ -85,6 +85,9 @@ Supported options for **search** include
> (--format=text0), as a JSON array (--format=json), or as
> an S-Expression list (--format=sexp).
>
> + Duplicate addresses are filtered out. Filtering can be
> + configured with the --filter-by option.
> +
> Note: Searching for **sender** should be much faster than
> searching for **recipients**, because sender addresses are
> cached directly in the database whereas other addresses
> @@ -151,6 +154,41 @@ Supported options for **search** include
> prefix. The prefix matches messages based on filenames. This
> option filters filenames of the matching messages.
>
> + ``--filter-by=``\ (**nameaddr**\ \|\ **name** \|\ **addr**\ \|\
> **addrfold**\ \|\ **nameaddrfold**\)
> +
> + Can be used with ``--output=sender`` or
> + ``--output=recipients`` to filter out duplicate addresses. The
> + filtering algorithm receives a sequence of email addresses and
> + outputs the same sequence without the addresses that are
> + considered a duplicate of a previously output address. What is
> + considered a duplicate depends on how the two addresses are
> + compared and this can be controlled with the follwing flags:
> +
> + **nameaddr** means that both name and address parts are
> + compared in case-sensitive manner. Therefore, all same looking
> + addresses strings are considered duplicate. This is the
> + default.
> +
> + **name** means that only the name part is compared (in
> + case-sensitive manner). For example, the addresses "John Doe
> + <me at example.com>" and "John Doe <john at doe.name>" will be
> + considered duplicate.
> +
> + **addr** means that only the address part is compared (in
> + case-sensitive manner). For example, the addresses "John Doe
> + <john at example.com>" and "Dr. John Doe <john at example.com>" will
> + be considered duplicate.
> +
> + **addrfold** is like **addr**, but comparison is done in
> + canse-insensitive manner. For example, the addresses "John Doe
> + <john at example.com>" and "Dr. John Doe <JOHN at EXAMPLE.COM>" will
> + be considered duplicate.
> +
> + **nameaddrfold** is like **nameaddr**, but address comparison
> + is done in canse-insensitive manner. For example, the
> + addresses "John Doe <john at example.com>" and "John Doe
> + <JOHN at EXAMPLE.COM>" will be considered duplicate.
> +
> EXIT STATUS
> ===========
>
> diff --git a/notmuch-search.c b/notmuch-search.c
> index ce3bfb2..47aa979 100644
> --- a/notmuch-search.c
> +++ b/notmuch-search.c
> @@ -34,6 +34,14 @@ typedef enum {
>
> #define OUTPUT_ADDRESS_FLAGS (OUTPUT_SENDER | OUTPUT_RECIPIENTS)
>
> +typedef enum {
> + FILTER_BY_NAMEADDR = 0,
> + FILTER_BY_NAME,
> + FILTER_BY_ADDR,
> + FILTER_BY_ADDRFOLD,
> + FILTER_BY_NAMEADDRFOLD,
> +} filter_by_t;
> +
> typedef struct {
> sprinter_t *format;
> notmuch_query_t *query;
> @@ -42,6 +50,7 @@ typedef struct {
> int offset;
> int limit;
> int dupe;
> + filter_by_t filter_by;
> } search_options_t;
>
> typedef struct {
> @@ -229,6 +238,52 @@ do_search_threads (search_options_t *opt)
> return 0;
> }
>
> +/* Returns TRUE iff name and/or addr is considered duplicite. */
A triviality; duplicite should be duplicate
> +static notmuch_bool_t
> +check_duplicite (const search_options_t *opt, GHashTable *addrs, const char
> *name, const char *addr)
I am not sure on style but maybe is_duplicate would be clearer?
Best wishes
Mark
> +{
> + notmuch_bool_t duplicite;
> + char *key;
> +
> + if (opt->filter_by == FILTER_BY_ADDRFOLD ||
> + opt->filter_by == FILTER_BY_NAMEADDRFOLD) {
> + gchar *folded = g_utf8_casefold (addr, -1);
> + addr = talloc_strdup (opt->format, folded);
> + g_free (folded);
> + }
> + switch (opt->filter_by) {
> + case FILTER_BY_NAMEADDR:
> + case FILTER_BY_NAMEADDRFOLD:
> + key = talloc_asprintf (opt->format, "%s <%s>", name, addr);
> + break;
> + case FILTER_BY_NAME:
> + key = talloc_strdup (opt->format, name); /* !name results in !key */
> + break;
> + case FILTER_BY_ADDR:
> + case FILTER_BY_ADDRFOLD:
> + key = talloc_strdup (opt->format, addr);
> + break;
> + default:
> + INTERNAL_ERROR("invalid --filter-by flags");
> + }
> +
> + if (opt->filter_by == FILTER_BY_ADDRFOLD ||
> + opt->filter_by == FILTER_BY_NAMEADDRFOLD)
> + talloc_free ((char*)addr);
> +
> + if (! key)
> + return FALSE;
> +
> + duplicite = g_hash_table_lookup_extended (addrs, key, NULL, NULL);
> +
> + if (! duplicite)
> + g_hash_table_insert (addrs, key, NULL);
> + else
> + talloc_free (key);
> +
> + return duplicite;
> +}
> +
> static void
> print_mailbox (const search_options_t *opt, const mailbox_t *mailbox)
> {
> @@ -263,7 +318,8 @@ print_mailbox (const search_options_t *opt, const
> mailbox_t *mailbox)
> }
>
> static void
> -process_address_list (const search_options_t *opt, InternetAddressList *list)
> +process_address_list (const search_options_t *opt, GHashTable *addrs,
> + InternetAddressList *list)
> {
> InternetAddress *address;
> int i;
> @@ -279,7 +335,7 @@ process_address_list (const search_options_t *opt,
> InternetAddressList *list)
> if (group_list == NULL)
> continue;
>
> - process_address_list (opt, group_list);
> + process_address_list (opt, addrs, group_list);
> } else {
> InternetAddressMailbox *mailbox = INTERNET_ADDRESS_MAILBOX
> (address);
> mailbox_t mbx = {
> @@ -287,13 +343,16 @@ process_address_list (const search_options_t *opt,
> InternetAddressList *list)
> .addr = internet_address_mailbox_get_addr (mailbox),
> };
>
> + if (check_duplicite (opt, addrs, mbx.name, mbx.addr))
> + continue;
> +
> print_mailbox (opt, &mbx);
> }
> }
> }
>
> static void
> -process_address_header (const search_options_t *opt, const char *value)
> +process_address_header (const search_options_t *opt, GHashTable *addrs,
> const char *value)
> {
> InternetAddressList *list;
>
> @@ -304,7 +363,13 @@ process_address_header (const search_options_t *opt,
> const char *value)
> if (list == NULL)
> return;
>
> - process_address_list (opt, list);
> + process_address_list (opt, addrs, list);
> +}
> +
> +static void
> +_my_talloc_free_for_g_hash (void *ptr)
> +{
> + talloc_free (ptr);
> }
>
> static int
> @@ -314,8 +379,13 @@ do_search_messages (search_options_t *opt)
> notmuch_messages_t *messages;
> notmuch_filenames_t *filenames;
> sprinter_t *format = opt->format;
> + GHashTable *addresses = NULL;
> int i;
>
> + if (opt->output & OUTPUT_ADDRESS_FLAGS)
> + addresses = g_hash_table_new_full (g_str_hash, g_str_equal,
> + _my_talloc_free_for_g_hash, NULL);
> +
> if (opt->offset < 0) {
> opt->offset += notmuch_query_count_messages (opt->query);
> if (opt->offset < 0)
> @@ -363,7 +433,7 @@ do_search_messages (search_options_t *opt)
> const char *addrs;
>
> addrs = notmuch_message_get_header (message, "from");
> - process_address_header (opt, addrs);
> + process_address_header (opt, addresses, addrs);
> }
>
> if (opt->output & OUTPUT_RECIPIENTS) {
> @@ -373,7 +443,7 @@ do_search_messages (search_options_t *opt)
>
> for (j = 0; j < ARRAY_SIZE (hdrs); j++) {
> addrs = notmuch_message_get_header (message, hdrs[j]);
> - process_address_header (opt, addrs);
> + process_address_header (opt, addresses, addrs);
> }
> }
> }
> @@ -381,6 +451,9 @@ do_search_messages (search_options_t *opt)
> notmuch_message_destroy (message);
> }
>
> + if (addresses)
> + g_hash_table_unref (addresses);
> +
> notmuch_messages_destroy (messages);
>
> format->end (format);
> @@ -447,6 +520,7 @@ notmuch_search_command (notmuch_config_t *config, int
> argc, char *argv[])
> .offset = 0,
> .limit = -1, /* unlimited */
> .dupe = -1,
> + .filter_by = FILTER_BY_NAMEADDR,
> };
> char *query_str;
> int opt_index, ret;
> @@ -490,6 +564,13 @@ notmuch_search_command (notmuch_config_t *config, int
> argc, char *argv[])
> { NOTMUCH_OPT_INT, &opt.offset, "offset", 'O', 0 },
> { NOTMUCH_OPT_INT, &opt.limit, "limit", 'L', 0 },
> { NOTMUCH_OPT_INT, &opt.dupe, "duplicate", 'D', 0 },
> + { NOTMUCH_OPT_KEYWORD, &opt.filter_by, "filter-by", 'b',
> + (notmuch_keyword_t []){ { "nameaddr", FILTER_BY_NAMEADDR },
> + { "name", FILTER_BY_NAME },
> + { "addr", FILTER_BY_ADDR },
> + { "addrfold", FILTER_BY_ADDRFOLD },
> + { "nameaddrfold", FILTER_BY_NAMEADDRFOLD },
> + { 0, 0 } } },
> { 0, 0, 0, 0, 0 }
> };
>
> @@ -500,6 +581,11 @@ notmuch_search_command (notmuch_config_t *config, int
> argc, char *argv[])
> if (! opt.output)
> opt.output = OUTPUT_SUMMARY;
>
> + if (opt.filter_by && !(opt.output & OUTPUT_ADDRESS_FLAGS)) {
> + fprintf (stderr, "Error: --filter-by can only be used with address
> output.\n");
> + return EXIT_FAILURE;
> + }
> +
> switch (format_sel) {
> case NOTMUCH_FORMAT_TEXT:
> opt.format = sprinter_text_create (config, stdout);
> diff --git a/test/T090-search-output.sh b/test/T090-search-output.sh
> index 947d572..841a721 100755
> --- a/test/T090-search-output.sh
> +++ b/test/T090-search-output.sh
> @@ -387,6 +387,93 @@ cat <<EOF >EXPECTED
> EOF
> test_expect_equal_file OUTPUT EXPECTED
>
> +test_begin_subtest "--output=sender"
> +notmuch search --output=sender '*' >OUTPUT
> +cat <<EOF >EXPECTED
> +Fran?ois Boulogne <boulogne.f at gmail.com>
> +Olivier Berger <olivier.berger at it-sudparis.eu>
> +Chris Wilson <chris at chris-wilson.co.uk>
> +Carl Worth <cworth at cworth.org>
> +Alexander Botero-Lowry <alex.boterolowry at gmail.com>
> +Keith Packard <keithp at keithp.com>
> +Jjgod Jiang <gzjjgod at gmail.com>
> +Rolland Santimano <rollandsantimano at yahoo.com>
> +Jan Janak <jan at ryngle.com>
> +Stewart Smith <stewart at flamingspork.com>
> +Lars Kellogg-Stedman <lars at seas.harvard.edu>
> +Alex Botero-Lowry <alex.boterolowry at gmail.com>
> +Ingmar Vanhassel <ingmar at exherbo.org>
> +Aron Griffis <agriffis at n01se.net>
> +Adrian Perez de Castro <aperez at igalia.com>
> +Israel Herraiz <isra at herraiz.org>
> +Mikhail Gusarov <dottedmag at dottedmag.net>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "--output=sender --format=json"
> +notmuch search --output=sender --format=json '*' >OUTPUT
> +cat <<EOF >EXPECTED
> +[{"name": "Fran?ois Boulogne", "address": "boulogne.f at gmail.com"},
> +{"name": "Olivier Berger", "address": "olivier.berger at it-sudparis.eu"},
> +{"name": "Chris Wilson", "address": "chris at chris-wilson.co.uk"},
> +{"name": "Carl Worth", "address": "cworth at cworth.org"},
> +{"name": "Alexander Botero-Lowry", "address": "alex.boterolowry at
> gmail.com"},
> +{"name": "Keith Packard", "address": "keithp at keithp.com"},
> +{"name": "Jjgod Jiang", "address": "gzjjgod at gmail.com"},
> +{"name": "Rolland Santimano", "address": "rollandsantimano at yahoo.com"},
> +{"name": "Jan Janak", "address": "jan at ryngle.com"},
> +{"name": "Stewart Smith", "address": "stewart at flamingspork.com"},
> +{"name": "Lars Kellogg-Stedman", "address": "lars at seas.harvard.edu"},
> +{"name": "Alex Botero-Lowry", "address": "alex.boterolowry at gmail.com"},
> +{"name": "Ingmar Vanhassel", "address": "ingmar at exherbo.org"},
> +{"name": "Aron Griffis", "address": "agriffis at n01se.net"},
> +{"name": "Adrian Perez de Castro", "address": "aperez at igalia.com"},
> +{"name": "Israel Herraiz", "address": "isra at herraiz.org"},
> +{"name": "Mikhail Gusarov", "address": "dottedmag at dottedmag.net"}]
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "--output=recipients"
> +notmuch search --output=recipients '*' >OUTPUT
> +cat <<EOF >EXPECTED
> +Allan McRae <allan at archlinux.org>
> +Discussion about the Arch User Repository (AUR) <aur-general at
> archlinux.org>
> +olivier.berger at it-sudparis.eu
> +notmuch at notmuchmail.org
> +notmuch <notmuch at notmuchmail.org>
> +Keith Packard <keithp at keithp.com>
> +Mikhail Gusarov <dottedmag at dottedmag.net>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "--output=sender --output=recipients"
> +notmuch search --output=sender --output=recipients '*' >OUTPUT
> +cat <<EOF >EXPECTED
> +Fran?ois Boulogne <boulogne.f at gmail.com>
> +Allan McRae <allan at archlinux.org>
> +Discussion about the Arch User Repository (AUR) <aur-general at
> archlinux.org>
> +Olivier Berger <olivier.berger at it-sudparis.eu>
> +olivier.berger at it-sudparis.eu
> +Chris Wilson <chris at chris-wilson.co.uk>
> +notmuch at notmuchmail.org
> +Carl Worth <cworth at cworth.org>
> +Alexander Botero-Lowry <alex.boterolowry at gmail.com>
> +Keith Packard <keithp at keithp.com>
> +Jjgod Jiang <gzjjgod at gmail.com>
> +Rolland Santimano <rollandsantimano at yahoo.com>
> +Jan Janak <jan at ryngle.com>
> +Stewart Smith <stewart at flamingspork.com>
> +Lars Kellogg-Stedman <lars at seas.harvard.edu>
> +notmuch <notmuch at notmuchmail.org>
> +Alex Botero-Lowry <alex.boterolowry at gmail.com>
> +Ingmar Vanhassel <ingmar at exherbo.org>
> +Aron Griffis <agriffis at n01se.net>
> +Adrian Perez de Castro <aperez at igalia.com>
> +Israel Herraiz <isra at herraiz.org>
> +Mikhail Gusarov <dottedmag at dottedmag.net>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> test_begin_subtest "sanitize output for quoted-printable line-breaks in
> author and subject"
> add_message "[subject]='two =?ISO-8859-1?Q?line=0A_subject?=
> headers'"
> diff --git a/test/T095-search-filter-by.sh b/test/T095-search-filter-by.sh
> new file mode 100755
> index 0000000..97d9a9b
> --- /dev/null
> +++ b/test/T095-search-filter-by.sh
> @@ -0,0 +1,64 @@
> +#!/usr/bin/env bash
> +test_description='duplicite address filtering in "notmuch search
> --output=recipients"'
> +. ./test-lib.sh
> +
> +add_message '[to]="Real Name <foo at example.com>, Real Name <bar at
> example.com>"'
> +add_message '[to]="Nickname <foo at example.com>"' '[cc]="Real Name <Bar at
> Example.COM>"'
> +add_message '[to]="Nickname <foo at example.com>"' '[bcc]="Real Name <Bar at
> Example.COM>"'
> +
> +test_begin_subtest "--output=recipients"
> +notmuch search --output=recipients "*" >OUTPUT
> +cat <<EOF >EXPECTED
> +Real Name <foo at example.com>
> +Real Name <bar at example.com>
> +Nickname <foo at example.com>
> +Real Name <Bar at Example.COM>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "--output=recipients --filter-by=nameaddr"
> +notmuch search --output=recipients --filter-by=nameaddr "*" >OUTPUT
> +# The same as above
> +cat <<EOF >EXPECTED
> +Real Name <foo at example.com>
> +Real Name <bar at example.com>
> +Nickname <foo at example.com>
> +Real Name <Bar at Example.COM>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "--output=recipients --filter-by=name"
> +notmuch search --output=recipients --filter-by=name "*" >OUTPUT
> +cat <<EOF >EXPECTED
> +Real Name <foo at example.com>
> +Nickname <foo at example.com>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "--output=recipients --filter-by=addr"
> +notmuch search --output=recipients --filter-by=addr "*" >OUTPUT
> +cat <<EOF >EXPECTED
> +Real Name <foo at example.com>
> +Real Name <bar at example.com>
> +Real Name <Bar at Example.COM>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "--output=recipients --filter-by=addrfold"
> +notmuch search --output=recipients --filter-by=addrfold "*" >OUTPUT
> +cat <<EOF >EXPECTED
> +Real Name <foo at example.com>
> +Real Name <bar at example.com>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_begin_subtest "--output=recipients --filter-by=nameaddrfold"
> +notmuch search --output=recipients --filter-by=nameaddrfold "*" >OUTPUT
> +cat <<EOF >EXPECTED
> +Real Name <foo at example.com>
> +Real Name <bar at example.com>
> +Nickname <foo at example.com>
> +EOF
> +test_expect_equal_file OUTPUT EXPECTED
> +
> +test_done
> --
> 2.1.1
>
> _______________________________________________
> notmuch mailing list
> notmuch at notmuchmail.org
> http://notmuchmail.org/mailman/listinfo/notmuch
