Re: multilingual notmuch (and Content-Language)
On Sun, Mar 18 2018, Daniel Kahn Gillmor wrote: > https://tools.ietf.org/html/rfc3282 describes a Content-Language: > header. https://tools.ietf.org/html/rfc8255 describes > a multipart/multilingual Content-Type. > > notmuch currently uses xapian with a hard-coded English stemmer which > works great for me as a monolingual American, but limits the > applicability of notmuch to Anglophiles (people who speak English). > That makes me sad. > > AIUI, xapian is pretty much committed to being a single-language > indexer. Have you seen the different stemmers it already has? Reference: https://xapian.org/docs/sourcedoc/html/dir_430c089e7e18d7ac6ff937a35cc3312c.html > But i just wanted to point out that it's possible that we > could be smarter about this in notmuch, and wanted to make a space for > possible design discussion. > > a few concrete suggestions (intended as brainstorming, feedback welcome): > > * if we know our index expects english, and we have a message part that >*is not* english (e.g. Content-Language: es), we could avoid indexing >that part. I'd prefer leaving the choice of default stemmer to the user. > * during indexing, we could add a property to each message when we >discover a Content-Language header. this would let you do something >like "notmuch search property:lang=es" to find all messages >explicitly tagged as spanish. > > * (pretty crazy) If we're willing to search in another language we >could add an additional xapian database configured that language, and >we could index identified parts in that language. Do we need to have separate DB if we can use different stemmers dynamically? > * for text parts without a Content-Language: header, we could do some >concrete heuristics to guess the language. For example, choose the >1000 most popular words for each language we might know about, and >look for their presence in the text. Choose the language that is >most heavily represented, and store it in the index as a property. >this could be combined with the suggestions above. +1 for heuristics. > what do you think? what ideas are missing from the branstorm above? I'd > love to hear from people with multilingual mailboxes about how we might > be able to make notmuch work better for them. As an actively bilingual person (English and Spanish), I love this idea. Servilio -- Servilio Afre Puentes Programmer/Analyst, SHARCNET project RHPCS | http://www.rhpcs.mcmaster.ca SHARCNET | https://sharcnet.ca Compute Ontario | http://computeontario.ca Compute/Calcul Canada | http://computecanada.ca 905-525-9140, x22540 ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: multilingual notmuch (and Content-Language)
On Sun 2018-03-18 21:32:35 +0200, Jani Nikula wrote: > On Sun, 18 Mar 2018, Daniel Kahn Gillmor wrote: >> * if we know our index expects english, and we have a message part that >>*is not* english (e.g. Content-Language: es), we could avoid indexing >>that part. > > Why would we do that? Search mostly works just fine for non-English > languages, it's just that the *stemming* is not right. > >> what do you think? what ideas are missing from the branstorm above? I'd >> love to hear from people with multilingual mailboxes about how we might >> be able to make notmuch work better for them. > > With my limited understanding of this, stemming happens both at indexing > and searching. Basically at indexing, the term generator indexes both > the full and the stemmed version of words. I'm wondering if we could > look at Content-Language (and missing that, heuristics), and (if the > user so desires) use multiple term generators with different stemmers on > a per document basis. Or, use non-stemming indexing for unidentified or > unsupported languages. How far would that take us? Then, perhaps, we > could also perform language specific queries? > > I don't know how feasible that is, or if it would require Xapian > changes. thanks, this is exactly the kind of promising idea i was hoping my dumb questions and half-baked suggestions would provoke :) Maybe Olly or someone else with deeper knowledge of xapian can weigh in about the feasibility of this proposal? --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: multilingual notmuch (and Content-Language)
On Sun, 18 Mar 2018, Daniel Kahn Gillmor wrote: > * if we know our index expects english, and we have a message part that >*is not* english (e.g. Content-Language: es), we could avoid indexing >that part. Why would we do that? Search mostly works just fine for non-English languages, it's just that the *stemming* is not right. > what do you think? what ideas are missing from the branstorm above? I'd > love to hear from people with multilingual mailboxes about how we might > be able to make notmuch work better for them. With my limited understanding of this, stemming happens both at indexing and searching. Basically at indexing, the term generator indexes both the full and the stemmed version of words. I'm wondering if we could look at Content-Language (and missing that, heuristics), and (if the user so desires) use multiple term generators with different stemmers on a per document basis. Or, use non-stemming indexing for unidentified or unsupported languages. How far would that take us? Then, perhaps, we could also perform language specific queries? I don't know how feasible that is, or if it would require Xapian changes. BR, Jani. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: multilingual notmuch (and Content-Language)
Daniel Kahn Gillmor writes: > AIUI, xapian is pretty much committed to being a single-language > indexer. But i just wanted to point out that it's possible that we > could be smarter about this in notmuch, and wanted to make a space for > possible design discussion. > More precisely, it uses a single _stemmer_ when generating terms and when parsing queries. Nothing says that these have to correspond to a single human language. The stemmer is also configured at runtime, so it could in principle be per database configurable. I mention the possibility of a custom stemmer because that also seems like a natural place to put things like unicode normalization and accent removal. d ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
multilingual notmuch (and Content-Language)
https://tools.ietf.org/html/rfc3282 describes a Content-Language: header. https://tools.ietf.org/html/rfc8255 describes a multipart/multilingual Content-Type. notmuch currently uses xapian with a hard-coded English stemmer which works great for me as a monolingual American, but limits the applicability of notmuch to Anglophiles (people who speak English). That makes me sad. AIUI, xapian is pretty much committed to being a single-language indexer. But i just wanted to point out that it's possible that we could be smarter about this in notmuch, and wanted to make a space for possible design discussion. a few concrete suggestions (intended as brainstorming, feedback welcome): * if we know our index expects english, and we have a message part that *is not* english (e.g. Content-Language: es), we could avoid indexing that part. * during indexing, we could add a property to each message when we discover a Content-Language header. this would let you do something like "notmuch search property:lang=es" to find all messages explicitly tagged as spanish. * (pretty crazy) If we're willing to search in another language we could add an additional xapian database configured that language, and we could index identified parts in that language. * for text parts without a Content-Language: header, we could do some concrete heuristics to guess the language. For example, choose the 1000 most popular words for each language we might know about, and look for their presence in the text. Choose the language that is most heavily represented, and store it in the index as a property. this could be combined with the suggestions above. what do you think? what ideas are missing from the branstorm above? I'd love to hear from people with multilingual mailboxes about how we might be able to make notmuch work better for them. Regards, --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
