Re: web interface to notmuch
Brian Sniffen writes: >> Assuming that you had a sanitize_this_html_part() function available to >> you, do you think it would be possible to make this safe? Have you >> considered proposing it for inclusion in contrib upstream? > > Okay, https://github.com/briansniffen/notmuch/tree/nmweb is now rebased > onto the notmuchmail.org head as of this morning. All of the changes > are under contrib/notmuch-web. > Maybe a minor point, but there is at least one existing "notmuch-web" [1]. I'm not sure if John Lenz (in copy) considers the project a going concern, but it might be worth thinking about search engine collision. > I haven't done this before, so: exactly how would you like this proposed > for upstream inclusion? It looks from > https://notmuchmail.org/contributing/ like you'd like documentation, > tests, NEWS, and then `git send-email`. Is that right? > > Do you want this crunched into one commit, "write an e-mail client"? I think Daniel covered this well, but let me reinforce that we want to review things in meaningful, not too large chunks. [1]: https://bitbucket.org/wuzzeb/notmuch-web ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
Hi Brian-- On Wed 2017-12-06 10:00:19 -0500, Brian Sniffen wrote: > Okay, https://github.com/briansniffen/notmuch/tree/nmweb is now rebased > onto the notmuchmail.org head as of this morning. All of the changes > are under contrib/notmuch-web. thanks for doing this! traditionally, we've encouraged code contributions to come in in separate patches (not one big lump), with each commit clearly justifying their changes. And yes, tests would make it much more likely that this sweet contribution doesn't bitrot! one thing i note is you've included a copy of jquery directly. i think we want to avoid having retrieved or generated code in the repository. if there's an external dependency, the repo should point to it but it shouldn't include it. notmuch doesn't include gmime, but it states that it has gmime as a dependency (with specific versions), and lets the builder/maintainer retrieve the dependency themselves. This makes licensing, source vetting, and maintenance easier by ensuring that it all stays loosely coupled. Presumably you aren't signing up for ensuring that any security vulnerabilities in jquery get pushed into the notmuch repo too :) Maybe you could do a similar explicitly-stated dependency for jquery instead of including it directly? debian ships jquery as libjs-jquery, for example (though you could also use an explicit fetch from the web for systems that aren't running an OS that provides a packaged form of jquery). other folks might have other preferences, i hope they'll chime in. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
> Assuming that you had a sanitize_this_html_part() function available to > you, do you think it would be possible to make this safe? Have you > considered proposing it for inclusion in contrib upstream? Okay, https://github.com/briansniffen/notmuch/tree/nmweb is now rebased onto the notmuchmail.org head as of this morning. All of the changes are under contrib/notmuch-web. I haven't done this before, so: exactly how would you like this proposed for upstream inclusion? It looks from https://notmuchmail.org/contributing/ like you'd like documentation, tests, NEWS, and then `git send-email`. Is that right? Do you want this crunched into one commit, "write an e-mail client"? Thanks, Brian ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Wed, 1 Nov 2017, 14:38 Brian Sniffen, wrote: > On Oct 31, 2017, at 5:32 PM, Matthew Lear wrote > > I’ll push some fixes to encoding to github later today. > I've just tested with e3ba123 and there is a significant improvement. Also, attachments can be accessed from html too. And it's even faster than before! Sweet :-) I've still got a couple of mails which don't throw exceptions but do have issues when rendered in the browser (encoding of the body generates unreadable characters). I'll try to send you some examples tomorrow. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
> On Oct 31, 2017, at 5:32 PM, Matthew Lear wrote: > > > FWIW this link > (https://nmweb.evenmere.org/show/CACMMjMLecmXopb8AATjE3UuCnNLOO%2B5Nmev5X8K-UostDEUdrQ%40mail.gmail.com) > has the tag attachment applied to the message, but there is no attachment > shown. And another > (https://nmweb.evenmere.org/show/87d31artti.fsf%40inf-8657.int-evry.fr). Funny thing, the text of those attachments was being emitted by the app server... but since an iframe tag used in showing text/html parts *wasn’t* being closed, that didn’t matter. I’ll push some fixes to encoding to github later today.___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
Comparing with the Haskell based notmuch-web, while slightly slower to render a browser page with the same search terms as nmweb, I can view the email which causes nmweb to throw the encoding exception just fine in it. I guess something in that implementation is able to handle encoding differently. Regards, Matt On Tue, 31 Oct 2017, 21:32 Matthew Lear, wrote: > On Tue, Oct 31, 2017 at 7:21 PM, Brian Sniffen wrote: > >> >> > I'm no Python expert, but from a quick google it would seem like the >> cause >> > of such an exception is related to not using utf-8. >> >> Neat. So to get there, this has to be a text/html part. It has to have >> been decoded, either with the declared content type or with ascii. If a >> \u201c (left double quote) showed up, it didn't get decoded as >> ascii---and indeed, it looks like the content-type specifies latin-1. >> But now when we try to encode back, using the same latin-1, it fails? >> That's really neat. >> >> > Brian - do you think something needs modifying in nmweb.py to cater for >> > this type of thing, or is this somehow related my own mailstore (not >> sure >> > why that would be as my messages haven't been modified). >> >> Lots of mail has busted encoding. I've done some defensive work against >> that---look at decodeAnyway and shed a tear for purity---but clearly not >> enough. Can you send me a message that causes the problem? >> > > I'll need to fix up the text in the message because it's confidential. > That should be easy enough to do. > I'll send it to you once I've done that. > > One other thing - it looks like accessing attachments should work, but > I've seen messages in my local set up here which have attachments shown, > but I've not been able to retrieve them. > Not sure what would cause that. Also some messages which are tagged as > having attachments, don't have them shown by nmweb. > > FWIW this link ( > https://nmweb.evenmere.org/show/CACMMjMLecmXopb8AATjE3UuCnNLOO%2B5Nmev5X8K-UostDEUdrQ%40mail.gmail.com) > has the tag attachment applied to the message, but there is no attachment > shown. And another ( > https://nmweb.evenmere.org/show/87d31artti.fsf%40inf-8657.int-evry.fr). > > Maybe text/plain only emails are the ones which aren't problematic w.r.t. > having their attachments shown? > Cheers, > -- Matt > > > ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Tue, Oct 31, 2017 at 7:21 PM, Brian Sniffen wrote: > > > I'm no Python expert, but from a quick google it would seem like the > cause > > of such an exception is related to not using utf-8. > > Neat. So to get there, this has to be a text/html part. It has to have > been decoded, either with the declared content type or with ascii. If a > \u201c (left double quote) showed up, it didn't get decoded as > ascii---and indeed, it looks like the content-type specifies latin-1. > But now when we try to encode back, using the same latin-1, it fails? > That's really neat. > > > Brian - do you think something needs modifying in nmweb.py to cater for > > this type of thing, or is this somehow related my own mailstore (not sure > > why that would be as my messages haven't been modified). > > Lots of mail has busted encoding. I've done some defensive work against > that---look at decodeAnyway and shed a tear for purity---but clearly not > enough. Can you send me a message that causes the problem? > I'll need to fix up the text in the message because it's confidential. That should be easy enough to do. I'll send it to you once I've done that. One other thing - it looks like accessing attachments should work, but I've seen messages in my local set up here which have attachments shown, but I've not been able to retrieve them. Not sure what would cause that. Also some messages which are tagged as having attachments, don't have them shown by nmweb. FWIW this link ( https://nmweb.evenmere.org/show/CACMMjMLecmXopb8AATjE3UuCnNLOO%2B5Nmev5X8K-UostDEUdrQ%40mail.gmail.com) has the tag attachment applied to the message, but there is no attachment shown. And another ( https://nmweb.evenmere.org/show/87d31artti.fsf%40inf-8657.int-evry.fr). Maybe text/plain only emails are the ones which aren't problematic w.r.t. having their attachments shown? Cheers, -- Matt ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
> just remove it), but along the way of searching and viewing mail, I've > encountered quite a few occurrences of failing to UnicodeEncode. An example > backtrace looks like this: > > Traceback (most recent call last): > File "/usr/lib/python2.7/dist-packages/web/application.py", line 239, in > process > return self.handle() > File "/usr/lib/python2.7/dist-packages/web/application.py", line 230, in > handle > return self._delegate(fn, self.fvars, args) > File "/usr/lib/python2.7/dist-packages/web/application.py", line 420, in > _delegate > return handle_class(cls) > File "/usr/lib/python2.7/dist-packages/web/application.py", line 396, in > handle_class > return tocall(*args) > File "/b/git/notmuch-brians.git/contrib/notmuch-web/nmweb.py", line 153, > in GET > sprefix=webprefix) > File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 989, > in render > return self.environment.handle_exception(exc_info, True) > File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 754, > in handle_exception > reraise(exc_type, exc_value, tb) > File "templates/show.html", line 1, in top-level template code > {% extends "base.html" %} > File "templates/base.html", line 32, in top-level template code > {% block content %} > File "templates/show.html", line 12, in block "content" > {% for part in format_message(m.get_filename(),mid): %}{{ part|safe > }}{% endfor %} > File "/b/git/notmuch-brians.git/contrib/notmuch-web/nmweb.py", line 245, > in format_message_walk > tags=safe_tags).encode(part.get_content_charset('ascii'))) > UnicodeEncodeError: 'latin-1' codec can't encode character u'\u201c' in > position 1141: ordinal not in range(256) > > 127.0.0.1:60968 - - [31/Oct/2017 17:00:02] "HTTP/1.1 GET /show/ > 665d8c5c2b024898ae21951c4b8b4...@co2pr05mb747.namprd05.prod.outlook.com" - > 500 Internal Server Error > > I'm no Python expert, but from a quick google it would seem like the cause > of such an exception is related to not using utf-8. Neat. So to get there, this has to be a text/html part. It has to have been decoded, either with the declared content type or with ascii. If a \u201c (left double quote) showed up, it didn't get decoded as ascii---and indeed, it looks like the content-type specifies latin-1. But now when we try to encode back, using the same latin-1, it fails? That's really neat. > Brian - do you think something needs modifying in nmweb.py to cater for > this type of thing, or is this somehow related my own mailstore (not sure > why that would be as my messages haven't been modified). Lots of mail has busted encoding. I've done some defensive work against that---look at decodeAnyway and shed a tear for purity---but clearly not enough. Can you send me a message that causes the problem? In the mean time, I think like 245 ought to be, appropriately indented: tags=safe_tags).encode(part.get_content_charset('ascii'), 'xmlcharrefreplace')) Thanks for the report---investigating it showed me that the search box doesn't tolerate that character either. -Brian ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
Hi Matthew Sorry for just chiming in here out of the blue. I don't really know anything on the code you are discussing, but I have some experience with python. Matthew Lear writes: > Traceback (most recent call last): > File "/usr/lib/python2.7/dist-packages/web/application.py", line 239, in > process > return self.handle() > File "/usr/lib/python2.7/dist-packages/web/application.py", line 230, in > handle > return self._delegate(fn, self.fvars, args) > File "/usr/lib/python2.7/dist-packages/web/application.py", line 420, in > _delegate > return handle_class(cls) > File "/usr/lib/python2.7/dist-packages/web/application.py", line 396, in > handle_class > return tocall(*args) > File "/b/git/notmuch-brians.git/contrib/notmuch-web/nmweb.py", line 153, > in GET > sprefix=webprefix) > File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 989, > in render > return self.environment.handle_exception(exc_info, True) > File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 754, > in handle_exception > reraise(exc_type, exc_value, tb) > File "templates/show.html", line 1, in top-level template code > {% extends "base.html" %} > File "templates/base.html", line 32, in top-level template code > {% block content %} > File "templates/show.html", line 12, in block "content" > {% for part in format_message(m.get_filename(),mid): %}{{ part|safe > }}{% endfor %} > File "/b/git/notmuch-brians.git/contrib/notmuch-web/nmweb.py", line 245, > in format_message_walk > tags=safe_tags).encode(part.get_content_charset('ascii'))) My guess is that the function part.get_content_charset is requesting the encoding used for a message, providing 'ascii' as a backup if not found. It is getting 'latin-1', which is hence tried for encoding output. > UnicodeEncodeError: 'latin-1' codec can't encode character u'\u201c' > in position 1141: ordinal not in range(256) Here is an interactive python session to reproduce: >>> u = u'\u201c' >>> u u'\u201c' >>> type(u) # (un-encoded) >>> u.encode('utf-8') '\xe2\x80\x9c' # utf-8 for encoding work fine >>> print u.encode('utf-8') “ >>> print u.encode('latin-1') Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'latin-1' codec can't encode character u'\u201c' in position 0: ordinal not in range(256) The character is not encodable with latin-1. So one should check that the function getting the encoding is doing a proper job and if so blame the message information. Just my 2 cents Best regards -- Tomas ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Fri, 27 Oct 2017, 18:52 Brian Sniffen, wrote: > Daniel Kahn Gillmor writes: > > > On Fri 2017-10-27 00:04:21 -0400, Brian Sniffen wrote: > >> With bleach integrated (all of five lines), I think this is safe enough > >> to let random notmuch users run it. > > > > hm, bleach might be a little too aggressive. > > > > jrollins just pointed toward: > > > > https://nmweb.evenmere.org/show/87innmvvam.fsf%40ligo.caltech.edu > > That's fixed in 53403ecd, and there's some examples of bleach on a rope > at > https://nmweb.evenmere.org/show/20141107190321.GL23609%40odin.tremily.us > > The mbox URL is linkified, the many other link-like texts aren't. > > > Next/prev links are at the bottom, and a thread listing. I haven't > thought through how to get the body delivered immediately, but speed > seems acceptable. Next up, some style revisions---and I'd love > proposals for something that looks less awful, or at least makes the > interface more clear. UI design is a strong anti-specialty for me. > I've been running this today - standalone on localhost port 80 with the built in CherryPy web server on my mail store. First impressions are it's terrific :-) As my intended 'target' mail store will be geared towards a 'internal work stuff knowledge collection', lots of emails contain html links to intranet pages and sites. I can adapt the bleach usage to suit (or just remove it), but along the way of searching and viewing mail, I've encountered quite a few occurrences of failing to UnicodeEncode. An example backtrace looks like this: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/web/application.py", line 239, in process return self.handle() File "/usr/lib/python2.7/dist-packages/web/application.py", line 230, in handle return self._delegate(fn, self.fvars, args) File "/usr/lib/python2.7/dist-packages/web/application.py", line 420, in _delegate return handle_class(cls) File "/usr/lib/python2.7/dist-packages/web/application.py", line 396, in handle_class return tocall(*args) File "/b/git/notmuch-brians.git/contrib/notmuch-web/nmweb.py", line 153, in GET sprefix=webprefix) File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 989, in render return self.environment.handle_exception(exc_info, True) File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 754, in handle_exception reraise(exc_type, exc_value, tb) File "templates/show.html", line 1, in top-level template code {% extends "base.html" %} File "templates/base.html", line 32, in top-level template code {% block content %} File "templates/show.html", line 12, in block "content" {% for part in format_message(m.get_filename(),mid): %}{{ part|safe }}{% endfor %} File "/b/git/notmuch-brians.git/contrib/notmuch-web/nmweb.py", line 245, in format_message_walk tags=safe_tags).encode(part.get_content_charset('ascii'))) UnicodeEncodeError: 'latin-1' codec can't encode character u'\u201c' in position 1141: ordinal not in range(256) 127.0.0.1:60968 - - [31/Oct/2017 17:00:02] "HTTP/1.1 GET /show/ 665d8c5c2b024898ae21951c4b8b4...@co2pr05mb747.namprd05.prod.outlook.com" - 500 Internal Server Error I'm no Python expert, but from a quick google it would seem like the cause of such an exception is related to not using utf-8. Brian - do you think something needs modifying in nmweb.py to cater for this type of thing, or is this somehow related my own mailstore (not sure why that would be as my messages haven't been modified). Cheers, -- Matt ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
Daniel Kahn Gillmor writes: > On Fri 2017-10-27 00:04:21 -0400, Brian Sniffen wrote: >> With bleach integrated (all of five lines), I think this is safe enough >> to let random notmuch users run it. > > hm, bleach might be a little too aggressive. > > jrollins just pointed toward: > > https://nmweb.evenmere.org/show/87innmvvam.fsf%40ligo.caltech.edu That's fixed in 53403ecd, and there's some examples of bleach on a rope at https://nmweb.evenmere.org/show/20141107190321.GL23609%40odin.tremily.us The mbox URL is linkified, the many other link-like texts aren't. Next/prev links are at the bottom, and a thread listing. I haven't thought through how to get the body delivered immediately, but speed seems acceptable. Next up, some style revisions---and I'd love proposals for something that looks less awful, or at least makes the interface more clear. UI design is a strong anti-specialty for me. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Fri 2017-10-27 00:04:21 -0400, Brian Sniffen wrote: > With bleach integrated (all of five lines), I think this is safe enough > to let random notmuch users run it. hm, bleach might be a little too aggressive. jrollins just pointed toward: https://nmweb.evenmere.org/show/87innmvvam.fsf%40ligo.caltech.edu which i'm pretty sure had actual content initially (id:87innmvvam@ligo.caltech.edu) but it starts with stdin redirection (using a left angle bracket) and then the rest of the message is gone :/ --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
I've had a play with this this morning. It's great! The speed and page loading efficiency is fantastic. Would be really nice if we could go next/previous in the thread (yes I know I'm complaining about one extra mouse click). Also, if I select a date via the drop down I need to delete the timestamp that appears prior to searching, otherwise there is a xapian error. This is definitely a candidate solution for me, though. Thanks Brian! Cheers, Matt On Fri, 27 Oct 2017, 05:24 Daniel Kahn Gillmor, wrote: > On Fri 2017-10-27 00:04:21 -0400, Brian Sniffen wrote: > > Thanks! The part I'm happiest about is the speed: > > amen, it feels very lightweight. > > > Very careful examination would have shown that the em-dashes between > > author and subject were red for matches. Now matches are in italics. > > cool. perhaps assigning a class to those elements and stashing some CSS > would make that easier for folks to experiment with (and probably reduce > the bytecount transfered)? > > or would that hurt the rendering time for some reason i'm unaware of? i > haven't thought about these mechanics as much as you have. > > > Yup. The thread object isn't accessible by then: it existed in the > > scope of the search query, and is gone by the time we show the message. > > get_replies isn't available. So what's the alternative? > > get_thread_id(), search for that thread id, identify this message *in* > > that thread id, and then link to the next message with a "next" link? > > While doing it, why not show the thread structure at the bottom of the > > message, I guess. > > yep, i think that's right. > > > With bleach integrated (all of five lines), I think this is safe enough > > to let random notmuch users run it. The worst they'll do is expose > > their mailstore on tcp/8080. Any interest in taking this into the > > upstream contrib directory? > > Yes, i think this should move into contrib/ upstream. And we should > think about what might be the appropriate way to package it for debian, > too. > > --dkg > ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Fri 2017-10-27 00:04:21 -0400, Brian Sniffen wrote: > Thanks! The part I'm happiest about is the speed: amen, it feels very lightweight. > Very careful examination would have shown that the em-dashes between > author and subject were red for matches. Now matches are in italics. cool. perhaps assigning a class to those elements and stashing some CSS would make that easier for folks to experiment with (and probably reduce the bytecount transfered)? or would that hurt the rendering time for some reason i'm unaware of? i haven't thought about these mechanics as much as you have. > Yup. The thread object isn't accessible by then: it existed in the > scope of the search query, and is gone by the time we show the message. > get_replies isn't available. So what's the alternative? > get_thread_id(), search for that thread id, identify this message *in* > that thread id, and then link to the next message with a "next" link? > While doing it, why not show the thread structure at the bottom of the > message, I guess. yep, i think that's right. > With bleach integrated (all of five lines), I think this is safe enough > to let random notmuch users run it. The worst they'll do is expose > their mailstore on tcp/8080. Any interest in taking this into the > upstream contrib directory? Yes, i think this should move into contrib/ upstream. And we should think about what might be the appropriate way to package it for debian, too. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
Daniel Kahn Gillmor writes: > On Wed 2017-10-25 18:03:01 -0400, Brian Sniffen wrote: >> That's inspiring! Now there's a demo of nmweb at >> >> https://nmweb.evenmere.org/ > > this is very nice, Brian. Thanks! The part I'm happiest about is the speed: this is as fast as I remember gmail being. The Secret Ingredient is HTTP chunked encoding, accessed through web.py's generators, and careful page design---almost every byte from the server is renderable as it arrives, and later bytes never disrupt placement of earlier objects. > Your URL highlighter seems a bit trigger-happy though: > >https://nmweb.evenmere.org/show/8760s7zr47.fsf%40zancas.localnet > > I don't think bremner was trying to link to http://index.cc ! As a wise soul once told me, use a library and then blame them. This is the Mozilla Bleach library, used for both sanitizing text/html parts and for linkifying text/plain parts. But since that supports filtering: sure, this can only linkify things starting with 'http[s]://' >> It's possible to get it to dump the whole mbox by clicking through the >> obvious links; please consider exploring at >> https://nmweb.evenmere.org/search/monkey instead. > > this is interesting because it shows me threads where some messages have > monkey in them, but i can't tell which messages actually have the > relevant search term. Maybe it could highlight the found messages? Very careful examination would have shown that the em-dashes between author and subject were red for matches. Now matches are in italics. > Also, once i'm looking at one message, i don't see an easy way to go > "next" in the thread. Yup. The thread object isn't accessible by then: it existed in the scope of the search query, and is gone by the time we show the message. get_replies isn't available. So what's the alternative? get_thread_id(), search for that thread id, identify this message *in* that thread id, and then link to the next message with a "next" link? While doing it, why not show the thread structure at the bottom of the message, I guess. With bleach integrated (all of five lines), I think this is safe enough to let random notmuch users run it. The worst they'll do is expose their mailstore on tcp/8080. Any interest in taking this into the upstream contrib directory? -Brian ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Wed 2017-10-25 18:03:01 -0400, Brian Sniffen wrote: > That's inspiring! Now there's a demo of nmweb at > > https://nmweb.evenmere.org/ this is very nice, Brian. Your URL highlighter seems a bit trigger-happy though: https://nmweb.evenmere.org/show/8760s7zr47.fsf%40zancas.localnet I don't think bremner was trying to link to http://index.cc ! > It's possible to get it to dump the whole mbox by clicking through the > obvious links; please consider exploring at > https://nmweb.evenmere.org/search/monkey instead. this is interesting because it shows me threads where some messages have monkey in them, but i can't tell which messages actually have the relevant search term. Maybe it could highlight the found messages? Also, once i'm looking at one message, i don't see an easy way to go "next" in the thread. see: you show off a cool trick, you get requests for more cool tricks :) > There are not many monkeys in the inbox. speak for your own inbox, please. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
That's inspiring! Now there's a demo of nmweb at https://nmweb.evenmere.org/ It's possible to get it to dump the whole mbox by clicking through the obvious links; please consider exploring at https://nmweb.evenmere.org/search/monkey instead. There are not many monkeys in the inbox. -Brian Vladimir Panteleev writes: > Hi, > > Sorry to barge in, I noticed this thread and thought I'd try to have a > go at setting up a test DFeed instance. > > Here it is: > > http://dfeed-notmuch.k3.1azy.net/ > > There is some more info on the help page: > > http://dfeed-notmuch.k3.1azy.net/help > > Posting is supported, but it is currently (intentionally) unconfigured > for now. > > What do you think? > > On 2017-10-21 22:21, Daniel Kahn Gillmor wrote: >> On Sat 2017-10-21 23:00:00 +0300, Jani Nikula wrote: >>> For the list archive, we could restrict to displaying text/plain only. >> >> and text/x-diff, surely :) >> >> But yeah, good point. >> >> Brian, what do you think about such a constraint? would that make your >> implementation safe enough to put on the public Internet for a read-only >> archive? >> >> --dkg >> >> >> >> ___ >> notmuch mailing list >> notmuch@notmuchmail.org >> https://notmuchmail.org/mailman/listinfo/notmuch >> > > -- > Best regards, > Vladimir ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
Thanks for doing that, and thanks to all for the feedback and input so far. For the interface I want to set up, I'd like the ability to enter notmuch search syntax in an input box, and also show tags applied to messages. The interface presented by the current version of notmuch-web ticks a lot of boxes for me. Speed of being able to enter free-form search syntax (or ideally, selecting from a list of favourite or predefined searches) and returning the results quickly, I think are critical for how I'd like the interface to be used. I see that the DFeed instance has an advanced search facility, but it's a few clicks away. I've not looked seriously at the other suggestions so far in this thread, though. Cheers, -- Matt On Tue, Oct 24, 2017 at 4:00 AM, Vladimir Panteleev < thecybersha...@gmail.com> wrote: > Hi, > > Sorry to barge in, I noticed this thread and thought I'd try to have a go > at setting up a test DFeed instance. > > Here it is: > > http://dfeed-notmuch.k3.1azy.net/ > > There is some more info on the help page: > > http://dfeed-notmuch.k3.1azy.net/help > > Posting is supported, but it is currently (intentionally) unconfigured for > now. > > What do you think? > > > On 2017-10-21 22:21, Daniel Kahn Gillmor wrote: > >> On Sat 2017-10-21 23:00:00 +0300, Jani Nikula wrote: >> >>> For the list archive, we could restrict to displaying text/plain only. >>> >> >> and text/x-diff, surely :) >> >> But yeah, good point. >> >> Brian, what do you think about such a constraint? would that make your >> implementation safe enough to put on the public Internet for a read-only >> archive? >> >> --dkg >> >> >> >> ___ >> notmuch mailing list >> notmuch@notmuchmail.org >> https://notmuchmail.org/mailman/listinfo/notmuch >> >> > -- > Best regards, > Vladimir > > ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
Hi, Sorry to barge in, I noticed this thread and thought I'd try to have a go at setting up a test DFeed instance. Here it is: http://dfeed-notmuch.k3.1azy.net/ There is some more info on the help page: http://dfeed-notmuch.k3.1azy.net/help Posting is supported, but it is currently (intentionally) unconfigured for now. What do you think? On 2017-10-21 22:21, Daniel Kahn Gillmor wrote: On Sat 2017-10-21 23:00:00 +0300, Jani Nikula wrote: For the list archive, we could restrict to displaying text/plain only. and text/x-diff, surely :) But yeah, good point. Brian, what do you think about such a constraint? would that make your implementation safe enough to put on the public Internet for a read-only archive? --dkg ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch -- Best regards, Vladimir ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Sat 2017-10-21 23:00:00 +0300, Jani Nikula wrote: > For the list archive, we could restrict to displaying text/plain only. and text/x-diff, surely :) But yeah, good point. Brian, what do you think about such a constraint? would that make your implementation safe enough to put on the public Internet for a read-only archive? --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Thu, 19 Oct 2017, Daniel Kahn Gillmor wrote: > On Thu 2017-10-19 11:01:53 -0400, Brian Sniffen wrote: >> I put together something like this, visible at >> https://github.com/briansniffen/notmuch/tree/nmweb/contrib/notmuch-web >> >> It's not much of a service. I am pretty sure it is exploitable---that >> content in text/html parts of messages can do Bad Things to your >> session. > > I think this is the crux of the problem, right? I was noticing the > other day that notmuch's own mail archives are published in pipermail, > which is *absolutely terrible* compared to dealing with a mailstore with > notmuch as a frontend. I'd love to be able to expose the archive to the > public this way. For the list archive, we could restrict to displaying text/plain only. BR, Jani. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
I haven't looked at the backing code in a while, but I really like the public-inbox [1] approach to archival. Since Gmane died, Git (and a few other projects [2]) have also been using the author's hosted version. I haven't looked at the backing code in a while, but it's live Perl, not a static site. It uses Xapian for search (like notmuch), but I was unable to talk Eric into using notmuch directly because of our lack of Perl bindings [3]. Still, it's a pretty similar idea, and it may be a good fit for you. Previous discussion on this list in [4,5]. Cheers, Trevor [1]: https://public-inbox.org/README.html [2]: https://public-inbox.org/hosted.html [3]: https://public-inbox.org/meta/20141027005553.ga10...@dcvr.yhbt.net/ [4]: id:20141107190321.gl23...@odin.tremily.us [5]: id:20160820062931.gy30...@odin.tremily.us -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Thu 2017-10-19 16:00:33 -0400, Brian Sniffen wrote: > I don’t think they can be sanitized. Web tech moves so fast. well, there are at least a handful of python modules that claim to do some sort of sanitization. in debian alone, we have at least: python3-django-html-sanitizer python3-feedparser python3-bleach python3-w3lib so, one approach would be to just adopt one of them, and then it's their fault if it breaks :) I'm not saying it's a great approach, but it seems better than the current situation where no sanitization is done at all. > But maybe they can be isolated. GMail uses a separate domain for the > content from the UI; I have hopes about response headers and iframe > attributes. That's an interesting approach too, though it doesn't isolate message A from message B, which is a distinct concern. The worry isn't just that the content could take over the UI, right? Maybe isolation and sanitization can be used in combination? even if neither of them are perfect, it'd be a damn sight better than pipermail :P > Also, if the whole site’s static—not just the nmweb part—you probably > can’t hurt much. depends on what kind of harm you're talking about -- i think the privacy harms are potentially pretty serious. The public library is static, but if reading one book meant that you ended up reporting on your future reading habits (of any book) to some unknown third party, that would be pretty bad. --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
> On Oct 19, 2017, at 12:55 PM, Daniel Kahn Gillmor > wrote: > >> On Thu 2017-10-19 11:01:53 -0400, Brian Sniffen wrote: >> I put together something like this, visible at >> https://github.com/briansniffen/notmuch/tree/nmweb/contrib/notmuch-web >> >> It's not much of a service. I am pretty sure it is exploitable---that >> content in text/html parts of messages can do Bad Things to your >> session. > > I think this is the crux of the problem, right? I was noticing the > other day that notmuch's own mail archives are published in pipermail, > which is *absolutely terrible* compared to dealing with a mailstore with > notmuch as a frontend. I'd love to be able to expose the archive to the > public this way. > > Assuming that you had a sanitize_this_html_part() function available to > you, do you think it would be possible to make this safe? Have you > considered proposing it for inclusion in contrib upstream? I don’t think they can be sanitized. Web tech moves so fast. But maybe they can be isolated. GMail uses a separate domain for the content from the UI; I have hopes about response headers and iframe attributes. Also, if the whole site’s static—not just the nmweb part—you probably can’t hurt much. ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
On Thu 2017-10-19 11:01:53 -0400, Brian Sniffen wrote: > I put together something like this, visible at > https://github.com/briansniffen/notmuch/tree/nmweb/contrib/notmuch-web > > It's not much of a service. I am pretty sure it is exploitable---that > content in text/html parts of messages can do Bad Things to your > session. I think this is the crux of the problem, right? I was noticing the other day that notmuch's own mail archives are published in pipermail, which is *absolutely terrible* compared to dealing with a mailstore with notmuch as a frontend. I'd love to be able to expose the archive to the public this way. Assuming that you had a sanitize_this_html_part() function available to you, do you think it would be possible to make this safe? Have you considered proposing it for inclusion in contrib upstream? --dkg signature.asc Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
Re: web interface to notmuch
I put together something like this, visible at https://github.com/briansniffen/notmuch/tree/nmweb/contrib/notmuch-web It's not much of a service. I am pretty sure it is exploitable---that content in text/html parts of messages can do Bad Things to your session. I haven't thought nearly hard enough about how it will deal with multiple users. But it's < 250 lines of Python, so perhaps you can adapt it to what you need. It uses web.py, so you *could* run it standalone, but you'll probably be happier with Apache or nginx or something in front of it, handling TLS termination and that sort of thing. It's only approach to sending mail is generating mailto: links that will open in whatever client the user has configured. -Brian Matthew Lear writes: > Hello all. A little side project at work involves me trying to put together > part of a knowledge share system where users can query and search email > stored and indexed centrally (by offlineimap & notmuch). My intention is to > provide a means to support multiple concurrent read-only accesses to the > notmuch database from users' web browsers so they can query and search mail. > > Consider a few different email addresses being plugged into various > systems, all receiving email on different topics. I'd like to build an > application which presents a web frontend which I can run on the server > which fetches and indexes the mail, and thus present a web interface to > search all mail using notmuch. > > notmuch-web has not seen much development for a few years. > noservice looks pretty nifty but I'm a little unsure of the status and if > it's missing anything fundamental. > > I think my requirements are pretty basic: > > * Read-only access > * Search and display mail only (no sending), including html mails > * Freeform entry of search terms in accordance with notmuch-search-terms(7). > > Would anybody have any ideas about the best way to undertake such a project? > > notmuch-web and noservice definitively look like they could be leveraged, > but I don't know if I'd be better trying to construct something from the > ground up which is better suited / tailored to my requirements (which are > much less than either of the above were intended to fulfil). > > A standalone app would be preferred rather than having to rely on a web > server, although I'm not picky about infrastructure. Web based programming > is not my forte so I'd appreciate any feedback relating also to > implementation, currently available open source web frameworks which could > be used / considered / leveraged, etc. > > Many thanks, > -- Matt > ___ > notmuch mailing list > notmuch@notmuchmail.org > https://notmuchmail.org/mailman/listinfo/notmuch ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch
web interface to notmuch
Hello all. A little side project at work involves me trying to put together part of a knowledge share system where users can query and search email stored and indexed centrally (by offlineimap & notmuch). My intention is to provide a means to support multiple concurrent read-only accesses to the notmuch database from users' web browsers so they can query and search mail. Consider a few different email addresses being plugged into various systems, all receiving email on different topics. I'd like to build an application which presents a web frontend which I can run on the server which fetches and indexes the mail, and thus present a web interface to search all mail using notmuch. notmuch-web has not seen much development for a few years. noservice looks pretty nifty but I'm a little unsure of the status and if it's missing anything fundamental. I think my requirements are pretty basic: * Read-only access * Search and display mail only (no sending), including html mails * Freeform entry of search terms in accordance with notmuch-search-terms(7). Would anybody have any ideas about the best way to undertake such a project? notmuch-web and noservice definitively look like they could be leveraged, but I don't know if I'd be better trying to construct something from the ground up which is better suited / tailored to my requirements (which are much less than either of the above were intended to fulfil). A standalone app would be preferred rather than having to rely on a web server, although I'm not picky about infrastructure. Web based programming is not my forte so I'd appreciate any feedback relating also to implementation, currently available open source web frameworks which could be used / considered / leveraged, etc. Many thanks, -- Matt ___ notmuch mailing list notmuch@notmuchmail.org https://notmuchmail.org/mailman/listinfo/notmuch