[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04 2014, "W. Trevor King" wrote:
>>
>> I don't know what to paste, so i paste this:
>>
>> $ python
>> Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>
> It looks like you left out:
>
> from __future__ import unicode_literals
>
> Can you try again with that line as the first command?
>
>> >>> data = {'from': '\u017b'}
>> >>> print(type(data['from']))
>>
>
> which is why your data is a 'str' and not a 'unicode' instance.
>
>> >>> string = ' {from}\n'.format(**data)
>> >>> print string
>> \u017b
Quick update before getting to sleep:
$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> from __future__ import unicode_literals
>>> data = {'from': '\u017b'}
>>> print(type(data['from']))
>>> string = ' {from}\n'.format(**data)
>>> print(string)
?
so getting success there
$ PYTHONPATH=$PWD/bindings/python/ ./devel/nmbug/nmbug-status
...
Traceback (most recent call last):
File "devel/nmbug/nmbug-status", line 318, in
page.write(database=db, views=config['views'])
File "devel/nmbug/nmbug-status", line 94, in write
self._write_view(database=database, view=view, stream=stream)
File "devel/nmbug/nmbug-status", line 113, in _write_view
self._write_threads(threads=threads, stream=stream)
File "devel/nmbug/nmbug-status", line 215, in _write_threads
).format(**message_display_data))
File "/usr/lib64/python2.6/codecs.py", line 351, in write
data, consumed = self.encode(object, self.errors)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
position 176: ordinal not in range(128)
I'll dig deeper tomorrow.
Tomi
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04 2014, "W. Trevor King" wrote:
>
> >>> from __future__ import unicode_literals
> >>> import codecs
> >>> import locale
> >>> import sys
> >>> print(locale.getpreferredencoding()) # same as yours
> UTF-8
> >>> print(sys.getdefaultencoding()) # same as yours
> ascii
> >>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
> >>> print(_ENCODING) # double-check default encodings
> UTF-8
> >>> byte_stream = sys.stdout # copied from Page.write
> >>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
> >>> data = {'from': '\u017b'} # fake the troublesome data
> >>> print(type(data['from'])) # double-check unicode_literals
>
> >>> string = ' {from}\n'.format(**data)
> >>> stream.write(string)
> ?
>
> It looks like you'll have the same _ENCODING as I do (UTF-8). That
> means your stream should be wrapped in a UTF-8 StreamWriter, so I
> don't understand why it's converting to ASCII. Can you run through
> the above on your troublesome machine and confirm that stream.write()
> is still raising the exception? If it doesn't work, can you just
> paste that whole run in your next email?
I don't know what to paste, so i paste this:
$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> data = {'from': '\u017b'}
>>> print(type(data['from']))
>>> string = ' {from}\n'.format(**data)
>>> print string
\u017b
and then:
>>> data = {'from': u'\u017b'}
>>> print(type(data['from']))
>>> string = ' {from}\n'.format(**data)
Traceback (most recent call last):
File "", line 1, in
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
>>> position 0: ordinal not in range(128)
... and ...
>>> import os
>>> print os.environ['LANG']
en_US.UTF-8
> Thanks,
> Trevor
Tomi
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04 2014, "W. Trevor King" wrote: > On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote: >> On Tue, Feb 04 2014, W. Trevor King wrote: >> > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote: >> >> File "devel/nmbug/nmbug-status", line 197, in _write_threads >> >> ).format(**message_display_data)) >> >> File "/usr/lib64/python2.6/codecs.py", line 351, in write >> >> data, consumed = self.encode(object, self.errors) >> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in >> >> position 176: ordinal not in range(128) >> > ? >> >> LANG=en_US.UTF-8 > > My current guess is that LANG is set in your terminal, but that you > ran nmbug-status from a cron job where it wasn't set. If that's the > case, try adding: > > export LANG=en_US.UTF-8 > > to your cron job (or replace with whichever encoding you like). Yes guess is good, but incorrect I run it on the command line. Now I did the following: $ git remote add tremily git://tremily.us/notmuch.git $ git fetch --all $ git checkout nmbug-status-python3 commit d64ef215b62fa74a96fdb4c93e1f6f7abf7c80c6 and then rerun nmbug-status -- and I am still having the same problem. I will investigate this further... > We don't care about the preferred language yet, but we could [1]. > There's actually not all that much that needs translating for > nmbug-status. Yes, that is unfortunate that David (Taavetti in Finnish) did an uneducated choice by choosing english as the preferred language. It should always have been written in Finnish so we would not have this conversation >;) > > Cheers, > Trevor > > [1]: http://docs.python.org/3/library/gettext.html Tomi
sanitization of args notmuch-cli in notmuch-emacs
On Tue, Feb 04 2014, David Bremner wrote: > Tomi Ollila writes: > >> >> Maybe the cli should be fixed ? (and/or make emacs MUA resilient to >> this kind of result) >> > > Would it make any sense to output errors in structured format? I guess > the downside is it would be harder for a human user to read. notmuch count --batch outputs just numbers separated by newline -- and with emacs stdout & stderr are in the same stream (in this case too) For that we'd need structured like notmuch count --batch --format=sexp and then write errors there... I don't think we can find SomeOne(tm) to do this -- or the ?berversion like 'notmuch execute' which takes sexp/json document in stdin and spits out sexp/json document for programs to parse ;D > d Romi
notmuch killed due to out of memory - how to move forward
On 4 February 2014 08:25, Jani Nikula wrote: > On Mon, 03 Feb 2014, Hamish Downer wrote: >> I recently deleted almost 3 old messages from my maildirs, and >> since I did that, notmuch new has not managed to complete. I have it >> running on a server with 1 GB of RAM, and the output is telling me >> >> Cleaned up 25515 of 29803 messages (1m 10s remaining). >> >> at the point it is killed. I have followed it up to this point with >> top and free and I can see memory usage growing to something like >> 700MB (RES) at which point the process is killed. > > Please try SIGINT, or ^C, on notmuch new before it gets killed. That > should be handled gracefully, making progress, and letting you chop up > and eventually finish the operation. Please let us know if this helps. That did it, thank you. Good to know Ctrl-C is gracefully handled. > Needless to say, we shouldn't use that much memory just to delete files > from the index. I have saved a copy of the index from before I fixed the problem. I am happy to re-run it with a rebuilt notmuch if that would be useful from the point of view of uncovering bugs. Otherwise I'll carry on as a happy notmuch user Hamish
sanitization of args notmuch-cli in notmuch-emacs
On Tue, 04 Feb 2014, David Bremner wrote: > Tomi Ollila writes: > >> >> Maybe the cli should be fixed ? (and/or make emacs MUA resilient to >> this kind of result) >> > > Would it make any sense to output errors in structured format? I guess > the downside is it would be harder for a human user to read. If we want to do something like this would something like --structured-errors be possible? Best wishes Mark
[PATCH 0/5] lib: make folder: prefix literal
Quoth Rob Browning on Jan 31 at 1:19 pm:
> Austin Clements writes:
>
> > folder: could work the way I suggested (simply the path to the file,
> > with {cur,new} stripped off).
>
> Hmm, so would notmuch try to guess whether or not it's dealing with a
> maildir++ tree, and if so convert folder:foo to a search of .foo, and/or
> folder:foo/bar to .foo.bar? Or would the user just need to know to say
> folder:.foo and folder:.foo.bar?
My opinion on this has changed over time, but I don't think we should
try to interpret Maildir++ trees specially. That is, the user would
have to say folder:.foo.bar if they're using Maildir++. The "." seems
as good as a "/" for a separator, so we might as well not translate
it. The leading "." is annoying, but *shrug* so is Maildir++.
> And if we're only planning special treatment for for maildir-like
> stores, then I wonder if the term should just be maildir:?
The simple algorithm of taking the relative path and stripping
{new,cur} (if present) does a good job of supporting both Maildir and
non-Maildir stores (while balancing this support with simplicity,
predictability, and usability).
> Though folder: would make more sense if the long-term goal was to have a
> "DTRT" term. But in that case, I wonder if it might eventually be
> expected to support mixed trees, i.e. say a tree containing maildir++
> and mh subdirs, and if so, how that should be handled.
The simple {new,cur}-stripping algorithm already does fairly well at
this. Worrying more about mixed Maildir++ and MH stores seems
unnecessary to me unless someone demonstrates and actual need.
> > many shells support "**" for recursive path matching and people are
> > already quite familiar with glob patterns for paths, so why not simply
> > adopt this?
>
> rsync too.
Ah, sure enough. Even better!
[PATCH 0/5] lib: make folder: prefix literal
Quoth Jani Nikula on Feb 01 at 4:54 pm:
> On Fri, 31 Jan 2014, Austin Clements wrote:
> > What if we introduce two prefixes, say folder: and path: (maybe dir:?)
> > to address both use cases, each as naturally as possible? Both would
> > be boolean prefixes because of the limitations of probabilistic
> > prefixes, but we could take advantage of Jani's idea of generating
> > several boolean terms.
>
> Agreed. On to details:
>
> > folder: could work the way I suggested (simply the path to the file,
> > with {cur,new} stripped off).
>
> What if the file is not in a folder named cur/new? I suggest indexing
> the folder as-is, if only for some backwards compatibility.
Agreed. I believe this will also support MH, if I understand MH
correctly (does anyone actually use MH?)
> What if there is not all of cur/new/tmp folders? I suggest ignoring
> that, and only look at the path to the file being indexed. This is
> simplest to implement, and it does not matter if the sibling directories
> come and go, and for this reason also unsurprising.
That sounds good to me.
> For top level cur/new, index the empty string "".
Yes.
> > path: would support file system search
> > uses. These seem more varied, but I think fall into exact match and
> > recursive match. Since I don't have this use case, I can't have any
> > strong opinions about syntax, but I'll throw out an idea: many shells
> > support "**" for recursive path matching and people are already quite
> > familiar with glob patterns for paths, so why not simply adopt this?
> > In other words, when adding the path "a/b/cur/x:2," add path: terms
> > "a/b/cur" and "a/b/**" and "a/**" and "**".
>
> Since folder: would cover the cur/new cases, I suggest the non-recursive
> variant of path: prefix is the exact filesystem folder name as-is (with
> the top level being the empty string ""). I presume this is what you
> meant too.
Yes. I suppose I didn't actually say it, but that's what I was
thinking.
> I kind of like the "/**" suffix for recursive, but there's two small
> wrinkles: 1) it needs quoting on the command line (unlike my original
> suggestion of just "/" suffix), and 2) what should the top level
> recursive search be? path:"**" or path:"/**" or path:"./**"? I guess the
> first one is most obvious?
The shell quoting is annoying, but depending on the shell, it should
at least give an error (zsh) or Just Work (apparently bash and sh pass
the unexpanded glob through if it doesn't match anything?).
> So here's what my original suggestions would become:
>
> >> Here's a thought. With boolean prefix folder:, we can devise a scheme
> >> where the folder: query defines what is to be matched.
> >>
> >> For example:
> >>
> >> folder:foo match files in foo, foo/new, and foo/cur.
>
> -> folder:foo
>
> >> folder:foo/match all files in all subdirectories under foo (this
> >>would handle Tomi's use case), including foo/new and foo/cur.
>
> -> path:"foo/**"
>
> >> folder:foo/. match in foo only, and specifically not in foo/cur or
> >> foo/new.
>
> -> path:foo
>
> >> folder:foo/new match in foo/new, and specifically not in foo/cur (this
> >>allows distinguishing between messages in cur and new).
>
> -> path:foo/new
>
> >> folder:/ match everything.
>
> -> path:"**"
>
> >> folder:/. match in top level maildir only.
>
> -> path:""
>
> >> folder:"" match in top level maildir, including cur/new.
>
> -> folder:""
>
>
> I'd like these details to be ironed out and agreed on before I send the
> next version.
This all looks good to me.
> BR,
> Jani.
notmuch killed due to out of memory - how to move forward
Jani Nikula writes: > On Mon, 03 Feb 2014, Hamish Downer wrote: >> Hello >> >> I recently deleted almost 3 old messages from my maildirs, and >> since I did that, notmuch new has not managed to complete. I have it >> running on a server with 1 GB of RAM, and the output is telling me >> >> Cleaned up 25515 of 29803 messages (1m 10s remaining). >> >> at the point it is killed. I have followed it up to this point with >> top and free and I can see memory usage growing to something like >> 700MB (RES) at which point the process is killed. >> >> I am using notmuch 0.17 (from the debian jessie package 0.17-3) on >> debian wheezy 64 bit. >> >> Any pointers as to what to do to get past this point would be greatly >> appreciated. If I have just deleted that many messages should I just >> expect notmuch to use that much memory? (Unfortunately the company >> supplying the server does not allow swap to be enabled so I can't fudge >> it that way) >> >> I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't >> produce a report (though I was unsure if I needed to rebuild notmuch >> before that would work). >> >> Happy to build notmuch from source if this might be an interesting bug >> worth tracking down. > > Please try SIGINT, or ^C, on notmuch new before it gets killed. That > should be handled gracefully, making progress, and letting you chop up > and eventually finish the operation. Please let us know if this helps. > > Needless to say, we shouldn't use that much memory just to delete files > from the index. > When I initially indexed my email archives into notmuch, I had problems with thermal cutouts, so I did something like this, which automated the workaround you mention: while :; do timeout -s INT 2m notmuch new sleep 600 done Of course, with memory issues as the constraint rather than temperature, the sleep should not be necessary. -- regards, kushal -- next part -- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 489 bytes Desc: not available URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140204/492dda0e/attachment.pgp>
[PATCH 0/5] lib: make folder: prefix literal
Austin Clements writes:
> The simple algorithm of taking the relative path and stripping
> {new,cur} (if present) does a good job of supporting both Maildir and
> non-Maildir stores (while balancing this support with simplicity,
> predictability, and usability).
Unless, of course, the user has legitimate folders named cur and new,
but perhaps that'll just end up a "don't do that then" FAQ...
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4
Re: [PATCH 0/5] lib: make folder: prefix literal
Austin Clements writes: > Agreed. I believe this will also support MH, if I understand MH > correctly (does anyone actually use MH?) When I started notmuch, I had all of my mail in one-message-per-file in various directories, (without these silly "cur" and "new" directories that maildir uses). At some point, I did a mass conversion of all of my directories to be roughly: ~/mail//MM So that I keep directories small by just delivering a month's worth of mail to each directory. This conversion, (and the delivery agent I am currently using, maildrop), happen to create the silly "cur" and "new" directories. So most of my mail still is in maildir format now. But I do have a few messages in non-maildir directories. These have generally come into being in cases such as someone providing me a message to demonstrate a notmuch bug or use case. So in cases like this I did things like: mkdir ~/mail/bug-description cp example-file ~/mail/bug-description I also have a few directories created similarly when I've copied some downloaded archives from a mailing list into my mail storage. (But often I've used mb2md for those so the conversion has accidentally created maildir directories). I don't know if the non-maildir directories I have are strict "mh format", (did it have filenames with sequential numbers? I don't recall). But my intention with notmuch from the beginning was to support any one-message-per-file layout without enforcing any particular naming of directories or files. And I would like to see that preserved. Since then, we have also supported various semantics when people do encode information in directories and filenames, (such as ignoring "cur"/"new" and interpreting maildir flags). This kind of thing does seem good. -Carl pgpZXkuCQyczA.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 2/2] emacs: Prefer Content-Description over filename for part buttons
On Tue, Feb 04 2014, "W. Trevor King" wrote: > On Mon, Feb 03, 2014 at 10:44:39PM +0200, Tomi Ollila wrote: >> having notmuch patch email as expected test output feels a bit >> confusing to me -- especially as 'git grep' may catch some of the >> (possibly future-outdated) content... > > There were two very similar patches with filename attachments in the > corpus, so I used them for the tests ;). I can convert that test to > use emacs_fcc_message or some other auto-generated content if you'd > prefer. You're right; corpus seems to be full of notmuch patch emails... ... and (although?) (just noticed)... $ grep '^ *[+-].*\' test/* test/*/* test/emacs.expected-output/attachment:+if (pw_buf_size == -1) pw_buf_size = 64; test/emacs.expected-output/attachment:+if (pw_buf_size == -1) pw_buf_size = 64; ... I am not giving much resistance to keep the patch as it is :D > > Cheers, > Trevor > Tomi
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04 2014, "W. Trevor King" wrote:
> On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
>> data['message-id-term'] = 'id:"{}"'.format(value)
>> ValueError: zero length field name in format
>
> Oops, Python 2.6 still needs explicit indexes ('{0}', not '{}'). It's
> an easy fix, so I'll queue it for v2. You're still going to need
> Python 2.7 or greater for collections.OrderedDict(). We could word
> around that too, but do we really care about 2.6? I don't expect that
> the installed nmbug-status userbase is so large and backward that
> upgrading to 2.7 will be that hard ;). 2.6 isn't even getting
> security fixes anymore [1], so I think it's time to migrate :p.
Probably not many cares about 2.6; I already use argparse and I can add
that OrderedDict() too. Still {0} is easy enough to do :D. I am running
this notmuch & nmbug in Scientific Linux 6.2 machine which has python 2.6
-- and this is the only machine where I can review your nmbug-status
changes ;)
>> File "devel/nmbug/nmbug-status", line 197, in _write_threads
>> ).format(**message_display_data))
>> File "/usr/lib64/python2.6/codecs.py", line 351, in write
>> data, consumed = self.encode(object, self.errors)
>> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
>> position 176: ordinal not in range(128)
>
> Hmm. __future__'s unicode_literals should be giving us a Unicode
> target, so I'm not sure why we'd have trouble injecting Unicode. This
> works fine for me on Python 2.7 and 3.3. Maybe you just have a funky
> encoding? What is your:
>
> $ locale
> LANG=en_US.UTF-8
> ?
> $ python -c 'import locale, sys; print(locale.getpreferredencoding() or
> sys.getdefaultencoding())'
> UTF-8
LANG=en_US.UTF-8
all other LC_* variables en_US.UTF-8 except
LC_TIME=en_GB.utf8
LC_ALL empty (naturally)
python -c 'import locale; print(locale.getpreferredencoding())'
UTF-8
python -c 'import sys; print(sys.getdefaultencoding())'
ascii
anyway, if this doesn't resolve out and there is no resistance to require
python 2.7 (that means from David) I can hack around this to get this
reviewed.
>
> Cheers,
> Trevor
Tomi
>
> [1]: http://www.python.org/download/releases/2.6.9/
Re: [PATCH 0/5] lib: make folder: prefix literal
Austin Clements writes:
> The simple algorithm of taking the relative path and stripping
> {new,cur} (if present) does a good job of supporting both Maildir and
> non-Maildir stores (while balancing this support with simplicity,
> predictability, and usability).
Unless, of course, the user has legitimate folders named cur and new,
but perhaps that'll just end up a "don't do that then" FAQ...
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 0/5] lib: make folder: prefix literal
Quoth Rob Browning on Jan 31 at 1:19 pm:
> Austin Clements writes:
>
> > folder: could work the way I suggested (simply the path to the file,
> > with {cur,new} stripped off).
>
> Hmm, so would notmuch try to guess whether or not it's dealing with a
> maildir++ tree, and if so convert folder:foo to a search of .foo, and/or
> folder:foo/bar to .foo.bar? Or would the user just need to know to say
> folder:.foo and folder:.foo.bar?
My opinion on this has changed over time, but I don't think we should
try to interpret Maildir++ trees specially. That is, the user would
have to say folder:.foo.bar if they're using Maildir++. The "." seems
as good as a "/" for a separator, so we might as well not translate
it. The leading "." is annoying, but *shrug* so is Maildir++.
> And if we're only planning special treatment for for maildir-like
> stores, then I wonder if the term should just be maildir:?
The simple algorithm of taking the relative path and stripping
{new,cur} (if present) does a good job of supporting both Maildir and
non-Maildir stores (while balancing this support with simplicity,
predictability, and usability).
> Though folder: would make more sense if the long-term goal was to have a
> "DTRT" term. But in that case, I wonder if it might eventually be
> expected to support mixed trees, i.e. say a tree containing maildir++
> and mh subdirs, and if so, how that should be handled.
The simple {new,cur}-stripping algorithm already does fairly well at
this. Worrying more about mixed Maildir++ and MH stores seems
unnecessary to me unless someone demonstrates and actual need.
> > many shells support "**" for recursive path matching and people are
> > already quite familiar with glob patterns for paths, so why not simply
> > adopt this?
>
> rsync too.
Ah, sure enough. Even better!
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04 2014, "W. Trevor King" wrote:
>>
>> I don't know what to paste, so i paste this:
>>
>> $ python
>> Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>
> It looks like you left out:
>
> from __future__ import unicode_literals
>
> Can you try again with that line as the first command?
>
>> >>> data = {'from': '\u017b'}
>> >>> print(type(data['from']))
>>
>
> which is why your data is a 'str' and not a 'unicode' instance.
>
>> >>> string = ' {from}\n'.format(**data)
>> >>> print string
>> \u017b
Quick update before getting to sleep:
$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> from __future__ import unicode_literals
>>> data = {'from': '\u017b'}
>>> print(type(data['from']))
>>> string = ' {from}\n'.format(**data)
>>> print(string)
Ż
so getting success there
$ PYTHONPATH=$PWD/bindings/python/ ./devel/nmbug/nmbug-status
...
Traceback (most recent call last):
File "devel/nmbug/nmbug-status", line 318, in
page.write(database=db, views=config['views'])
File "devel/nmbug/nmbug-status", line 94, in write
self._write_view(database=database, view=view, stream=stream)
File "devel/nmbug/nmbug-status", line 113, in _write_view
self._write_threads(threads=threads, stream=stream)
File "devel/nmbug/nmbug-status", line 215, in _write_threads
).format(**message_display_data))
File "/usr/lib64/python2.6/codecs.py", line 351, in write
data, consumed = self.encode(object, self.errors)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
position 176: ordinal not in range(128)
I'll dig deeper tomorrow.
Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 0/5] lib: make folder: prefix literal
Quoth Jani Nikula on Feb 01 at 4:54 pm:
> On Fri, 31 Jan 2014, Austin Clements wrote:
> > What if we introduce two prefixes, say folder: and path: (maybe dir:?)
> > to address both use cases, each as naturally as possible? Both would
> > be boolean prefixes because of the limitations of probabilistic
> > prefixes, but we could take advantage of Jani's idea of generating
> > several boolean terms.
>
> Agreed. On to details:
>
> > folder: could work the way I suggested (simply the path to the file,
> > with {cur,new} stripped off).
>
> What if the file is not in a folder named cur/new? I suggest indexing
> the folder as-is, if only for some backwards compatibility.
Agreed. I believe this will also support MH, if I understand MH
correctly (does anyone actually use MH?)
> What if there is not all of cur/new/tmp folders? I suggest ignoring
> that, and only look at the path to the file being indexed. This is
> simplest to implement, and it does not matter if the sibling directories
> come and go, and for this reason also unsurprising.
That sounds good to me.
> For top level cur/new, index the empty string "".
Yes.
> > path: would support file system search
> > uses. These seem more varied, but I think fall into exact match and
> > recursive match. Since I don't have this use case, I can't have any
> > strong opinions about syntax, but I'll throw out an idea: many shells
> > support "**" for recursive path matching and people are already quite
> > familiar with glob patterns for paths, so why not simply adopt this?
> > In other words, when adding the path "a/b/cur/x:2," add path: terms
> > "a/b/cur" and "a/b/**" and "a/**" and "**".
>
> Since folder: would cover the cur/new cases, I suggest the non-recursive
> variant of path: prefix is the exact filesystem folder name as-is (with
> the top level being the empty string ""). I presume this is what you
> meant too.
Yes. I suppose I didn't actually say it, but that's what I was
thinking.
> I kind of like the "/**" suffix for recursive, but there's two small
> wrinkles: 1) it needs quoting on the command line (unlike my original
> suggestion of just "/" suffix), and 2) what should the top level
> recursive search be? path:"**" or path:"/**" or path:"./**"? I guess the
> first one is most obvious?
The shell quoting is annoying, but depending on the shell, it should
at least give an error (zsh) or Just Work (apparently bash and sh pass
the unexpanded glob through if it doesn't match anything?).
> So here's what my original suggestions would become:
>
> >> Here's a thought. With boolean prefix folder:, we can devise a scheme
> >> where the folder: query defines what is to be matched.
> >>
> >> For example:
> >>
> >> folder:foo match files in foo, foo/new, and foo/cur.
>
> -> folder:foo
>
> >> folder:foo/match all files in all subdirectories under foo (this
> >>would handle Tomi's use case), including foo/new and foo/cur.
>
> -> path:"foo/**"
>
> >> folder:foo/. match in foo only, and specifically not in foo/cur or
> >> foo/new.
>
> -> path:foo
>
> >> folder:foo/new match in foo/new, and specifically not in foo/cur (this
> >>allows distinguishing between messages in cur and new).
>
> -> path:foo/new
>
> >> folder:/ match everything.
>
> -> path:"**"
>
> >> folder:/. match in top level maildir only.
>
> -> path:""
>
> >> folder:"" match in top level maildir, including cur/new.
>
> -> folder:""
>
>
> I'd like these details to be ironed out and agreed on before I send the
> next version.
This all looks good to me.
> BR,
> Jani.
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: sanitization of args notmuch-cli in notmuch-emacs
On Tue, 04 Feb 2014, David Bremner wrote: > Tomi Ollila writes: > >> >> Maybe the cli should be fixed ? (and/or make emacs MUA resilient to >> this kind of result) >> > > Would it make any sense to output errors in structured format? I guess > the downside is it would be harder for a human user to read. If we want to do something like this would something like --structured-errors be possible? Best wishes Mark ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 08:40:18PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, W. Trevor King wrote:
> >
> > >>> from __future__ import unicode_literals
> > >>> import codecs
> > >>> import locale
> > >>> import sys
> > >>> print(locale.getpreferredencoding()) # same as yours
> > UTF-8
> > >>> print(sys.getdefaultencoding()) # same as yours
> > ascii
> > >>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
> > >>> print(_ENCODING) # double-check default encodings
> > UTF-8
> > >>> byte_stream = sys.stdout # copied from Page.write
> > >>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
> > >>> data = {'from': '\u017b'} # fake the troublesome data
> > >>> print(type(data['from'])) # double-check unicode_literals
> >
> > >>> string = ' {from}\n'.format(**data)
> > >>> stream.write(string)
> > Ż
> >
> > It looks like you'll have the same _ENCODING as I do (UTF-8). That
> > means your stream should be wrapped in a UTF-8 StreamWriter, so I
> > don't understand why it's converting to ASCII. Can you run through
> > the above on your troublesome machine and confirm that stream.write()
> > is still raising the exception? If it doesn't work, can you just
> > paste that whole run in your next email?
>
> I don't know what to paste, so i paste this:
>
> $ python
> Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
It looks like you left out:
from __future__ import unicode_literals
Can you try again with that line as the first command?
> >>> data = {'from': '\u017b'}
> >>> print(type(data['from']))
>
which is why your data is a 'str' and not a 'unicode' instance.
> >>> string = ' {from}\n'.format(**data)
> >>> print string
> \u017b
>
> and then:
>
> >>> data = {'from': u'\u017b'}
This works around the lack of unicode_literals with an explicit u''.
> >>> print(type(data['from']))
>
> >>> string = ' {from}\n'.format(**data)
> Traceback (most recent call last):
> File "", line 1, in
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
However, without unicode_literals or an explicit u'', you're format
string '…{from}' is a str (it should be a 'unicode' instance with
unicode_literals).
> >>> import os
> >>> print os.environ['LANG']
> en_US.UTF-8
That's good anyway ;). Thanks for digging into this :).
Cheers,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 08:40:18PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, W. Trevor King wrote:
> >
> > >>> from __future__ import unicode_literals
> > >>> import codecs
> > >>> import locale
> > >>> import sys
> > >>> print(locale.getpreferredencoding()) # same as yours
> > UTF-8
> > >>> print(sys.getdefaultencoding()) # same as yours
> > ascii
> > >>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
> > >>> print(_ENCODING) # double-check default encodings
> > UTF-8
> > >>> byte_stream = sys.stdout # copied from Page.write
> > >>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
> > >>> data = {'from': '\u017b'} # fake the troublesome data
> > >>> print(type(data['from'])) # double-check unicode_literals
> >
> > >>> string = ' {from}\n'.format(**data)
> > >>> stream.write(string)
> > ?
> >
> > It looks like you'll have the same _ENCODING as I do (UTF-8). That
> > means your stream should be wrapped in a UTF-8 StreamWriter, so I
> > don't understand why it's converting to ASCII. Can you run through
> > the above on your troublesome machine and confirm that stream.write()
> > is still raising the exception? If it doesn't work, can you just
> > paste that whole run in your next email?
>
> I don't know what to paste, so i paste this:
>
> $ python
> Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
> [GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
It looks like you left out:
from __future__ import unicode_literals
Can you try again with that line as the first command?
> >>> data = {'from': '\u017b'}
> >>> print(type(data['from']))
>
which is why your data is a 'str' and not a 'unicode' instance.
> >>> string = ' {from}\n'.format(**data)
> >>> print string
> \u017b
>
> and then:
>
> >>> data = {'from': u'\u017b'}
This works around the lack of unicode_literals with an explicit u''.
> >>> print(type(data['from']))
>
> >>> string = ' {from}\n'.format(**data)
> Traceback (most recent call last):
> File "", line 1, in
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
However, without unicode_literals or an explicit u'', you're format
string '?{from}' is a str (it should be a 'unicode' instance with
unicode_literals).
> >>> import os
> >>> print os.environ['LANG']
> en_US.UTF-8
That's good anyway ;). Thanks for digging into this :).
Cheers,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20140204/89b717bd/attachment.pgp>
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04 2014, "W. Trevor King" wrote:
>
> >>> from __future__ import unicode_literals
> >>> import codecs
> >>> import locale
> >>> import sys
> >>> print(locale.getpreferredencoding()) # same as yours
> UTF-8
> >>> print(sys.getdefaultencoding()) # same as yours
> ascii
> >>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
> >>> print(_ENCODING) # double-check default encodings
> UTF-8
> >>> byte_stream = sys.stdout # copied from Page.write
> >>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
> >>> data = {'from': '\u017b'} # fake the troublesome data
> >>> print(type(data['from'])) # double-check unicode_literals
>
> >>> string = ' {from}\n'.format(**data)
> >>> stream.write(string)
> Ż
>
> It looks like you'll have the same _ENCODING as I do (UTF-8). That
> means your stream should be wrapped in a UTF-8 StreamWriter, so I
> don't understand why it's converting to ASCII. Can you run through
> the above on your troublesome machine and confirm that stream.write()
> is still raising the exception? If it doesn't work, can you just
> paste that whole run in your next email?
I don't know what to paste, so i paste this:
$ python
Python 2.6.6 (r266:84292, Nov 21 2013, 12:39:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> data = {'from': '\u017b'}
>>> print(type(data['from']))
>>> string = ' {from}\n'.format(**data)
>>> print string
\u017b
and then:
>>> data = {'from': u'\u017b'}
>>> print(type(data['from']))
>>> string = ' {from}\n'.format(**data)
Traceback (most recent call last):
File "", line 1, in
UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
>>> position 0: ordinal not in range(128)
... and ...
>>> import os
>>> print os.environ['LANG']
en_US.UTF-8
> Thanks,
> Trevor
Tomi
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04 2014, "W. Trevor King" wrote: > On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote: >> On Tue, Feb 04 2014, W. Trevor King wrote: >> > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote: >> >> File "devel/nmbug/nmbug-status", line 197, in _write_threads >> >> ).format(**message_display_data)) >> >> File "/usr/lib64/python2.6/codecs.py", line 351, in write >> >> data, consumed = self.encode(object, self.errors) >> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in >> >> position 176: ordinal not in range(128) >> > … >> >> LANG=en_US.UTF-8 > > My current guess is that LANG is set in your terminal, but that you > ran nmbug-status from a cron job where it wasn't set. If that's the > case, try adding: > > export LANG=en_US.UTF-8 > > to your cron job (or replace with whichever encoding you like). Yes guess is good, but incorrect I run it on the command line. Now I did the following: $ git remote add tremily git://tremily.us/notmuch.git $ git fetch --all $ git checkout nmbug-status-python3 commit d64ef215b62fa74a96fdb4c93e1f6f7abf7c80c6 and then rerun nmbug-status -- and I am still having the same problem. I will investigate this further... > We don't care about the preferred language yet, but we could [1]. > There's actually not all that much that needs translating for > nmbug-status. Yes, that is unfortunate that David (Taavetti in Finnish) did an uneducated choice by choosing english as the preferred language. It should always have been written in Finnish so we would not have this conversation >;) > > Cheers, > Trevor > > [1]: http://docs.python.org/3/library/gettext.html Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: notmuch killed due to out of memory - how to move forward
On 4 February 2014 08:25, Jani Nikula wrote: > On Mon, 03 Feb 2014, Hamish Downer wrote: >> I recently deleted almost 3 old messages from my maildirs, and >> since I did that, notmuch new has not managed to complete. I have it >> running on a server with 1 GB of RAM, and the output is telling me >> >> Cleaned up 25515 of 29803 messages (1m 10s remaining). >> >> at the point it is killed. I have followed it up to this point with >> top and free and I can see memory usage growing to something like >> 700MB (RES) at which point the process is killed. > > Please try SIGINT, or ^C, on notmuch new before it gets killed. That > should be handled gracefully, making progress, and letting you chop up > and eventually finish the operation. Please let us know if this helps. That did it, thank you. Good to know Ctrl-C is gracefully handled. > Needless to say, we shouldn't use that much memory just to delete files > from the index. I have saved a copy of the index from before I fixed the problem. I am happy to re-run it with a rebuilt notmuch if that would be useful from the point of view of uncovering bugs. Otherwise I'll carry on as a happy notmuch user Hamish ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: sanitization of args notmuch-cli in notmuch-emacs
On Tue, Feb 04 2014, David Bremner wrote: > Tomi Ollila writes: > >> >> Maybe the cli should be fixed ? (and/or make emacs MUA resilient to >> this kind of result) >> > > Would it make any sense to output errors in structured format? I guess > the downside is it would be harder for a human user to read. notmuch count --batch outputs just numbers separated by newline -- and with emacs stdout & stderr are in the same stream (in this case too) For that we'd need structured like notmuch count --batch --format=sexp and then write errors there... I don't think we can find SomeOne(tm) to do this -- or the überversion like 'notmuch execute' which takes sexp/json document in stdin and spits out sexp/json document for programs to parse ;D > d Romi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote: > On Tue, Feb 04 2014, W. Trevor King wrote: > > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote: > >> File "devel/nmbug/nmbug-status", line 197, in _write_threads > >> ).format(**message_display_data)) > >> File "/usr/lib64/python2.6/codecs.py", line 351, in write > >> data, consumed = self.encode(object, self.errors) > >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in > >> position 176: ordinal not in range(128) > > … > > LANG=en_US.UTF-8 My current guess is that LANG is set in your terminal, but that you ran nmbug-status from a cron job where it wasn't set. If that's the case, try adding: export LANG=en_US.UTF-8 to your cron job (or replace with whichever encoding you like). We don't care about the preferred language yet, but we could [1]. There's actually not all that much that needs translating for nmbug-status. Cheers, Trevor [1]: http://docs.python.org/3/library/gettext.html -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote: > On Tue, Feb 04 2014, W. Trevor King wrote: > > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote: > >> File "devel/nmbug/nmbug-status", line 197, in _write_threads > >> ).format(**message_display_data)) > >> File "/usr/lib64/python2.6/codecs.py", line 351, in write > >> data, consumed = self.encode(object, self.errors) > >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in > >> position 176: ordinal not in range(128) > > ? > > LANG=en_US.UTF-8 My current guess is that LANG is set in your terminal, but that you ran nmbug-status from a cron job where it wasn't set. If that's the case, try adding: export LANG=en_US.UTF-8 to your cron job (or replace with whichever encoding you like). We don't care about the preferred language yet, but we could [1]. There's actually not all that much that needs translating for nmbug-status. Cheers, Trevor [1]: http://docs.python.org/3/library/gettext.html -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140204/f17c99ed/attachment.pgp>
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 07:50:32AM -0800, W. Trevor King wrote:
> That's two votes for Python 2.6 and none for later versions, so I'll
> just try a bit harder for 2.6 compatibility ;).
As a v2 preview, and to help with further review, I've posted my
current v1+ version of this branch at:
git://tremily.us/notmuch.git nmbug-status-python3
Changes since v1:
* '{}' → '{0}' for Python 2's str.format().
* Added an OrderedDict stub for Python 2.6 (and earlier, but they'll
be missing json).
This v1+ works on my local box with Python 2.6.9. I can push it as v2
now, but it's a long series and I don't want to spam the list just for
Python 2.6 fixups. If folks want a v2 pushed to the list, just let me
know.
Cheers,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 07:50:32AM -0800, W. Trevor King wrote:
> That's two votes for Python 2.6 and none for later versions, so I'll
> just try a bit harder for 2.6 compatibility ;).
As a v2 preview, and to help with further review, I've posted my
current v1+ version of this branch at:
git://tremily.us/notmuch.git nmbug-status-python3
Changes since v1:
* '{}' ? '{0}' for Python 2's str.format().
* Added an OrderedDict stub for Python 2.6 (and earlier, but they'll
be missing json).
This v1+ works on my local box with Python 2.6.9. I can push it as v2
now, but it's a long series and I don't want to spam the list just for
Python 2.6 fixups. If folks want a v2 pushed to the list, just let me
know.
Cheers,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20140204/636b7af4/attachment.pgp>
Re: notmuch killed due to out of memory - how to move forward
Jani Nikula writes: > On Mon, 03 Feb 2014, Hamish Downer wrote: >> Hello >> >> I recently deleted almost 3 old messages from my maildirs, and >> since I did that, notmuch new has not managed to complete. I have it >> running on a server with 1 GB of RAM, and the output is telling me >> >> Cleaned up 25515 of 29803 messages (1m 10s remaining). >> >> at the point it is killed. I have followed it up to this point with >> top and free and I can see memory usage growing to something like >> 700MB (RES) at which point the process is killed. >> >> I am using notmuch 0.17 (from the debian jessie package 0.17-3) on >> debian wheezy 64 bit. >> >> Any pointers as to what to do to get past this point would be greatly >> appreciated. If I have just deleted that many messages should I just >> expect notmuch to use that much memory? (Unfortunately the company >> supplying the server does not allow swap to be enabled so I can't fudge >> it that way) >> >> I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't >> produce a report (though I was unsure if I needed to rebuild notmuch >> before that would work). >> >> Happy to build notmuch from source if this might be an interesting bug >> worth tracking down. > > Please try SIGINT, or ^C, on notmuch new before it gets killed. That > should be handled gracefully, making progress, and letting you chop up > and eventually finish the operation. Please let us know if this helps. > > Needless to say, we shouldn't use that much memory just to delete files > from the index. > When I initially indexed my email archives into notmuch, I had problems with thermal cutouts, so I did something like this, which automated the workaround you mention: while :; do timeout -s INT 2m notmuch new sleep 600 done Of course, with memory issues as the constraint rather than temperature, the sleep should not be necessary. -- regards, kushal pgpFUhTbipR_x.pgp Description: PGP signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
notmuch killed due to out of memory - how to move forward
On Mon, 03 Feb 2014, Hamish Downer wrote: > Hello > > I recently deleted almost 3 old messages from my maildirs, and > since I did that, notmuch new has not managed to complete. I have it > running on a server with 1 GB of RAM, and the output is telling me > > Cleaned up 25515 of 29803 messages (1m 10s remaining). > > at the point it is killed. I have followed it up to this point with > top and free and I can see memory usage growing to something like > 700MB (RES) at which point the process is killed. > > I am using notmuch 0.17 (from the debian jessie package 0.17-3) on > debian wheezy 64 bit. > > Any pointers as to what to do to get past this point would be greatly > appreciated. If I have just deleted that many messages should I just > expect notmuch to use that much memory? (Unfortunately the company > supplying the server does not allow swap to be enabled so I can't fudge > it that way) > > I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't > produce a report (though I was unsure if I needed to rebuild notmuch > before that would work). > > Happy to build notmuch from source if this might be an interesting bug > worth tracking down. Please try SIGINT, or ^C, on notmuch new before it gets killed. That should be handled gracefully, making progress, and letting you chop up and eventually finish the operation. Please let us know if this helps. Needless to say, we shouldn't use that much memory just to delete files from the index. BR, Jani.
sanitization of args notmuch-cli in notmuch-emacs
Tomi Ollila writes: > > Maybe the cli should be fixed ? (and/or make emacs MUA resilient to > this kind of result) > Would it make any sense to output errors in structured format? I guess the downside is it would be harder for a human user to read. d
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
Tomi Ollila writes: > anyway, if this doesn't resolve out and there is no resistance to require > python 2.7 (that means from David) I can hack around this to get this > reviewed. Currently the production copy of nmbug-status is running with python2.6 on Debian squeeze. I _should_ upgrade that machine to wheezy, but I'm not sure if there are any complications preventing that. d
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, "W. Trevor King" wrote:
> > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
> >> File "devel/nmbug/nmbug-status", line 197, in _write_threads
> >> ).format(**message_display_data))
> >> File "/usr/lib64/python2.6/codecs.py", line 351, in write
> >> data, consumed = self.encode(object, self.errors)
> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
> >> position 176: ordinal not in range(128)
> >
> > Hmm. __future__'s unicode_literals should be giving us a Unicode
> > target, so I'm not sure why we'd have trouble injecting Unicode. This
> > works fine for me on Python 2.7 and 3.3. Maybe you just have a funky
> > encoding? …
>
> LANG=en_US.UTF-8
> all other LC_* variables en_US.UTF-8 except
> LC_TIME=en_GB.utf8
> LC_ALL empty (naturally)
>
> python -c 'import locale; print(locale.getpreferredencoding())'
> UTF-8
> python -c 'import sys; print(sys.getdefaultencoding())'
> ascii
That's very strange. On Python 2.6.9, with the same encodings:
>>> from __future__ import unicode_literals
>>> import codecs
>>> import locale
>>> import sys
>>> print(locale.getpreferredencoding()) # same as yours
UTF-8
>>> print(sys.getdefaultencoding()) # same as yours
ascii
>>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
>>> print(_ENCODING) # double-check default encodings
UTF-8
>>> byte_stream = sys.stdout # copied from Page.write
>>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
>>> data = {'from': '\u017b'} # fake the troublesome data
>>> print(type(data['from'])) # double-check unicode_literals
>>> string = ' {from}\n'.format(**data)
>>> stream.write(string)
Ż
It looks like you'll have the same _ENCODING as I do (UTF-8). That
means your stream should be wrapped in a UTF-8 StreamWriter, so I
don't understand why it's converting to ASCII. Can you run through
the above on your troublesome machine and confirm that stream.write()
is still raising the exception? If it doesn't work, can you just
paste that whole run in your next email?
Thanks,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 12:30:30PM +0200, Tomi Ollila wrote:
> On Tue, Feb 04 2014, "W. Trevor King" wrote:
> > On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
> >> File "devel/nmbug/nmbug-status", line 197, in _write_threads
> >> ).format(**message_display_data))
> >> File "/usr/lib64/python2.6/codecs.py", line 351, in write
> >> data, consumed = self.encode(object, self.errors)
> >> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
> >> position 176: ordinal not in range(128)
> >
> > Hmm. __future__'s unicode_literals should be giving us a Unicode
> > target, so I'm not sure why we'd have trouble injecting Unicode. This
> > works fine for me on Python 2.7 and 3.3. Maybe you just have a funky
> > encoding? ?
>
> LANG=en_US.UTF-8
> all other LC_* variables en_US.UTF-8 except
> LC_TIME=en_GB.utf8
> LC_ALL empty (naturally)
>
> python -c 'import locale; print(locale.getpreferredencoding())'
> UTF-8
> python -c 'import sys; print(sys.getdefaultencoding())'
> ascii
That's very strange. On Python 2.6.9, with the same encodings:
>>> from __future__ import unicode_literals
>>> import codecs
>>> import locale
>>> import sys
>>> print(locale.getpreferredencoding()) # same as yours
UTF-8
>>> print(sys.getdefaultencoding()) # same as yours
ascii
>>> _ENCODING = locale.getpreferredencoding() or sys.getdefaultencoding()
>>> print(_ENCODING) # double-check default encodings
UTF-8
>>> byte_stream = sys.stdout # copied from Page.write
>>> stream = codecs.getwriter(encoding=_ENCODING)(stream=byte_stream)
>>> data = {'from': '\u017b'} # fake the troublesome data
>>> print(type(data['from'])) # double-check unicode_literals
>>> string = ' {from}\n'.format(**data)
>>> stream.write(string)
?
It looks like you'll have the same _ENCODING as I do (UTF-8). That
means your stream should be wrapped in a UTF-8 StreamWriter, so I
don't understand why it's converting to ASCII. Can you run through
the above on your troublesome machine and confirm that stream.write()
is still raising the exception? If it doesn't work, can you just
paste that whole run in your next email?
Thanks,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20140204/5b72876f/attachment.pgp>
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 09:07:45AM -0400, David Bremner wrote: > Tomi Ollila writes: > > anyway, if this doesn't resolve out and there is no resistance to require > > python 2.7 (that means from David) I can hack around this to get this > > reviewed. > > Currently the production copy of nmbug-status is running with python2.6 > on Debian squeeze. That's two votes for Python 2.6 and none for later versions, so I'll just try a bit harder for 2.6 compatibility ;). Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
[PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04, 2014 at 09:07:45AM -0400, David Bremner wrote: > Tomi Ollila writes: > > anyway, if this doesn't resolve out and there is no resistance to require > > python 2.7 (that means from David) I can hack around this to get this > > reviewed. > > Currently the production copy of nmbug-status is running with python2.6 > on Debian squeeze. That's two votes for Python 2.6 and none for later versions, so I'll just try a bit harder for 2.6 compatibility ;). Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140204/717b1f33/attachment-0001.pgp>
Re: sanitization of args notmuch-cli in notmuch-emacs
Tomi Ollila writes: > > Maybe the cli should be fixed ? (and/or make emacs MUA resilient to > this kind of result) > Would it make any sense to output errors in structured format? I guess the downside is it would be harder for a human user to read. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
Tomi Ollila writes: > anyway, if this doesn't resolve out and there is no resistance to require > python 2.7 (that means from David) I can hack around this to get this > reviewed. Currently the production copy of nmbug-status is running with python2.6 on Debian squeeze. I _should_ upgrade that machine to wheezy, but I'm not sure if there are any complications preventing that. d ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 2/2] emacs: Prefer Content-Description over filename for part buttons
On Tue, Feb 04 2014, "W. Trevor King" wrote: > On Mon, Feb 03, 2014 at 10:44:39PM +0200, Tomi Ollila wrote: >> having notmuch patch email as expected test output feels a bit >> confusing to me -- especially as 'git grep' may catch some of the >> (possibly future-outdated) content... > > There were two very similar patches with filename attachments in the > corpus, so I used them for the tests ;). I can convert that test to > use emacs_fcc_message or some other auto-generated content if you'd > prefer. You're right; corpus seems to be full of notmuch patch emails... ... and (although?) (just noticed)... $ grep '^ *[+-].*\' test/* test/*/* test/emacs.expected-output/attachment:+if (pw_buf_size == -1) pw_buf_size = 64; test/emacs.expected-output/attachment:+if (pw_buf_size == -1) pw_buf_size = 64; ... I am not giving much resistance to keep the patch as it is :D > > Cheers, > Trevor > Tomi ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH 00/17] nmbug-status: Python-3-compabitility and general refactoring
On Tue, Feb 04 2014, "W. Trevor King" wrote:
> On Mon, Feb 03, 2014 at 11:10:23PM +0200, Tomi Ollila wrote:
>> data['message-id-term'] = 'id:"{}"'.format(value)
>> ValueError: zero length field name in format
>
> Oops, Python 2.6 still needs explicit indexes ('{0}', not '{}'). It's
> an easy fix, so I'll queue it for v2. You're still going to need
> Python 2.7 or greater for collections.OrderedDict(). We could word
> around that too, but do we really care about 2.6? I don't expect that
> the installed nmbug-status userbase is so large and backward that
> upgrading to 2.7 will be that hard ;). 2.6 isn't even getting
> security fixes anymore [1], so I think it's time to migrate :p.
Probably not many cares about 2.6; I already use argparse and I can add
that OrderedDict() too. Still {0} is easy enough to do :D. I am running
this notmuch & nmbug in Scientific Linux 6.2 machine which has python 2.6
-- and this is the only machine where I can review your nmbug-status
changes ;)
>> File "devel/nmbug/nmbug-status", line 197, in _write_threads
>> ).format(**message_display_data))
>> File "/usr/lib64/python2.6/codecs.py", line 351, in write
>> data, consumed = self.encode(object, self.errors)
>> UnicodeEncodeError: 'ascii' codec can't encode character u'\u017b' in
>> position 176: ordinal not in range(128)
>
> Hmm. __future__'s unicode_literals should be giving us a Unicode
> target, so I'm not sure why we'd have trouble injecting Unicode. This
> works fine for me on Python 2.7 and 3.3. Maybe you just have a funky
> encoding? What is your:
>
> $ locale
> LANG=en_US.UTF-8
> …
> $ python -c 'import locale, sys; print(locale.getpreferredencoding() or
> sys.getdefaultencoding())'
> UTF-8
LANG=en_US.UTF-8
all other LC_* variables en_US.UTF-8 except
LC_TIME=en_GB.utf8
LC_ALL empty (naturally)
python -c 'import locale; print(locale.getpreferredencoding())'
UTF-8
python -c 'import sys; print(sys.getdefaultencoding())'
ascii
anyway, if this doesn't resolve out and there is no resistance to require
python 2.7 (that means from David) I can hack around this to get this
reviewed.
>
> Cheers,
> Trevor
Tomi
>
> [1]: http://www.python.org/download/releases/2.6.9/
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
notmuch killed by out of memory - how to move forward
Hello I recently deleted almost 3 old messages from my maildirs, and since I did that, notmuch new has not managed to complete. I have it running on a server with 1 GB of RAM, and the output is telling me Cleaned up 25515 of 29803 messages (1m 10s remaining). at the point it is killed. I have followed it up to this point with top and free and I can see memory usage growing to something like 700MB (RES) at which point the process is killed. I am using notmuch 0.17 (from the debian jessie package) on debian wheezy 64 bit. Any pointers as to what to do to get past this point would be greatly appreciated. If I have just deleted that many messages should I just expect notmuch to use that much memory? I guess I could enable swap and hope that allows enough memory to get past this operation. (Just checked and swap is not an option with this host). I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't produce a report (though I was unsure if I needed to rebuild notmuch before that would work). Happy to build notmuch from source if this might be an interesting bug worth tracking down. Hamish ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
notmuch new crashing due to out of memory - how to deal with it
Hello I recently deleted almost 3 old messages from my maildirs, and since I did that, notmuch new has not managed to complete. I have it running on a server with 1 GB of RAM, and the output is telling me Cleaned up 25515 of 29803 messages (1m 10s remaining). at the point it is killed. I have followed it up to this point with top and free and I can see memory usage growing to something like 700MB (RES) at which point the process is killed. I am using notmuch 0.17 (from the debian jessie package) on debian wheezy 64 bit. Any pointers as to what to do to get past this point would be greatly appreciated. If I have just deleted that many messages should I just expect notmuch to use that much memory? I guess I could enable swap and hope that allows enough memory to get past this operation. I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't produce a report (though I was unsure if I needed to rebuild notmuch before that would work). Happy to build notmuch from source if this might be an interesting bug worth tracking down. Hamish ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: notmuch killed due to out of memory - how to move forward
On Mon, 03 Feb 2014, Hamish Downer wrote: > Hello > > I recently deleted almost 3 old messages from my maildirs, and > since I did that, notmuch new has not managed to complete. I have it > running on a server with 1 GB of RAM, and the output is telling me > > Cleaned up 25515 of 29803 messages (1m 10s remaining). > > at the point it is killed. I have followed it up to this point with > top and free and I can see memory usage growing to something like > 700MB (RES) at which point the process is killed. > > I am using notmuch 0.17 (from the debian jessie package 0.17-3) on > debian wheezy 64 bit. > > Any pointers as to what to do to get past this point would be greatly > appreciated. If I have just deleted that many messages should I just > expect notmuch to use that much memory? (Unfortunately the company > supplying the server does not allow swap to be enabled so I can't fudge > it that way) > > I did also try running it with NOTMUCH_TALLOC_REPORT, but that didn't > produce a report (though I was unsure if I needed to rebuild notmuch > before that would work). > > Happy to build notmuch from source if this might be an interesting bug > worth tracking down. Please try SIGINT, or ^C, on notmuch new before it gets killed. That should be handled gracefully, making progress, and letting you chop up and eventually finish the operation. Please let us know if this helps. Needless to say, we shouldn't use that much memory just to delete files from the index. BR, Jani. ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
