[PATCH v3] nmbug: Translate to Python
On Tue, Aug 05, 2014 at 10:24:10PM -0300, David Bremner wrote:
> I have a local commit that deletes a couple tags; when I attempt to
> merge I get complaints about local changes to files.
>
> error: Your local changes to the following files would be overwritten by
> merge:
> tags/1406859003-11561-2-git-send-email-amdragon at mit.edu/needs-review
> tags/1406859003-11561-3-git-send-email-amdragon at mit.edu/needs-review
> Please, commit your changes or stash them before you can merge.
> ?
> Calling the perl version of nmbug successfully creates a little
> diamond merge
The Perl version has:
git ( { GIT_WORK_TREE => $tempwork }, 'checkout', '-f', 'HEAD');
git ( { GIT_WORK_TREE => $tempwork }, 'merge', $commit);
But the Python version only has:
_git(
args=['merge', reference],
additional_env={'GIT_WORK_TREE': workdir},
wait=True)
I suppose we need the checkout to populate the working directory, but
I'm not sure we want to force the checkout. Do we expect to have
unmerged entries in the index?
I'll add an unforced checkout here in v4. I'll add it to pull() too,
now that it's decoupled from nmbug's merge implementation.
Cheers,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20140806/d4b359f5/attachment.pgp>
[PATCH v3] nmbug: Translate to Python
On Tue, Aug 05, 2014 at 10:16:10PM -0300, David Bremner wrote: > David Bremner writes: > > I did notice that merging was noticably noisier than I remembered. > > rejected pushes also seem noisier than before; I'm not sure the > python backtrace adds anything here. > > To nmbug at nmbug.tethera.net:nmbug-tags > ! [rejected]master -> master (fetch first) > error: failed to push some refs to 'nmbug at nmbug.tethera.net:nmbug-tags' > hint: Updates were rejected because the remote contains work that you do > hint: not have locally. This is usually caused by another repository pushing > hint: to the same ref. You may want to first integrate the remote changes > hint: (e.g., 'git pull ...') before pushing again. > hint: See the 'Note about fast-forwards' in 'git push --help' for details. > Traceback (most recent call last): > File "/home/bremner/config/scripts/nmbug", line 766, in > args.func(**kwargs) > File "/home/bremner/config/scripts/nmbug", line 463, in push > _git(args=args, wait=True) > File "/home/bremner/config/scripts/nmbug", line 210, in _git > return _spawn(args=args, **kwargs) > File "/home/bremner/config/scripts/nmbug", line 193, in _spawn > args=args, status=status, stdout=stdout, stderr=stderr) > __main__.SubprocessError: ['git', '--git-dir', '/home/bremner/.nmbug', > 'push'] exited with 1 I can drop the backtrace and just print the SubprocessError, and just show the traceback if the logging is set to ?debug?. If that sounds reasonable, I'll to it in v4. I'm not sure what the current nmbug shows in this case. Do you want me to prune the ?hint? lines too? I usually find those to be pretty informative, but the're not directly applicable to folks using nmbug who prefer to ignore the underlying Git layer. Ideally, we'd translate them to apply to nmbug (??(e.g. nmbug pull ...) before pushing??), but I can't think of a maintainable way to do that. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: OpenPGP digital signature URL: <http://notmuchmail.org/pipermail/notmuch/attachments/20140806/8c1fc8ee/attachment.pgp>
[PATCH v3] nmbug: Translate to Python
On Mon, Aug 04, 2014 at 09:14:46PM -0300, David Bremner wrote:
> W. Trevor King writes:
> > * Commands are no longer split into "most common", "other useful",
> > and "less common" sets. If we need something like this, I'd
> > prefer workflow examples highlighting common commands in the
> > module docstring (available with 'nmbug --help').
>
> I don't feel strongly about this, but I remember implementing it by
> request in the first version. OTOH, I think you shortened up the
> main help string when you split it.
I didn't intentionally remove any information. I think it's just
shorter because command-specific details are now in the
command-specific docstring/help.
> We may want to think about a seperate man page as a follow project.
Works for me, but I think it would look a lot like the wiki page [1].
> One thing I did notice is that there is no hint to call nmbug
> {command} --help in the main docstring.
I'll add that in v4.
> > +#!/usr/bin/env python
> > +# Copyright (c) 2011 David Bremner
> > +# License: same as notmuch
>
> You should add your self, update the date, and probably explicitly
> state the license, as in Carl's patch for nmbug-status.
Will do in v4.
> > +__version__ = '0.2'
>
> Do we need/want a version distinct from that of notmuch?
nmbug is very loosely bound to the notmuch core. To me it feels like
a separate project that happens to share the same version control
repository. I'm happy to synchronize versions, but then we have to
remember to bump the nmbug version for each notmuch release.
> > +def _hex_quote(string, safe='+@=:,'):
>
> I'm not sure I really understand what makes a function/variable
> "private" and hence prefixed with _ in your translation.
The public interface is what I thought was reasonably stable for folks
who want to call nmbug as a Python library. It's basically the
command-line functionality, with a few other helpers that seemed
important enough to be worth preserving. If we keep a separate nmbug
version, I'd cut major releases for anything that broke compatibility
on a public function. I don't need to use nmbug as a library myself,
so this might all be over-engineered ;).
> > +status, tree, stderr = _git(
>
> as a non-native speaker of python, I find this a bit hard to read.
> How about adding some parens to make the multiple return more clear,
> so
>
> (status, tree, stderr) = _git(
That's legal, but I rarely see the parenthesized version in the wild.
For examples showing the unparenthesized version, see [2,3].
Parentheses are optional for Python tuples [4], so you'd only want
them if ?=? had a higher precedence than ?,?. That's my argument for
the unparenthesized version, but feel free to overrule me ;).
> I did notice that merging was noticably noisier than I remembered.
>
> > +output = _collections.defaultdict(
> > +lambda : _collections.defaultdict( # {tag: status_string}
> > +lambda : ' ')) # default local status
>
> The initial comment is confusing (to me) because it looks like code.
> The two layers of defaultdict are a bit confusing too; it would help
> to mention the key of the outer dictionary.
In v4, I'll use a leading comment for the whole structure, instead of
interleaving the comments. I'll also mention explicitly that the
outer layer is keyed by message id.
> I guess it makes sense to go for a relatively straight translation
> as a first step; I did wonder about whether using the python
> bindings to notmuch would speed things up. Any ideas about how to
> even figure out where the bottlenecks are?
You could profile the Python script [5]. With my usual workflow, the
existing implementation isn't generating too many subprocesses.
Loading the database is probably slow though, so I'd expect reasonable
gains everywhere we call notmuch more than once. I expect checkout
out working directories to also be slow, for the few command that do
that on behalf of Git.
Cheers,
Trevor
[1]: http://notmuchmail.org/nmbug/
[2]:
https://docs.python.org/3.4/tutorial/introduction.html#first-steps-towards-programming
[3]: http://legacy.python.org/dev/peps/pep-3132/
[4]: https://docs.python.org/3.4/reference/expressions.html#parenthesized-forms
[5]: https://docs.python.org/3.4/library/profile.html
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
-- next part ------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL:
<http://notmuchmail.org/pipermail/notmuch/attachments/20140806/ff36993e/attachment.pgp>
[PATCH] Add configurable changed tag to messages that have been changed on disk
Quoth Gaute Hope on Aug 06 at 11:02 am: > On Fri, Aug 1, 2014 at 8:55 PM, Austin Clements wrote: > > I have a prototype implementation of message modification times on my > > lastmod-v1 branch at > > > > https://github.com/aclements/notmuch/tree/lastmod-v1 > > > > It builds on my database features series that's currently awaiting > > review [1]. > > > > The series uses a monotonic revision number, rather than wall-clock > > time, for reasons related to Xapian's concurrent control and detailed > > in the main commit's commit message. The implementation isn't quite > > useful from the CLI yet because I haven't added any way to query the > > database's current revision number. (I'm still thinking about how I > > want to do this, since search/show don't have a good way to deliver > > "additional" information right now. I might just add the last > > modification for each individual message/max of all messages in a > > thread, similar to what Thomas Jost's patch did long ago.) > > > > [1] id:1406859003-11561-1-git-send-email-amdragon at mit.edu > > > Hi, > > this should allow me to do what I wish to accomplish. The message > deletion is still a problem though, I can see two options at the moment: > > a) output during notmuch new to a hook or a list somewhere deleted files. >if list: notmuch will not handle this list, only append to it and > the user must >purge it when it is safe to do so. > >if hook: for my purposes I would just create a hook appending to the >list. as a minimum I think thread_id, message_id and revision number >should be included. > > > b) maintain a full list of deleted / dead messages. a user initiated >purge can clean this from the database. a tag could be used for this, >so that clients can ignore unlinked/deleted/dead messages. this >differs from a 'deleted' message (IMAP/Maildir context) that has not >yet been expunged so there is confusion to be avoided. > >a garbage collection function and interface must also be set up, but >this one is probably simple. > > > in most cases I think a) would be sufficient, and probably much easier > to do. it might be slow in cases where large amounts of messages have been > deleted, but this is seldom the case for me at least. I have a separate branch (also sitting on top of the features branch) that implements "ghost" messages. The main intent is to fix a bug we currently have in threading, but it puts us in a good position to maintain state for messages we don't have the content of, including last modification times for deleted messages and pre-seeded tags for undelivered messages (useful for pre-tagging sent messages as sent, nmbug, notmuch insert, etc.)
[PATCH] Add configurable changed tag to messages that have been changed on disk
On Fri, Aug 1, 2014 at 8:55 PM, Austin Clements wrote: > I have a prototype implementation of message modification times on my > lastmod-v1 branch at > > https://github.com/aclements/notmuch/tree/lastmod-v1 > > It builds on my database features series that's currently awaiting > review [1]. > > The series uses a monotonic revision number, rather than wall-clock > time, for reasons related to Xapian's concurrent control and detailed > in the main commit's commit message. The implementation isn't quite > useful from the CLI yet because I haven't added any way to query the > database's current revision number. (I'm still thinking about how I > want to do this, since search/show don't have a good way to deliver > "additional" information right now. I might just add the last > modification for each individual message/max of all messages in a > thread, similar to what Thomas Jost's patch did long ago.) > > [1] id:1406859003-11561-1-git-send-email-amdragon at mit.edu Hi, this should allow me to do what I wish to accomplish. The message deletion is still a problem though, I can see two options at the moment: a) output during notmuch new to a hook or a list somewhere deleted files. if list: notmuch will not handle this list, only append to it and the user must purge it when it is safe to do so. if hook: for my purposes I would just create a hook appending to the list. as a minimum I think thread_id, message_id and revision number should be included. b) maintain a full list of deleted / dead messages. a user initiated purge can clean this from the database. a tag could be used for this, so that clients can ignore unlinked/deleted/dead messages. this differs from a 'deleted' message (IMAP/Maildir context) that has not yet been expunged so there is confusion to be avoided. a garbage collection function and interface must also be set up, but this one is probably simple. in most cases I think a) would be sufficient, and probably much easier to do. it might be slow in cases where large amounts of messages have been deleted, but this is seldom the case for me at least. cheers, gaute
[PATCH v3 02/13] util: Const version of strtok_len
Austin Clements writes: > Because of limitations in the C type system, we can't a strtok_len > that can work on both const string and non-const strings. The C > library solves this by taking a const char* and returning a char* > in functions like this (e.g., strchr), but that's not const-safe. > Solve it by introducing strtok_len_c, a version of strtok_len for > const strings. pushed the first two patches in the series d
[PATCH v2] new: Report and abort on upgrade failure
Austin Clements writes: > Previously the return status of notmuch_database_upgrade went > completely unchecked. pushed. d
Re: [PATCH] Add configurable changed tag to messages that have been changed on disk
On Fri, Aug 1, 2014 at 8:55 PM, Austin Clements amdra...@mit.edu wrote: I have a prototype implementation of message modification times on my lastmod-v1 branch at https://github.com/aclements/notmuch/tree/lastmod-v1 It builds on my database features series that's currently awaiting review [1]. The series uses a monotonic revision number, rather than wall-clock time, for reasons related to Xapian's concurrent control and detailed in the main commit's commit message. The implementation isn't quite useful from the CLI yet because I haven't added any way to query the database's current revision number. (I'm still thinking about how I want to do this, since search/show don't have a good way to deliver additional information right now. I might just add the last modification for each individual message/max of all messages in a thread, similar to what Thomas Jost's patch did long ago.) [1] id:1406859003-11561-1-git-send-email-amdra...@mit.edu Hi, this should allow me to do what I wish to accomplish. The message deletion is still a problem though, I can see two options at the moment: a) output during notmuch new to a hook or a list somewhere deleted files. if list: notmuch will not handle this list, only append to it and the user must purge it when it is safe to do so. if hook: for my purposes I would just create a hook appending to the list. as a minimum I think thread_id, message_id and revision number should be included. b) maintain a full list of deleted / dead messages. a user initiated purge can clean this from the database. a tag could be used for this, so that clients can ignore unlinked/deleted/dead messages. this differs from a 'deleted' message (IMAP/Maildir context) that has not yet been expunged so there is confusion to be avoided. a garbage collection function and interface must also be set up, but this one is probably simple. in most cases I think a) would be sufficient, and probably much easier to do. it might be slow in cases where large amounts of messages have been deleted, but this is seldom the case for me at least. cheers, gaute ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v3] nmbug: Translate to Python
On Mon, Aug 04, 2014 at 09:14:46PM -0300, David Bremner wrote:
W. Trevor King writes:
* Commands are no longer split into most common, other useful,
and less common sets. If we need something like this, I'd
prefer workflow examples highlighting common commands in the
module docstring (available with 'nmbug --help').
I don't feel strongly about this, but I remember implementing it by
request in the first version. OTOH, I think you shortened up the
main help string when you split it.
I didn't intentionally remove any information. I think it's just
shorter because command-specific details are now in the
command-specific docstring/help.
We may want to think about a seperate man page as a follow project.
Works for me, but I think it would look a lot like the wiki page [1].
One thing I did notice is that there is no hint to call nmbug
{command} --help in the main docstring.
I'll add that in v4.
+#!/usr/bin/env python
+# Copyright (c) 2011 David Bremner
+# License: same as notmuch
You should add your self, update the date, and probably explicitly
state the license, as in Carl's patch for nmbug-status.
Will do in v4.
+__version__ = '0.2'
Do we need/want a version distinct from that of notmuch?
nmbug is very loosely bound to the notmuch core. To me it feels like
a separate project that happens to share the same version control
repository. I'm happy to synchronize versions, but then we have to
remember to bump the nmbug version for each notmuch release.
+def _hex_quote(string, safe='+@=:,'):
I'm not sure I really understand what makes a function/variable
private and hence prefixed with _ in your translation.
The public interface is what I thought was reasonably stable for folks
who want to call nmbug as a Python library. It's basically the
command-line functionality, with a few other helpers that seemed
important enough to be worth preserving. If we keep a separate nmbug
version, I'd cut major releases for anything that broke compatibility
on a public function. I don't need to use nmbug as a library myself,
so this might all be over-engineered ;).
+status, tree, stderr = _git(
as a non-native speaker of python, I find this a bit hard to read.
How about adding some parens to make the multiple return more clear,
so
(status, tree, stderr) = _git(
That's legal, but I rarely see the parenthesized version in the wild.
For examples showing the unparenthesized version, see [2,3].
Parentheses are optional for Python tuples [4], so you'd only want
them if ‘=’ had a higher precedence than ‘,’. That's my argument for
the unparenthesized version, but feel free to overrule me ;).
I did notice that merging was noticably noisier than I remembered.
+output = _collections.defaultdict(
+lambda : _collections.defaultdict( # {tag: status_string}
+lambda : ' ')) # default local status
The initial comment is confusing (to me) because it looks like code.
The two layers of defaultdict are a bit confusing too; it would help
to mention the key of the outer dictionary.
In v4, I'll use a leading comment for the whole structure, instead of
interleaving the comments. I'll also mention explicitly that the
outer layer is keyed by message id.
I guess it makes sense to go for a relatively straight translation
as a first step; I did wonder about whether using the python
bindings to notmuch would speed things up. Any ideas about how to
even figure out where the bottlenecks are?
You could profile the Python script [5]. With my usual workflow, the
existing implementation isn't generating too many subprocesses.
Loading the database is probably slow though, so I'd expect reasonable
gains everywhere we call notmuch more than once. I expect checkout
out working directories to also be slow, for the few command that do
that on behalf of Git.
Cheers,
Trevor
[1]: http://notmuchmail.org/nmbug/
[2]:
https://docs.python.org/3.4/tutorial/introduction.html#first-steps-towards-programming
[3]: http://legacy.python.org/dev/peps/pep-3132/
[4]: https://docs.python.org/3.4/reference/expressions.html#parenthesized-forms
[5]: https://docs.python.org/3.4/library/profile.html
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v3] nmbug: Translate to Python
On Tue, Aug 05, 2014 at 10:16:10PM -0300, David Bremner wrote: David Bremner writes: I did notice that merging was noticably noisier than I remembered. rejected pushes also seem noisier than before; I'm not sure the python backtrace adds anything here. To nm...@nmbug.tethera.net:nmbug-tags ! [rejected]master - master (fetch first) error: failed to push some refs to 'nm...@nmbug.tethera.net:nmbug-tags' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details. Traceback (most recent call last): File /home/bremner/config/scripts/nmbug, line 766, in module args.func(**kwargs) File /home/bremner/config/scripts/nmbug, line 463, in push _git(args=args, wait=True) File /home/bremner/config/scripts/nmbug, line 210, in _git return _spawn(args=args, **kwargs) File /home/bremner/config/scripts/nmbug, line 193, in _spawn args=args, status=status, stdout=stdout, stderr=stderr) __main__.SubprocessError: ['git', '--git-dir', '/home/bremner/.nmbug', 'push'] exited with 1 I can drop the backtrace and just print the SubprocessError, and just show the traceback if the logging is set to ‘debug’. If that sounds reasonable, I'll to it in v4. I'm not sure what the current nmbug shows in this case. Do you want me to prune the ‘hint’ lines too? I usually find those to be pretty informative, but the're not directly applicable to folks using nmbug who prefer to ignore the underlying Git layer. Ideally, we'd translate them to apply to nmbug (“…(e.g. nmbug pull ...) before pushing…”), but I can't think of a maintainable way to do that. Cheers, Trevor -- This email may be signed or encrypted with GnuPG (http://www.gnupg.org). For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy signature.asc Description: OpenPGP digital signature ___ notmuch mailing list notmuch@notmuchmail.org http://notmuchmail.org/mailman/listinfo/notmuch
Re: [PATCH v3] nmbug: Translate to Python
On Tue, Aug 05, 2014 at 10:24:10PM -0300, David Bremner wrote:
I have a local commit that deletes a couple tags; when I attempt to
merge I get complaints about local changes to files.
error: Your local changes to the following files would be overwritten by
merge:
tags/1406859003-11561-2-git-send-email-amdra...@mit.edu/needs-review
tags/1406859003-11561-3-git-send-email-amdra...@mit.edu/needs-review
Please, commit your changes or stash them before you can merge.
…
Calling the perl version of nmbug successfully creates a little
diamond merge
The Perl version has:
git ( { GIT_WORK_TREE = $tempwork }, 'checkout', '-f', 'HEAD');
git ( { GIT_WORK_TREE = $tempwork }, 'merge', $commit);
But the Python version only has:
_git(
args=['merge', reference],
additional_env={'GIT_WORK_TREE': workdir},
wait=True)
I suppose we need the checkout to populate the working directory, but
I'm not sure we want to force the checkout. Do we expect to have
unmerged entries in the index?
I'll add an unforced checkout here in v4. I'll add it to pull() too,
now that it's decoupled from nmbug's merge implementation.
Cheers,
Trevor
--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
signature.asc
Description: OpenPGP digital signature
___
notmuch mailing list
notmuch@notmuchmail.org
http://notmuchmail.org/mailman/listinfo/notmuch
