[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3  added the comment:

A new patch, https://bugs.python.org/issue45235 has clobbered this patch.

It has also exposed the inadequate unittesting for the case(s) where the 'dest' 
of main namespace, subparser namespace, user provided namespace overlap.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3  added the comment:

I just realized if the subparser argument used 

default=argparse.SUPPRESS

it would not clobber values (default or user) set by the main parser.

(a 'store_true' would have to be replaced with a 'store_const' to do this)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[hai shi]

hai shi  added the comment:

How about use a flag(such USING_OUT_NAMESPACE) to identify we use namespace or 
not?

For example:
subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
for key, value in vars(subnamespace).items():
if USING_OUT_NAMESPACE and not hasattr(namespace, key):
setattr(namespace, key, value)

--
nosy: +shihai1991

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3  added the comment:

A variation on the problem I reported in 

https://bugs.python.org/issue9351#msg229968

is that a custom Namespace class as documented in 

https://docs.python.org/3/library/argparse.html#the-namespace-object

will not be used by the subparsers.  Only the main parser uses that custom 
Namespace object; the subparsers continue to use the default 
'argparse.Namespace()'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

Changes by paul j3 :


--
Removed message: http://bugs.python.org/msg274336

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3 added the comment:

In http://bugs.python.org/issue27859 I've demonstrated how a subclass of 
_SubParsersAction can be used to revert the behavior to pre 2.7.9, passing the 
main namespace to the subparser.

The only other change is to the parser registry:

parser.register('action', 'parsers', MyParserAction)

So there's no need to change the argparse.py file itself.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3 added the comment:

Another example where the old way would have worked better

http://bugs.python.org/issue27859

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3 added the comment:

Another example of this patch causing backward compatibility problems

http://bugs.python.org/issue24251

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Rémi Rampin]

Rémi Rampin added the comment:

To me this is much more than a compatibility problem. The way it worked before 
made a lot of sense, and just felt like the correct solution to accept a flag 
in multiple places.

Having a --verbose flag is something everybody should consider (Python has a 
decent builtin logging module), and anybody providing it would definitely want 
to accept it before and after subcommands (or at least, for every subcommand).

The only way right now is to not only create different arguments with 
add_argument(), for each parser, but you also need to provide different 
destination names (and then do something shitty like verbosity = 
args.verb_main+args.verb_subcommand). This bug makes argparse completely 
unusable for any real-life application that uses subparsers (in addition to 
breaking existing programs). And it breaks silently too, simply amazing!

Of course there is very little point in fixing this now. Since this affects 
multiple released versions of Python, I have to use a work-around anyway (until 
I can move from argparse to something that won't decide to break someday for 
the hell of it).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Rémi Rampin]

Changes by Rémi Rampin remiram...@gmail.com:


--
nosy: +remram

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Changaco]

Changaco added the comment:

The broken patch made it into the 2.7.9 release, causing argparse to silently 
ignore arguments, cf http://bugs.python.org/issue23058

--
nosy: +Changaco

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
resolution: fixed - 
stage: commit review - needs patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[R. David Murray]

R. David Murray added the comment:

If I understand you correctly, that would mean that if the namespace keyword is 
not used, we'd have the fixed behavior, but if the namespace keyword is used, 
we'd have the backward compatible behavior?  If I'm understanding correctly, 
that sounds like a good solution to me (coupled with backing this out of the 
maint versions).

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3 added the comment:

I'm exploring modifying that patch by adding a 'subnamespace' attribute to the 
subparsers Action object.  This would give the user, or the main parser control 
over the namespace that is passed to a subparser.

For example: 

subnamespace = getattr(self, 'subnamespace', False)
if subnamespace is None:
pass
elif not  subnamespace:
subnamespace = namespace
# if None, subparser will use a fresh one
subnamespace, arg_strings = parser.parse_known_args(arg_strings, 
subnamespace)
for key, value in vars(subnamespace).items():
setattr(namespace, key, value)

As written here, no value (or any False) would use the main parser namespace, 
as done with existing code.

A 'None' value, would use the a new empty namespace, as proposed in this patch.

Or the user could directly define the namespace, e.g.

sp = parser.add_subparsers()
sp.subnamespace = Namespace(foo=4)

That could also be convenient if the user is using a custom Namespace class.

'parse_known_args' could also set this attribute, based on whether the user has 
given it a namespace or not.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3 added the comment:

A possible problem with this patch is that it overrides any Namespace given by 
the user.  In the test example:

parser.parse_args(['X'], namespace=argparse.Namespace(foo=3))

the result is 'foo=2', the setdefault from the subparser.  Previously, and with 
a single level parser, the result would be 'foo=3'.  This is also true for an 
ordinary argument default in the subparser.

It could be argued that a user is unlikely to use both the 'namespace=...' 
parameter, and 'setdefault' for the same variable, especially one that isn't an 
argument 'dest'.  But the fact that the  custom namespace does not override 
regular subparser argument defaults bothers me.

Also, should an untested change like this be applied to 3.4 and 2.7?  This is 
not a backward compatible bug fix.

The test case also touches on a real bug in the recent releases - subparsers 
are now optional.  'parser.parse_args([])' runs without complaint about the 
missing 'X'.  Actually it is convenient for testing this case.  But it is still 
an incompatibility with earlier behavior

http://bugs.python.org/issue9253

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[R. David Murray]

R. David Murray added the comment:

Yeah, I hesitated a bit about the backports, but didn't visualize the scenario 
you are suggesting and thought it would be safe.  Perhaps it should be backed 
out of 2.7 and 3.4.

For 3.5, do you have any thoughts about how to make namespace= play nicely with 
this problem?

--
status: closed - open

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[R. David Murray]

Changes by R. David Murray rdmur...@bitdance.com:


--
stage: resolved - commit review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Roundup Robot]

Roundup Robot added the comment:

New changeset e9cb45ccf42b by R David Murray in branch '3.4':
#9351: set_defaults on subparser is no longer ignored if set on parent.
https://hg.python.org/cpython/rev/e9cb45ccf42b

New changeset b35a811d4420 by R David Murray in branch 'default':
Merge: #9351: set_defaults on subparser is no longer ignored if set on parent.
https://hg.python.org/cpython/rev/b35a811d4420

New changeset 1a3143752db2 by R David Murray in branch '2.7':
#9351: set_defaults on subparser is no longer ignored if set on parent.
https://hg.python.org/cpython/rev/1a3143752db2

--
nosy: +python-dev

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[R. David Murray]

R. David Murray added the comment:

Thanks, Jyrki.

--
nosy: +r.david.murray
resolution:  - fixed
stage: commit review - resolved
status: open - closed
versions: +Python 3.4, Python 3.5 -Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

paul j3 added the comment:

This is not getting much attention for several reasons:

- there's quite a backlog of argparse patches and issues

- 'set_defaults' is not commonly used.  Setting default in 'add_argument' seems 
more common.

- defining the same argument for both the parser and subparser can create other 
difficulties.

- in Bethard's example, 'set_default' sets an attribute that has no connection 
to any argument.  It's a cleaver trick that few users will think to use, and 
probably won't be of much value.

The idea proposed here of using a subnamespace for the subparser is 
interesting.  It reminds me of a StackOverflow question about implementing 
nested namespaces.

http://stackoverflow.com/questions/18668227

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[paul j3]

Changes by paul j3 ajipa...@gmail.com:


--
nosy: +paul.j3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Mikael Knutsson]

Mikael Knutsson added the comment:

Just wanted to drop in here to let you know that I hit this behaviour recently 
when writing a small tool using both configparser and argparse and the 
workaround proves rather annoying (custom namespace object or similar).

Would be awesome if this moved forward with the proposed patch!

--
nosy: +mikn

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Jyrki Pulliainen]

Jyrki Pulliainen added the comment:

Yeah, I tried figuring out something more clever, as this, in the current form, 
has a bit too hackish feeling in it, but I couldn't find a proper tool for the 
job.

Anyway, attached a patch with the getattr removed.

--
Added file: http://bugs.python.org/file27793/issue9351.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Jyrki Pulliainen]

Changes by Jyrki Pulliainen jy...@dywypi.org:


Removed file: http://bugs.python.org/file27793/issue9351.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Jyrki Pulliainen]

Changes by Jyrki Pulliainen jy...@dywypi.org:


Added file: http://bugs.python.org/file27795/issue9351.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Jyrki Pulliainen]

Changes by Jyrki Pulliainen jy...@dywypi.org:


Added file: http://bugs.python.org/file27794/issue9351.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Jyrki Pulliainen]

Changes by Jyrki Pulliainen jy...@dywypi.org:


Removed file: http://bugs.python.org/file27677/issue9351.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Jyrki Pulliainen]

Changes by Jyrki Pulliainen jy...@dywypi.org:


Removed file: http://bugs.python.org/file27794/issue9351.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Petri Lehtinen]

Petri Lehtinen added the comment:

LGTM. Steven?

--
stage: needs patch - commit review

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Petri Lehtinen]

Petri Lehtinen added the comment:

+for key in vars(subnamespace):
+setattr(namespace, key, getattr(subnamespace, key))

There might be even more clever ways to achieve this, but what about at least 
saying for key, value in vars(subnamespace).items(), and then using the value 
accordingly, to avoid the getattr() call?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Ezio Melotti]

Changes by Ezio Melotti ezio.melo...@gmail.com:


--
nosy: +ezio.melotti

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Jyrki Pulliainen]

Jyrki Pulliainen added the comment:

I've attached a proposed fix where the subparser parses to an empty namespace 
and that namespace is then merged with the parent namespace

--
keywords: +patch
nosy: +nailor, petri.lehtinen
Added file: http://bugs.python.org/file27677/issue9351.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue9351] argparse set_defaults on subcommands should override top level set_defaults

[Steven Bethard]

New submission from Steven Bethard steven.beth...@gmail.com:

If you use set_defaults on a subparser, but a default exists on the top level 
parser, the subparser defaults are ignored:

 parser = argparse.ArgumentParser()
 xparser = parser.add_subparsers().add_parser('X')
 parser.set_defaults(foo=1)
 xparser.set_defaults(foo=2)
 parser.parse_args(['X'])
Namespace(foo=1)

This is counter to what people probably expect, that the subparser, when 
selected, would override the top level parser.

The behavior is because of the following code in parse_known_args:

for dest in self._defaults:
if not hasattr(namespace, dest):
setattr(namespace, dest, self._defaults[dest])

This happens before the subparser sees the namespace object, and so the 
subparser sees that no defaults need to be filled in.

--
components: Library (Lib)
messages: 111324
nosy: bethard
priority: normal
severity: normal
stage: needs patch
status: open
title: argparse set_defaults on subcommands should override top level 
set_defaults
type: behavior
versions: Python 2.7, Python 3.2

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue9351
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com