[issue45235] argparse does not preserve namespace with subparser defaults

[Guido van Rossum]

Change by Guido van Rossum :


--
nosy:  -gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Frost Ming]

Frost Ming  added the comment:

Per the review comments of @jiasli, I worked out a second PR to fix this issue.

This fix has less side-effect and better backward compatibility. I will leave 
the two PRs open. Any feedback is welcome.

- Frost

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Frost Ming]

Change by Frost Ming :


--
pull_requests: +28441
pull_request: https://github.com/python/cpython/pull/30219

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Frost Ming]

Frost Ming  added the comment:

Hi, I noticed this bug because of the regression of Python 3.9.8. And I 
proposed a better approach in PR 29574.

Maybe the folks here can have a look. Thanks

- Frost

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Paul, should this be closed or do you think there is still a namespace issue to 
be resolved?

--
assignee: rhettinger -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Frost Ming]

Change by Frost Ming :


--
nosy: +Frost Ming
nosy_count: 10.0 -> 11.0
pull_requests: +27818
pull_request: https://github.com/python/cpython/pull/29574

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset e4c5a5eabadd1dcd0b522ffbd70157cd95506ad1 by Miss Islington (bot) 
in branch '3.10':
bpo-45235:  Revert an argparse bugfix that caused a regression (GH-29525) 
(GH-29530)
https://github.com/python/cpython/commit/e4c5a5eabadd1dcd0b522ffbd70157cd95506ad1


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:

I've restored the prior state of affairs.

Leaving this issue open because it still isn't clear what should be guaranteed 
or whether further improvements need to be made.

--
priority: high -> normal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset 587ff7f50bcbfd8346c6d5db459a1628a350c04d by Miss Islington (bot) 
in branch '3.9':
bpo-45235:  Revert an argparse bugfix that caused a regression (GH-29525) 
(GH-29531)
https://github.com/python/cpython/commit/587ff7f50bcbfd8346c6d5db459a1628a350c04d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[miss-islington]

Change by miss-islington :


--
pull_requests: +27781
pull_request: https://github.com/python/cpython/pull/29531

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[miss-islington]

Change by miss-islington :


--
pull_requests: +27780
pull_request: https://github.com/python/cpython/pull/29530

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset 807f839bbfd5805fb76eb3436c9252a0441296eb by Raymond Hettinger in 
branch 'main':
bpo-45235:  Revert an argparse bugfix that caused a regression (GH-29525)
https://github.com/python/cpython/commit/807f839bbfd5805fb76eb3436c9252a0441296eb


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Only 3.9 needs an expedited rerelease.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Pablo Galindo Salgado]

Pablo Galindo Salgado  added the comment:

The next bugfixe release of 3.10 is the 6th of December

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:

How long until the next bugfix releases for 3.9 and 3.10?  To avoid further 
snowballing, it would be great to have this reversion pushed out soonish.

--
nosy: +pablogsal
priority: normal -> high

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Change by Raymond Hettinger :


--
pull_requests: +27775
stage: needs patch -> patch review
pull_request: https://github.com/python/cpython/pull/29525

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Łukasz Langa]

Łukasz Langa  added the comment:

Go for it, Raymond.

--
nosy: +lukasz.langa
resolution: fixed -> 
stage: test needed -> needs patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:

Unless anyone objects, I'll revert this across all affected branches.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Guido van Rossum]

Guido van Rossum  added the comment:

This has caused a regression, also reported here:
https://github.com/Azure/azure-cli/issues/20269 Global Arguments stop working 
in Python 3.9.8

Can we please get some attention from @rhettinger and @ambv?

--
nosy: +gvanrossum

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Łukasz Wałejko]

Łukasz Wałejko  added the comment:

I can also confirm that there is an regression in Python 3.9.8 regarding 
argparse

For example, using watchdog 2.1.6 package

Python 3.9.7 (correct behaviour)
In [1]: from watchdog import watchmedo
In [2]: watchmedo.cli.parse_args(["auto-restart", "echo", "123"])
Out[2]: Namespace(command='echo', command_args=['123'], directories=None, 
patterns='*', ignore_patterns='', ignore_directories=False, recursive=False, 
timeout=1.0, signal='SIGINT', debug_force_polling=False, kill_after=10.0, 
func=)

Python 3.9.8 (incorrect behaviour)
In [1]: from watchdog import watchmedo
In [2]: watchmedo.cli.parse_args(["auto-restart", "echo", "123"])
Out[2]: Namespace(command='auto-restart', command_args=['123'], 
directories=None, patterns='*', ignore_patterns='', ignore_directories=False, 
recursive=False, timeout=1.0, signal='SIGINT', debug_force_polling=False, 
kill_after=10.0, func=)

--
nosy: +lwalejko

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[RhinosF1]

Change by RhinosF1 :


--
nosy: +RhinosF1

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[dgw]

dgw  added the comment:

Can confirm that this patch DOES cause backward compatibility issues, as 
paul.j3's gut feeling said it could. One of my projects, testing against 
3.6-3.9, now fails its test suite on Python 3.9.8, which includes this change. 
Arguments passed to a subparser are indeed ignored in lieu of default values. 
We are tracking the problem in our own issue, 
https://github.com/sopel-irc/sopel/issues/2210

I, for one, am not amused that 7-year-old behavior was "clobbered" (as 
previously described in this thread) in a patch release.

--
nosy: +dgw

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[paul j3]

paul j3  added the comment:

I should study previous posts in more detail, but here are some thoughts on 
correctly handling user namespace.

At the start of `parse_known_args`, there's a

if namespace is None:
namespace = Namespace()

We need to hang on to a copy of this namespace, e.g. call it

import copy
orig_namespace = copy.copy(namespace)

In _SubParsersAction.__call__, pass this copy to the subparser (instead of 
None):

subnamespace, arg_strings = parser.parse_known_args(arg_strings, 
orig_namespace)
for key, value in vars(subnamespace).items():
setattr(namespace, key, value)

Prior to 9351, the main namespace was passed to the subparser

namespace, arg_strings = parser.parse_known_args(arg_strings, namespace)

The trick is to get orig_namespace from the main parse_known_args to 
SubParsersAction.__call__ method.

in a 9351 post I explore the idea of allowing the user to specify a 
'sub_namespace' for the subparser.

https://bugs.python.org/msg230056

In any case, this is a complicated issue that needs careful thought and more 
extensive testing.  I didn't entirely like the original 9351 change, but since 
that's been part of argparse for many years, we need to very careful about 
clobbering it.

--
status: pending -> open

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[paul j3]

Change by paul j3 :


--
stage: resolved -> test needed
status: closed -> pending

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[paul j3]

paul j3  added the comment:

parser = argparse.ArgumentParser()
sub = parser.add_subparsers()
example_subparser = sub.add_parser("example")
example_subparser.add_argument("--flag", default=10)
print(parser.parse_args(["example","--flag=15"], 
argparse.Namespace(flag=20)))

still returns flag=20

User input should override values set by the provided Namespace.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[paul j3]

paul j3  added the comment:

I just downloaded this `argparse.py`.

This change makes it impossible to use a subparser argument if it is defined in 
the user provided namespace, or by the main parser.  It blocks not only 
subparser default, but also user input.

It has reverted the 9351 patch which dates to 2014.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[paul j3]

paul j3  added the comment:

I haven't studied or tested this change, but it looks like a partial retraction 
of

https://bugs.python.org/issue9351 
argparse set_defaults on subcommands should override top level set_defaults

Originally the main namespace was passed to the subparser.  Steven Bethard 
changed it so that the subparser got a fresh namespace, and all values were 
copied to the main namespace.

I and others raised the question of how it affected user provided values 

https://bugs.python.org/issue27859
argparse - subparsers does not retain namespace

Looks like this patch tries to solve some problems by moving the self._defaults 
step to the end of parser_know_args.  I view that change with some trepidation. 
 Handling defaults is tricky enough, with setting them at the start, but then 
only passing them through 'type' at the end if they still match the original 
strings.

Mostly I've been telling StackOverflow questioners that it best not to use the 
same argument 'dest' in both the main and subparsers.  Flags can be the same, 
but the 'dest' should be different to avoid conflicts over which default has 
priority.

Again, I haven't tested this change, but I have a gut feeling it could have 
backward compatibility issues.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[paul j3]

Change by paul j3 :


--
nosy: +paul.j3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Change by Raymond Hettinger :


--
assignee:  -> rhettinger
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset a18d52269ab6071a605d6c72f6af585a4c533ca4 by Miss Islington (bot) 
in branch '3.9':
bpo-45235: Fix argparse overrides namespace with subparser defaults (GH-28420) 
(GH-28443)
https://github.com/python/cpython/commit/a18d52269ab6071a605d6c72f6af585a4c533ca4


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset 6e4101add568910409294554c8e863226a66bb64 by Miss Islington (bot) 
in branch '3.10':
bpo-45235: Fix argparse overrides namespace with subparser defaults (GH-28420) 
(GH-28442)
https://github.com/python/cpython/commit/6e4101add568910409294554c8e863226a66bb64


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[miss-islington]

Change by miss-islington :


--
pull_requests: +26846
pull_request: https://github.com/python/cpython/pull/28443

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Raymond Hettinger]

Raymond Hettinger  added the comment:


New changeset a6e8db5e8e6780411db749d404715dbe021647c7 by Adam Schwalm in 
branch 'main':
bpo-45235: Fix argparse overrides namespace with subparser defaults (GH-28420)
https://github.com/python/cpython/commit/a6e8db5e8e6780411db749d404715dbe021647c7


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[miss-islington]

Change by miss-islington :


--
nosy: +miss-islington
nosy_count: 2.0 -> 3.0
pull_requests: +26845
pull_request: https://github.com/python/cpython/pull/28442

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Adam Schwalm]

Change by Adam Schwalm :


--
keywords: +patch
pull_requests: +26832
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/28420

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue45235] argparse does not preserve namespace with subparser defaults

[Adam Schwalm]

New submission from Adam Schwalm :

The following snippet demonstrates the problem. If a subparser flag has a 
default set, argparse will override the existing value in the provided 
'namespace' if the flag does not appear (e.g., if the default is used):

import argparse
parser = argparse.ArgumentParser()
sub = parser.add_subparsers()
example_subparser = sub.add_parser("example")
example_subparser.add_argument("--flag", default=10)
print(parser.parse_args(["example"], argparse.Namespace(flag=20)))

This should return 'Namespace(flag=20)' because 'flag' already exists in the 
namespace, but instead it returns 'Namespace(flag=10)'. This intended behavior 
is described and demonstrated in the second example here: 
https://docs.python.org/3/library/argparse.html#default

Lib's behavior is correct for the non-subparser cause.

--
components: Library (Lib)
messages: 402060
nosy: ALSchwalm
priority: normal
severity: normal
status: open
title: argparse does not preserve namespace with subparser defaults
type: behavior
versions: Python 3.10, Python 3.11, Python 3.6, Python 3.7, Python 3.8, Python 
3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com