[issue14191] argparse doesn't allow optionals within positionals

[Tadek Kijkowski]

Change by Tadek Kijkowski :


--
pull_requests: +23191
pull_request: https://github.com/python/cpython/pull/24367

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Tadek Kijkowski]

Change by Tadek Kijkowski :


--
nosy: +monkeyman79
nosy_count: 9.0 -> 10.0
pull_requests: +23084
pull_request: https://github.com/python/cpython/pull/24259

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[R. David Murray]

R. David Murray added the comment:

Thanks Paul.  By the way, if you want your "real name" in What's New, just let 
me know what it is and I'll make the change.

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[R. David Murray]

R. David Murray added the comment:


New changeset 0f6b9d230674da784ca79a0cf1a03d2af5a8b6a8 by R. David Murray in 
branch 'master':
bpo-14191 Add parse_intermixed_args. (#3319)
https://github.com/python/cpython/commit/0f6b9d230674da784ca79a0cf1a03d2af5a8b6a8


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[R. David Murray]

R. David Murray added the comment:

I got an offline agreement from Zach Ware, and nobody here at the sprint has 
objected (though I don't know if anyone else looked), so I'll go ahead and 
finish the PR.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

I would dearly love to discard my private copies of argparse subclasses to 
implement this that I have been using for the last 5 years. Please, some other 
committer, concur here!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[R. David Murray]

R. David Murray added the comment:

I've turned intermixed.patch into a PR.  I tweaked the documentation so that it 
does not refer to the details of the implementation.  I tweaked the 
implementation to have the 'try' start before the code that modifies the state, 
and did the line wrapping to <80 columns.

I have one concern about this patch: parse_intermixed_args doesn't support all 
of the argparse features.  It feels to me like it is worthwhile to add anyway, 
and maybe someone will figure out how to support more features later.  But I'd 
like at least one other committer to concur :)

Given concurrence I'll add news and what's new entries.

paul.j3: I'm trying to review argparse patches during this core sprint week, so 
if you have anything in particular you want me to focus on, please let me know.

--
stage:  -> patch review
type: behavior -> enhancement
versions: +Python 3.7 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[R. David Murray]

Changes by R. David Murray :


--
pull_requests: +3345

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[dg1727]

Changes by dg1727 :


--
nosy: +dg1727

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

I encountered a conflict when merging this patch with 
http://bugs.python.org/issue15112.  In my first testcase, 'cmd' and 'rest' 
failed the 'required' test in phase one of 'intermixed'.  That's because 15112 
postponed parsing them (with nargs=0/suppressed).  

I got around that by temporarily setting the 'required' attribute to False.

The whole issue of when (or even whether) a positional that is satisfied with 0 
arguments, is consumed, is a bit messy.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Martin Panter]

Martin Panter added the comment:

It sounds like this bug might cover Issue 15112, which is only concerned with 
options between different positional parameters.

--
nosy: +vadmium

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

Above in
http://bugs.python.org/issue14191#msg187051
I proposed a patch that is quite close to bethard's patch in 
http://bugs.python.org/issue15112#msg166173

Both modify the same place, doing the same (pop items off arg_counts).  The 
logic is a little different.  I'd have to look at them more carefully to see 
whether one is more robust.  Thanks for linking that issue.

The main issue here is different, allowing for complete intermixing of 
optionals and positionals.  Over at 15112 the issue is intermixing of optionals 
and 'whole' positionals.  My 187051 patch belongs over there.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

Changes by paul j3 ajipa...@gmail.com:


Added file: http://bugs.python.org/file29880/mixed.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

Changes by paul j3 ajipa...@gmail.com:


Removed file: http://bugs.python.org/file29880/mixed.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

It's everything I intend to add.  Now I'm just waiting for a committer to act, 
either with suggested changes, or a merge.  I'm watching more than a dozen 
argparse patches.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

So I read over your code again, and even read the documentation this time, and 
it all looks good, and I know it works good because I've been using the code. I 
tried to send a notice through Reitveld, and maybe did, but I don't know where 
it went, so I'll say this much here, too.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

Paul, is this ready to merge, or are you thinking of more refinements?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

This is a refinement of the patch with Message188609.

In parse_known_intermixed_args, the temporary capture of formatted usage has 
been put in a try/finally structure.

Positionals are now 'deactivated' with

action.nargs = SUPPRESS
action.default = SUPPRESS

To use this, a 'nargs==SUPPRESS' case has been added to the relevant methods.  
In _get_args_pattern() it acts just like 'nargs=0'.  In '_get_values()' it 
returns 'value=SUPPRESS'.  The net effect is that, in take_action(), 'action' 
is not invoked, and that positional is not added to the namespace.

Previously I used nargs=0, which put a [] value in the namespace, which then 
had to be deleted.

I have not added anything about this SUPPRESS option to the documentation 
(PARSER isn't there either).

When the parser uses incompatible features (e.g. REMAINDER), this now raises a 
TypeError.  The effect is similar to giving add_argument incompatible 
definitions.  The 'fallback' that I used earlier can be implemented with a 
simple 'try/except TypeError'.  Other parsing errors go through the usual 
ArgumentParser.error() method.

test_argparse.py has been changed to handle this TypeError.

--
Added file: http://bugs.python.org/file30422/intermixed.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

These sound like good refinements. You've been thinking.  By making the 
fallback happen externally, it simplifies the implementation of 
parse_intermixed_args, and forces the application to accept responsibility for 
calling it with a consistent set of arguments, or calling something else. I 
like that. I don't really see the fallback as a particularly useful feature, so 
pushing it outside the stdlib, yet still making it simple to implement for any 
that do find it to be useful, seems like a good tradeoff.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

Changes by paul j3 ajipa...@gmail.com:


Removed file: http://bugs.python.org/file29880/mixed.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

OK, I've been running with the new code most the day, and it seems functional 
in my testing.

I only sort of follow your discussion about the custom action class caveat, 
probably because I haven't used custom action classes... I tried once, but 
failed to achieve my goal, as it was more ambitious than they presently 
support. If the [] value is significantly problematical in some manner, could 
positional nargs be set to a sentinal value that would avoid the assignment of 
the [] value? I realize that would require code changes in some other function 
or functions, in addition to the added new functions, so that would make the 
patch a bit more intrusive.

If _fallback helps some folks with understanding errors clearly, I won't object 
to it. I guess it would only be programmers that would be confused, because 
they would be the ones interpreting the errors... and with adequate testing, 
should fix the programming errors before the users get a chance to be confused.

So maybe your next .patch will be ready to ship!

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

'parse_fallback_args()' function is only in the 'test_intermixed.py' file, not 
the patch.  It should be in the 'if __name__' section of that file, along with 
the modified 'exit()' method, since it is part of these testing suit, not meant 
to be imported.  'test_intermixed.py' is more of an example and discussion 
tool, not a formal test.

I added the '_fallback' optional argument because the normal exit from 
'parse_args' using SystemExit is remarkably uninformative.  It's hard to 
distinguish between the 'fallback' errors, and the ones generated by 
'parse_known_args' (that have more to do with the argument strings).  One is a 
programming error, the other a user generated error.

It is possible to redefine ArgumentParser.error() so it gives more information, 
for example by raising an Exception(message).  I have added to 
test_intermixed.py an alternative 'parse_fallback_args' that uses such a 
modified error rather than the _fallback option.  But that ends up modifying 
the normal parsing error generation as well.

I used the 'fallback' idea to test 'parse_intermixed_args' against the whole 
set 'test_argparse.py' tests.  It would nice to have a way of doing that 
automatically anytime other features are added to 'parse_args'.  But I can't 
think of a clean way of doing that.

Regarding earlier versions of these files - I do not see a way of deleting 
them.  

I have attached a modified test_intermixed.py that has these changes.  I also 
modified how 'parse_known_intermixed_args' restores the original value of 
self.usage, using an outer 'try:finally:' block.  I need to make a note to 
myself to put that in the patch.

--
Added file: http://bugs.python.org/file30204/test_intermixed.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

paul j3: Regarding earlier versions of these files - I do not see a way of 
deleting them. 

Click on edit, then there is an option to unlink. I don't know if they ever 
actually get deleted, but it clears out the clutter when looking for the latest 
version.

Will check out the newest code shortly.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

I should note one caveat:

As a consequence of setting nargs to 0 for the first 'parse_know_args' step, 
all positional entries in the namespace get an empty list value ([]).  This is 
produced by 'ArgumentParser._get_values'.  With the builtin action classes this 
does not cause any problems.

However a custom action class might have problems with this [] value.
For example in 'test_argparse.py', TestActionUserDefined the PositionalAction 
class does check the values and throws an error with this [] value.

The positional arguments are removed from the namespace before it is passed on 
to the 2nd 'parse_known_args', so these [] in the first don't affect the final 
namespace.

I don't think anything about this should be added to main documentation, since 
it could confuse most readers.  I might add a note of warning to the code 
itself.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

Paul, thanks for your continued work.

I had reworked your prior patch into a subclass of Argument Parser, and 
tweaking the code to get parse_intermixed_args to adjust the behaviors I had 
reported.

Now substituting exactly your more flexible new code into my subclass from your 
latest test_intermixed.py (you should delete your old patches), I can quickly 
confirm that it works with my applications that used to use my wrapper class, 
and expect and use intermixed functionality.

I also read through all your code and comments and it looks good to me.

Regarding parse_fallback_args, I do not see documentation for it. If that is 
intentional, you might want to add comments in the code regarding its use for 
testing only... and might want to rename it to _parse_fallback_args. I 
personally don't see a lot of value to the function, or the new parameter; 
tests for parse_intermixed_args and parse_known_intermixed_args should be (and 
have been, thanks) added to the tests for argparse, and should suffice for 
testing. In non-test code, I see no benefit: either the user uses features that 
are incompatible with parse_intermixed_args, and thus uses the other features 
of argparse, or the user, for compatibility reasons, needs to use 
parse_intermixed_args, and thus is prevented from successfully using the 
incompatible features. If I'm missing some benefit of parse_fallback_args, it 
should be explained in either the documentation or the comments.

Regarding the terminology: both intermixed and interspersed would be correct 
English words to describe the use case. So would intermingled :)

Because Stephen blessed intermixed, and because it is used by getopt 
documentation (getopt has not been deprecated, optparse has), it seems to be 
the best term to use. Should optparse someday be removed, along with its 
documentation, the use of the term interspersed would also disappear, leaving 
more consistency in terminology.

Alternative:

Because optparse uses interspersed in an API, we cannot fix it to use 
intermixed. However, we could fix the uses of intermixed to be 
interspersed prior to or at the time of accepting your patch to argparse... 
afterwards, it would be too late.  Personally, I see no problem with the use of 
both terms in the documentation, and intermixed is the shortest, so I have a 
slight preference for that.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

This is a revision of the test_intermixed.py that I submitted earlier.  Now 
`parse_intermixed_args` acts like `parse_args', and calls 
`parse_known_intermixed_args`.  Again it is form that can exercise the idea 
without modifying `argparse.py`.

If the parser has incompatible features (REMAINDER, PARSER, or certain 
exclusive groups), it raises an error.  However to facilitate testing I 
included a `_fallback` backdoor.  If not default None it will be called instead 
of raising the error.

While making documentation changes, I got to wondering whether 'interspersed' 
would be a better term than 'intermixed'.  optparse has an 'interspersed' 
option and api.  However the getopt documentation does use 'intermixed'.

--
Added file: http://bugs.python.org/file30159/test_intermixed.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

This is the formal patch corresponding to the `test_intermixed.py`.  It 
includes changes to `argparse.rst`, plus tests in `test_argparse.py`.  These 
tests are near the end, after those for `parse_known_args`.  They are roughly 
equivalent to the examples in `test_intermixed.py`.
 -
The new documentation section is:

Some users expect to freely intermix optional and positional argument strings. 
For example, optparse, by default, allows interspersed argument strings. GNU 
getopt() permutes the argument strings so non-options are at the end. The 
parse_intermixed_args() method emulates this behavior by first calling 
parse_known_args() with just the optional arguments being active. It is then 
called a second time to parse the list of remaining argument strings using the 
positional arguments.

parse_intermixed_args() raises an error if the parser uses features that are 
incompatible with this two step parsing. These include subparsers, 
argparse.REMAINDER, and mutually exclusive groups that include both optionals 
and positionals.

In this example, parse_known_args() returns an unparsed list of arguments [‘2’, 
‘3’], while parse_intermixed_args() returns rest=[1, 2, 3].

 parser = argparse.ArgumentParser()
 parser.add_argument('--foo')
 parser.add_argument('cmd')
 parser.add_argument('rest', nargs='*', type=int)
 parser.parse_known_args('cmd1 1 --foo bar 2 3'.split())
(Namespace(cmd='cmd1', foo='bar', rest=[1]), ['2', '3']) 
 parser.parse_intermixed_args('cmd1 1 --foo bar 2 3'.split())
Namespace(cmd='cmd1', foo='bar', rest=[1, 2, 3])

parse_known_intermixed_args() method, returns a two item tuple containing the 
populated namespace and the list of remaining argument strings. 
parse_intermixed_args() raises an error if there are any remaining unparsed 
argument strings.

--
Added file: http://bugs.python.org/file30160/intermixed.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

Very nice, Paul.

I tested that with some of my applications, and some of my test cases. All of 
them initially failed, because you have parse_intermixed_args returning 
parameters like parse_known_args instead of like parse_args. Now I can 
understand that might be a little confusing in msg166175, but note that the 
implementation is like a call to parse_known_args followed by a call to 
parse_args... strongly implying that the return should be like parse_args.

After tweaking your implementation in that regard, then I was able to get all 
the same applications and test cases to pass, although I haven't tried all my 
applications and all my test cases, as yet.

Your techniques for disabling particular parameters are pretty clever.

I think the difficult cases should raise an error. 

Firstly, parse_intermixed_args is intended to be for functional compatibility 
with optparse functionality, which doesn't support the difficult cases, 
therefore use of the difficult cases would require additional restrictions on 
the allowed order of options on the command line, beyond what optparse 
supports... this would be an application interface change, and as part of that 
interface change, should such happen, the flexibility of intermixing optionals 
and positionals can be restricted.

Secondly, if an understanding of how to define the use parse_intermixed_args 
with one or more of the difficult cases is reached, replacing an error case 
with a functional case is possible, but replacing one silent functionality with 
a different one is a backwards compatibility problem. Throwing an error avoids 
limiting a future definition of these cases.

The freedom of mixing optionals and positionals that would available in the now 
deprecated optparse does seem to be restored by this patch.

I look forward to seeing a revised patch, this is a very promising solution to 
this bug.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

Yes, http://bugs.python.org/msg166175 does use 'parse_args' in the second call. 
 But I think things would be more flexible if we had a second function:

def parse_???(self, args=None, namespace=None):
args, argv = self.parse_intermixed_args(args, namespace)
if argv:
msg = _('unrecognized arguments: %s')
self.error(msg % ' '.join(argv))
return args

But then what would a be a good pair of names?

parse??? and parse_intermixed_args
versus
parse_intermixed_args and parse_known_intermixed_args
or
something else?

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

Yes, a second function would give more flexibility.

Due to the approval in msg166175 to use the name parse_intermixed_args for 
the functionality described there, it would probably be best to use that name 
for that functionality.

So then we are left naming your current function something else. 
parse_known_intermixed_args certainly is descriptive, and fits the naming 
conventions of the other methods in the class. Quite long, unfortunately... but 
then I doubt it will get used much. I am using parse_intermixed_args regularly 
(via my wrapper class), and it is quite long enough.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

The attached file has a 'parse_intermixed_args()' that has the same API as 
'parse_known_args()'.

It follows the two parse step model

args, remaining_args = optionals.parse_known_args()
args, extras = positionals.parse_known_args(remaining_args, args)

except that the 'optionals parser' is self with the positional arguments 
'deactivated' by setting their nargs to 0.  Similarly the 'positionals parser' 
is self with the optional arguments set to 'required=false'.

Here it is in a standalone test module illustrating its functionality and 
limitations.  I could provide a patch, but this form might easier to test in 
your own code.

When used to run test_argparse.py, it had problems in the cases where the 
distinction between positionals and optionals is blurred.  

For example, PARSER and REMAINDER are supposed to grab everything that follows 
regardless of what it looks like.  I choose to fall back on a single 
'parse_know_args' call.  Raising an error would the alternative.

Similarly, a mutually exclusive group that includes a positional is difficult 
to handle.  Again I fall back on the single step.

So the two issues to be discussed are:

- does it provide the desired freedom to mix optionals and positionals?

- in the difficult cases, should it punt, or raise an error?

--
Added file: http://bugs.python.org/file29982/test_intermixed.py

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

Paul, your comments are interesting, but your proposed patch doesn't actually 
solve the problem.

So here I am typing away at my command prompt, and I type in a couple optional 
parameters I know I'll need and start on the sequence of positional ones, and 
half way through I remember oh, I need another optional paremeter so I type 
it in next before I forget, and then go on with the positional ones.

No prior Unix-style argument parsing mechanism that I have ever seen or heard 
of would be confused by that, but argparse is.

This bug is about providing a facility in argparse that supports intermixing 
optional parameters into strings of positional parameters, just like all prior 
Unix-style argument parsing mechanisms, so that an application can be ported to 
use argparse without breaking command lines that their users have stored in 
command files. Otherwise argparse is not an upgrade path for an application, 
yet optparse has been deprecated.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

I should clarify, before someone jumps in: some particular applications do 
implement restrictions on order of optional and positional arguments; I'm aware 
of that. getopt easily supported application defined order restrictions, 
because it processed arguments sequentially, and the processing loop was user 
code. optparse, as has been pointed out, parses the optionals, and leaves a 
single list of positionals, combined from between all the optionals, for the 
user code to process in any manner, but would actually make it harder for user 
code to implement order restrictions. argparse goes the other way, taking over 
all the user parsing (which is a good thing), but not providing sufficient 
features to implement flexible mixing of optional and positional arguments.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

Glenn
Take a look at http://bugs.python.org/issue15427

I took a stab at changing the documentation, including recommending 
parse_known_args when porting optparse uses.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman added the comment:

Docs look good as mentioned there, for the current behavior, although it would 
be good to improve the behavior.  

Note that I have supplied a wrapper (t18a.py) (if it hasn't bit-rotted for 3.4, 
I'm still using 3.3) that provides the needed functionality. The problem is, 
that I have no clue how to modify the internals of argparse to allow it to 
simply be a method of the current argparse class.  One could achieve the goal 
by renaming the current argparse class to _argparse, and renaming my wrapper 
class to be the real argparse, and that would work, but would seem to be 
inefficient.

It would be nice if someone could move the needed functionality, a new API 
called parse_intermixed_args, already approved by msg166175, that does the same 
thing as my wrapper does, but without the wrapper class. This would be a cure 
to the problem, and it could be tested against my wrapper class by comparison 
to ensure the needed functionality is provided. I'd be glad to help with 
testing and understanding the requirements, but don't have time to figure out 
the internals of argparse at present.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

This patch permits the mixing of optionals with positionals, with the caveat 
that a particular positional cannot be split up.

If:

parser = ArgumentParser()
parser.add_argument('-f','--foo')
parser.add_argument('cmd')
parser.add_argument('rest', nargs='*')

'-f1 cmd 1 2 3', 
'cmd -f1 1 2 3', 
'cmd 1 2 3 -f1' 

all give {cmd='cmd', rest=['1','2','3'], foo='1'}.  

But 'cmd 1 -f1 2 3', does not recognize ['2','3'].

Previously 'cmd -f1 1 2 3' would return rest=[], and not recognize 
['1','2','3'].  With this change the nargs='*' behaves more like nargs='+', 
surviving to parse the 2nd group of positional strings.

The trick is to modify arg_counts in consume_positionals(), removing matches 
that don't do anything (don't consume argument strings). 

if 'O' in arg_strings_pattern[start_index:]:
# if there is an optional after this, remove
# 'empty' positionals from the current match
while len(arg_counts)1 and arg_counts[-1]==0:
arg_counts = arg_counts[:-1]

This change passes all of the existing test_argparse.py tests.  It also passes 
the optparse tests that I added in http://bugs.python.org/issue9334#msg184987
I added 4 cases to illustrate this change.

--
Added file: http://bugs.python.org/file29880/mixed.patch

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[paul j3]

paul j3 added the comment:

Glenn
I looked at your t18a.py test case

parser = ArgumentParser()
parser.add_argument('--foo', dest='foo')
parser.add_argument('--bar', dest='bar')
parser.add_argument('foz')
parser.add_argument('baz', nargs='*')

and parse variations on 'a b c d --foo x --bar 1'

I think your main problem is with the 'baz', nargs='*'.  If nargs was changed 
to '+', 'a --foo x b c d --bar 1' would work, returning {foz='a', 
bar=['b','c','d']}.

argparse alternates between consuming positional and optionals until it runs 
out of arguments or argument strings.  But with '*', both 'foz' and 'baz' are 
consumed with the first set of positional strings {foz='a', baz=[]}.  When it 
gets to 'b c d' there are no more positional arguments to consume, so they get 
put into 'extras'.

With nargs='+', 'a b --foo x c d --bar 1' would assign {foz='a', baz=[b]}, and 
extras=['c','d'].

So while optionals can be interspersed with positionals, they can't be placed 
within the set of strings intended for one positional. That seems to me to very 
reasonable (why break up 'b c d'?).  And as your file demonstrates, you can 
fall back on parse_known_args to handle the extras.

--
nosy: +paul.j3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Andrew McNabb]

Changes by Andrew McNabb amcn...@mcnabbs.org:


--
nosy: +amcnabb

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Steven Bethard]

Steven Bethard steven.beth...@gmail.com added the comment:

I created Issue 15427 for the parse_args documentation bug. So let's make this 
issue just about parsing intermixed arguments.

Yes, if someone would like to provide a patch for this, please create a method 
parse_intermixed_args rather than adding a boolean flag parameter. It should 
be basically equivalent to this code:

args, remaining_args = optionals.parse_known_args()
args = positionals.parse_args(remaining_args, args)

Except that it should give proper error messages. There should be some tests to 
make sure both that it parses things as expected and that it gives error 
messages as expected.

--
assignee: docs@python - 
components:  -Documentation
title: argparse: nargs='*' doesn't get out-of-order positional parameters - 
argparse doesn't allow optionals within positionals
versions: +Python 3.4 -Python 2.7, Python 3.2, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue14191] argparse doesn't allow optionals within positionals

[Glenn Linderman]

Glenn Linderman v+pyt...@g.nevcal.com added the comment:

So my t18a.py wraps Argparse, because the externals are documented and I could 
understand that. Given enough time, I might be able to understand the internals 
too... it is just Python...

Seems like the internals separate positionals and optionals into two 
subparsers, the logic needed is to save the positionals, temporarily replace 
them with an empty group, and call parse_known_args, then restore the 
positionals, and call parse_known_args again.

What I haven't figured out, is how that would affect the help message building, 
and how to make the empty group of positionals without affecting the help 
message.

It's also not clear that it is possible, with current internals, to substitute 
an empty group... the only features for creating a group seem to always add it 
to a list of _action_groups.  So that muddies the water, just like the help 
message... everything seems intertwined internally.

A little guidance here, would let me code... I could probably whack and slash, 
and keep extra lists of arguments and sort of make it work like my wrapper 
does, but I doubt that would be an acceptable patch. So I would be happy, with 
guidance, to try to create the code, and I can do a diff, but beyond that, 
I'm sort of at a loss.

So I'd be happy to work with someone to create the patch, but I've also never 
created a patch, nor test cases, for Python (or any other open source project). 
Theoretically, it is straightforward, and documented; as a practical matter, it 
isn't likely that I'll find time to figure out all that methodology and 
actually create a patch, in the near future (although it is on my list of 
things to learn, in the fullness of time, and after the first one, I'm sure 
subsequent ones would be easier).

Or if someone else wants to code it, I'd be happy to look it over, test it with 
my environment and applications that I'm currently using with my wrapper.

Given this guidance, I've tweaked my wrapper to have parse_intermixed_args 
rather than changing the behavior of parse_args as I have been, and will tweak 
the apps correspondingly, so I'll be in a position to test any code created for 
this issue.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue14191
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com