[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Change by Tadek Kijkowski :


--
pull_requests: +23190
pull_request: https://github.com/python/cpython/pull/24367

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

> So in the big picture, the purpose of this change is to treat the inputs like 
> a kind of state-machine.

Not necessarily that. Simple parsers should be easy to write, complicated 
parsers should be _possible_ to write. Users should be able to do whatever they 
wish by providing custom actions. Like writing Conway's Life in Excel or 
building Turing Machine using bricks and conveyors. Problem with positional 
parameter is a blocker. Fact that you can: 1. mix positional parameters with 
options in general, 2. have unlimited number of positional parameters, but 3. 
can't have 1 and 2 at the same time, makes great number of things impossible. 
For me that looks like a bug, not a feature.

Capture actions are second priority for me, because if this PR goes through, 
user will be able to write his own actions.

> While the proposed change to the core parser is (apparently) minor, it does 
> occur at the center of the action. That is not the place we want any (new) 
> bugs or backward incompatibility.

The core change is apparently minor, and also actually minor. It boils out to 
this:

@@ -2010,17 +2020,23 @@ def consume_positionals(start_index):
 match_partial = self._match_arguments_partial
 selected_pattern = arg_strings_pattern[start_index:]
 arg_counts = match_partial(positionals, selected_pattern)
+action_index = 0

 # slice off the appropriate arg strings for each Positional
 # and add the Positional and its args to the list
-for action, arg_count in zip(positionals, arg_counts):
+for arg_count in arg_counts:
+action = positionals[action_index]
 args = arg_strings[start_index: start_index + arg_count]
 start_index += arg_count
 take_action(action, args)
+# if positional action nargs is '**',
+# never remove it from actions list
+if action.nargs != AS_MANY_AS_POSSIBLE:
+action_index += 1

 # slice off the Positionals that we just parsed and return the
 # index at which the Positionals' string args stopped
-positionals[:] = positionals[len(arg_counts):]
+positionals[:] = positionals[action_index:]
 return start_index

It's not hard to determine that if all action.nargs != AS_MANY_AS_POSSIBLE, old 
and new code do exactly the same.
Besides that's what tests and code review is for. As for backward 
compatibility, nargs='**' was illegal until now,
and for all parameters with nargs!='**' the code behaves exactly as before.

> And the full implementation requires a new Action subclass that is quite 
> different from existing ones.

No, that's an extra. That's why it's separate issue and separate PR. This issue 
is about giving users ability to write their own actions.

> Given how different this is from the normal argparse parsing (and the POSIX 
> parsing argparse seeks to emulate), I question the wisdom of adding this, in 
> part or whole, to the stock distribution.  It could certainly be published as 
> a pypi.  That already has a number of  parsers, some built on argparse, 
> others stand alone.
>
> I also wonder whether it would be simpler to do this kind of parsing directly 
> from sys.argv.  Just step through that list, consuming the values and flags 
> in sequence.

In other words, "if argparse doesn't meet your requirements, you can write your 
own parser". I don't think that's the way to go.

Both GNU getopt and GNU argp_parse allow mixing options with positional 
arguments without reordering. For getopt it's: options argument string begins 
with a hyphen, for argp_parse it's ARGP_IN_ORDER flag. I don't see why argparse 
wouldn't allow that. Original POSIX getopt didn't even have longopts.

> Sorry to sound like a wet blanket, but I've seen too many seemingly innocent 
> patches that caused unforeseen problems down the line.

Yeah, that can be said about any project at any time - changes can introduce 
bugs, so let's just keep it as it is. I appreciate your ideas, but your 
criticism does sound unsubstantiated: "POSIX getopts didn't have it", "it can 
be done on pypi", "enhancements introduce bugs".

I think this change should be applied, because it is small change in code which 
removes significant obstacle in writing advanced option parsers.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Glenn Linderman]

Glenn Linderman  added the comment:

paul j3 said:

Given how different this is from the normal argparse parsing (and the POSIX 
parsing argparse seeks to emulate), I question the wisdom of adding this, in 
part or whole, to the stock distribution.  It could certainly be published as a 
pypi.  That already has a number of  parsers, some built on argparse, others 
stand alone.


I say:

This has been a deficiency of argparse from day one. Tadek's solution seems to 
enable addressing the deficiency in a backward-compatible manner. Do you, paul, 
find any test failures? Or any incompatibilities that may not be in the test 
cases? If not, then it certainly does seem like a wet-blanket comment.


paul j3 forther said:

I also wonder whether it would be simpler to do this kind of parsing directly 
from sys.argv.  Just step through that list, consuming the values and flags in 
sequence.  


I say:

The whole point of argparse, and of deprecation of the prior optparse was to 
make more functionality available in a more powerful API. Increasing the power 
of the API seems to be consistent with the purpose of argparse. It took me some 
time and use cases to discover the limitations of argparse, and although 
parse_intermixed_args solved the use cases I had, I was well aware that it 
didn't solve cases of every existing Unix utility.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[paul j3]

paul j3  added the comment:

Sometimes patches have unforeseen benefits.  My earlier patch for this issue, 
parse_intermixed_args, has been useful beyond the OP's case.

https://stackoverflow.com/questions/50916124/allow-positional-command-line-arguments-with-nargs-to-be-seperated-by-a-flag

https://bugs.python.org/issue15112
argparse: nargs='*' positional argument doesn't accept any items if preceded by 
an option and another positional

With

 usage: test.py [-h] [-a A] b [c]

and 

 test.py B -a A C

has problems because the optional '[c]' positional is used up when 'b' is 
processed.   Intermixed gets around this by first processing '-a', and then 
handling 'b [c]' together.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[paul j3]

paul j3  added the comment:

So in the big picture, the purpose of this change is to treat the inputs like a 
kind of state-machine.

In the bigger example, the `**` positional values are processed one by one, 
using the interspersed optionals to set state parameters.

`args.sources` then ends up as a list of dicts, each of the form:

{'name': 'input3.txt', 'frobnicate': None, 'massage_level': 5}

where 'name' is the positional value, and 'frobnicate' and 'massage_level' are 
the latest values (default or user supplied).  The optionals can be specified 
in any order or repeatedly.

While the proposed change to the core parser is (apparently) minor, it does 
occur at the center of the action.  That is not the place we want any (new) 
bugs or backward incompatibility.  And the full implementation requires a new 
Action subclass that is quite different from existing ones.

Given how different this is from the normal argparse parsing (and the POSIX 
parsing argparse seeks to emulate), I question the wisdom of adding this, in 
part or whole, to the stock distribution.  It could certainly be published as a 
pypi.  That already has a number of  parsers, some built on argparse, others 
stand alone.

I also wonder whether it would be simpler to do this kind of parsing directly 
from sys.argv.  Just step through that list, consuming the values and flags in 
sequence.  

Sorry to sound like a wet blanket, but I've seen too many seemingly innocent 
patches that caused unforeseen problems down the line.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Glenn Linderman]

Glenn Linderman  added the comment:

Yes I think this is a useful enabling step toward enhanced functionality, as is.

But I think the learning curve to achieve the enhanced functionality is a bit 
high for most people, as it requires too much knowledge of argparse internals, 
so I really look forward to your followon work on 'capture' actions. which will 
be an enhancement that will not require internals knowledge, and will resolve a 
day-one deficiency in argparse in a nicely backward-compatible manner.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

I went with Paul's suggestion and added nargs='**' instead of greedy, so at 
this moment this PR doesn't introduce any changes in any public signatures.

I'm working on 'capture' actions, but I want that to be a separate PR and a 
separate issue because - first - it may be harder to accept as it is bigger 
change in both code and documentation and - second - it actually is a separate 
feature, which could be useful for optional arguments even without nargs='**' 
implemented.

So, what do you all think about this PR as it now? Docs and tests are updated.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

Paul,

> The basic logic of argparse is to accept optionals in any order, and 
> positionals in strict positional order.  'nargs' allows us to pair any number 
> of strings with each optional's flag.

I started this issue because that was insufficient for my project.

> I wonder if adding a new nargs, such as '**' would be better than than the 
> new parameter 'greedy_star=True'.

Yeah, I considered that earlier, but I thought that would be too much effort to 
review all the places where it would have to be treated as equal to '*'. But 
now I agree with you, I'll try to change that.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Glenn Linderman]

Glenn Linderman  added the comment:

Paul said:
I haven't had a chance to study your longer posts, but it seems to me that the 
AddFruitAction example could just as well be implemented with 

parser.add_argument('--color', nargs='*', action='append')

with post parsing processing to create the 'fruits' dicts from the appended 
lists.  


That was also my first reaction, Paul, when I read Tadek's proposal. But I 
quickly realized that particularly with the feature of "capture and reset to 
default" that the number of instances of a particular optional and the number 
of positionals do not always match, and there is no way to correlate them 
later, even if they are all captured and saved.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Glenn Linderman]

Glenn Linderman  added the comment:

This sounds very good to me.  Might also want action='store_capture' for a 
single positional definition?

capture could be a string, or any iterable of strings (tuple comes to mind)

capture_once have similar value as capture, but I wonder if the naming would be 
better as capture_reset to indicate the value is reset to default, rather than 
just captured.


Tadek said:
There could be new predefined action, e.g. 'extend_with_capture' (or just 
'extend_capture') which could be used like this:

  parser.add_argument('fruits', nargs='*', action='extend_capture', 
capture_once=['color'])

Where 'capture' would be one of: a list of parameters to capture (could be a 
string instead of a list if it's just one parameter) or '*' to capture all 
optional parameters and 'capture_once' would be similar list of parameters to 
capture and reset to default.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[paul j3]

paul j3  added the comment:

I haven't had a chance to study your longer posts, but it seems to me that the 
AddFruitAction example could just as well be implemented with 

parser.add_argument('--color', nargs='*', action='append')

with post parsing processing to create the 'fruits' dicts from the appended 
lists.  

The basic logic of argparse is to accept optionals in any order, and 
positionals in strict positional order.  'nargs' allows us to pair any number 
of strings with each optional's flag.

While custom Action classes can implement interactions between arguments based 
on values in the namespace, it is usually easier to do this after parsing.  

But back to your core change, I wonder if adding a new nargs, such as '**' 
would be better than than the new parameter 'greedy_star=True'.  I prefer not 
to add parameters that are hard to document.  At this point in the argparse 
development, changes should be minimally invasive.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

>> In the docs fruits example, though, I think the input and output are 
>> inconsistent: you have a brown banana reported as yellow?

Yeah, I noticed that and fixed already :)


I agree with your ideas. My goal was just to provide bare minimum that would 
allow users to implement this functionality on their own, but turning it into 
something useful out of the box sounds like a good idea.

There could be new predefined action, e.g. 'extend_with_capture' (or just 
'extend_capture') which could be used like this:

  parser.add_argument('fruits', nargs='*', action='extend_capture', 
capture_once=['color'])

Where 'capture' would be one of: a list of parameters to capture (could be a 
string instead of a list if it's just one parameter) or '*' to capture all 
optional parameters and 'capture_once' would be similar list of parameters to 
capture and reset to default.

>> Does the namespace still contain the default value for an optional after it 
>> has been used?  It looks like it might

I think it does - not the namespace, but ArgumentParser, which is also 
available in Action.__call__, has 'get_defaults'.

>> I'm not sure how that plays with optionals that use append or append_const 
>> storage methods, nor do I fully understand the comment in "def 
>> _copy_items(items):"

I don't see any danger here, the action will just have to make sure to capture 
copy of the array, rather than reference to it. The same applies to 'extend'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Glenn Linderman]

Glenn Linderman  added the comment:

for more helpful  => far more helpful

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Glenn Linderman]

Glenn Linderman  added the comment:

OK, I think I see what you are doing here. Thanks for your responses.

And probably it is the bare-bones minimum feature that allows 
user-implementation of "as complex as desired" combinations of optionals and 
context-sensitive positionals.

In the docs fruits example, though, I think the input and output are 
inconsistent: you have a brown banana reported as yellow?

I guess I could think of enhancements to simplify common cases. For example, 
each positional might want to track/save values for some number of optionals. 
So one could write custom code to do that, like your fruit basket, or one could 
imagine a canned approach, where the list of optionals was passed as a 
parameter when defining the positionals.  This could be built as a wrapper for 
add_argument, that would call a more general version of the AddFruit Action.  
Perhaps a special parameter could also indicate saving the values of all 
optionals at the point of parsing the positional, giving each positional a 
complete set of optionals that apply.

Your feature of having the optional apply to only one positional would be a 
harder thing, especially if you wanted to have both kinds of optionals: that 
apply to the next positional, or that apply to all following positionals. But 
hmm.  Maybe not much harder: just have two lists of optionals passed to the 
positional: a list of each kind.

Does the namespace still contain the default value for an optional after it has 
been used?  It looks like it might, so your fruit example could be reduced to 
not needing the DEFAULT_COLOR variable, but rather change references to it to 
use 

  self._get_default_metavar_for_positional(action) 

?  That avoids using a global variable, and is for more helpful for the "list 
of optionals to be reset on each use" rather than re-creating the more complex 
data structure required to hold all the defaults.

I'm not sure how that plays with optionals that use append or append_const 
storage methods, nor do I fully understand the comment in "def 
_copy_items(items):"

I think if you could avoid the global variable, your implementation would be 
stronger. And if you provide, either as a sample subclass, or argparse new 
feature, the ability to do the things I mention here, it would be even 
stronger, as it would allow the user to deal with the more powerful 
capabilities without needing as much understanding of argparse internals.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

This is, I think, smallest functional example for matching optional parameters 
with positionals - fruits.py:

  import argparse
  
  DEFAULT_COLOR="plain"
  
  class AddFruitAction(argparse.Action):
 def __call__(self, parser, namespace, values, option_string=None):
for fruit in values:
   namespace.fruits.append({'name': fruit, 'color': namespace.color})
   namespace.color = DEFAULT_COLOR
  
  def show_fruits(fruits):
 for fruit in fruits:
print(f"{fruit['color']} {fruit['name']}")
  
  parser = argparse.ArgumentParser(greedy=True)
  parser.add_argument('--color', default=DEFAULT_COLOR)
  parser.add_argument('fruits', nargs='*', action=AddFruitAction, default=[])
  args = parser.parse_args()
  show_fruits(args.fruits)

It starts with 'namespace.color' set to 'DEFAULT_COLOR' - 
'default=DEFAULT_COLOR' takes care of that, and with 'namespace.fruits' set to 
empty list - via 'default=[]'.

For each group of positional command-line arguments, AddFruitAction is called 
with one or more fruit names in 'value'. The method iterates over them adding 
series of dicts to the 'fruits' list. The 'namespace.color' is immediately 
reset to default value, because we want the 'color' to apply to one following 
'fruit'. If we wanted the 'color' to apply to all following 'fruits' the action 
class could just leave it alone.

After parsing is done, we get our namespace assigned to args, with list of 
color + fruit name pairs in 'args.fruits'.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

>> Is it up to the special handler of the positional parameter to read and save 
>> the values of the optional parameters specified so far?

Yes, in order to get anything more that just concatenated list of positional 
parameters, one has to provide specialized action class. Basic functionality is 
easy to implement, as you can see in the fruits.py example.

>> ... how are you saving that context?

Entire context is stored in the namespace object. Action class has access to 
all previous parameters via the namespace and it can modify the namespace as 
needed. Standard 'store' action just adds value to the namespace, but 
specialized action can anything.

>> What about the ones unspecified so far?

They are obviously not available for the action processing positional 
parameters preceeding them, but will be present in the resulting namespace and 
can be processed after parsing is done.

>> Can one even read and save the default value?

I'm not sure what do you mean, but it all depend on how action class deals with 
it.

>> And where does it get stored? Is the value of the positional parameter 
>> turned into a dict or class containing its own value + the saved optional 
>> parameters?

It all gets stored in the namespace. Turning positionals + optionals into a 
dict or class seems most reasonable, but it all depends on how action class 
will deal with them.

>> I think I have seen Unix command line programs that had rich semantics 
>> similar to what you are proposing, where the sequence of repeated optional 
>> args could affect handling of later positionals but not earlier ones.

For example 'tar' with '-C' parameter which applies to all files following it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Glenn Linderman]

Glenn Linderman  added the comment:

On 1/25/2021 12:43 PM, Tadek Kijkowski wrote:
> Tadek Kijkowski  added the comment:
>
> I added tests and docs to the PR. How does it look now?
Could you send me the docs privately?

I'm trying to understand what you are suggesting, without reading the code.

I realize that you are trying to apply different values of optional 
parameters to positional parameters... how are you saving that context? 
Is it up to the special handler of the positional parameter to read and  
save the values of the optional parameters specified so far? What about 
the ones unspecified so far? Can one even read and save the default value?

And where does it get stored? Is the value of the positional parameter 
turned into a dict or class containing its own value + the saved 
optional parameters?

I do agree that the even though the parse_intermixed_args was sufficient 
for the use case I had at the time, and has been sufficient for me so 
far, that it  not fully flexible, and I think I have seen Unix command 
line programs that had rich semantics similar to what you are proposing, 
where the sequence of repeated optional args could affect handling of 
later positionals but not earlier ones.

So I applaud your efforts here, but simply reading the issue, and having 
forgotten most of the internal workings of argument parser since getting 
the compromise going, I think reading your docs would help clarify it 
for me.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

I added tests and docs to the PR. How does it look now?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Glenn Linderman]

Change by Glenn Linderman :


--
nosy: +v+python

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

>> I tried to checkout python sources, but even with -depth 1 it took too long 
>> for my patience and I gave up after 'du -sh .git' showed 2G.

Ignore that. I must have been doing something worng.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Tadek Kijkowski  added the comment:

>> Your example is incomplete.  That is _CustomAction?  What namespace does 
>> your patch produce.

>> In your example I'd suggest making '--tracks' a nargs=2 and 'append' option. 
>>  'choices' would have to be replaced with a either a custom 'type', a custom 
>> Action, or post-parsing testing.

My use case and example were oversimplified. The real life use case would be a 
program that processes multiple sources, with various processing parameters 
which may change between sources and writes result to destination:

example.py:

#! /usr/bin/python3

import argparse

class _AddSourceAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, const=None,
 default=None, type=None, choices=None, required=False,
 help=None, metavar=None):
super(_AddSourceAction, self).__init__(
option_strings=option_strings, dest=dest, nargs=nargs,
const=const, default=default, type=type,
choices=choices, required=required, help=help,
metavar=metavar)

def __call__(self, parser, namespace, values, option_string=None):
for source in values:
namespace.sources.append({'name': source,
'frobnicate': namespace.frobnicate,
'massage_level': namespace.massage_level})

def process_files(dest, sources):
for source in sources:
frob = source["frobnicate"]
print("processing %s, %sfrobnication, massage level %d " %
  (source["name"],
  "default " if frob is None else "no " if frob == False else "",
  source["massage_level"]))
print("saving output to %s" % dest)

parser = argparse.ArgumentParser()
if hasattr(parser, "greedy_star"):
setattr(parser, "greedy_star", True)
parser.add_argument('destination')
parser.add_argument('--frobnicate', action='store_true', dest='frobnicate')
parser.add_argument('--no-frobnicate', action='store_false', dest='frobnicate')
parser.add_argument('--massage-level', type=int, default=5)
parser.add_argument('sources', metavar="source", nargs='*', 
action=_AddSourceAction)
parser.set_defaults(sources=[])
parser.set_defaults(frobnicate=None)
args = parser.parse_args()

process_files(args.destination, args.sources)

Without patch:

$ ./example.py output.txt --massage-level 4 input1.txt input2.txt 
--massage-level 5 input3.txt --frobnicate input4.txt input5.txt
example.py: error: unrecognized arguments: input1.txt input2.txt input3.txt 
input4.txt input5.txt

With patch:

$ PYTHONPATH=patch ./example.py output.txt --massage-level 4 input1.txt 
input2.txt --massage-level 5 input3.txt --frobnicate input4.txt input5.txt
processing input1.txt, default frobnication, massage level 4
processing input2.txt, default frobnication, massage level 4
processing input3.txt, default frobnication, massage level 5
processing input4.txt, frobnication, massage level 5
processing input5.txt, frobnication, massage level 5
saving output to output.txt

Note that I avoided passing greedy_star to constructor, to avoid error with 
unpatched argparse. Maybe that should be recommended way to enable this 
feature? N.b. I don't quite like this name 'greedy_star' seems to informal for 
me, but I didn't come up with anything better.

In this example frobnicate and massage_level apply to all following sources, 
but _AddSourceAction could be modified to reset their values, so they would 
only apply to first following source.

>> It's been a while since I worked on the intermixed patch, but as I recall 
>> the OP was happy with the fix.  Also I don't recall any talk about 'matching 
>> "optionals" with "positionals" with they relate to. Was that part of the 
>> discussion?
>> On Stackoverflow I have seen questions about pairing arguments, and answered 
>> some.  I don't recall the best answers.  The tough version is when some of 
>> the grouped inputs may be missing 

The use case above seems quite basic and common to me. I'm surprises that I 
actually didn't find any question on stackoverflow about this exact use case.

>> One question I frequently ask posters who want to parse complex inputs is: 
>> 'how are going to explain this to your users?'  'What kind of usage and help 
>> do you want see?'

I would (and I will, because I'm really working on a project which will do 
something like this) put all options which apply to source files into separate 
option group and add help string to this option group which explains that those 
options may be specified between source files and that they apply to all (or 
just first) following source files.

>> Your patch is incomplete, without documentation or tests.

Yeah, I can work on that if there is any chance that this will go anywhere. 
This is an effort from my side however, because I have quite slow mobile link 
at this moment. I tried to checkout python sources, but even with -depth 1 it 
took too long for my patience and I gave up after 'du 

[issue42973] argparse: mixing optional and positional arguments... not again

[paul j3]

paul j3  added the comment:

Your patch is incomplete, without documentation or tests.

Your example is incomplete.  That is _CustomAction?  What namespace does your 
patch produce.

It's been a while since I worked on the intermixed patch, but as I recall the 
OP was happy with the fix.  Also I don't recall any talk about 'matching 
"optionals" with "positionals" with they relate to. Was that part of the 
discussion?

On Stackoverflow I have seen questions about pairing arguments, and answered 
some.  I don't recall the best answers.  The tough version is when some of the 
grouped inputs may be missing 

In your example I'd suggest making '--tracks' a nargs=2 and 'append' option.  
'choices' would have to be replaced with a either a custom 'type', a custom 
Action, or post-parsing testing.

I haven't had time yet to test your patch.

One question I frequently ask posters who want to parse complex inputs is: 'how 
are going to explain this to your users?'  'What kind of usage and help do you 
want see?'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Raymond Hettinger]

Change by Raymond Hettinger :


--
nosy: +paul.j3, r.david.murray

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

Change by Tadek Kijkowski :


--
keywords: +patch
pull_requests: +23083
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/24259

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue42973] argparse: mixing optional and positional arguments... not again

[Tadek Kijkowski]

New submission from Tadek Kijkowski :

I have following use case for argparse:

./prog --tracks=80 image1 --tracks=40 image2 --tracks=80 image3 ...

I expected that the following parser would be able process that command line:

parser = argparse.ArgumentParser()
add = parser.add_argument
add('--tracks', choices=['40', '80'])
add('image', nargs='*', action=_CustomAction)
parser.parse_args()

But it stops with 'unrecognized arguments: image2' error. It turns out that 
there is no way to prepare parser which could handle this command line.

There was a long discussion about this issue in #14191. There were two 
approaches proposed:
- Allow 'image' action to trigger multiple times with additional encountered 
parameters.
- Parse all optionals first and all positionals in second pass.

The first approach was represented by a patch by user guilherme-pg. This patch 
was slightly criticized (undeservedly IMO) and pretty much ignored.

The discussion then focused on the second approach which ended up with 
introduction of `parse_intermixed_args` into codebase.

The problem with `parse_intermixed_args` is that it defeats the purpose of 
having intermixed arguments. When the arguments are effectively reordered, 
there is no way to match "optionals" with "positionals" with they relate to.

In my option the Guilherme's approach was the correct way to go, I would say it 
was elegant. The only complaint against that patch is that it unnecessarily 
modified _StoreAction class - instead of that, the way to go is to use "extend" 
or custom action.

I allowed myself to simplify the changes a bit for readability. The patch 
doesn't change default functionality, for the cost of changing public API by 
adding new argument to ArgumentParser constructor.

With the changes applied, I can parse my command line with this code:

parser = argparse.ArgumentParser(greedy_star=True)
add = parser.add_argument
add('--tracks', choices=['40', '80'])
add('image', nargs='*', action=_CustomAction)
parser.parse_args()

--
components: Library (Lib)
messages: 385301
nosy: monkeyman79
priority: normal
severity: normal
status: open
title: argparse: mixing optional and positional arguments... not again
type: enhancement
versions: Python 3.10

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com