Re: [Python-ideas] PEP: Dict addition and subtraction

[Chris Barker via Python-ideas]

Just in case I'm not the only one that had a hard time finding the latest
version of this PEP, here it is in the PEPS Repo:

https://github.com/python/peps/blob/master/pep-0584.rst

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[David Mertz]

One of the nice things in wrapt is that Dumpleton lets you use the same
decorator for functions, regular methods, static methods, and class
methods.  Does yours handle that sort of "polymorphism"?

FWIW, thanks for the cool work with your libraries!

I don't think I will want the specific with-or-without parens feature,
since it feels too implicit.  Typing `@deco_factory()` really isn't too
much work for me to use the two characters extra.  But given that I feel
the option is an antipattern, I don't want to add core language features to
make the pattern easier.  Both you and Graham Dumpleton have found
workarounds to get that behavior when it is wanted, but I don't want it to
be "too easy."

FWIW... I think I'd be tempted to use a metaclass approach so that both the
class and instance are callable.  The class would be called with a single
function argument (i.e. a decorator), but if called with any other
signature it would manufacture a callable instance that was parameterized
by the initialization arguments (i.e. a decorator factory).  Actually, I
haven't looked at your actual code, maybe that's what you do.

Best, David...

On Tue, Mar 12, 2019 at 12:44 PM Sylvain MARIE  wrote:

> Thanks David,
>
>
>
> > I did write the book _Functional Programming in Python_, so I'm not
> entirely unfamiliar with function wrappers.
>
>
>
> Nice ! I did not realize ; good job here, congrats! ;)
>
>
>
> --
>
> I carefully read the documentation you pointed at,
> https://wrapt.readthedocs.io/en/latest/decorators.html#decorators-with-optional-arguments
>
> This is the example shown by the author:
>
>
>
> def with_optional_arguments(wrapped=None, myarg1=1, myarg2=2):
>
> if wrapped is None:
>
> return functools.partial(with_optional_arguments,
>
> myarg1=myarg1, myarg2=myarg2)
>
>
>
> @wrapt.decorator
>
> def wrapper(wrapped, instance, args, kwargs):
>
> return wrapped(*args, **kwargs)
>
>
>
> return wrapper(wrapped)
>
>
>
> As you can see:
>
>- the developer has to explicitly handle the no-parenthesis case (the
>first two lines of code).
>- And in the next lines of the doc you see his recommendations “For
>this to be used in this way, it is a requirement that the decorator
>arguments be supplied as keyword arguments. If using Python 3, the
>requirement to use keyword only arguments can again be enforced using the
>keyword only argument syntax.”
>- Finally, but this is just a comment: this is not “flat” mode but
>nested mode (the code returns a decorator that returns a function wrapper)
>
>
>
> So if I’m not misleading, the problem is not really solved. Or at least,
> not the way I would like the problem to be solved : it is solved here (a)
> only if the developer takes extra care and (b) reduces the way the
> decorator can be used (no positional args). This is precisely because I was
> frustrated by all these limitations that depend on the desired signature
> that I wrote decopatch. As a developer I do not want to care about which
> trick to use in which situation (mandatory args, optional args,
> var-positional args..). My decorators may change signature during the
> development cycle, and if I frequently had to change trick during
> development as I changed the signature - that is a bit tiring.
>
>
>
> --
>
> Concerning creation of signature-preserving wrappers: @wrapt.decorator is
> not signature preserving, I just checked it. You can check it with the
> following experiment:
>
>
>
> *def *dummy(wrapped):
> @wrapt.decorator
> *def *wrapper(wrapped, instance, args, kwargs):
> print(*"wrapper called"*)
> *return *wrapped(*args, **kwargs)
> *return *wrapper(wrapped)
>
>
>
> @dummy
> *def *function(a, b):
> *pass*
>
>
>
> If you call
>
>
>
> function(1)
>
>
>
> you will see that “wrapper called” is displayed before the TypeError is
> raised…
>
>
>
> The signature-preserving equivalent of @wrapt.decorator,
> @decorator.decorator, is the source of inspiration for makefun. You can
> see `makefun` as a generalization of the core of `decorator`.
>
>
>
> --
>
> > I'm not sure I ever want them (decopatch and makefun) independently in
> practice
>
>
>
> I totally understand.
>
> But some projects actually need makefun and not decopatch because their
> need is different: they just want to create a function dynamically. This is
> low-level tooling, really.
>
> So at least now there is a clear separation of concerns (and dedicated
> issues management/roadmap, which is also quite convenient. Not to mention
> readability !).
>
> To cover your concern: decopatch depends on makefun, so both come at the
> same time when you install decopatch, and decopatch by default relies on
> makefun when you use it in “double-flat” mode to create wrappers as
> explained here https://smarie.github.io/python-decopatch/#even-simpler
>
> --
>
>
>
> Thanks again for this discussion! It is challenging but it is necessary,
> to make sure I did 

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[Sylvain MARIE via Python-ideas]

Thanks David,

> I did write the book _Functional Programming in Python_, so I'm not entirely 
> unfamiliar with function wrappers.

Nice ! I did not realize ; good job here, congrats! ;)

--
I carefully read the documentation you pointed at, 
https://wrapt.readthedocs.io/en/latest/decorators.html#decorators-with-optional-arguments
This is the example shown by the author:

def with_optional_arguments(wrapped=None, myarg1=1, myarg2=2):
if wrapped is None:
return functools.partial(with_optional_arguments,
myarg1=myarg1, myarg2=myarg2)

@wrapt.decorator
def wrapper(wrapped, instance, args, kwargs):
return wrapped(*args, **kwargs)

return wrapper(wrapped)

As you can see:

  *   the developer has to explicitly handle the no-parenthesis case (the first 
two lines of code).
  *   And in the next lines of the doc you see his recommendations “For this to 
be used in this way, it is a requirement that the decorator arguments be 
supplied as keyword arguments. If using Python 3, the requirement to use 
keyword only arguments can again be enforced using the keyword only argument 
syntax.”
  *   Finally, but this is just a comment: this is not “flat” mode but nested 
mode (the code returns a decorator that returns a function wrapper)

So if I’m not misleading, the problem is not really solved. Or at least, not 
the way I would like the problem to be solved : it is solved here (a) only if 
the developer takes extra care and (b) reduces the way the decorator can be 
used (no positional args). This is precisely because I was frustrated by all 
these limitations that depend on the desired signature that I wrote decopatch. 
As a developer I do not want to care about which trick to use in which 
situation (mandatory args, optional args, var-positional args..). My decorators 
may change signature during the development cycle, and if I frequently had to 
change trick during development as I changed the signature - that is a bit 
tiring.

--
Concerning creation of signature-preserving wrappers: @wrapt.decorator is not 
signature preserving, I just checked it. You can check it with the following 
experiment:


def dummy(wrapped):
@wrapt.decorator
def wrapper(wrapped, instance, args, kwargs):
print("wrapper called")
return wrapped(*args, **kwargs)
return wrapper(wrapped)


@dummy
def function(a, b):
pass

If you call


function(1)

you will see that “wrapper called” is displayed before the TypeError is raised…

The signature-preserving equivalent of @wrapt.decorator, @decorator.decorator, 
is the source of inspiration for makefun. You can see `makefun` as a 
generalization of the core of `decorator`.

--
> I'm not sure I ever want them (decopatch and makefun) independently in 
> practice

I totally understand.
But some projects actually need makefun and not decopatch because their need is 
different: they just want to create a function dynamically. This is low-level 
tooling, really.
So at least now there is a clear separation of concerns (and dedicated issues 
management/roadmap, which is also quite convenient. Not to mention readability 
!).
To cover your concern: decopatch depends on makefun, so both come at the same 
time when you install decopatch, and decopatch by default relies on makefun 
when you use it in “double-flat” mode to create wrappers as explained here 
https://smarie.github.io/python-decopatch/#even-simpler
--

Thanks again for this discussion! It is challenging but it is necessary, to 
make sure I did not answer a non-existent need ;)
Kind regards

--
Sylvain

De : David Mertz 
Envoyé : mardi 12 mars 2019 15:30
À : Sylvain MARIE 
Cc : Steven D'Aprano ; python-ideas 

Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators


[External email: Use caution with links and attachments]




The documentation for wrapt mentions:

Decorators With Optional Arguments

Although opinion can be mixed about whether the pattern is a good one, if the 
decorator arguments all have default values, it is also possible to implement 
decorators which have optional arguments.
As Graham hints in his docs, I think repurposing decorator factories as 
decorators is an antipattern. Explicit is better than implicit.

While I *do* understands that what decotools and makefun do are technically 
independent, I'm not sure I ever want them independently in practice. I did 
write the book _Functional Programming in Python_, so I'm not entirely 
unfamiliar with function wrappers.
On Tue, Mar 12, 2019, 10:18 AM David Mertz 
mailto:me...@gnosis.cx>> wrote:
The wrapt module I linked to (not funtools.wraps) provides all the capabilities 
you mention since 2013. It allows mixed use of decorators as decorator 
factories. It has a flat style.

There are some minor API difference between your libraries and wrapt, but the 
concept is very similar. Since yours is something new, I imagine you perceive 
some win over what 

Re: [Python-ideas] Suggestions: dict.flow_update and dict.__add__

[Christopher Barker]

On Fri, Mar 8, 2019 at 1:52 AM Jonathan Fine  wrote:

> I've just learnt something new. Look at
>
> >>> from operator import iadd
> >>> lst = [1, 2, 3]
> >>> iadd(lst, 'hi')
> [1, 2, 3, 'h', 'i']
> >>> lst
> [1, 2, 3, 'h', 'i']
>
> This shows that the proposals dict.flow_update and dict.__iadd__ are
> basically the same. (I think this is quite important for understanding
> the attraction of fluent programming. We ALREADY like and use it, in
> the form of augmented assignment of mutables.)
>

well, no -- the fact that __iadd__ acts like it does is essentially an
accident of implementation, and calling __iadd__ directly is frowned upon
(and I'm not totally sure if it is guaranteed to keep working that way by
the language spec). And, in fact, it DOESN'T act like flow_merge method --
as it both mutates the original object, and returns itself -- which I think
is a no-no in fluent programming, yes? (certainly in functional programming)

In [10]: list1 = [1,2,3]

In [11]: list2 = [4,5,6]

In [12]: list3 = list1.__iadd__(list2)

In [13]: list3
Out[13]: [1, 2, 3, 4, 5, 6]

In [14]: list1
Out[14]: [1, 2, 3, 4, 5, 6]

In [15]: list1 is list3
Out[15]: True


This also shows that
>combined = defaults.copy()
>combined.update(options)
> could, if the proposal is accepted, be written as
>defaults.copy().__iadd__(options)
>

did you mean:

combined = defaults.copy().__iadd__(options)

because the way you wrote it, you are making a copy, mutating it, and then
throwing it away...

in which case, yes it could, but it would not be recommended, and I can't
see the advantage of it over:

combined = defaults + options

or even, if you REALLY want to use __ methods:

combined = defaults.__add__(options)


In [17]: list3 = list1.__add__(list2)

In [18]: list1
Out[18]: [1, 2, 3]

In [19]: list3
Out[19]: [1, 2, 3, 4, 5, 6]


> I got the idea from the withdrawn PEP (thank you, Nick Coghlan, for
> writing it):
> PEP 577 -- Augmented Assignment Expressions
> https://www.python.org/dev/peps/pep-0577/
>

Interestingly (to me) it was withdrawn for different reasons than what I
would think -- mutating and assigning at once is dangerous.

-CHB


-- 
Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Dict joining using + and +=

[Christopher Barker]

> This question is probably on its own a valid argument against the
> proposal. When it comes to dicts (and not Mappings in general) {**d1,
> **d2} or d.update() already have clearly-defined semantics.


Actually, in my mind, this is an argument for an operator (or method) —
besides being obtuse, the {**d1,**d2} syntax only creates actual dicts. If
we had an operator defined for mappings in general, it would be easier to
duck type dicts.

I think this is pretty compelling, actually. And also an argument for aging
the operation return the type it was invoked on, rather than always a dict.

I can’t find the latest draft of the PEP, so I’m not sure if this is
discussed there. But it should be.

-CHB
-- 
Christopher Barker, PhD

Python Language Consulting
  - Teaching
  - Scientific Software Development
  - Desktop GUI and Web Development
  - wxPython, numpy, scipy, Cython
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Preallocated tuples and dicts for function calls

[Jeroen Demeyer]

On 2019-03-08 22:16, Martin Bammer wrote:

Hi,

what about the idea that the interpreter preallocates and preinitializes
the

tuples and dicts for function calls where possible when loading a module?


The basic premise here is wrong: function calls using the METH_FASTCALL 
convention don't need to allocate any temporary tuple/dict for function 
calls. Of course, not all function calls use METH_FASTCALL, but most of 
them where performance matters do.

___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Left arrow and right arrow operators

[Nick Timkovich]

In general, there is lots of code out in the wild that can't be updated for
whatever reason, e.g. the person that knows Python left and it needs to
continue to work. Weak argument, but cost-benefit I think it comes out
ahead. In your example there isn't a reason I can tell why swapping the
operands isn't what should be done as Calvin mentioned. The onus is on you
to positively demonstrate you require both directions, not him to
negatively demonstrate it's never required.

I suggest you confine your proposal to `->` only, as it's currently illegal
syntax. You would also want the reflected `__r*__` equivalent of
`__arrow__` or `__rarrow__` (`__rrarrow__` if you also need the
left-arrow...)

Perhaps broadening the use of it, functions may be able to use it as a pipe
operator, e.g. Elixir:
https://elixir-lang.org/getting-started/enumerables-and-streams.html#the-pipe-operator

On Mon, Mar 11, 2019 at 2:58 PM francismb  wrote:

> Hi Greg,
>
> On 3/9/19 1:42 AM, Greg Ewing wrote:
> > Do you really want
> > to tell them that all their code is now wrong?
> Of course not, at least not so promptly. But, would it be still a
> problem if the update to a new version (let say from 3.X to next(3.X))
> is done through some kind of updater/re-writer/evolver. In that case the
> evolver could just add the blanks. What do you think ? Could it work?
>
> Thanks in advance!
> --francis
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[David Mertz]

The documentation for wrapt mentions:

Decorators With Optional Arguments


Although opinion can be mixed about whether the pattern is a good one, if
the decorator arguments all have default values, it is also possible to
implement decorators which have optional arguments.
As Graham hints in his docs, I think repurposing decorator factories as
decorators is an antipattern. Explicit is better than implicit.

While I *do* understands that what decotools and makefun do are technically
independent, I'm not sure I ever want them independently in practice. I did
write the book _Functional Programming in Python_, so I'm not entirely
unfamiliar with function wrappers.

On Tue, Mar 12, 2019, 10:18 AM David Mertz  wrote:

> The wrapt module I linked to (not funtools.wraps) provides all the
> capabilities you mention since 2013. It allows mixed use of decorators as
> decorator factories. It has a flat style.
>
> There are some minor API difference between your libraries and wrapt, but
> the concept is very similar. Since yours is something new, I imagine you
> perceive some win over what wrapt does.
>
> On Tue, Mar 12, 2019, 9:52 AM Sylvain MARIE  wrote:
>
>> David, Steven,
>>
>> Thanks for your interest !
>>
>> As you probably know, decorators and function wrappers are *completely
>> different concepts*. A decorator can directly return the decorated function
>> (or class), it does not have to return a wrapper. Even more, it can
>> entirely replace the decorated item with something else (not even a
>> function or class!). Try it: it is possible to write a decorator to replace
>> a function with an integer, even though it is probably not quite useful :)
>>
>> `decopatch` helps you write decorators, whatever they are. It "just"
>> solves the annoying issue of having to handle the no-parenthesis and
>> with-parenthesis calls. In addition as a 'goodie', it proposes two
>> development styles: *nested* (you have to return a function) and *flat*
>> (you directly write what will happen when the decorator is applied to
>> something).
>> --
>> Now about creating signature-preserving function wrappers (in a
>> decorator, or outside a decorator - again, that's not related). That use
>> case is supposed to be covered by functools.wrapt. Unfortunately as
>> explained here
>> https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697
>> this is not the case because with functools.wrapt:
>>  - the wrapper code will execute even when the provided arguments are
>> invalid.
>>  - the wrapper code cannot easily access an argument using its name, from
>> the received *args, **kwargs. Indeed one would have to handle all cases
>> (positional, keyword, default) and therefore to use something like
>> Signature.bind().
>>
>> For this reason I proposed a replacement in `makefun`:
>> https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers
>> --
>> Now bridging the gap. Of course a very interesting use cases for
>> decorators is to create decorators that create a signature-preserving
>> wrapper. It is possible to combine decopatch and makefun for this:
>> https://smarie.github.io/python-decopatch/#3-creating-function-wrappers .
>> Decopatch even proposes a "double-flat" development style where you
>> directly write the wrapper body, as explained in the doc.
>>
>> Did I answer your questions ?
>> Thanks again for the quick feedback !
>> Best,
>>
>> Sylvain
>>
>> -Message d'origine-
>> De : Python-ideas 
>> De la part de Steven D'Aprano
>> Envoyé : mardi 12 mars 2019 12:30
>> À : python-ideas@python.org
>> Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators
>>
>> [External email: Use caution with links and attachments]
>>
>> 
>>
>>
>>
>> On Tue, Mar 12, 2019 at 09:36:41AM +, Sylvain MARIE via Python-ideas
>> wrote:
>>
>> > I therefore proposed
>> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma
>> > rie.github.io%2Fpython-makefun%2Fdata=02%7C01%7Csylvain.marie%40s
>> > e.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae
>> > 68fef%7C0%7C0%7C636879872385158085sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2
>> > Fekk35bnYGvmEFJyCXaLDyLm9I%3Dreserved=0 . In particular it
>> > provides an equivalent of `@functools.wraps` that is truly
>> > signature-preserving
>>
>> Tell us more about that please. I'm very interested in getting decorators
>> preserve the original signature.
>>
>>
>> --
>> Steven
>> ___
>> Python-ideas mailing list
>> Python-ideas@python.org
>>
>> 

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[David Mertz]

The wrapt module I linked to (not funtools.wraps) provides all the
capabilities you mention since 2013. It allows mixed use of decorators as
decorator factories. It has a flat style.

There are some minor API difference between your libraries and wrapt, but
the concept is very similar. Since yours is something new, I imagine you
perceive some win over what wrapt does.

On Tue, Mar 12, 2019, 9:52 AM Sylvain MARIE  wrote:

> David, Steven,
>
> Thanks for your interest !
>
> As you probably know, decorators and function wrappers are *completely
> different concepts*. A decorator can directly return the decorated function
> (or class), it does not have to return a wrapper. Even more, it can
> entirely replace the decorated item with something else (not even a
> function or class!). Try it: it is possible to write a decorator to replace
> a function with an integer, even though it is probably not quite useful :)
>
> `decopatch` helps you write decorators, whatever they are. It "just"
> solves the annoying issue of having to handle the no-parenthesis and
> with-parenthesis calls. In addition as a 'goodie', it proposes two
> development styles: *nested* (you have to return a function) and *flat*
> (you directly write what will happen when the decorator is applied to
> something).
> --
> Now about creating signature-preserving function wrappers (in a decorator,
> or outside a decorator - again, that's not related). That use case is
> supposed to be covered by functools.wrapt. Unfortunately as explained here
> https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697
> this is not the case because with functools.wrapt:
>  - the wrapper code will execute even when the provided arguments are
> invalid.
>  - the wrapper code cannot easily access an argument using its name, from
> the received *args, **kwargs. Indeed one would have to handle all cases
> (positional, keyword, default) and therefore to use something like
> Signature.bind().
>
> For this reason I proposed a replacement in `makefun`:
> https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers
> --
> Now bridging the gap. Of course a very interesting use cases for
> decorators is to create decorators that create a signature-preserving
> wrapper. It is possible to combine decopatch and makefun for this:
> https://smarie.github.io/python-decopatch/#3-creating-function-wrappers .
> Decopatch even proposes a "double-flat" development style where you
> directly write the wrapper body, as explained in the doc.
>
> Did I answer your questions ?
> Thanks again for the quick feedback !
> Best,
>
> Sylvain
>
> -Message d'origine-
> De : Python-ideas 
> De la part de Steven D'Aprano
> Envoyé : mardi 12 mars 2019 12:30
> À : python-ideas@python.org
> Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators
>
> [External email: Use caution with links and attachments]
>
> 
>
>
>
> On Tue, Mar 12, 2019 at 09:36:41AM +, Sylvain MARIE via Python-ideas
> wrote:
>
> > I therefore proposed
> > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma
> > rie.github.io%2Fpython-makefun%2Fdata=02%7C01%7Csylvain.marie%40s
> > e.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae
> > 68fef%7C0%7C0%7C636879872385158085sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2
> > Fekk35bnYGvmEFJyCXaLDyLm9I%3Dreserved=0 . In particular it
> > provides an equivalent of `@functools.wraps` that is truly
> > signature-preserving
>
> Tell us more about that please. I'm very interested in getting decorators
> preserve the original signature.
>
>
> --
> Steven
> ___
> Python-ideas mailing list
> Python-ideas@python.org
>
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=XcYfEginmDF7kIpGGA0XxDZKpUn9e4p2zPFk7UAruYg%3Dreserved=0
> Code of Conduct:
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=20ZrtVQZbpQ54c96veSXIOfEK7rKy0ggj0omTZg3ri8%3Dreserved=0
>
> __
> This email has been scanned by the Symantec Email Security.cloud service.
> __
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Add method unittest.TestProgram to add custom arguments

[Rémi Lapeyre]

Hi everybody,

I would like to add a new method to unittest.TestProgram that would let
the user add its own arguments to TestProgram so its behavior can be
changed from the command line, and make them accessible from a TestCase.

This would allow for more customization when running the tests.

Currently there is no way to do this, so users either have to rely on
environment variables or parse sys.argv themselves like what is proposed
in https://stackoverflow.com/a/46061210.

This approach have several disadvantages:
  - it's more work to check and parse the argument
  - it's not discoverable from the help
  - everyone have its own way of doing this

Here's two cases where this would be useful for me:
  - in one project, I have an option to run tests in "record mode". In
  this mode, all tests are executed but the results are saved to files
  instead of checking their validity. In subsequent runs, results are
  compared to those files to make sure they are correct:

def _test_payload(obj, filename):
import json, os

filename = f"test/results/{filename}"
if "RECORD_PAYLOADS" in os.environ:
with open(filename, "w") as f:
json.dump(obj, f, indent=2)
else:
with open(filename) as f:
assert obj == json.load(f)

  This let us updating tests very quickly and just checking they are
  correct by looking at the diff. This is very useful for testing an
  API with many endpoints.

  As you can see, we use an environment variable to change TestCase
  behavior but it is awkward to run tests as
  `RECORD_PAYLOADS=1 python test_api.py -v` instead of a more
  straightforward `python test_api.py -v --record-payloads`.

  - in https://bugs.python.org/issue18765 (unittest needs a way to launch 
pdb.post_mortem or other debug hooks)
  Gregory P. Smith and Michael Foord propose to make TestCase call
  pdb.post_mortem() on failure. This also requires a way to change the
  behavior as this would not be wanted in CI.

Just subclassing TestProgram would not be sufficient as the result of the 
parsing
would not be accessible from the TestCase.

One argument against making such change is that it would require adding
a new attribute to TestCase which would not be backward compatible and
which could break existing code but I think this can be manageable if we
choose an appropriate name for it.

I attached an implementation of this feature (also available at 
https://github.com/remilapeyre/cpython/tree/unittest-custom-arguments)
and look forward for your comments.

What do you think about this?

---
 Lib/unittest/case.py|   7 +-
 Lib/unittest/loader.py  |  73 +
 Lib/unittest/main.py|  49 +-
 Lib/unittest/test/test_discovery.py | 225 
 Lib/unittest/test/test_program.py   |  34 -
 5 files changed, 331 insertions(+), 57 deletions(-)

diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py
index a157ae8a14..5698e3a640 100644
--- a/Lib/unittest/case.py
+++ b/Lib/unittest/case.py
@@ -1,6 +1,7 @@
 """Test case implementation"""

 import sys
+import argparse
 import functools
 import difflib
 import logging
@@ -416,7 +417,7 @@ class TestCase(object):

 _class_cleanups = []

-def __init__(self, methodName='runTest'):
+def __init__(self, methodName='runTest', command_line_arguments=None):
 """Create an instance of the class that will use the named test
method when executed. Raises a ValueError if the instance does
not have a method with the specified name.
@@ -436,6 +437,10 @@ class TestCase(object):
 self._testMethodDoc = testMethod.__doc__
 self._cleanups = []
 self._subtest = None
+if command_line_arguments is None:
+self.command_line_arguments = argparse.Namespace()
+else:
+self.command_line_arguments = command_line_arguments

 # Map types to custom assertEqual functions that will compare
 # instances of said type in more detail to generate a more useful
diff --git a/Lib/unittest/loader.py b/Lib/unittest/loader.py
index ba7105e1ad..d5f0a97213 100644
--- a/Lib/unittest/loader.py
+++ b/Lib/unittest/loader.py
@@ -5,6 +5,7 @@ import re
 import sys
 import traceback
 import types
+import itertools
 import functools
 import warnings

@@ -81,7 +82,7 @@ class TestLoader(object):
 # avoid infinite re-entrancy.
 self._loading_packages = set()

-def loadTestsFromTestCase(self, testCaseClass):
+def loadTestsFromTestCase(self, testCaseClass, 
command_line_arguments=None):
 """Return a suite of all test cases contained in testCaseClass"""
 if issubclass(testCaseClass, suite.TestSuite):
 raise TypeError("Test cases should not be derived from "
@@ -90,12 +91,21 @@ class TestLoader(object):
 testCaseNames = self.getTestCaseNames(testCaseClass)
 if not testCaseNames and 

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[Sylvain MARIE via Python-ideas]

David
I realize that you were pointing at the 'wrapt'  library not the 
functools.wrapt library.
>From what I have tried with that library, it does not solve the issue solved 
>with decopatch.

Sylvain 

-Message d'origine-
De : Sylvain MARIE 
Envoyé : mardi 12 mars 2019 14:53
À : Steven D'Aprano ; python-ideas@python.org; David Mertz 

Objet : RE: [Python-ideas] Problems (and solutions?) in writing decorators

David, Steven,

Thanks for your interest !

As you probably know, decorators and function wrappers are *completely 
different concepts*. A decorator can directly return the decorated function (or 
class), it does not have to return a wrapper. Even more, it can entirely 
replace the decorated item with something else (not even a function or class!). 
Try it: it is possible to write a decorator to replace a function with an 
integer, even though it is probably not quite useful :)

`decopatch` helps you write decorators, whatever they are. It "just" solves the 
annoying issue of having to handle the no-parenthesis and with-parenthesis 
calls. In addition as a 'goodie', it proposes two development styles: *nested* 
(you have to return a function) and *flat* (you directly write what will happen 
when the decorator is applied to something).
--
Now about creating signature-preserving function wrappers (in a decorator, or 
outside a decorator - again, that's not related). That use case is supposed to 
be covered by functools.wrapt. Unfortunately as explained here 
https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697
 this is not the case because with functools.wrapt:
 - the wrapper code will execute even when the provided arguments are invalid.
 - the wrapper code cannot easily access an argument using its name, from the 
received *args, **kwargs. Indeed one would have to handle all cases 
(positional, keyword, default) and therefore to use something like 
Signature.bind().

For this reason I proposed a replacement in `makefun`: 
https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers
--
Now bridging the gap. Of course a very interesting use cases for decorators is 
to create decorators that create a signature-preserving wrapper. It is possible 
to combine decopatch and makefun for this: 
https://smarie.github.io/python-decopatch/#3-creating-function-wrappers .
Decopatch even proposes a "double-flat" development style where you directly 
write the wrapper body, as explained in the doc.

Did I answer your questions ?
Thanks again for the quick feedback !
Best,

Sylvain 

-Message d'origine-
De : Python-ideas  De la 
part de Steven D'Aprano Envoyé : mardi 12 mars 2019 12:30 À : 
python-ideas@python.org Objet : Re: [Python-ideas] Problems (and solutions?) in 
writing decorators

[External email: Use caution with links and attachments]





On Tue, Mar 12, 2019 at 09:36:41AM +, Sylvain MARIE via Python-ideas wrote:

> I therefore proposed
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma
> rie.github.io%2Fpython-makefun%2Fdata=02%7C01%7Csylvain.marie%40s
> e.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae
> 68fef%7C0%7C0%7C636879872385158085sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2
> Fekk35bnYGvmEFJyCXaLDyLm9I%3Dreserved=0 . In particular it 
> provides an equivalent of `@functools.wraps` that is truly 
> signature-preserving

Tell us more about that please. I'm very interested in getting decorators 
preserve the original signature.


--
Steven
___
Python-ideas mailing list
Python-ideas@python.org
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=XcYfEginmDF7kIpGGA0XxDZKpUn9e4p2zPFk7UAruYg%3Dreserved=0
Code of Conduct: 
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=20ZrtVQZbpQ54c96veSXIOfEK7rKy0ggj0omTZg3ri8%3Dreserved=0

__
This email has been scanned by the Symantec Email Security.cloud service.
__
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[Sylvain MARIE via Python-ideas]

David, Steven,

Thanks for your interest !

As you probably know, decorators and function wrappers are *completely 
different concepts*. A decorator can directly return the decorated function (or 
class), it does not have to return a wrapper. Even more, it can entirely 
replace the decorated item with something else (not even a function or class!). 
Try it: it is possible to write a decorator to replace a function with an 
integer, even though it is probably not quite useful :)

`decopatch` helps you write decorators, whatever they are. It "just" solves the 
annoying issue of having to handle the no-parenthesis and with-parenthesis 
calls. In addition as a 'goodie', it proposes two development styles: *nested* 
(you have to return a function) and *flat* (you directly write what will happen 
when the decorator is applied to something).
--
Now about creating signature-preserving function wrappers (in a decorator, or 
outside a decorator - again, that's not related). That use case is supposed to 
be covered by functools.wrapt. Unfortunately as explained here 
https://stackoverflow.com/questions/308999/what-does-functools-wraps-do/55102697#55102697
 this is not the case because with functools.wrapt:
 - the wrapper code will execute even when the provided arguments are invalid.
 - the wrapper code cannot easily access an argument using its name, from the 
received *args, **kwargs. Indeed one would have to handle all cases 
(positional, keyword, default) and therefore to use something like 
Signature.bind().

For this reason I proposed a replacement in `makefun`: 
https://smarie.github.io/python-makefun/#signature-preserving-function-wrappers 
--
Now bridging the gap. Of course a very interesting use cases for decorators is 
to create decorators that create a signature-preserving wrapper. It is possible 
to combine decopatch and makefun for this: 
https://smarie.github.io/python-decopatch/#3-creating-function-wrappers .
Decopatch even proposes a "double-flat" development style where you directly 
write the wrapper body, as explained in the doc.

Did I answer your questions ?
Thanks again for the quick feedback !
Best,

Sylvain 

-Message d'origine-
De : Python-ideas  De la 
part de Steven D'Aprano
Envoyé : mardi 12 mars 2019 12:30
À : python-ideas@python.org
Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators

[External email: Use caution with links and attachments]





On Tue, Mar 12, 2019 at 09:36:41AM +, Sylvain MARIE via Python-ideas wrote:

> I therefore proposed
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsma
> rie.github.io%2Fpython-makefun%2Fdata=02%7C01%7Csylvain.marie%40s
> e.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae
> 68fef%7C0%7C0%7C636879872385158085sdata=nB9p9V%2BJ7gk%2Fsc%2BA5%2
> Fekk35bnYGvmEFJyCXaLDyLm9I%3Dreserved=0 . In particular it 
> provides an equivalent of `@functools.wraps` that is truly 
> signature-preserving

Tell us more about that please. I'm very interested in getting decorators 
preserve the original signature.


--
Steven
___
Python-ideas mailing list
Python-ideas@python.org
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=XcYfEginmDF7kIpGGA0XxDZKpUn9e4p2zPFk7UAruYg%3Dreserved=0
Code of Conduct: 
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C579232e7e10e475314c708d6a6de9d23%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636879872385158085sdata=20ZrtVQZbpQ54c96veSXIOfEK7rKy0ggj0omTZg3ri8%3Dreserved=0

__
This email has been scanned by the Symantec Email Security.cloud service.
__
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[Steven D'Aprano]

On Tue, Mar 12, 2019 at 09:36:41AM +, Sylvain MARIE via Python-ideas wrote:

> I therefore proposed 
> https://smarie.github.io/python-makefun/ . In particular it provides 
> an equivalent of `@functools.wraps` that is truly signature-preserving

Tell us more about that please. I'm very interested in getting 
decorators preserve the original signature.


-- 
Steven
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[David Mertz]

What advantage do you perceive decopatch to have over wrapt? (
https://github.com/GrahamDumpleton/wrapt)

On Tue, Mar 12, 2019, 5:37 AM Sylvain MARIE via Python-ideas <
python-ideas@python.org> wrote:

> Dear python enthusiasts,
>
> Writing python decorators is indeed quite a tideous process, in particular
> when you wish to add arguments, and in particular in two cases : all
> optional arguments, and one mandatory argument. Indeed in these two cases
> there is a need to disambiguate between no-parenthesis and with-parenthesis
> usage.
>
> After having struggled with this pattern for two years in various open
> source and industrial projects, I ended up writing a library to hopefully
> solve this once and for all: https://smarie.github.io/python-decopatch/ .
> It is extensively tested (203 tests) against many combinations of
> signature/calls.
> I would gladly appreciate any feedback !
>
> Please note that there is a "PEP proposal draft" in the project page
> because I belive that the best a library can do will always be a poor
> workaround, where the interpreter or stdlib could really fix it properly.
> Sorry for not providing too much implementation details in that page, my
> knowledge of the python interpreter is unfortunately quite limited.
>
> --
>
> Finally there is an additional topic around decorators : people tend to
> believe that decorators and function wrappers are the same, which is
> absolutely not the case. I used the famous `decorator` lib in many projects
> but I was not satisfied because it was solving both issues at the same
> time, maintaining the confusion. I therefore proposed
> https://smarie.github.io/python-makefun/ . In particular it provides an
> equivalent of `@functools.wraps` that is truly signature-preserving (based
> on the same recipe than `decorator`).
> Once again, any feedback would be gladly appreciated !
>
> Kind regards
>
> Sylvain
>
> -Message d'origine-
> De : Python-ideas 
> De la part de Greg Ewing
> Envoyé : vendredi 26 octobre 2018 00:04
> À : python-ideas 
> Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators
>
> [External email: Use caution with links and attachments]
>
> 
>
>
>
> Jonathan Fine wrote:
> > I also find writing decorators a bit
> > hard. It seems to be something I have to learn anew each time I do it.
> > Particularly for the pattern
> >
> > @deco(arg1, arg2) def fn(arg3, arg4):
>  > # function body
> >
> > Perhaps doing something with partial might help here. Anyone here
> > interested in exploring this?
> >
>
> I can't think of a way that partial would help. But would you find it
> easier if you could do something like this?
>
>  class deco(Decorator):
>
>  def __init__(self, arg1, arg2):
>  self.arg1 = arg1
>  self.arg2 = arg2
>
>  def invoke(self, func, arg3, arg4):
>  # function body
>
> Implementation:
>
>  class Decorator:
>
>  def __call__(self, func):
>  self._wrapped_function = func
>  return self._wrapper
>
>  def _wrapper(self, *args, **kwds):
>  return self.invoke(self._wrapped_function, *args, **kwds)
>
> --
> Greg
> ___
> Python-ideas mailing list
> Python-ideas@python.org
>
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C295cecd58fc3461f42c108d63ac5e03e%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C63676101605889sdata=Z8OET1CZZWnmN5czi0rZ1X57%2FDd4a9IDbbSujsDNWzk%3Dreserved=0
> Code of Conduct:
> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C295cecd58fc3461f42c108d63ac5e03e%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C63676101615894sdata=u0hvM5cR%2BR1Vni%2BV48WoNF%2FpriCaOG5%2BFAXayaTGsYY%3Dreserved=0
>
> __
> This email has been scanned by the Symantec Email Security.cloud service.
> __
> ___
> Python-ideas mailing list
> Python-ideas@python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

Re: [Python-ideas] Problems (and solutions?) in writing decorators

[Sylvain MARIE via Python-ideas]

Dear python enthusiasts,

Writing python decorators is indeed quite a tideous process, in particular when 
you wish to add arguments, and in particular in two cases : all optional 
arguments, and one mandatory argument. Indeed in these two cases there is a 
need to disambiguate between no-parenthesis and with-parenthesis usage.

After having struggled with this pattern for two years in various open source 
and industrial projects, I ended up writing a library to hopefully solve this 
once and for all: https://smarie.github.io/python-decopatch/ . It is 
extensively tested (203 tests) against many combinations of signature/calls.
I would gladly appreciate any feedback !

Please note that there is a "PEP proposal draft" in the project page because I 
belive that the best a library can do will always be a poor workaround, where 
the interpreter or stdlib could really fix it properly. 
Sorry for not providing too much implementation details in that page, my 
knowledge of the python interpreter is unfortunately quite limited.

--

Finally there is an additional topic around decorators : people tend to believe 
that decorators and function wrappers are the same, which is absolutely not the 
case. I used the famous `decorator` lib in many projects but I was not 
satisfied because it was solving both issues at the same time, maintaining the 
confusion. I therefore proposed https://smarie.github.io/python-makefun/ . In 
particular it provides an equivalent of `@functools.wraps` that is truly 
signature-preserving (based on the same recipe than `decorator`).
Once again, any feedback would be gladly appreciated ! 

Kind regards

Sylvain

-Message d'origine-
De : Python-ideas  De la 
part de Greg Ewing
Envoyé : vendredi 26 octobre 2018 00:04
À : python-ideas 
Objet : Re: [Python-ideas] Problems (and solutions?) in writing decorators

[External email: Use caution with links and attachments]





Jonathan Fine wrote:
> I also find writing decorators a bit
> hard. It seems to be something I have to learn anew each time I do it.
> Particularly for the pattern
>
> @deco(arg1, arg2) def fn(arg3, arg4):
 > # function body
>
> Perhaps doing something with partial might help here. Anyone here 
> interested in exploring this?
>

I can't think of a way that partial would help. But would you find it easier if 
you could do something like this?

 class deco(Decorator):

 def __init__(self, arg1, arg2):
 self.arg1 = arg1
 self.arg2 = arg2

 def invoke(self, func, arg3, arg4):
 # function body

Implementation:

 class Decorator:

 def __call__(self, func):
 self._wrapped_function = func
 return self._wrapper

 def _wrapper(self, *args, **kwds):
 return self.invoke(self._wrapped_function, *args, **kwds)

--
Greg
___
Python-ideas mailing list
Python-ideas@python.org
https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideasdata=02%7C01%7Csylvain.marie%40se.com%7C295cecd58fc3461f42c108d63ac5e03e%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C63676101605889sdata=Z8OET1CZZWnmN5czi0rZ1X57%2FDd4a9IDbbSujsDNWzk%3Dreserved=0
Code of Conduct: 
https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2Fdata=02%7C01%7Csylvain.marie%40se.com%7C295cecd58fc3461f42c108d63ac5e03e%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C63676101615894sdata=u0hvM5cR%2BR1Vni%2BV48WoNF%2FpriCaOG5%2BFAXayaTGsYY%3Dreserved=0

__
This email has been scanned by the Symantec Email Security.cloud service.
__
___
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/