So should I just scratch this and rewrite a PEP for an extensible assertion
mechanism?
On Fri, Sep 24, 2021, at 14:04, Noam Tenne wrote:
> Hi All,
>
> Following the discussions on "Power Assertions: Is it PEP-able?", I've
> drafted this PEP.
> Your comments are most welcome.
>
> PEP: 9999
> Title: Power Assertion
> Author: Noam Tenne <n...@10ne.org>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 24-Sep-2021
>
>
> Abstract
> ========
>
> This PEP introduces a language enhancement named "Power Assertion".
>
> The Power Assertion is inspired by a similar feature found in the
> `Groovy language`_ and is an extension to the core lib's ``assert``
> keyword.
>
> When an assertion expression evaluates to ``False``, the output shows
> not only the failure, but also a breakdown of the evaluated expression
> from the inner part to the outer part.
>
> .. _Groovy language:
> http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#_power_assertions
>
>
> Motivation
> =========
>
> Every test boils down to the binary statement "Is this true or false?",
> whether you use the built-in assert keyword or a more advanced
> assertion method provided by a testing framework.
> When an assertion fails, the output is binary too —
> "Expected x, but got y".
>
> There are helpful libraries like Hamcrest which give you a more
> verbose breakdown of the difference and answer the question
> "What exactly is the difference between x and y?".
> This is extremely helpful, but it still focuses on the difference
> between the values.
>
> Keep in mind that a given state is normally an outcome of a series of
> states, that is, one outcome is a result of multiple conditions and causes.
> This is where Power Assertion comes in. It allows us to better
> understand what led to the failure.
>
>
> The Community Wants This
> ------------------------
>
> As mentioned in the abstract, this feature was borrowed from Groovy.
> It is a very popular feature within the Groovy community, also used by
> projects such as `Spock`_.
>
> On top of that, it is very much needed in the Python community as well:
>
> * `Power Assertion was explicitly requested`_ as a feature in the
> `Nimoy`_ testing framework
> * There's a `similar feature in pytest`_
>
> And when `discussed in the python-ideas`_ mailing list, the responses
> were overall positive:
>
> * "This is cool." - Guido van Rossum
> * "I was actually thinking exactly the opposite: this would more
> useful in production than in testing."
> - 2QdxY4RzWzUUiLuE＠potatochowder.com
> * "What pytest does is awesome. I though about implementing it in the
> standard compiler since seen it the first time." - Serhiy Storchaka
>
> .. _Spock: https://spockframework.org/
> .. _Power Assertion was explicitly requested:
> https://stackoverflow.com/a/58536986/198825
> .. _similar feature in pytest:
> https://docs.pytest.org/en/latest/how-to/assert.html
> .. _discussed in the python-ideas:
> https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4BMPG5EOB3A2ELVEWQPYRENRXHM/
>
>
> Rational
> ========
>
> Code Example
> ------------
>
> ::
>
> class SomeClass:
> def __init__(self):
> self.val = {'d': 'e'}
>
> def __str__(self):
> return str(self.val)
>
> sc = SomeClass()
>
> assert sc.val['d'] == 'f'
>
> This routine will result in the output:
>
> ::
>
> Assertion failed:
>
> sc.val['d'] == f
> | | |
> | e False
> |
> {'d': 'e'}
>
>
> Display
> -------
>
> In the output above we can see the value of every part of the
> expression from left to right, mapped to their expression fragment
> with the pipe (``|``).
> The number of rows that are printed depend on the value of each
> fragment of the expression.
> If the value of a fragment is longer than the actual fragment
> (``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``)
> will be printed on a new line which will appear above.
> Values are appended to the same line until it overflows in length to
> horizontal position of the next fragment.
>
> This way of presentation is clearer and more human friendly than the
> output offered by pytest's solution.
>
> The information that’s displayed is dictated by the type.
> If the type is a constant value, it will be displayed as is.
> If the type implements `__str__`, then the return value of that will
> be displayed.
>
>
> Mechanics
> ---------
>
> Reference Implementation
> ''''''''''''''''''''''''
>
> The reference implementation uses AST manipulation because this is
> the only way that this level of involvement can be achieved by a
> third party library.
>
> It iterates over every subexpression in the assert statement.
> Subexpressions are parts of the expression separated by a lookup
> (``map[key]``), an attribute reference (``key.value``) or a binary
> comparison operator (``==``).
>
> It then builds an AST in the structure of a tree to maintain
> the order of the operations in the original code, and tracks the
> original code of the subexpression together with the AST code of the
> subexpression and the original indices.
>
> It then rewrites the AST of the original expression to call a
> specialised assertion function, which accepts the tree as a parameter.
>
> At runtime the expression is executed. If it fails, a rendering
> function is called to render the assertion message as per the example
> above.
>
>
> Actual Implementation
> '''''''''''''''''''''
>
> To be discussed.
>
> In the python-ideas mailing list, Serhiy Storchaka suggests:
>
> It needs a support in the compiler. The condition expression should be
> compiled to keep all immediate results of subexpressions on the stack.
> If the final result is true, immediate results are dropped. If it is
> false, the second argument of assert is evaluated and its value together
> with all immediate results of the first expression, together with
> references to corresponding subexpressions (as strings, ranges or AST
> nodes) are passed to the special handler. That handler can be
> implemented in a third-party library, because formatting and outputting
> a report is a complex task. The default handler can just raise an
> AttributeError.
>
>
> Caveats
> -------
>
> It is important to note that expressions with side effects are affected by
> this feature. This is because in order to display this information, we must
> store references to the instances and not just the values.
>
>
> Reference Implementation
> ========================
>
> There's a `complete implementation`_ of this enhancement in the
> `Nimoy`_ testing framework.
>
> It uses AST manipulation to remap the expression to a `data structure`_
> at compile time, so that it can then be `evaluated and printed`_ at runtime.
>
> .. _Nimoy: https://browncoat-ninjas.github.io/nimoy/
> .. _complete implementation:
> https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta
> .. _data structure:
> https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expression_transformer.py#L77
> .. _evaluated and printed:
> https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/power.py
>
>
> Copyright
> =========
>
> This document is placed in the public domain or under the
> CC0-1.0-Universal license, whichever is more permissive.
>
>
>
> ..
> Local Variables:
> mode: indented-text
> indent-tabs-mode: nil
> sentence-end-double-space: t
> fill-column: 70
> coding: utf-8
> End:
>
> *Attachments:*
> * pep-9999.rst_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/DO3ETJ4COXAFJKNYZP3IAGFX5N64CILD/
Code of Conduct: http://python.org/psf/codeofconduct/
