Hi All, Following the discussions on "Power Assertions: Is it PEP-able?", I've drafted this PEP. Your comments are most welcome.
PEP: 9999 Title: Power Assertion Author: Noam Tenne <n...@10ne.org> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 24-Sep-2021 Abstract ======== This PEP introduces a language enhancement named "Power Assertion". The Power Assertion is inspired by a similar feature found in the `Groovy language`_ and is an extension to the core lib's ``assert`` keyword. When an assertion expression evaluates to ``False``, the output shows not only the failure, but also a breakdown of the evaluated expression from the inner part to the outer part. .. _Groovy language: http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#_power_assertions Motivation ========= Every test boils down to the binary statement "Is this true or false?", whether you use the built-in assert keyword or a more advanced assertion method provided by a testing framework. When an assertion fails, the output is binary too — "Expected x, but got y". There are helpful libraries like Hamcrest which give you a more verbose breakdown of the difference and answer the question "What exactly is the difference between x and y?". This is extremely helpful, but it still focuses on the difference between the values. Keep in mind that a given state is normally an outcome of a series of states, that is, one outcome is a result of multiple conditions and causes. This is where Power Assertion comes in. It allows us to better understand what led to the failure. The Community Wants This ------------------------ As mentioned in the abstract, this feature was borrowed from Groovy. It is a very popular feature within the Groovy community, also used by projects such as `Spock`_. On top of that, it is very much needed in the Python community as well: * `Power Assertion was explicitly requested`_ as a feature in the `Nimoy`_ testing framework * There's a `similar feature in pytest`_ And when `discussed in the python-ideas`_ mailing list, the responses were overall positive: * "This is cool." - Guido van Rossum * "I was actually thinking exactly the opposite: this would more useful in production than in testing." - 2QdxY4RzWzUUiLuE＠potatochowder.com * "What pytest does is awesome. I though about implementing it in the standard compiler since seen it the first time." - Serhiy Storchaka .. _Spock: https://spockframework.org/ .. _Power Assertion was explicitly requested: https://stackoverflow.com/a/58536986/198825 .. _similar feature in pytest: https://docs.pytest.org/en/latest/how-to/assert.html .. _discussed in the python-ideas: https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4BMPG5EOB3A2ELVEWQPYRENRXHM/ Rational ======== Code Example ------------ :: class SomeClass: def __init__(self): self.val = {'d': 'e'} def __str__(self): return str(self.val) sc = SomeClass() assert sc.val['d'] == 'f' This routine will result in the output: :: Assertion failed: sc.val['d'] == f | | | | e False | {'d': 'e'} Display ------- In the output above we can see the value of every part of the expression from left to right, mapped to their expression fragment with the pipe (``|``). The number of rows that are printed depend on the value of each fragment of the expression. If the value of a fragment is longer than the actual fragment (``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``) will be printed on a new line which will appear above. Values are appended to the same line until it overflows in length to horizontal position of the next fragment. This way of presentation is clearer and more human friendly than the output offered by pytest's solution. The information that’s displayed is dictated by the type. If the type is a constant value, it will be displayed as is. If the type implements `__str__`, then the return value of that will be displayed. Mechanics --------- Reference Implementation '''''''''''''''''''''''' The reference implementation uses AST manipulation because this is the only way that this level of involvement can be achieved by a third party library. It iterates over every subexpression in the assert statement. Subexpressions are parts of the expression separated by a lookup (``map[key]``), an attribute reference (``key.value``) or a binary comparison operator (``==``). It then builds an AST in the structure of a tree to maintain the order of the operations in the original code, and tracks the original code of the subexpression together with the AST code of the subexpression and the original indices. It then rewrites the AST of the original expression to call a specialised assertion function, which accepts the tree as a parameter. At runtime the expression is executed. If it fails, a rendering function is called to render the assertion message as per the example above. Actual Implementation ''''''''''''''''''''' To be discussed. In the python-ideas mailing list, Serhiy Storchaka suggests: It needs a support in the compiler. The condition expression should be compiled to keep all immediate results of subexpressions on the stack. If the final result is true, immediate results are dropped. If it is false, the second argument of assert is evaluated and its value together with all immediate results of the first expression, together with references to corresponding subexpressions (as strings, ranges or AST nodes) are passed to the special handler. That handler can be implemented in a third-party library, because formatting and outputting a report is a complex task. The default handler can just raise an AttributeError. Caveats ------- It is important to note that expressions with side effects are affected by this feature. This is because in order to display this information, we must store references to the instances and not just the values. Reference Implementation ======================== There's a `complete implementation`_ of this enhancement in the `Nimoy`_ testing framework. It uses AST manipulation to remap the expression to a `data structure`_ at compile time, so that it can then be `evaluated and printed`_ at runtime. .. _Nimoy: https://browncoat-ninjas.github.io/nimoy/ .. _complete implementation: https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta .. _data structure: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expression_transformer.py#L77 .. _evaluated and printed: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/power.py Copyright ========= This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:
pep-9999.rst
Description: Binary data
_______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TNMUISK7Y24E62B2AIBIPNPFRDVLTWIH/ Code of Conduct: http://python.org/psf/codeofconduct/
