[Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules
I'm in the process of trying to disentangle http://bugs.python.org/issue27137 which points out some of the behavioural differences that arise when falling back from the original C implementation of functools.partial to the pure Python emulation that uses a closure. That issue was opened due to a few things that work with the C implementation that fail with the Python implementation: - the C version can be pickled (and hence used with multiprocessing) - the C version can be subclassed - the C version can be used in "isinstance" checks - the C version behaves as a static method, the Python version as a normal instance method While I'm planning to accept the patch that converts the pure Python version to a full class that matches the semantics of the C version in these areas as well as in its core behaviour, that last case is one where the pure Python version merely exhibits different behaviour from the C version, rather than failing outright. Given that the issues that arose in this case weren't at all obvious up front, what do folks think of the idea of updating PEP 399 to explicitly prohibit class/function mismatches between accelerator modules and their pure Python counterparts? The rationale for making such a change is that when it comes to true drop-in API compatibility, we have reasonable evidence that "they're both callables" isn't sufficient once the complexities of real world applications enter the picture. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/09/2016 09:50 AM, Nick Coghlan wrote: > Given that the issues that arose in this case weren't at all obvious > up front, what do folks think of the idea of updating PEP 399 to > explicitly prohibit class/function mismatches between accelerator > modules and their pure Python counterparts? > > The rationale for making such a change is that when it comes to true > drop-in API compatibility, we have reasonable evidence that "they're > both callables" isn't sufficient once the complexities of real world > applications enter the picture. +1. Might need some clarification: - - "C functions can fall back to ". - - "C classes must fall back to Python classes". Outlining the constraints in the PEP (identical pickling semantics, sublcassability, etc.) might be important, too. Tres. - -- === Tres Seaver +1 540-429-0999 [email protected] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJXgRCpAAoJEPKpaDSJE9HYNhwP+gN1xGSZlEvrxz5SGrqTneUx 5WDh2oUJzlTFHDrbSMTeGcpoYviWPLWFy0Hw7PBgRhrlA/TS7WA5/4ABde+2Zs0w PN5AaaXZrkGAHvcQZkBzcEY9ITSpeb+GSmLG4Eih30UAuPFnM3M1UYSjEGjVZV23 tYeDTcmORNaBcQDPG8HiidfOArBKTcz8Jd1IimFrYEOFGSsk6DxPWffJ3EkR6qFj FwsktPDZT113AiztrkLt1s8vyLj8JdzkGKJO+fSfOsp70NZCRy1SKi6tJjHfd4e5 qf9qgi9yh39y+VktBV0o83+gkaGlOKIPRCqOYdkOQHl59RT0YWBoRuXrVtmE00fl QoePBxJjVszlzknULLOXptv0B0sv0ZhsXPgID3hhZ0Z78LQ1RG/9fquGGbfOFfe0 qiPR4LZKCTatP4jvxV3PVKJ9NdXb8OKmfF7oEO7t8WZBJUMtpeuKOw5Qj6Am1pTA UUtDuCXeP0rPVE6Nj5p3NuhkVWuW9eX+7v4XhUC+t4c74PeDo+Fx+LjZF/D3WGVr yD2fcoL16mZ/+LWbxblVkhmsNQpyogtZfj/yvnLctMlGfvseMV9tPOe4GG5QLexW HRl3fSMRIi6MjYnxQyeF/vp+eWd6ApK9EIFqYcLWO+AjzXeZ8uS8+ezGzA7ZUvyG GKJB/ThZHTxuszh7kUgq =mRpk -END PGP SIGNATURE- ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules
On 07/09/2016 06:50 AM, Nick Coghlan wrote: Given that the issues that arose in this case weren't at all obvious up front, what do folks think of the idea of updating PEP 399 to explicitly prohibit class/function mismatches between accelerator modules and their pure Python counterparts? So this is basically a doc change and a reminder to test both the C version and the Python version to ensure identical semantics? -- ~Ethan~ ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules
On 7/9/2016 9:50 AM, Nick Coghlan wrote: I'm in the process of trying to disentangle http://bugs.python.org/issue27137 which points out some of the behavioural differences that arise when falling back from the original C implementation of functools.partial to the pure Python emulation that uses a closure. That issue was opened due to a few things that work with the C implementation that fail with the Python implementation: - the C version can be pickled (and hence used with multiprocessing) - the C version can be subclassed - the C version can be used in "isinstance" checks - the C version behaves as a static method, the Python version as a normal instance method While I'm planning to accept the patch that converts the pure Python version to a full class that matches the semantics of the C version in these areas as well as in its core behaviour, that last case is one where the pure Python version merely exhibits different behaviour from the C version, rather than failing outright. Given that the issues that arose in this case weren't at all obvious up front, what do folks think of the idea of updating PEP 399 to explicitly prohibit class/function mismatches between accelerator modules and their pure Python counterparts? +1 I would put it positively that both version have to use matching classes or matching functions. I assumed that this was already true. The rationale for making such a change is that when it comes to true drop-in API compatibility, we have reasonable evidence that "they're both callables" isn't sufficient once the complexities of real world applications enter the picture. This same issue has come up in the itertools docs. The C code callables are iterator classes that return iterator instances. The didactic doc 'equivalents' are generator functions that return generator instances. The result is functionally the same in that the respective iterators yield the same sequence of objects. In this, the doc equivalents concisely fulfill their purpose. But non-iterator behaviors of C class instances and generators are different, and this confused some people. Python-coded itertools would have to be iterator classes to be really equivalent. In May, Raymond added 'Roughly' to 'equivalent'. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules
On Sat, Jul 09, 2016 at 11:50:59PM +1000, Nick Coghlan wrote: > I'm in the process of trying to disentangle > http://bugs.python.org/issue27137 which points out some of the > behavioural differences that arise when falling back from the original > C implementation of functools.partial to the pure Python emulation > that uses a closure. [...] > Given that the issues that arose in this case weren't at all obvious > up front, what do folks think of the idea of updating PEP 399 to > explicitly prohibit class/function mismatches between accelerator > modules and their pure Python counterparts? > > The rationale for making such a change is that when it comes to true > drop-in API compatibility, we have reasonable evidence that "they're > both callables" isn't sufficient once the complexities of real world > applications enter the picture. Are these problems common enough, or serious enough, to justify a blanket ban on mismatches (a "MUST NOT" rather than a "SHOULD NOT" in RFC 2119 terminology)? The other side of the issue is that requiring exact correspondence is considerably more difficult and may be over-kill for some uses. Hypothetically speaking, if I have an object that only supports pickling by accident, and I replace it with one that doesn't support pickling, shouldn't it be my decision whether that counts as a functional regression (a bug) or a reliance on an undocumented and accidental implementation detail? -- Steve ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules
On 10 July 2016 at 05:10, Steven D'Aprano wrote: > The other side of the issue is that requiring exact correspondence is > considerably more difficult and may be over-kill for some uses. > Hypothetically speaking, if I have an object that only supports pickling > by accident, and I replace it with one that doesn't support pickling, > shouldn't it be my decision whether that counts as a functional > regression (a bug) or a reliance on an undocumented and accidental > implementation detail? That's the proposed policy change and the reason I figured it needed a python-dev discussion, as currently it's up to folks adding the Python equivalents (or the C accelerators) to decide on a case by case basis whether or not to care about compatibility for: - string representations - pickling (and hence multiprocessing support) - subclassing - isinstance checks - descriptor behaviour The main way for such discrepancies to arise is for the Python implementation to be a function (or closure), while the C implementation is a custom stateful callable. The problem with the current "those are just technical implementation details" approach is that a lot of Pythonistas learn standard library API behaviour and capabilities through a mix of experimentation and introspection rather than reading the documentation, so if CPython uses the accelerated version by default, then most folks aren't going to notice the discrepancies until they (or their users) are trying to debug a problem like "my library works fine in CPython, but breaks when used with multiprocessing on PyPy" or "my doctests fail when running under MicroPython". For non-standard library code, those kinds of latent compatibility defects are fine, but PEP 399 is specifically about setting design & development policy for the *standard library* to help improve cross-implementation compatibility not just of the standard library itself, but of code *using* the standard library in ways that work on CPython in its default configuration. One example of a practical consequence of the change in policy would be to say that if you don't want to support subclassing, then don't give the *type* a public name - hide it behind a factory function, the way contextlib.contextmanager hides contextlib._GeneratorContextManager. That example also shows that accidentally making a type public (but still undocumented) isn't necessarily a commitment to keeping it public forever - that type was originally contextlib.GeneratorContextManager, so when it was pointed out it wasn't documented, I had to choose between supporting it as a public API (which I didn't want to do) and adding the leading underscore to its name to better reflect its implementation detail status. In a similar fashion, the policy I'm proposing here also wouldn't require that discrepancies always be resolved in favour of enhancing the Python version to match the C version - in some cases, if the module maintainer genuinely doesn't want to support a particular behaviour, then they can make sure that behaviour isn't available for the C version either. The key change is that it would become officially *not* OK for the feature set of the C version to be a superset of the feature set of the Python version - either the C version has to be constrained to match the Python one, or the Python one enhanced to match the C one, rather than leaving the latent compatibility defect in place as a barrier to adoption for alternate runtimes. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposal: explicitly disallow function/class mismatches in accelerator modules
On Sat, 9 Jul 2016 at 06:52 Nick Coghlan wrote: > I'm in the process of trying to disentangle > http://bugs.python.org/issue27137 which points out some of the > behavioural differences that arise when falling back from the original > C implementation of functools.partial to the pure Python emulation > that uses a closure. > > That issue was opened due to a few things that work with the C > implementation that fail with the Python implementation: > > - the C version can be pickled (and hence used with multiprocessing) > - the C version can be subclassed > - the C version can be used in "isinstance" checks > - the C version behaves as a static method, the Python version as a > normal instance method > > While I'm planning to accept the patch that converts the pure Python > version to a full class that matches the semantics of the C version in > these areas as well as in its core behaviour, that last case is one > where the pure Python version merely exhibits different behaviour from > the C version, rather than failing outright. > > Given that the issues that arose in this case weren't at all obvious > up front, what do folks think of the idea of updating PEP 399 to > explicitly prohibit class/function mismatches between accelerator > modules and their pure Python counterparts? > I think flat-out prohibiting won't work in the Python -> C case as you can do things such as closures and such that I don't know if we provide the APIs to mimic through the C API. I'm fine saying we "strongly encourage mirroring the design between the pure Python and accelerated version for various reasons". -Brett > > The rationale for making such a change is that when it comes to true > drop-in API compatibility, we have reasonable evidence that "they're > both callables" isn't sufficient once the complexities of real world > applications enter the picture. > > Regards, > Nick. > > -- > Nick Coghlan | [email protected] | Brisbane, Australia > ___ > Python-Dev mailing list > [email protected] > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list [email protected] https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
