Re: [Python-Dev] Update on Cygwin support (was: Clarifying Cygwin support in CPython)
On 26 July 2018 at 02:13, Erik Bray wrote: > I think a new approach that might be more practical for actually > getting this platform re-supported, is to go ahead and add a CI build, > and just skip all known failing test modules. This is what I've done > in a new PR to add a Cygwin build on AppVeyor: > > https://github.com/python/cpython/pull/8463 > > This is far from ideal of course, and should not mean the platform is > "supported". But now I and others can go through and fix the > remaining test failures, re-enable those modules in the CI > configuration, and actually obtain some meaningful results, which will > hopefully encourage the core committers to accept fixes for the > platform. I believe the PEP 538 & 540 locale handling tests are amongst those that are still a bit sketchy (or outright broken?) on Cygwin, and I think having an advisory CI bot would definitely help with that. (Cygwin/MinGW are an interesting hybrid that really highlight the fact that neither "POSIX implies not Windows" nor "Windows implies the Win32 API" are entirely valid assumptions) So your suggested approach seems like a plausible way forward to me. The main potentially viable alternative I see would be to set up the *buildbot* first, and then devote the custom builder branch to the task of Cygwin testing for a while: https://devguide.python.org/buildbots/#custom-builders However, I think the overall UX of that would be worse than going down the advisory CI path (especially since it wouldn't really help with the aspect of parallel development introducing new Cygwin failures). Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tests failing on Windows with TESTFN
On 30/07/2018 16:41, Nick Coghlan wrote: On 29 July 2018 at 03:20, Tim Golden wrote: I think that was my starting point: rather than develop increasingly involved and still somewhat brittle mechanisms, why not do what you'd naturally do with a new test and use tempfile? I was expecting someone to come forward to highlight some particular reason why the TESTFN approach is superior, but apart from a reference to the possibly cost of creating a new temporary file per test, no-one really has. For higher level modules, "just use tempfile to create a new temporary directory, then unlink it at the end" is typically going to be a good answer (modulo the current cleanup issues that Jeremy is reporting, but ideally those will be fixed rather than avoided, either by improving the way the module is being used, or fixing any underlying defects). For lower level modules though, adding a test suite dependency on tempfile introduces a non-trivial chance of adding an operational dependency from a module's test suite back to the module itself. When that happens, module regressions may show up as secondary failures in tempfile that then need to be debugged, rather than as specific unit test failures that point you towards exactly what you broke. Cheers, Nick. Thanks Nick; I hadn't thought about the possible interdependency issue. I think for the moment my approach will be to switch to support.unlink wherever possible to start with. Before introducing other (eg tempfile) changes, this should at least narrow the issues down. I've made a start on that (before inadvertently blowing away all the changes since my hours-previous commit!) If further changes are necessary then I'll probably look case-by-case to see whether a tempfile or some other solution would help. That said, that's potentially quite a lot of change -- at least in terms of files changed if not strictly of functionality. So I'm thinking of trickle-feeding the changes through as people will understandably baulk at a patchbomb (PR-bomb?) hitting the codebase all at once. TJG ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fuzzing the Python standard library
On 18 July 2018 at 17:49, Steve Holden wrote:
> On Tue, Jul 17, 2018 at 11:44 PM, Paul G wrote:
>>
>> In many languages numeric types can't hold arbitrarily large values, and I
>> for one hadn't really previously recognized that if you read in a numeric
>> value with an exponent that it would be represented *exactly* in memory (and
>> thus one object with a very compact representation can take up huge amounts
>> of memory). It's also not *inconceivable* that under the hood Python would
>> represent fractions.Fraction("1.64E664644") "lazily" in some fashion so
>> that it did not consume all the memory on disk.
>>
> Sooner or later you are going to need the digits of the number to perform a
> computation. Exactly when would you propose the deferred evaluation should
> take place?
>
> There are already occasional inquiries about the effects of creation of such
> large numbers and their unexpected effects, so they aren't completely
> unknown. At the same time, this isn't exactly a mainstream "bug", as
> evidenced by the fact that such issues are relatively rare.
It does mean that if Fraction is being used with untrusted inputs
though, it *does* make sense to put a reasonable upper bound on
permitted exponents.
The default decimal context caps expression results at an exponent of
less than 1 million for example:
>>> +decimal.Decimal("1e999_999")
Decimal('1E+99')
>>> +decimal.Decimal("1e1_000_000")
Traceback (most recent call last):
File "", line 1, in
decimal.Overflow: []
That's already large enough to result in a ~415k integer that takes a
minute or so for my machine to create if I call int() on it.
So I think it does make sense to at least describe how to use the
decimal module to do some initial sanity checking on potentially
exponential inputs, even if the fractions module itself never gains
native support for processing untrusted inputs.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Reserve one type slot for Cython
On 2018-07-30 15:35, INADA Naoki wrote: As repeatedly said, PEP 580 is very complicated protocol when just implementing callable object. Can you be more concrete what you find complicated? Maybe I can improve the PEP to explain it more. Also, I'm open to suggestions to make it less complicated. It is optimized for implementing custom method object, although almost only Cython want the custom method type. For the record, Numba also seems interested in the PEP: https://groups.google.com/a/continuum.io/d/msg/numba-users/2G6k2R92MIM/P-cFKW7xAgAJ I'm not sure adding such complicated protocol almost only for Cython. If CyFunction can be implemented behind PEP 576, it may be better. I recall my post https://mail.python.org/pipermail/python-dev/2018-July/154238.html explaining the main difference between PEP 576 and PEP 580. I would like to stress that PEP 580 was designed for maximum performance, both today and for future extensions (such as calling with native C types). * PEP 576 and 580 are not strictly mutually exclusive; PEP 576 may be accepted in addition to PEP 580 I don't think that this is a good idea: you will mostly end up with the disadvantages of both approaches. Jeroen. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Error message for wrong number of arguments
On 30 July 2018 at 22:12, Jeroen Demeyer wrote: > I think it has been argued before that it's a feature that self is counted. > So I consider the error message for list().append a bug. This is one of the > many oddities I noticed while working on improving built-in functions. > > Would you agree to change the error message for built-in methods to be > closer to Python methods? I would, and I think it would make sense for the PEP to cite improving consistency (and reducing code duplication?) in that regard as an advantage of the PEP. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing C API
On 30 July 2018 at 21:23, Victor Stinner wrote: > Or maybe test__capi.py so you can more easily discover > test_unicode_cami while working on Unicode. You can use -m "test_*_capi" to > run all C API tests. Missing word there: -m test "test*_capi" I think between this approach and expanding test_capi to be a test package, not just a test file, I think it would be possible to make these parts of the test a fair bit more discoverable and maintainable. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tests failing on Windows with TESTFN
On 29 July 2018 at 03:20, Tim Golden wrote: > I think that was my starting point: rather than develop increasingly > involved and still somewhat brittle mechanisms, why not do what you'd > naturally do with a new test and use tempfile? I was expecting someone to > come forward to highlight some particular reason why the TESTFN approach is > superior, but apart from a reference to the possibly cost of creating a new > temporary file per test, no-one really has. For higher level modules, "just use tempfile to create a new temporary directory, then unlink it at the end" is typically going to be a good answer (modulo the current cleanup issues that Jeremy is reporting, but ideally those will be fixed rather than avoided, either by improving the way the module is being used, or fixing any underlying defects). For lower level modules though, adding a test suite dependency on tempfile introduces a non-trivial chance of adding an operational dependency from a module's test suite back to the module itself. When that happens, module regressions may show up as secondary failures in tempfile that then need to be debugged, rather than as specific unit test failures that point you towards exactly what you broke. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Error message for wrong number of arguments
Hello, I noticed an inconsistency in the error messages for the number of arguments to a method call. For Python methods, the "self" argument is counted. For built-in methods, the "self" argument is *not* counted: >>> class mylist(list): ... def append(self, val): super().append(val) >>> f = list().append >>> f(1,2) Traceback (most recent call last): File "", line 1, in TypeError: append() takes exactly one argument (2 given) >>> g = mylist().append >>> g(1,2) Traceback (most recent call last): File "", line 1, in TypeError: append() takes 2 positional arguments but 3 were given I think it has been argued before that it's a feature that self is counted. So I consider the error message for list().append a bug. This is one of the many oddities I noticed while working on improving built-in functions. Would you agree to change the error message for built-in methods to be closer to Python methods? Jeroen. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 576/579/580 benchmark: mistune
Like previous SageMath bench, this is caused by Cython's specialization; __Pyx_PyObject_CallOneArg. It specializing calling PyFunction and PyCFunction, but it doesn't specialized for calling CyFunction. Cython can optimize both benchmark with binding, but without PEP 576 nor 580, by adding calling code specialized for CyFunction to __Pyx_PyObject_CallOneArg. Of course, this specialization doesn't help (1) calling CyFunction from Python case and (2) calling CyFunction from C case. Please don't attack me about it. I just survey and report where the overhead in both benchmark with binding=True in this mail. I will send another mail about PEP 576 vs 580. But I'm not good English writer. It's difficult to explain my opinion on such sensitive topic. Regards, On Fri, Jul 27, 2018 at 10:38 PM Jeroen Demeyer wrote: > > Hello all, > > since my latest benchmark for PEP 580 [1] involved SageMath, which is > quite a big project, I instead propose a much simpler benchmark > involving mistune. > > mistune [2] is a Markdown parser implemented in the Python language. It > optionally allows Cython compilation. It doesn't use any kind of > optimization beyond that, but I created a branch [3] to use extension > types instead of Python classes. > > Cython can either use built-in functions/methods or a custom class > (which is not optimized but which would be optimized with PEP 580). > > I benchmarked mistune with custom classes [3] (binding=True, the > default) and with built-in functions/methods [4] (binding=False). This > is the median time of 5 runs: > > Binding=True: 9.063s > Binding=False: 8.658s > > So this shows again that PEP 580 improves performance in actual > real-world use cases. > > > Jeroen. > > > > [1] https://mail.python.org/pipermail/python-dev/2018-July/154740.html > [2] https://github.com/lepture/mistune > [3] https://github.com/jdemeyer/mistune/tree/cython_pxd > [4] https://github.com/jdemeyer/mistune/tree/cython_pxd_nobinding > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com -- INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing C API
Or maybe test__capi.py so you can more easily discover test_unicode_cami while working on Unicode. You can use -m "test_*_capi" to run all C API tests. Victor Le lundi 30 juillet 2018, Victor Stinner a écrit : > Buildbots have a timeout of 15 min per test. I suggest to use multiple test_capi_.py files rather than a directory which behaves as a single test. Or regrtest should be modified to implement timeout differently. > > Victor > > Le dimanche 29 juillet 2018, Serhiy Storchaka a écrit : >> Currently C API is not completely covered by tests. Tests for particular parts of C API are scattered through different files. There are files completely purposed for testing C API (like test_capi.py, test_getargs2.py), there are classes (usually having "CAPI" in the name) in different files for testing C API specific for unicode, marshal. Argument parsing tests are split between two files, test_capi.py, test_getargs2.py. >> >> I need to add new tests for new features, and I'm going to add new tests for existing C API. But first I'm going to reorganize tests. Add a new directory Lib/test/test_capi, and move all C API tests into it, grouped by function prefixes. test_getargs.py for testing PyArg_*(), test_unicode.py for testing PyUnicode_*(), etc. Tests that use the _testcapi module, but don't test specific C API, will left on place. >> >> The benefit is that it will be easier to run all C API tests at once, and only them, and it will be clearer what C API is covered by tests. The disadvantage is that you will need to run several files for testing marshal for example. >> >> What are your thoughts? >> >> ___ >> Python-Dev mailing list >> Python-Dev@python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com >> ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] [PEP 576/580] Reserve one type slot for Cython
As repeatedly said, PEP 580 is very complicated protocol when just implementing callable object. It is optimized for implementing custom method object, although almost only Cython want the custom method type. I'm not sure adding such complicated protocol almost only for Cython. If CyFunction can be implemented behind PEP 576, it may be better. On the other hand, most complexity of PEP 580 is not newly added. Most of them are in PyCFunction, method_descriptor, and some calling APIs already. PEP 580 just restructure them completely to be reusable from Cython. So I agree that PEP 580 is better when thinking from Cython's side. --- I'm not sure which way we should go yet. But my current idea is: * Implement PEP 580 as semi-public APIs only for tools like Cython. * Other Python implementation may not support it in foreseeable future. So such tools should support legacy implementation too. * PEP 576 and 580 are not strictly mutually exclusive; PEP 576 may be accepted in addition to PEP 580, for simpler FASTCALL-able object support. Especially for extension author prefer C to Cython (including stdlib). * If this happened, PEP 580 can remove one abstraction; tp_ccalloffset is offset of PyCCallRoot instead of pointer to it. Py_TPFLAGS_FUNCTION_DESCRIPTOR will be removed from PEP 576 too. Regards, -- INADA Naoki ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
> On 30 Jul 2018, at 10:20, Victor Stinner wrote: > > The API leaking all implementation details will remain available as an opt-in > option for Cython, cffi and debug tools. But this API will only be usable on > the "slow" Python runtime, the one which keeps maximum backward > compatibility. To get new optimizations, you have to use Py_INCREF() and > avoid accessing C strucuture fields, which may or may not need to modify your > code. > > Hum, please, join the capi-sig mailing list, since I already explained that > in my long reply to Stefan on capi-sig ;-) Interesting. I didn’t know that list exists. I’ll respond to your message on that list. Ronald ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing C API
Buildbots have a timeout of 15 min per test. I suggest to use multiple test_capi_.py files rather than a directory which behaves as a single test. Or regrtest should be modified to implement timeout differently. Victor Le dimanche 29 juillet 2018, Serhiy Storchaka a écrit : > Currently C API is not completely covered by tests. Tests for particular parts of C API are scattered through different files. There are files completely purposed for testing C API (like test_capi.py, test_getargs2.py), there are classes (usually having "CAPI" in the name) in different files for testing C API specific for unicode, marshal. Argument parsing tests are split between two files, test_capi.py, test_getargs2.py. > > I need to add new tests for new features, and I'm going to add new tests for existing C API. But first I'm going to reorganize tests. Add a new directory Lib/test/test_capi, and move all C API tests into it, grouped by function prefixes. test_getargs.py for testing PyArg_*(), test_unicode.py for testing PyUnicode_*(), etc. Tests that use the _testcapi module, but don't test specific C API, will left on place. > > The benefit is that it will be easier to run all C API tests at once, and only them, and it will be clearer what C API is covered by tests. The disadvantage is that you will need to run several files for testing marshal for example. > > What are your thoughts? > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 576/579/580 benchmark: mistune
On 2018-07-30 13:11, INADA Naoki wrote: Like previous SageMath bench, this is caused by Cython's specialization; __Pyx_PyObject_CallOneArg. It specializing calling PyFunction and PyCFunction, but it doesn't specialized for calling CyFunction. Yes, I saw that too. But this is exactly what CPython does (it optimizes PyFunction and PyCFunction but not CyFunction), so I would still argue that the benchmark is fair. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Error message for wrong number of arguments
On 2018-07-30 20:22, Chris Barker wrote: is it possible for the interpreter to know when this error is generated that this is a bound method expecting a "self", rather than an arbitrary function with n parameters? That would be quite hard. The error message is generated by the underlying function. At that point, the information of how it was called (as bound method or not) is already gone. Jeroen. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Error message for wrong number of arguments
On Mon, Jul 30, 2018 at 11:39 AM, Jeroen Demeyer wrote: > On 2018-07-30 20:22, Chris Barker wrote: > >> is it possible for the interpreter to know when this error is >> generated that this is a bound method expecting a "self", rather than an >> arbitrary function with n parameters? >> > > That would be quite hard. The error message is generated by the underlying > function. At that point, the information of how it was called (as bound > method or not) is already gone. Thanks, I figured as much. Oh well. This is a serious usability issue -- but what can you do? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Tests failing on Windows with TESTFN
On Mon, Jul 30, 2018 at 8:46 AM, Tim Golden wrote: > On 30/07/2018 16:41, Nick Coghlan wrote: >> >> On 29 July 2018 at 03:20, Tim Golden wrote: >>> >>> I think that was my starting point: rather than develop increasingly >>> involved and still somewhat brittle mechanisms, why not do what you'd >>> naturally do with a new test and use tempfile? I was expecting someone to >>> come forward to highlight some particular reason why the TESTFN approach >>> is >>> superior, but apart from a reference to the possibly cost of creating a >>> new >>> temporary file per test, no-one really has. >> >> >> For higher level modules, "just use tempfile to create a new temporary >> directory, then unlink it at the end" is typically going to be a good >> answer (modulo the current cleanup issues that Jeremy is reporting, >> but ideally those will be fixed rather than avoided, either by >> improving the way the module is being used, or fixing any underlying >> defects). If there's a desire to use tempfile, another option is to have tempfile create the temp files inside the temporary directory the test harness creates specifically for testing -- using the "dir" argument to many of tempfile's functions. Here is where the process-specific temp directory is created for testing (inside test.libregrtest.main): https://github.com/python/cpython/blob/9045199c5aaeac9b52537581be127d999b5944ee/Lib/test/libregrtest/main.py#L511 This would also facilitate the clean-up of any leftover temp files. Again, I think it would be best to use any tempfile functions behind one or more test-support functions, so the choice of location, etc. can be changed centrally without needing to modify code everywhere. --Chris >> >> For lower level modules though, adding a test suite dependency on >> tempfile introduces a non-trivial chance of adding an operational >> dependency from a module's test suite back to the module itself. When >> that happens, module regressions may show up as secondary failures in >> tempfile that then need to be debugged, rather than as specific unit >> test failures that point you towards exactly what you broke. >> >> Cheers, >> Nick. >> > > Thanks Nick; I hadn't thought about the possible interdependency issue. > > I think for the moment my approach will be to switch to support.unlink > wherever possible to start with. Before introducing other (eg tempfile) > changes, this should at least narrow the issues down. I've made a start on > that (before inadvertently blowing away all the changes since my > hours-previous commit!) > > If further changes are necessary then I'll probably look case-by-case to see > whether a tempfile or some other solution would help. > > That said, that's potentially quite a lot of change -- at least in terms of > files changed if not strictly of functionality. So I'm thinking of > trickle-feeding the changes through as people will understandably baulk at a > patchbomb (PR-bomb?) hitting the codebase all at once. > > TJG > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Error message for wrong number of arguments
On Mon, Jul 30, 2018 at 5:12 AM, Jeroen Demeyer wrote: > I think it has been argued before that it's a feature that self is > counted. I suppose it is, as it's technically correct, but it's also a HUGE source of confusion, particularly for newbies. IF this is being touched anyway, is it possible for the interpreter to know when this error is generated that this is a bound method expecting a "self", rather than an arbitrary function with n parameters? In which case, it would be really nice if the error message replaced that somehow, maybe something like: >>> g(1,2) Traceback (most recent call last): File "", line 1, in TypeError: append() takes 1 positional argument in addition to the automatically added instance object -- but 2 were given in addition to the object instance. Man -- hard to come up with good wording for that -- but SOMETHING that lets users know what they actually did wrong would be good :-) If it's not do-able, then still +1 on making builtins consistent. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Accessing mailing list archives
hi all, I've been trying to figure out how to access the archives programmatically. I'm sure this is easy once you know, but googling various things hasn't worked. What I want to do is graph the number of messages about PEP 572 by time. (or has someone already done that?) I installed GNU Mailman, and downloaded the gzip'ed archives for a number of months and unzipped them, and I suspect that there's some way to get them all into a single database, but it hasn't jumped out at me. If I count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines in one of those text files, I get slightly different numbers for each. Alternatively, they're maybe *already* in a database, and I just need API access to do the querying? Can someone help me out? Bob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [PEP 576/580] Reserve one type slot for Cython
Jeroen Demeyer schrieb am 30.07.2018 um 16:40: > On 2018-07-30 15:35, INADA Naoki wrote: >> As repeatedly said, PEP 580 is very complicated protocol >> when just implementing callable object. > > Can you be more concrete what you find complicated? Maybe I can improve the > PEP to explain it more. Also, I'm open to suggestions to make it less > complicated. > >> It is optimized for implementing custom method object, although >> almost only Cython want the custom method type. > > For the record, Numba also seems interested in the PEP: > https://groups.google.com/a/continuum.io/d/msg/numba-users/2G6k2R92MIM/P-cFKW7xAgAJ To add to that record, I (briefly) talked to Ronan Lamy about this at EuroPython and PyPy could also be interested in generalising the call protocol, especially with the future goal of extending it into a C level call protocol that their JIT could understand and build a cffi-like interface on. Stefan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing C API
Actually, I mean -m test --match 'test_*_capi' where --match can also be written -m. It may catch false positive since the filter is also applied to test case names and test method names. Maybe 'test_*_capi.*' is better, I didn't try. You may using use "ls Lib/test/test_*_capi.py > tests; ./python -m test --fromfile tests". There are different options. By the way, running the full test suite just takes 5 min on my laptop, it isn't so long ;-) Victor Le lundi 30 juillet 2018, Nick Coghlan a écrit : > On 30 July 2018 at 21:23, Victor Stinner wrote: >> Or maybe test__capi.py so you can more easily discover >> test_unicode_cami while working on Unicode. You can use -m "test_*_capi" to >> run all C API tests. > > Missing word there: -m test "test*_capi" > > I think between this approach and expanding test_capi to be a test > package, not just a test file, I think it would be possible to make > these parts of the test a fair bit more discoverable and maintainable. > > Cheers, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
>> Last year, I gave a talk at the Language Summit (during Pycon) to >> explain that CPython should become 2x faster to remain competitive. >> IMHO all attempts to optimize Python (CPython forks) have failed >> because they have been blocked by the C API which implies strict >> constraints. > > Well, that's your opinion, but did you prove it? I discussed with many Python developers who agree with me that the C API blocked CPython forks. For example, I heard that Pyston was very fast and very promising before starting to support the C API. The C API requires that your implementations make almost all the same design choices that CPython made 25 years ago (C structures, memory allocators, reference couting, specific GC implementation, GIL, etc.). More efficient technonogies appeared in the meanwhile. Multiple PyPy developers told me that cpyext remains a blocker issue to use PyPy. I am not sure how I am supposed to "prove" these facts. Oh, by the way, I will not promise anything about any potential performance gain. When I write "2x faster", I mean that our current approach for optimization failed to make Python 2x faster over the last 10 years. Python 3 is more or less as fast, or a little bit faster, than Python 2. But Python 2 must not be used as an example of performance. People hesitate between Go, Javascript and Python. And Python is not the winner in term of performance. > The one area I know of where the C API is a strong barrier to > improvement is removing the GIL, and I'd rather let Larry speak about > it. IMHO Gilectomy is going to fail without our help to change the C API. Again, I don't want to promise anything here. Removing reference counting inside CPython is a giant project. But at least, I know that borrowed references are very complex to support if CPython doesn't use reference counting. I heard that PyPy has issues to implement borrowed references. If we succeed to remove them, PyPy should benefit directly of that work. Note: PyPy will still have to support borrowed references for C extensions usong the old C API. But I expect that PyPy will be more reliable, maybe even faster, to run C extensions using the new C API without reference counting. I have to confess that helping Larry is part of my overall plan. But I dislke making promise that I cannot do, and I dislike working on giant multiyear Python roject. My hope is to have a working new C API next year which will hide some implementation details, but not all of them. I want to work incrementally using popular C extensions in the feedback loop. Building a new C API is useless if nobody can use it. But it will take time to adjust the backward compatibility cursor. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accessing mailing list archives
Hi Bob, I wrote a basic script to compute the number of emails per PEP. It requires to download gzipped mbox files from the web page of archives per month, then ungzip them: https://github.com/vstinner/misc/blob/master/python/parse_mailman_mbox_peps.py Results: https://mail.python.org/pipermail/python-committers/2018-April/005310.html Victor Le lundi 30 juillet 2018, Bob Purvy a écrit : > hi all, > I've been trying to figure out how to access the archives programmatically. I'm sure this is easy once you know, but googling various things hasn't worked. What I want to do is graph the number of messages about PEP 572 by time. (or has someone already done that?) > I installed GNU Mailman, and downloaded the gzip'ed archives for a number of months and unzipped them, and I suspect that there's some way to get them all into a single database, but it hasn't jumped out at me. If I count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines in one of those text files, I get slightly different numbers for each. > Alternatively, they're maybe already in a database, and I just need API access to do the querying? Can someone help me out? > Bob ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] USE_STACKCHECK and running out of stack
On Sat, Jul 28, 2018 at 7:51 AM Ronald Oussoren via Python-Dev < python-dev@python.org> wrote: > Hi, > > I’m looking at PyOS_CheckStack because this feature might be useful on > macOS (and when I created bpo-33955 for this someone ran with it and > created a patch). > > Does anyone remember why the interpreter raises MemoryError and not > RecursionError when PyOS_CheckStack detects that we’re about to run out of > stack space? > Running out of C stack makes more sense to me as a MemoryError, you don't know that recursion was the primary cause and the fundamental problem is that you are (about to) run out of a finite memory resource. All you know is that at least one C frame on the C call stack may have required a lot of local stack space. Just as likely as there being tons of regular sized frames. RecursionError is only raised when CPython's arbitrary Python recursion depth limit is reached. It is not raised for memory reasons. In small thread stack space environments (64k, etc) I tended to find problematic code to be C code with local arrays that consumed several kilobytes each rather than actual recursion. Python 2's own import system was notorious for that so if your server is calling service handlers in threads with a small stack and some of those are doing new imports at handler time rather than having done all imports before serving began: boom. stack overflow. -gps ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing C API
> On Jul 29, 2018, at 4:53 AM, Serhiy Storchaka wrote: > > The benefit is that it will be easier to run all C API tests at once, and > only them, and it will be clearer what C API is covered by tests. The > disadvantage is that you will need to run several files for testing marshal > for example. > > What are your thoughts? I prefer the current organization that keeps the various tests together with the category being tested. I almost never need to run the C API tests all at once, but I do need to see all the tests for an object in one place. When maintaining something like marshal, it would be easy to miss some of the tests if they are in a separate file. IMO, the proposed change would hinder future maintenance and fly in the face of our traditional code organization. Raymond ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
Hi Victor, On Sun, 29 Jul 2018 21:47:51 +0200 Victor Stinner wrote: > > I just sent an email to the capi-sig mailing list. Since this mailing > list was idle for months, I copy my email here to get a wider > audience. But if possible, I would prefer that you join me on capi-sig > to reply ;-) > > -- > > Hi, > > Last year, I gave a talk at the Language Summit (during Pycon) to > explain that CPython should become 2x faster to remain competitive. > IMHO all attempts to optimize Python (CPython forks) have failed > because they have been blocked by the C API which implies strict > constraints. Well, that's your opinion, but did you prove it? Which CPython forks did you examine that failed because of the C API? The one area I know of where the C API is a strong barrier to improvement is removing the GIL, and I'd rather let Larry speak about it. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Python on a fork-less POSIX-like OS
On 7/30/2018 4:26 AM, Barath Aron wrote: On 07/30/2018 10:23 AM, Victor Stinner wrote: Python 3.8 will support os.posix_spawn(). I would like to see it used whenever possible instead of fork+exec, since it's faster and it can be safer on some platforms. Pablo Salgado is your guy for that. Victor Awesome! Will this backported to 2.7? Or people should forget 2.7? Normally, enhancements are not backported anywhere. New API == new version of Python. You can, and people and organizations do, compile your own customized version. Official (pydev) bugfixes for 2.7 end 1 Jan 2020. Many projects have already stopped or will stop before then fixing 2.7 versions of their packages. But some people will probably use it for at least another decade. Do what is best for you. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Python on a fork-less POSIX-like OS
Python 3.8 will support os.posix_spawn(). I would like to see it used whenever possible instead of fork+exec, since it's faster and it can be safer on some platforms. Pablo Salgado is your guy for that. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing C API
30.07.18 09:46, Raymond Hettinger пише: I prefer the current organization that keeps the various tests together with the category being tested. I almost never need to run the C API tests all at once, but I do need to see all the tests for an object in one place. When maintaining something like marshal, it would be easy to miss some of the tests if they are in a separate file. IMO, the proposed change would hinder future maintenance and fly in the face of our traditional code organization. What about moving just test_capi.py, test_getargs2.py and test_structmembers.py into Lib/test/test_capi? They are not related to specific types or modules. For references: An issue on the tracker: https://bugs.python.org/issue34272. A PR that moves C API into the test_capi subdirectory: https://github.com/python/cpython/pull/8551. Currently it just moves test_capi.py, test_getargs2.py and test_structmembers.py. An initial version that moved also other C API tests: https://github.com/python/cpython/pull/8551/commits/eb16b9ee9eb36c9965c2d852461bb7838f3f6dfa. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Let's change to C API!
The API leaking all implementation details will remain available as an opt-in option for Cython, cffi and debug tools. But this API will only be usable on the "slow" Python runtime, the one which keeps maximum backward compatibility. To get new optimizations, you have to use Py_INCREF() and avoid accessing C strucuture fields, which may or may not need to modify your code. Hum, please, join the capi-sig mailing list, since I already explained that in my long reply to Stefan on capi-sig ;-) Victor Le lundi 30 juillet 2018, Chris Angelico a écrit : > On Mon, Jul 30, 2018 at 10:46 AM, Victor Stinner wrote: >> 2018-07-29 23:41 GMT+02:00 Jeroen Demeyer : >>> For example, you mention that you want to make Py_INCREF() a function call >>> instead of a macro. But since Py_INCREF is very common, I would guess that >>> this would make performance worse (not by much maybe but surely measurable). >> >> For the very specific case of Py_INCREF(), yes, I agree that >> performance is an issue. But I don't see how I would hide completely >> the PyObject structure without converting Py_INCREF() macro with a >> function call. (I have reasons to want to hide everything, explained >> in the project.) >> >> The open question is if the cost of using function calls for >> Py_INCREF/DECREF versus the benefit of having the ability to modify >> deeply CPython internals. >> >> I'm not sure that it's worth to bet at this point, it's too early, and >> we can decide that later. Moreover, it's also possible to keep >> Py_INCREF() as a macro in the "backward compatible" mode, but require >> a function call in the mode which hides all implementation details >> (the one where you can experiment deep CPython internals changes). >> > > If the macro and function are absolutely 100% compatible, it would be > possible to set compilation to use the function by default, and have a > directive that switches to using the macro. It'd improve performance > at the price of locking you to the exact CPython build. So within > CPython itself, there'd be no performance cost (ergo if you mess with > the internals, you have to recompile all of CPython), most extension > libraries would pay a small (probably immeasurable) price for > compatibility, and a small handful could opt to improve performance at > the price of breaking if anything changes. > > ChrisA > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Testing C API
> On Jul 30, 2018, at 12:06 AM, Serhiy Storchaka wrote: > > 30.07.18 09:46, Raymond Hettinger пише: >> I prefer the current organization that keeps the various tests together with >> the category being tested. I almost never need to run the C API tests all >> at once, but I do need to see all the tests for an object in one place. >> When maintaining something like marshal, it would be easy to miss some of >> the tests if they are in a separate file. IMO, the proposed change would >> hinder future maintenance and fly in the face of our traditional code >> organization. > > What about moving just test_capi.py, test_getargs2.py and > test_structmembers.py into Lib/test/test_capi? They are not related to > specific types or modules That would be reasonable. Raymond ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Python on a fork-less POSIX-like OS
On 07/30/2018 10:23 AM, Victor Stinner wrote: Python 3.8 will support os.posix_spawn(). I would like to see it used whenever possible instead of fork+exec, since it's faster and it can be safer on some platforms. Pablo Salgado is your guy for that. Victor Awesome! Will this backported to 2.7? Or people should forget 2.7? Aron ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Using Python on a fork-less POSIX-like OS
Supporting a new platform requires a lot work. It would be more reasonable for you to first try to get a good support of the master branch before start thinking how to support Python versions. Python 2.7 in 2018? Really? Tick tock: https://pythonclock.org/ http://python3statement.org/ Usually, we don't support new platforms in CPython without strong support of a core developer. See the PEP 11 for more conditions like buildbot requirement. I suggest you to start working on a fork of CPython and maintain your changes in a branch. Git rebase makes it easy. Victor Le lundi 30 juillet 2018, Barath Aron a écrit : > On 07/30/2018 10:23 AM, Victor Stinner wrote: >> >> Python 3.8 will support os.posix_spawn(). I would like to see it used whenever possible instead of fork+exec, since it's faster and it can be safer on some platforms. Pablo Salgado is your guy for that. >> >> Victor > > Awesome! Will this backported to 2.7? Or people should forget 2.7? > > Aron > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
