Re: [Python-Dev] Summary of Python tracker Issues
Brett Cannon wrote: Do we have a graph of the historical trend of the number of bugs (or at least the historical details stored somewhere)? I think we have had a net decrease in open bugs the last couple of weeks and it would be neat to see an absolute and relative graph of the overall trend since Python 3.3.0 was released. Also might make a nice motivator to try to close issues faster. =) Otherwise is the code public for this somewhere? I assume it's making an XML-RPC call or something every week to get the results, but if I decide to do a little App Engine app to store historical data and do a graph I would rather not have to figure all of this out from scratch. =) Although I could I guess also parse the email if I wanted to ignore all other emails. A few months ago I made a script that downloads all python-dev mailman archives, scans them to find the summary messages, parses the messages and creates a graph using matplotlib. The script is available at https://gist.github.com/2723809. ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] cpython: Issue #16592: stringlib_bytes_join doesn't raise MemoryError on allocation
On Sun, Dec 2, 2012 at 6:19 PM, Jeff Allen ja...py@farowl.co.uk wrote: On 02/12/2012 07:08, Nick Coghlan wrote: On Sun, Dec 2, 2012 at 4:56 PM, christian.heimes python-check...@python.org wrote: ... http://hg.python.org/cpython/rev/9af5a2611202 diff --git a/Misc/NEWS b/Misc/NEWS ... +- Issue #16592: stringlib_bytes_join doesn't raise MemoryError on allocation + failure. Please don't write NEWS entries in past tense like this - they're annoyingly ambiguous, as it isn't clear whether the entry is describing the reported problem or the fix for the problem. Describing just the new behaviour or the original problem and the fix is much easier to follow. For example: - Issue #16592: stringlib_bytes_join now correctly raises MemoryError on allocation failure. - Issue #16592: stringlib_bytes_join was triggering SystemError on allocation failure. It now correctly raises MemoryError. Issue titles for actual bugs generally don't make good NEWS entries, as they're typically a summary of the problem rather than the solution (RFE's are different, as there the issue title is often a good summary of the proposed change) You mean please do (re-)write such statements in the past tense, when the news is that the statement is no longer true. I agree about the ambiguity that arises here, but there's a simple alternative to re-writing. Surely all that has been forgotten here is an enclosing The following issues have been resolved:? I think there's a lot to be said for cut and paste of actual titles on grounds of accuracy and speed (and perhaps scriptability). Readability matters - ambiguous release notes don't help anyone, and, like code, release notes are read by many more people than write them. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keyword meanings [was: Accept just PEP-0426]
How to use Obsoletes: The author of B decides A is obsolete. A releases an empty version of itself that Requires: B B Obsoletes: A The package manager says These packages are obsolete: A. Would you like to remove them? User says OK. On Wed, Nov 21, 2012 at 2:54 AM, Stephen J. Turnbull step...@xemacs.orgwrote: PJ Eby writes: On Wed, Nov 21, 2012 at 12:00 AM, Stephen J. Turnbull step...@xemacs.orgwrote: What I care about is when I'm using Gorgon, and there's something better (or worse, correct) to use in my application. Hence my suggestion for an Obsoleted-By field, in which Gorgon would be able to suggest alternatives. My bad, my precise intention was to follow up on your idea (which, credit where credit is due, I had *not* hit upon independently). I should have made that clear. (I really shouldn't be answering English email at a Japanese-speaking conference, my brain thinks it knows what it's doing but shirazuni 日 本文化が染み込む) It might be a good idea to have a just-like-Amazon While-This-Package-Is-Great-You-Might-Also-Consider: field. Yeah, that's basically what Obsoleted-By is for. Well, Obsoleted-By is pretty strong language for suggesting possible alternatives. But I suspect that few projects would really want to be suggesting competitors' products *or* their own oldie-but-still-goodie that they'd really like to obsolete ASAP (put an Obsoleted-By line in every Python 2 distribution, anyone? :-) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dholth%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Proposing Argument Clinic, a new way of specifying arguments to builtins for CPython
Say there, the Python core development community! Have I got a question for you! *ahem* Which of the following four options do you dislike least? ;-) 1) CPython continues to provide no function signature objects (PEP 362) or inspect.getfullargspec() information for any function implemented in C. 2) We add new hand-coded data structures representing the metadata necessary for function signatures for builtins. Which means that, when defining arguments to functions in C, we'd need to repeat ourselves *even more* than we already do. 3) Builtin function arguments are defined using some seriously uncomfortable and impenetrable C preprocessor macros, which produce all the various types of output we need (argument processing code, function signature metadata, possibly the docstrings too). 4) Builtin function arguments are defined in a small DSL; these are expanded to code and data using a custom compile-time preprocessor step. All the core devs I've asked said given all that, I'd prefer the hairy preprocessor macros. But by the end of the conversation they'd changed their minds to prefer the custom DSL. Maybe I'll make a believer out of you too--read on! I've named this DSL preprocessor Argument Clinic, or Clinic for short**. Clinic works similarly to Ned Batchelder's brilliant Cog tool: http://nedbatchelder.com/code/cog/ You embed the input to Clinic in a comment in your C file, and the output is written out immediately after that comment. The output's overwritten every time the preprocessor is run. In short it looks something like this: /*[clinic] input to the DSL [clinic]*/ ... output from the DSL, overwritten every time ... /*[clinic end:checksum]*/ The input to the DSL includes all the metadata about the function that we need for the function signature: * the name of the function, * the return annotation (if any), * each parameter to the function, including * its name, * its type (in C), * its default value, * and a per-parameter docstring; * and the docstring for the function as a whole. The resulting output contains: * the docstring for the function, * declarations for all your parameters, * C code handling all argument processing for you, * and a #define'd methoddef structure for adding the function to the module. I discussed this with Mark HotPy Shannon, and he suggested we break our existing C functions into two. We put the argument processing into its own function, generated entirely by Clinic, and have the implementation in a second function called from the first. I like this approach simply because it makes the code cleaner. (Note that this approach should not cause any overhead with a modern compiler, as both functions will be static.) But it also provides an optimization opportunity for HotPy: it could read the metadata, and when generating the JIT'd code it could skip building the PyObjects and argument tuple (and possibly keyword argument dict), and the subsequent unpacking/decoding, and just call the implementation function directly, giving it a likely-measurable speed boost. And we can go further! If we add a new extension type API allowing you to register both functions, and external modules start using it, sophisticated Python implementations like PyPy might be able to skip building the tuple for extension type function calls--speeding those up too! Another plausible benefit: alternate implementations of Python could read the metadata--or parse the input to Clinic themselves--to ensure their reimplementations of the Python standard library conform to the same API! Clinic can also run general-purpose Python code (/*[python]). All output from print is redirected into the output section after the Python code. As you've no doubt already guessed, I've made a prototype of Argument Clinic. You can see it--and some sample conversions of builtins using it for argument processing--at this BitBucket repo: https://bitbucket.org/larry/python-clinic I don't claim that it's fabulous, production-ready code. But it's a definite start! To save you a little time, here's a preview of using Clinic for dbm.open(). The stuff at the same indent as a declaration are options; see the clinic.txt in the repo above for full documentation. /*[clinic] dbm.open - mapping basename=dbmopen const char *filename; The filename to open. const char *flags=r; How to open the file. r for reading, w for writing, etc. int mode=0666; default=0o666 If creating a new file, the mode bits for the new file (e.g. os.O_RDWR). Returns a database object. [clinic]*/ PyDoc_STRVAR(dbmopen__doc__, dbm.open(filename[, flags=\'r\'[, mode=0o666]]) - mapping\n \n filename\n The filename to open.\n \n flags\n How to open the file. \r\ for reading, \w\ for writing, etc.\n \n mode\n
Re: [Python-Dev] Proposing Argument Clinic, a new way of specifying arguments to builtins for CPython
On Mon, Dec 3, 2012 at 2:29 PM, Larry Hastings la...@hastings.org wrote: Say there, the Python core development community! Have I got a question for you! *ahem* Which of the following four options do you dislike least? ;-) 1) CPython continues to provide no function signature objects (PEP 362) or inspect.getfullargspec() information for any function implemented in C. yuck on #1, though this is what happens by default if we don't do anything nice. 2) We add new hand-coded data structures representing the metadata necessary for function signatures for builtins. Which means that, when defining arguments to functions in C, we'd need to repeat ourselves *even more* than we already do. yuck on #2. 3) Builtin function arguments are defined using some seriously uncomfortable and impenetrable C preprocessor macros, which produce all the various types of output we need (argument processing code, function signature metadata, possibly the docstrings too). Likely painful to maintain. C++ templates would likely be easier. 4) Builtin function arguments are defined in a small DSL; these are expanded to code and data using a custom compile-time preprocessor step. All the core devs I've asked said given all that, I'd prefer the hairy preprocessor macros. But by the end of the conversation they'd changed their minds to prefer the custom DSL. Maybe I'll make a believer out of you too--read on! It always strikes me that C++ could be such a DSL that could likely be used for this purpose rather than defining and maintaining our own yet another C preprocessor step. But I don't have suggestions and we're not allowing C++ so... nevermind. :) I've named this DSL preprocessor Argument Clinic, or Clinic for short**. Clinic works similarly to Ned Batchelder's brilliant Cog tool: http://nedbatchelder.com/code/**cog/http://nedbatchelder.com/code/cog/ You embed the input to Clinic in a comment in your C file, and the output is written out immediately after that comment. The output's overwritten every time the preprocessor is run. In short it looks something like this: /*[clinic] input to the DSL [clinic]*/ ... output from the DSL, overwritten every time ... /*[clinic end:checksum]*/ The input to the DSL includes all the metadata about the function that we need for the function signature: * the name of the function, * the return annotation (if any), * each parameter to the function, including * its name, * its type (in C), * its default value, * and a per-parameter docstring; * and the docstring for the function as a whole. The resulting output contains: * the docstring for the function, * declarations for all your parameters, * C code handling all argument processing for you, * and a #define'd methoddef structure for adding the function to the module. I discussed this with Mark HotPy Shannon, and he suggested we break our existing C functions into two. We put the argument processing into its own function, generated entirely by Clinic, and have the implementation in a second function called from the first. I like this approach simply because it makes the code cleaner. (Note that this approach should not cause any overhead with a modern compiler, as both functions will be static.) But it also provides an optimization opportunity for HotPy: it could read the metadata, and when generating the JIT'd code it could skip building the PyObjects and argument tuple (and possibly keyword argument dict), and the subsequent unpacking/decoding, and just call the implementation function directly, giving it a likely-measurable speed boost. And we can go further! If we add a new extension type API allowing you to register both functions, and external modules start using it, sophisticated Python implementations like PyPy might be able to skip building the tuple for extension type function calls--speeding those up too! Another plausible benefit: alternate implementations of Python could read the metadata--or parse the input to Clinic themselves--to ensure their reimplementations of the Python standard library conform to the same API! Clinic can also run general-purpose Python code (/*[python]). All output from print is redirected into the output section after the Python code. As you've no doubt already guessed, I've made a prototype of Argument Clinic. You can see it--and some sample conversions of builtins using it for argument processing--at this BitBucket repo: https://bitbucket.org/larry/**python-clinichttps://bitbucket.org/larry/python-clinic I don't claim that it's fabulous, production-ready code. But it's a definite start! To save you a little time, here's a preview of using Clinic for dbm.open(). The stuff at the same indent as a declaration are options; see the clinic.txt in the repo above for full
Re: [Python-Dev] Proposing Argument Clinic, a new way of specifying arguments to builtins for CPython
+1 to what Greg said. -- Sent from my phone, thus the relative brevity :) ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing Argument Clinic, a new way of specifying arguments to builtins for CPython
On 12/3/2012 3:42 PM, Gregory P. Smith wrote: All the core devs I've asked said given all that, I'd prefer the hairy preprocessor macros. But by the end of the conversation they'd changed their minds to prefer the custom DSL. Maybe I'll make a believer out of you too--read on! It always strikes me that C++ could be such a DSL that could likely be used for this purpose rather than defining and maintaining our own yet another C preprocessor step. But I don't have suggestions and we're not allowing C++ so... nevermind. :) C++ has enough power to delude many (including me) into thinking that it could be used this way but in my experience, it isn't quite there. There isn't quite enough distinction between various integral types to achieve the goals I once had, anyway... and that was some 15 years ago... but for compatibility reasons, I doubt it has improved in that area. Glenn ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing Argument Clinic, a new way of specifying arguments to builtins for CPython
On 12/03/2012 03:42 PM, Gregory P. Smith wrote: On Mon, Dec 3, 2012 at 2:29 PM, Larry Hastings la...@hastings.org mailto:la...@hastings.org wrote: Default values for arguments are represented in C as strings; the conversion process attempts eval() on the string, and if that works it uses the result, otherwise it simply passes through the string. I think passing on the string if that doesn't work is wrong. It could lead to a behavior change not realized until runtime due to some other possibly unrelated thing causing the eval to fail. Good point. I amend my proposal to say: we make this explicit rather than implicit. We declare an additional per-parameter flag that says don't eval this, just pass through the string. In absence of this flag, the struct-to-Signature-izer runs eval on the string and complains noisily if it fails. * Right now Clinic paves over the PyArg_ParseTuple API for you. If we convert CPython to use Clinic everywhere, theoretically we could replace the parsing API with something cleaner and/or faster. Does anyone have good ideas (and time, and energy) here? By paves over do you mean that Clinic is currently using the ParseTuple API in its generated code? Yes. Specifically, it uses ParseTuple for positional-only argument processing, and ParseTupleAndKeywords for all others. You can see the latter in the sample output in my original email. Yes, we should do better. But don't hold Clinic up on that. As I have not! But the biggest unresolved question... is this all actually a terrible idea? No it is not. I like it. \o/ //arry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing Argument Clinic, a new way of specifying arguments to builtins for CPython
On Dec 03, 2012, at 02:29 PM, Larry Hastings wrote: 4) Builtin function arguments are defined in a small DSL; these are expanded to code and data using a custom compile-time preprocessor step. All the core devs I've asked said given all that, I'd prefer the hairy preprocessor macros. But by the end of the conversation they'd changed their minds to prefer the custom DSL. Maybe I'll make a believer out of you too--read on! The biggest question with generated code is always the effect on debugging. How horrible will it be when I have to step through argument parsing to figure out what's going wrong? -Barry ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Accept just PEP-0426
On Tue, Nov 20, 2012 at 11:01 PM, Nick Coghlan ncogh...@gmail.com wrote: On Wed, Nov 21, 2012 at 1:20 PM, Nick Coghlan ncogh...@gmail.com wrote: On Wed, Nov 21, 2012 at 1:10 PM, PJ Eby p...@telecommunity.com wrote: Conversely, if you have already installed a package that says it Obsoletes another package, this does *not* tell you that the obsolete package shouldn't still be installed! A replacement project doesn't necessarily share the same API, and may exist in a different package namespace altogether. Then that's a bug in the metadata of the project misusing Obsoletes, and should be reported as such. If the new package is not a drop-in replacement, then it has no business claiming to obsolete the other package. I think one of the big reasons this kind of use is rare in the Python community is that project name changes are almost always accompanied by *package* name changes, and as soon as you change the package name, you're changing the public API, and thus it is no longer appropriate to use Provides or Obsoletes, as the renamed project is no longer a drop-in replacement for the original. I realised that my comments above are more about the appropriate use of Provides, rather than Obsoletes. For a practically useful Obsoletes, I think I'm inclined to agree with you, as Obsoleted-By provides a way for a maintainer to explicitly declare that a project is no longer receiving updates, and users should migrate to the replacement project if they want to continue to receive fixes and improvements. The current version of Obsoletes is, as Daniel describes, really only useful as documentation. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia A few more changes to try to address some of the confusion about Requires-Dist: without re-designing the entire requirements system. PEP-426 was written only to add extras support to the format. The other changes, re-writing much of the PEP, have been an unfortunate side-effect. The file format's keys are case-insensitive. The version number should be in PEP 386 form. There are too many non-PEP-386 versions now and in the future to make it a must. Distribution (requirement) names are noted as being distinct from ``import x`` module names. Parenthetical explanation has balanced parens. bundled has been struck from the PEP. diff -r 55c706023fa2 -r 026aebf2265d pep-0426.txt --- a/pep-0426.txt Sun Nov 18 19:55:10 2012 +0200 +++ b/pep-0426.txt Mon Dec 03 20:36:13 2012 -0500 @@ -34,9 +34,9 @@ The syntax defined in this PEP is for use with Python distribution metadata files. The file format is a simple UTF-8 encoded Key: value -format with no maximum line length, followed by a blank line and an -arbitrary payload. The keys are case-insensitive. It is parseable by -the ``email`` module with an appropriate ``email.policy.Policy()``. +format with case-insensitive keys and no maximum line length, followed by +a blank line and an arbitrary payload. It is parseable by the ``email`` +module with an appropriate ``email.policy.Policy()``. When ``metadata`` is a Unicode string, ```email.parser.Parser().parsestr(metadata)`` is a serviceable parser. @@ -94,7 +94,7 @@ ::: A string containing the distribution's version number. This -field must be in the format specified in PEP 386. +field should be in the format specified in PEP 386. Example:: @@ -283,12 +283,13 @@ Each entry contains a string naming some other distutils project required by this distribution. -The format of a requirement string is identical to that of a -distutils project name (e.g., as found in the ``Name:`` field. -optionally followed by a version declaration within parentheses. +The format of a requirement string is identical to that of a distribution +name (e.g., as found in the ``Name:`` field) optionally followed by a +version declaration within parentheses. -The distutils project names should correspond to names as found -on the `Python Package Index`_. +The distribution names should correspond to names as found on the `Python +Package Index`_; often the same as, but distinct from, the module names +as accessed with ``import x``. Version declarations must follow the rules described in `Version Specifiers`_ @@ -305,7 +306,8 @@ Like Requires-Dist, but names dependencies needed while the distributions's distutils / packaging `setup.py` / `setup.cfg` is run. -Commonly used to generate a manifest from version control. +Commonly used to bring in extra compiler support or a package needed +to generate a manifest from version control. Examples:: @@ -318,17 +320,19 @@ Provides-Dist (multiple use) -Each entry contains a string naming a Distutils project which -is contained within this distribution. This field *must* include -the project identified in the ``Name`` field, followed by the -version : Name (Version). +Each entry contains a string naming a requirement that is
Re: [Python-Dev] Proposing Argument Clinic, a new way of specifying arguments to builtins for CPython
On 12/03/2012 02:37 PM, Barry Warsaw wrote: The biggest question with generated code is always the effect on debugging. How horrible will it be when I have to step through argument parsing to figure out what's going wrong? Right now, it's exactly like the existing solution. The generated function looks more or less like the top paragraph of the old code did; it declares variables, with defaults where appropriate, it calls PyArg_ParseMumbleMumble, if that fails it returns NULL, and otherwise it calls the impl function. There *was* an example of generated code in my original email; I encourage you to go back and take a look. For more you can look at the bitbucket repo; the output of the DSL is checked in there, as would be policy if we went with Clinic. TBH I think debuggability is one of the strengths of this approach. Unlike C macros, here all the code is laid out in front of you, formatted for easy reading. And it's not terribly complicated code. If we change the argument parsing code to use some new API, one hopes we will have the wisdom to make it /easier/ to read than PyArg_*. //arry/ ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Proposing Argument Clinic, a new way of specifying arguments to builtins for CPython
On Tue, Dec 4, 2012 at 8:37 AM, Barry Warsaw ba...@python.org wrote: On Dec 03, 2012, at 02:29 PM, Larry Hastings wrote: 4) Builtin function arguments are defined in a small DSL; these are expanded to code and data using a custom compile-time preprocessor step. All the core devs I've asked said given all that, I'd prefer the hairy preprocessor macros. But by the end of the conversation they'd changed their minds to prefer the custom DSL. Maybe I'll make a believer out of you too--read on! The biggest question with generated code is always the effect on debugging. How horrible will it be when I have to step through argument parsing to figure out what's going wrong? That's the advantage of the Cog-style approach that modifies the C source files in place and records checksums so the generator can easily tell when the code needs to be regenerated, either because it was changed via hand editing or because the definition changed. Yes, it violates the guideline of don't check in generated code, but it makes debugging sane. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
