[Python-Dev] Re: PEP 617: New PEG parser for CPython
On 4/04/20 9:29 am, Brett Cannon wrote: I think "needs" is a bit strong. It would be nice, though. Regardless, as long as this is a net improvement over the status quo I don't see this being rejected on the grounds that an LR or LALR parser would be better since we have a working PEG parser today. :) Even if the section only says "We didn't consider any alternatives, because...", I still think it should be there. -- Greg ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/URT2FASB34SF2VDC57ZCWEEL2RLMIGN7/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 617: New PEG parser for CPython
Greg Ewing wrote: > On 3/04/20 7:10 am, Guido van Rossum wrote: > > Since last fall's core sprint in London, Pablo > > Galindo Salgado, > > Lysandros Nikolaou and myself have been working on a new parser for > > CPython. We are now far enough along that we present a PEP we've written: > > https://www.python.org/dev/peps/pep-0617/ > > Was any consideration given to other types of parser, such > as LR or LALR? > LR parsers handle left recursion naturally, and don't suffer > from any of the drawbacks mentioned in the PEP such as taking > exponential time or requiring all the source to be loaded > into memory. > I think there needs to be a section in the PEP justifying the > choice of PEG over the alternatives. I think "needs" is a bit strong. It would be nice, though. Regardless, as long as this is a net improvement over the status quo I don't see this being rejected on the grounds that an LR or LALR parser would be better since we have a working PEG parser today. :) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/NZAFQSEJD6SCXLSFNTNMPEJZM5XJ7TKL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Any thoughts about a control flow optimizer for CPython?
03.04.20 18:13, joannah nanjekye пише: From my CS theory, a control flow graph models a program flow and one of its main characteristics is it has one entry and exit point. IIRC, CPython’s compilation process involves generation of a control flow graph. Contrary to peephole optimizations, optimizations on the control flow graph are more global allowing us to have complex and global optimizations like branch and checkpoint eliminations etc. I have seen several implementations of control flow optimizations. The one I am familiar with is the V8 control flow optimizer. I tried to investigate this for one of my directed courses last fall but I want to know if there are people who have been thinking about this for CPython and what their thoughts are. I implemented the CFG optimizer, but the problem was that it removed bytecode which served a marker for the lineno setter in the frame object. The lineno setter checked if it valid to jump to the specified instruction by counting instructions specific to some sybtax constructions. And if you have unconditional break, continue or return in the with or try block, the code for normal leaving the block was removed, and the lineno setter was not able to determine when we jump in or out of the block. Until we radically change the algorithm of the lineno setter it is hard to implement a true CFG optimizer. The peepholer contains special exceptions for such codes, but the CFG optimizer is more aggressive. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TIJ3B2H4KLQZMBTAGN754YHTUGXY3J5F/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Summary of Python tracker Issues
ACTIVITY SUMMARY (2020-03-27 - 2020-04-03) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open7401 (+44) closed 44526 (+40) total 51927 (+84) Open issues with patches: 2909 Issues opened (65) == #23574: datetime: support leap seconds https://bugs.python.org/issue23574 reopened by vstinner #35455: Solaris thread_time doesn't work with current implementation https://bugs.python.org/issue35455 reopened by BTaskaya #37207: Use PEP 590 vectorcall to speed up calls to range(), list() an https://bugs.python.org/issue37207 reopened by petr.viktorin #39562: Asynchronous comprehensions don't work in asyncio REPL https://bugs.python.org/issue39562 reopened by vstinner #39951: Ignore specific errors when closing ssl connections https://bugs.python.org/issue39951 reopened by Dima.Tisnek #40068: test_threading: ThreadJoinOnShutdown.test_reinit_tls_after_for https://bugs.python.org/issue40068 reopened by vstinner #40093: ThreadPoolExecutor with wait=True shuts down too early https://bugs.python.org/issue40093 opened by fireattack #40094: Add os.waitstatus_to_exitcode() function https://bugs.python.org/issue40094 opened by vstinner #40095: Incorrect st_ino returned for ReFS on Windows 10 https://bugs.python.org/issue40095 opened by mbri...@gmail.com #40098: dir() does not return the list of valid attributes for the obj https://bugs.python.org/issue40098 opened by serhiy.storchaka #40100: IDLE: Undesired behavior of emoji in editor. https://bugs.python.org/issue40100 opened by Alex-Python-Programmer #40101: lib2to3 fails on default convert functionality https://bugs.python.org/issue40101 opened by BTaskaya #40102: Improve XLC support for function attributes https://bugs.python.org/issue40102 opened by BTaskaya #40103: ZipFile.extractall is not multiprocess safe with regard to dir https://bugs.python.org/issue40103 opened by nathansilberman #40105: Updating zip comment doesn't truncate the zip file https://bugs.python.org/issue40105 opened by nergall2 #40106: multiprocessor spawn https://bugs.python.org/issue40106 opened by mouse07410 #40107: pathlib: make `_Accessor.open()` return a file object and not https://bugs.python.org/issue40107 opened by barneygale #40110: multiprocessing.Pool.imap() should be lazy https://bugs.python.org/issue40110 opened by kousu #40111: Introspect ssl context: read ALPN and NPN protocols https://bugs.python.org/issue40111 opened by Dima.Tisnek #40113: Turtle demo https://bugs.python.org/issue40113 opened by Davide Golinelli #40116: Regression in memory use of shared key dictionaries for "compa https://bugs.python.org/issue40116 opened by Mark.Shannon #40118: os.stat in linux shows the wrong inode https://bugs.python.org/issue40118 opened by omer sela #40119: ensurepip should use different pattern for pip/setuptool wheel https://bugs.python.org/issue40119 opened by anthon #40120: Undefined C behavior going beyond end of struct via a [1] arra https://bugs.python.org/issue40120 opened by gregory.p.smith #40124: Clearer assertion error https://bugs.python.org/issue40124 opened by pgjones #40126: Incorrect error handling in unittest.mock https://bugs.python.org/issue40126 opened by bmclarnon #40127: Documentation of SSL library https://bugs.python.org/issue40127 opened by Christophe Nanteuil #40128: IDLE Show completions pop-up not working on macOS https://bugs.python.org/issue40128 opened by darthur90 #40129: Add test classes for custom __index__, __int__, __float__ and https://bugs.python.org/issue40129 opened by serhiy.storchaka #40132: Mechanism to control who owns package names on PyPI? https://bugs.python.org/issue40132 opened by ChrisRands #40133: Provide additional matchers for unittest.mock https://bugs.python.org/issue40133 opened by Flameeyes #40134: Inconsistent ANSI escape code handling on Windows 10 https://bugs.python.org/issue40134 opened by daverove #40135: multiprocessing: test_shared_memory_across_processes() cannot https://bugs.python.org/issue40135 opened by vstinner #40137: TODO list when PEP 573 "Module State Access from C Extension M https://bugs.python.org/issue40137 opened by vstinner #40138: Windows implementation of os.waitpid() truncates the exit stat https://bugs.python.org/issue40138 opened by vstinner #40139: mimetypes module racy https://bugs.python.org/issue40139 opened by ukl #40142: Modify _PyObject_GC_TRACK() to ensure that newly tracked objec https://bugs.python.org/issue40142 opened by vstinner #40143: shutil.rmtree will frequently fail on Windows under heavy load https://bugs.python.org/issue40143 opened by Alexander Riccio #40145: Pyshellext room for binary size improvement https://bugs.python.org/issue40145 opened by Alexander Riccio #40147: Move checking for duplicated keywords to the compiler
[Python-Dev] Re: Any thoughts about a control flow optimizer for CPython?
> I'm not familiar with it. Is there a summary online? I followed mostly the implementation here : https://github.com/v8/v8/tree/4b9b23521e6fd42373ebbcb20ebe03bf445494f9/src/compiler > started playing with moving the CFG optimizations from the peepholer My doubts have been on whether it was too radical to completely replace the peephole optimizer with this CFG optimization pass. If replacing the peephole optimizer is not a problem, how are we going to ensure compatibility for the related peephole optimizer C-API? Just have a stub? > I'd be happy to offer advice and review any PRs. My schedule may lighten this summer so I will try to look more but I wanted to know if we think its worth investigating first. If anyone wanted to make contributions to this, is such an implementation trivial enough to be just a PR? I guess what am trying to say; I hope if I one tried to open a PR, he/she wont be told, first write a PEP. Best, Joannah On Fri, Apr 3, 2020 at 1:15 PM Mark Shannon wrote: > Hi Joannah, > > On 03/04/2020 4:13 pm, joannah nanjekye wrote: > > Hey all, > > > > From my CS theory, a control flow graph models a program flow and one > > of its main characteristics is it has one entry and exit point. IIRC, > > CPython's CFG has multiple exit points, but that shouldn't matter. > > > CPython’s compilation process involves generation of a control flow > graph. > I started playing with moving the CFG optimizations from the peepholer > to their own pass back at the core sprints in London. > I've worked on it a bit since, but not for a while. > My branch is here: > > https://github.com/python/cpython/compare/master...markshannon:cfg-optimizer > > One thing that is blocking progress is the line-number table format. > There is no way to say that an instruction does not have a line number > attached to it. That makes handling and transforming of compiler > generated code (implicit `return None`s, for example) harder. > > See https://bugs.python.org/issue39537 > It's not an insurmountable obstacle, but it is something to be aware of. > > > > > Contrary to peephole optimizations, optimizations on the control flow > > graph are more global allowing us to have complex and global > > optimizations like branch and checkpoint eliminations etc. > > Checkpoints for tracing or concurrent GC are not something we need to > worry about. > > Because Python is so dynamic, there is little we can do in the way of > further optimizations in the bytecode compiler apart from control flow > improvements. > However, Python does have some fairly complex control flow, so there is > room for improvement. > > > > > > > I have seen several implementations of control flow optimizations. The > > one I am familiar with is the V8 control flow optimizer. > > I'm not familiar with it. Is there a summary online? > > > > > > > I tried to investigate this for one of my directed courses last fall but > > I want to know if there are people who have been thinking about this for > > CPython and what their thoughts are. > > There won't be much of a speed up, maybe 2 or 3%, but adding an explicit > CFG optimisation pass will improve maintainability as we won't need to > regenerate the CFG after bytecode layout. > > I'd be happy to offer advice and review any PRs. > > Cheers, > Mark. > > > > > -- > > //Best, > > Joannah Nanjekye > > /"You think you know when you learn, are more sure when you can write, > > even more when you can teach, but certain when you can program." > > Alan J. Perlis/ > > > > ___ > > Python-Dev mailing list -- python-dev@python.org > > To unsubscribe send an email to python-dev-le...@python.org > > https://mail.python.org/mailman3/lists/python-dev.python.org/ > > Message archived at > https://mail.python.org/archives/list/python-dev@python.org/message/I376QAAWUHVLO5WGFFNOBTZAUPAVZKCB/ > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > -- Best, Joannah Nanjekye *"You think you know when you learn, are more sure when you can write, even more when you can teach, but certain when you can program." Alan J. Perlis* ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7HPSFVHR64ELVRE2OXVNOAQED3WIHAYX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Any thoughts about a control flow optimizer for CPython?
Hi Joannah, On 03/04/2020 4:13 pm, joannah nanjekye wrote: Hey all, From my CS theory, a control flow graph models a program flow and one of its main characteristics is it has one entry and exit point. IIRC, CPython's CFG has multiple exit points, but that shouldn't matter. CPython’s compilation process involves generation of a control flow graph. I started playing with moving the CFG optimizations from the peepholer to their own pass back at the core sprints in London. I've worked on it a bit since, but not for a while. My branch is here: https://github.com/python/cpython/compare/master...markshannon:cfg-optimizer One thing that is blocking progress is the line-number table format. There is no way to say that an instruction does not have a line number attached to it. That makes handling and transforming of compiler generated code (implicit `return None`s, for example) harder. See https://bugs.python.org/issue39537 It's not an insurmountable obstacle, but it is something to be aware of. Contrary to peephole optimizations, optimizations on the control flow graph are more global allowing us to have complex and global optimizations like branch and checkpoint eliminations etc. Checkpoints for tracing or concurrent GC are not something we need to worry about. Because Python is so dynamic, there is little we can do in the way of further optimizations in the bytecode compiler apart from control flow improvements. However, Python does have some fairly complex control flow, so there is room for improvement. I have seen several implementations of control flow optimizations. The one I am familiar with is the V8 control flow optimizer. I'm not familiar with it. Is there a summary online? I tried to investigate this for one of my directed courses last fall but I want to know if there are people who have been thinking about this for CPython and what their thoughts are. There won't be much of a speed up, maybe 2 or 3%, but adding an explicit CFG optimisation pass will improve maintainability as we won't need to regenerate the CFG after bytecode layout. I'd be happy to offer advice and review any PRs. Cheers, Mark. -- //Best, Joannah Nanjekye /"You think you know when you learn, are more sure when you can write, even more when you can teach, but certain when you can program." Alan J. Perlis/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/I376QAAWUHVLO5WGFFNOBTZAUPAVZKCB/ Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/MHBRWBDDWDXOBXODITF35XEES47UZM5L/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Any thoughts about a control flow optimizer for CPython?
Hey all, >From my CS theory, a control flow graph models a program flow and one of its main characteristics is it has one entry and exit point. IIRC, CPython’s compilation process involves generation of a control flow graph. Contrary to peephole optimizations, optimizations on the control flow graph are more global allowing us to have complex and global optimizations like branch and checkpoint eliminations etc. I have seen several implementations of control flow optimizations. The one I am familiar with is the V8 control flow optimizer. I tried to investigate this for one of my directed courses last fall but I want to know if there are people who have been thinking about this for CPython and what their thoughts are. -- Best, Joannah Nanjekye *"You think you know when you learn, are more sure when you can write, even more when you can teach, but certain when you can program." Alan J. Perlis* ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/I376QAAWUHVLO5WGFFNOBTZAUPAVZKCB/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Problem with instantiating a C extension class from another class
Hello,
(I've asked this question before on python-list but only got scarce
and ultimately unhelpful answers. Although this isn't about Python
development itself, I'm assuming that there is more CPython
knowledge on this list than on the other, so please bear with me.)
I've written a C extension module, each defining a class
(called Series and CLHist, respectively). I can import both modules in
Python and use the classes. Everything works fine, also according to
valgrind.
Now I want to return a Series instance directly from a CLHist
method, and that immediately crashes Python with a segfault.
Here's a bit of code and just the _new, _init, and _finalize methods
which I've sprinkled with debugging code. When I instantiate a Series
object from Python, I see output from all three methods. When I call
pyseries_new() from C, none of the three functions are called and it
crashes.
At some point I thought maybe I should call PyType_Ready() from the C
side before instantiating Series. In a "standard" (system) Python
installation, it printed the desired debugging output and crashed
later. Under a debugging Python installation I built it outputs
nothing except an interesting post-mortem "Fatal Python error: UNREF
invalid object" (see below).
Here's my C code with the boilerplate stuff left out.
typedef struct {
PyObject_HEAD
struct clh_series *series;
} Series;
static PyObject *Series_new(PyTypeObject *type,
PyObject *args, PyObject *kw) {
Series *self;
self = (Series *) type->tp_alloc(type, 0);
fprintf(stderr, "Series_new(%p)\n", self);
self->series = NULL;
return (PyObject*)self;
}
static int Series_init(Series *self, PyObject *args, PyObject *kw) {
fprintf(stderr, "Series_init(%p)\n", self);
self->series = NULL;
return 0;
}
static void Series_finalize(PyObject *self) {
fprintf(stderr, "Series_finalize(%p)\n", self);
clh_series_free(((Series*)self)->series);
}
/* To create a new Series object directly from C */
PyObject *pyseries_new(struct clh_series *series) {
Series *pyseries;
pyseries = PyObject_New(Series, _type);
PyObject_Init((PyObject *)pyseries, _type);
pyseries->series = series;
return (PyObject *) pyseries;
}
Here's the debugging output:
* ob
object :
type: tuple
refcount: 0
address : 0x7f1a4e2f97a8
* op->_ob_prev->_ob_next
object : <_Series object at 0x7f1a4e2ee9d0>
type: _Series
refcount: 1
address : 0x7f1a4e2ee9d0
* op->_ob_next->_ob_prev
object :
type: tuple
refcount: 0
address : 0x7f1a4e2f97a8
Fatal Python error: UNREF invalid object
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/ILMRMQ64P7H6KHH2BZQPDYGXW773MOII/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 617: New PEG parser for CPython
>The only thing I'm missing from the PEP is more detail about how the > cross-language nature of the parser actions are handled. The example covers > just C, and the description of the actions says they're C expressions. The > only mention of Python code generation is for alternatives without actions. > Is the intent that the actions are cross-language, or translated to Python > somehow, or is the support for generating a Python-based parser merely for >. debugging, as that action suggests? Oh, good point. Thanks for pointing that out. We certainly need to explain that a bit better. The current situation is that actions support both Python and C code. They are basically pieces of code that will be included in the resulting program, no matter on what language is written in. For instance, we use the Python generator to generate the code that parses the grammar for the generator itself. The output is written in Python and the metagrammar uses actions written in Python: https://github.com/we-like-parsers/cpython/blob/pegen/Tools/peg_generator/pegen/metagrammar.gram So regarding the usage of Python code generation, is certainly useful for debugging but is actually used by the generator itself to bootstrap a section of it (the one that parses grammars). The feeling of bootstrapping parsers never gets old and is one of the most fun parts to do :) I will prepare a PR soon to complement the section about actions in the PEP. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/62FBT5KVUES5W4SXF22WAXXE2E32Y6WT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 617: New PEG parser for CPython
> That paragraph seems rather confused. I think what it might be > trying to say is that a PEG parser allows you to write productions > with overlapping first sets (which would be "ambiguous" for an > LL parser), but still somehow guarantees that a unique parse tree > is produced. The latter suggests that the grammar as a whole still > needs to be unambiguous. We may need to rephrase this to make it a bit more clear, but this is trying to say that PEG grammars cannot be ambiguous in the same sense as context-free grammars are normally said to be ambiguous. Notice that an ambiguous grammar is normally defined (for instance here https://en.wikipedia.org/wiki/Ambiguous_grammar) only for context-free grammars as a grammar with more than one possible parse tree. In the PEG formalism as Guido explained in the previous email there is only one possible parse tree because the parser always chooses the first option. As a consequence of this (and as a particular case of this) and as you mention, the PEG formalism allows writing productions with overlapping first sets. Also, notice that first sets are mainly relevant for LL(k) parsers and the like because those need to *deduce* which alternative to follow given multiple choices in production while PEG will always try in order. In general, the argument is that because of how PEG works, it will only be one parse tree and this makes the grammar "not ambiguous" under the typical definition for ambiguity for context-free grammars (having multiple parse trees). ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/4D3B2NM2JMV2UKIT6EV5Q2A6XK2HXDEH/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: PEP 617: New PEG parser for CPython
Thanks, Guido, Pablo, Lysandros, that's a great PEP. Also thanks to everyone else working on the PEG parser over the last year, like Emily. I know it's a lot of work but as someone who's intimately aware of the headaches caused by the LL(1) parser, I greatly appreciate it :). The only thing I'm missing from the PEP is more detail about how the cross-language nature of the parser actions are handled. The example covers just C, and the description of the actions says they're C expressions. The only mention of Python code generation is for alternatives without actions. Is the intent that the actions are cross-language, or translated to Python somehow, or is the support for generating a Python-based parser merely for debugging, as that action suggests? -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QCLQTDFCUYJWCZOUYKPYTN5DVUGAATFW/ Code of Conduct: http://python.org/psf/codeofconduct/
