[Python-Dev] Re: Set PyEval_SetTrace to method from class

2020-04-05 Thread Robert Collins 
Hi Leandro, it seems that your email is probably more about how to use the
existing PyEval_SetTrace API, not about developing or changing Python
itself. May I direct you to the "Got a Python problem or question?" section
from https://www.python.org/about/help/ web page, where you are likely to
get a good answer.

Cheers,
Rob

On Sun, 5 Apr 2020 at 15:09, Leandro Müller 
wrote:

> Hello.
>
> I need to set a PyEval_SetTrace with a new class instance.
> All examples use simple function.
> PyEval_SetTrace(trace_trampoline, obj);
>
> Following my problem:
> I start three threads, but I need to check trace C just one thread.
> If I run a while on C trace on thread, it stops all threads.
>
> So, I've been thinking to run Trace with a new instance class C.
>
> Any suggestions?
>
>
> Att.
>
> Leandro Müller
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/CDH7BTAYPJ6QRTXI32QRJVEPKWXEPYF4/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/ZJ32HUJP7T6MCZCZ2UP2GYON2PO37ROJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Any thoughts about a control flow optimizer for CPython?

2020-04-05 Thread Ned Batchelder 

On 4/3/20 11:13 AM, joannah nanjekye wrote:

Hey all,
From my CS theory, a control flow graph models a program flow and one 
of its main characteristics is it has one entry and exit point. IIRC, 
CPython’s compilation process involves generation of a control flow graph.


Contrary to peephole optimizations, optimizations on the  control flow 
graph are more global allowing us to have complex and global 
optimizations like branch and checkpoint eliminations etc.



I have seen several implementations of control flow optimizations. The 
one I am familiar with is the V8 control flow optimizer.



I tried to investigate this for one of my directed courses last fall 
but I want to know if there are people who have been thinking about 
this for CPython and what their thoughts are.




Please make it possible to disable any optimizations.  Sometimes 
programs are run to understand the program, not for speed (for example, 
during debugging, or coverage measurement.)


--Ned.


--
//Best,
Joannah Nanjekye
/"You think you know when you learn, are more sure when you can write, 
even more when you can teach, but certain when you can program."

Alan J. Perlis/

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/I376QAAWUHVLO5WGFFNOBTZAUPAVZKCB/
Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/UGEVQQWB32DQHBONJKTL2Q5K2YPCPIN2/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-05 Thread Jelle Zijlstra 
El jue., 2 abr. 2020 a las 11:19, Guido van Rossum ()
escribió:

> Since last fall's core sprint in London, Pablo Galindo Salgado, Lysandros
> Nikolaou and myself have been working on a new parser for CPython. We are
> now far enough along that we present a PEP we've written:
>
> https://www.python.org/dev/peps/pep-0617/
>
> Hopefully the PEP speaks for itself. We are hoping for a speedy resolution
> so we can land the code we've written before 3.9 beta 1.
>
> The current CPython parser usually just produces "SyntaxError: invalid
syntax" for any error, while other languages that I work with usually say
something more precise like 'expected x, got y'. What will the error
messages in the PEG parser look like? Making syntax errors more informative
can be a nice improvement to usability.


> If people insist I can post a copy of the entire PEP here on the list, but
> since a lot of it is just background information on the old LL(1) and the
> new PEG parsing algorithms, I figure I'd spare everyone the need of reading
> through that. Below is a copy of the most relevant section from the PEP.
> I'd also like to point out the section on performance (which you can find
> through the above link) -- basically performance is on a par with that of
> the old parser.
>
> ==
> Migration plan
> ==
>
> This section describes the migration plan when porting to the new
> PEG-based parser
> if this PEP is accepted. The migration will be executed in a series of
> steps that allow
> initially to fallback to the previous parser if needed:
>
> 1.  Before Python 3.9 beta 1, include the new PEG-based parser machinery
> in CPython
> with a command-line flag and environment variable that allows
> switching between
> the new and the old parsers together with explicit APIs that allow
> invoking the
> new and the old parsers independently. At this step, all Python APIs
> like ``ast.parse``
> and ``compile`` will use the parser set by the flags or the
> environment variable and
> the default parser will be the current parser.
>
> 2.  After Python 3.9 Beta 1 the default parser will be the new parser.
>
> 3.  Between Python 3.9 and Python 3.10, the old parser and related code
> (like the
> "parser" module) will be kept until a new Python release happens
> (Python 3.10). In
> the meanwhile and until the old parser is removed, **no new Python
> Grammar
> addition will be added that requires the peg parser**. This means that
> the grammar
> will be kept LL(1) until the old parser is removed.
>
> 4.  In Python 3.10, remove the old parser, the command-line flag, the
> environment
> variable and the "parser" module and related code.
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> 
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/HOZ2RI3FXUEMAT4XAX4UHFN4PKG5J5GR/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/SRVOVLZWEMZX4LSUXFDZCQO6CV27QEGO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-05 Thread Greg Ewing 

On 6/04/20 2:08 am, Jelle Zijlstra wrote:
The current CPython parser usually just produces "SyntaxError: invalid 
syntax" for any error, while other languages that I work with usually 
say something more precise like 'expected x, got y'. What will the error 
messages in the PEG parser look like? Making syntax errors more 
informative can be a nice improvement to usability.


And related to that, how precisely will it be able to pinpoint the
location of the error? The backtracking worries me a bit in that
regard. I can imagine it trying all possible ways to parse the
input and then only being able to say "Something is wrong somewhere
in this file."

--
Greg
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/AW3FIDN6BTD2UANVBPT4LO76PALKVLJN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-05 Thread Guido van Rossum 
> On 6/04/20 2:08 am, Jelle Zijlstra wrote:
> > The current CPython parser usually just produces "SyntaxError: invalid
> > syntax" for any error, while other languages that I work with usually
> > say something more precise like 'expected x, got y'. What will the error
> > messages in the PEG parser look like? Making syntax errors more
> > informative can be a nice improvement to usability.
>

Unfortunately they look pretty much the same. We're actually currently
trying to improve the error messages for situations where the old parser
produces something specialized (mostly because the LL(1) grammar can't
express something and the check is done in a later pass).


> On Sun, Apr 5, 2020 at 7:55 AM Greg Ewing 
> wrote:
> And related to that, how precisely will it be able to pinpoint the
> location of the error? The backtracking worries me a bit in that
> regard. I can imagine it trying all possible ways to parse the
> input and then only being able to say "Something is wrong somewhere
> in this file."
>

There's no need to worry about this: in almost all cases the error
indicator points to the same spot in the source code as with the old
parser. I was worried about this too, but it really doesn't seem to be a
problem -- I think this might be different with highly ambiguous grammars,
but since Python's grammar is still *mostly* LL(1), it looks like we're
fine.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/CSFQGEYL6NVWRWQLJ3B5AXKV3ZAFWA65/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How to enable tracemalloc for the test suite?

2020-04-05 Thread Victor Stinner 
Le dim. 5 avr. 2020 à 04:02, Skip Montanaro  a écrit :
> The command is
>
> % PYTHONTRACEMALLOC=5 ./python ./Tools/scripts/run_tests.py -R
> 5:50:reflog.txt test_rattlesnake
> (...)
> FWIW, the register branch of my CPython fork:
>
> https://github.com/smontanaro/cpython/tree/register

gdb traceback:

(gdb) where
(...)
#2  0x0056671d in fatal_error_exit (status=-1) at
Python/pylifecycle.c:2183
(...)
#8  0x0047d61e in PyObject_Free (ptr=0x7d3e40
<_Py_FalseStruct>) at Objects/obmalloc.c:709
#9  0x00498a9b in object_dealloc (self=False) at
Objects/typeobject.c:3824
#10 0x0047c5aa in _Py_Dealloc (op=False) at Objects/object.c:2206
#11 0x005f771c in _Py_DECREF (filename=0x741e7d
"./Modules/_io/iobase.c", lineno=275, op=False) at
./Include/object.h:425
#12 0x005f7f16 in iobase_finalize (self=<_io.TextIOWrapper at
remote 0x7fffeab2a6e0>) at ./Modules/_io/iobase.c:275
#13 0x00477755 in PyObject_CallFinalizer
(self=<_io.TextIOWrapper at remote 0x7fffeab2a6e0>) at
Objects/object.c:193
#14 0x004777ea in PyObject_CallFinalizerFromDealloc
(self=<_io.TextIOWrapper at remote 0x7fffeab2a6e0>) at
Objects/object.c:211
#15 0x005f7fd8 in _PyIOBase_finalize (self=<_io.TextIOWrapper
at remote 0x7fffeab2a6e0>) at ./Modules/_io/iobase.c:319
#16 0x00609533 in textiowrapper_dealloc (self=0x7fffeab2a6e0)
at ./Modules/_io/textio.c:1433
(...)
#48 0x00595fa7 in delete_garbage (tstate=0x82dc80,
gcstate=0x82c648, collectable=0x7fffd2c0, old=0x82c690) at
Modules/gcmodule.c:1005
#49 0x005965c8 in collect (tstate=0x82dc80, generation=2,
n_collected=0x0, n_uncollectable=0x0, nofail=1) at
Modules/gcmodule.c:1275
#50 0x005980e7 in _PyGC_CollectNoFail () at Modules/gcmodule.c:2099
#51 0x00546da8 in _PyImport_Cleanup (tstate=0x82dc80) at
Python/import.c:633
#52 0x00564f89 in Py_FinalizeEx () at Python/pylifecycle.c:1416
#53 0x0041ed1b in Py_RunMain () at Modules/main.c:634
#54 0x0041ed90 in pymain_main (args=0x7fffd500) at
Modules/main.c:662
#55 0x0041ee0a in Py_BytesMain (argc=6, argv=0x7fffd628)
at Modules/main.c:686
#56 0x0041d786 in main (argc=6, argv=0x7fffd628) at
./Programs/python.c:16

(gdb) p *res
$6 = {
  ob_refcnt = 0,
  ob_type = 0x7d3ca0 
}

(gdb) p *(PyLongObject*)res
$7 = {
  ob_base = {
ob_base = {
  ob_refcnt = 0,
  ob_type = 0x7d3ca0 
},
ob_size = 0
  },
  ob_digit = {0}
}


(gdb) p res == ((PyObject *) &_Py_FalseStruct)
$8 = 1


Py_FinalizeEx() cleans up all modules with _PyImport_Cleanup(). When
Python destroys a _io.TextIOWrapper instance, Py_False is destroyed...

Py_False must never me deallocated: it's statically allocated.
Py_False reference counter should not decrease to 0.

It seems like your Python changes use Py_False "somewhere" without
Py_INCREF(Py_False).

--

It seems like you are working on a register-based bytecode, cool. I
recall that you told me something like that ;-)

It seems like executing the register-based bytecode of the following
function is enough to steal one reference to False (use False without
INCREF):

def _long_block(s, b):
if s > b:
return s
return b

Maybe it's COMPARE_OP_REG() which calls SETLOCAL(dst, False).

Maybe it's JUMP_IF_FALSE_REG() which has suspicious code: DECREF
without clearing GETLOCAL(src).

PyObject *cond = GETLOCAL(src);
..
Py_DECREF(cond);

Maybe SETLOCAL(src, NULL); is need somewhere.

Maybe it's something else.

Replacing stack-based bytecode with register-based bytecode requires
to rethink the lifetime of registers... I had a hard time to fix my
old "registervm" project to fix the register lifetime: I added
CLEAR_REG bytecode to explicitly clear a register. Using a stack, all
"CLEAR_REG" operation are implicit. You have to make them explicit.
Hopefully, a compiler can easily reuse registers and remove most
CLEAR_REG.

--

My old project (2012, OMG, I'm getting old!):
https://faster-cpython.readthedocs.io/registervm.html
https://hg.python.org/sandbox/registervm/file/tip/REGISTERVM.txt

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/OJSBGOQMYQYCCICE2GOHER2GTIJLWNRJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-05 Thread Pablo Galindo Salgado 
> The only thing I'm missing from the PEP is more detail about how the
cross-language nature of the parser actions are handled.

Expanded the "actions" section in the PEP here: 
https://github.com/python/peps/pull/1357
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/EJMHASPUOAW7R2BKJCCVI4BGQRLN3ZRX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-05 Thread Guido van Rossum 
The tl;dr is that actions specified in the grammar are specific to the
target language. So if you want to use the pegen tool to generate both
Python and C code for the same grammar, you would need two grammar files
with the same grammar but different actions. Since our goal here is just to
generate a parser for use in CPython that's not a problem. Other PEG parser
generators make different choices, e.g. TatSu puts semantics actions in a
separate file (https://tatsu.readthedocs.io/en/stable/semantics.html).

On Sun, Apr 5, 2020 at 11:06 AM Pablo Galindo Salgado 
wrote:

> > The only thing I'm missing from the PEP is more detail about how the
> cross-language nature of the parser actions are handled.
>
> Expanded the "actions" section in the PEP here:
> https://github.com/python/peps/pull/1357
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/EJMHASPUOAW7R2BKJCCVI4BGQRLN3ZRX/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/UHGCXQGQYPJIOM34VEEMV74OPNBXNNW5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How to enable tracemalloc for the test suite?

2020-04-05 Thread Skip Montanaro 
> It seems like your Python changes use Py_False "somewhere" without
> Py_INCREF(Py_False).

> Maybe it's COMPARE_OP_REG() which calls SETLOCAL(dst, False).

Yes, this was the problem. Thanks for the fix. Too much blind
adherence on my part to the existing COMPARE_OP logic. I've even
written (relatively speaking) tomes about it in both my in-progress
PEP as well as in various comments throughout the code. I don't think
I had all that sorted out in my mind before implementing the first few
instructions.

Fortunately, I'm not too far into implementing the actual
instructions. I should be able to easily go back and desk check the
others.

> Replacing stack-based bytecode with register-based bytecode requires
> to rethink the lifetime of registers... I had a hard time to fix my
> old "registervm" project to fix the register lifetime: I added
> CLEAR_REG bytecode to explicitly clear a register. Using a stack, all
> "CLEAR_REG" operation are implicit. You have to make them explicit.
> Hopefully, a compiler can easily reuse registers and remove most
> CLEAR_REG.

I'm trying it the simplest way I can think of. Registers are exactly
like local variables, so SETLOCAL Py_XDECREFs whatever is already
there before overwriting it with a new value. At the end of
_PyEval_EvalFrameDefault if the code object's co_flags includes the
(new) CO_REGISTER flag, it loops over the stack/register space calling
Py_CLEAR. The stack/register space is also Py_CLEAR'd when the frame
is first allocated.

Skip
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/OD2ZNQRVDN652JZAPPFYJV67KRXHIMTH/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Problem with instantiating a C extension class from another class

2020-04-05 Thread Nick Coghlan 
Hi Musbur,

While python-dev is specifically for core development, the "specific
interest group" mailing lists are for both change proposals and existing
usage questions.

I've cc'ed capi-sig on this reply.

Cheers,
Nick.

On Sat., 4 Apr. 2020, 2:02 am Musbur,  wrote:

> Hello,
>
> (I've asked this question before on python-list but only got scarce
> and ultimately unhelpful answers. Although this isn't about Python
> development itself, I'm assuming that there is more CPython
> knowledge on this list than on the other, so please bear with me.)
>
> I've written a C extension module, each defining a class
> (called Series and CLHist, respectively). I can import both modules in
> Python and use the classes. Everything works fine, also according to
> valgrind.
>
> Now I want to return a Series instance directly from a CLHist
> method, and that immediately crashes Python with a segfault.
>
> Here's a bit of code and just the _new, _init, and _finalize methods
> which I've sprinkled with debugging code. When I instantiate a Series
> object from Python, I see output from all three methods. When I call
> pyseries_new() from C, none of the three functions are called and it
> crashes.
>
> At some point I thought maybe I should call PyType_Ready() from the C
> side before instantiating Series. In a "standard" (system) Python
> installation, it printed the desired debugging output and crashed
> later. Under a debugging Python installation I built it outputs
> nothing except an interesting post-mortem "Fatal Python error: UNREF
> invalid object" (see below).
>
> Here's my C code with the boilerplate stuff left out.
>
> typedef struct {
> PyObject_HEAD
> struct clh_series *series;
> } Series;
>
> static PyObject *Series_new(PyTypeObject *type,
> PyObject *args, PyObject *kw) {
> Series *self;
>
> self = (Series *) type->tp_alloc(type, 0);
> fprintf(stderr, "Series_new(%p)\n", self);
> self->series = NULL;
> return (PyObject*)self;
> }
>
> static int Series_init(Series *self, PyObject *args, PyObject *kw) {
> fprintf(stderr, "Series_init(%p)\n", self);
> self->series = NULL;
> return 0;
> }
>
> static void Series_finalize(PyObject *self) {
> fprintf(stderr, "Series_finalize(%p)\n", self);
> clh_series_free(((Series*)self)->series);
> }
>
> /* To create a new Series object directly from C */
> PyObject *pyseries_new(struct clh_series *series) {
> Series *pyseries;
>
> pyseries = PyObject_New(Series, &series_type);
> PyObject_Init((PyObject *)pyseries, &series_type);
> pyseries->series = series;
> return (PyObject *) pyseries;
> }
>
> Here's the debugging output:
>
> * ob
> object  : 
> type: tuple
> refcount: 0
> address : 0x7f1a4e2f97a8
> * op->_ob_prev->_ob_next
> object  : <_Series object at 0x7f1a4e2ee9d0>
> type: _Series
> refcount: 1
> address : 0x7f1a4e2ee9d0
> * op->_ob_next->_ob_prev
> object  : 
> type: tuple
> refcount: 0
> address : 0x7f1a4e2f97a8
> Fatal Python error: UNREF invalid object
> ___
> Python-Dev mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/[email protected]/message/ILMRMQ64P7H6KHH2BZQPDYGXW773MOII/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/PLSKLZQYP64CG6F7DR3VZBXOFQ3YA3A6/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-05 Thread Greg Ewing 

On 6/04/20 4:48 am, Guido van Rossum wrote:
There's no need to worry about this: in almost all cases the error 
indicator points to the same spot in the source code as with the old 
parser.


I'm curious about how that works. From the description in the PEP,
it seems that none of the individual parsing functions can report
an error, because there might be another branch higher up that
succeeds. Does it keep track of the maximum distance it got through
the source or something like that?

--
Greg
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/THY5LIBHBSVZTUYMGZUYJDDXRASU4T6Z/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: How to enable tracemalloc for the test suite?

2020-04-05 Thread Tim Peters 
For posterity, just recording best guesses for the other mysteries left hanging:

- PYTHONTRACEMALLOC didn't work for you because Victor's traceback
showed that Py_FinalizeEx was executing _PyImport_Fini,, one statement
_after_ it disabled tracemalloc via _PyTraceMalloc_Fini.

- The address passed to free() was for the struct representing the
False singleton, which is static   Since it wasn't obtained from a
malloc/realloc call to begin with, the free() debug code didn't find
anything it expected to find.  As noted earlier, if tracemalloc had
still been active, it wouldn't have found that address in its
database, and so would have produced no output at all.

So - that was easy ;-)
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/BRPNFTDWNJKVZS3KUKWYPN46KGER6LDU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 617: New PEG parser for CPython

2020-04-05 Thread Guido van Rossum 
On Sun, Apr 5, 2020 at 5:16 PM Greg Ewing 
wrote:

> On 6/04/20 4:48 am, Guido van Rossum wrote:
> > There's no need to worry about this: in almost all cases the error
> > indicator points to the same spot in the source code as with the old
> > parser.
>
> I'm curious about how that works. From the description in the PEP,
> it seems that none of the individual parsing functions can report
> an error, because there might be another branch higher up that
> succeeds. Does it keep track of the maximum distance it got through
> the source or something like that?
>

I guess you could call it that. There is a small layer of abstraction
between the actual tokenizer (which cannot go back) and the generated
parser functions. This abstraction buffers tokens. When a parser function
wants a token it calls into this abstraction, and that either satisfies it
from its buffer, or if there is no lookahead in the buffer left, calls the
actual tokenizer. When a parser function fails, it calls into the
abstraction layer to back up to a previous point (which I call the "mark").

(A simplified version of this layer is shown in my blog post,
https://medium.com/@gvanrossum_83706/building-a-peg-parser-d4869b5958fb --
the class Tokenizer.)

When an error bubbles all the way up, we report a SyntaxError pointing to
the farthest token that the abstraction has buffered (self.pos in the blog
post).

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/U47A3SALOBQMWTVBPHDFD5OXCYXF7QSY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Question about bytecode stability

2020-04-05 Thread Jonathan Goble 
I'm getting ready to get back into an old personal project with some new
ideas. Before I do, I need to ask a question regarding the stability of
CPython bytecode. Before you say it, I am well aware and fully understand
that the opcodes may change in value and semantics between 3.x and 3.(x+1)
without notice, and that's acceptable to me.

My question is, are the opcodes guaranteed stable across the lifetime of a
single 3.x release? In other words, are they guaranteed to not change
values or semantics between 3.x.y and 3.x.(y+1)? Reading through the list
of opcodes in the dis documentation, it seems that all changes have
occurred in 3.x.0, so it seems the answer would be yes, but then the
"CPython implementation detail" paragraph at the top doesn't specify that
and is a little vague on whether that's true or not.

(For context, my personal project is a BASIC implementation in Python that
would compile to Python bytecode.)
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/Z6RRM4WQAKBKMLPBCOMB47NDPEROVTTZ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Question about bytecode stability

2020-04-05 Thread Chris Angelico 
On Mon, Apr 6, 2020 at 1:53 PM Jonathan Goble  wrote:
>
> I'm getting ready to get back into an old personal project with some new 
> ideas. Before I do, I need to ask a question regarding the stability of 
> CPython bytecode. Before you say it, I am well aware and fully understand 
> that the opcodes may change in value and semantics between 3.x and 3.(x+1) 
> without notice, and that's acceptable to me.
>
> My question is, are the opcodes guaranteed stable across the lifetime of a 
> single 3.x release? In other words, are they guaranteed to not change values 
> or semantics between 3.x.y and 3.x.(y+1)? Reading through the list of opcodes 
> in the dis documentation, it seems that all changes have occurred in 3.x.0, 
> so it seems the answer would be yes, but then the "CPython implementation 
> detail" paragraph at the top doesn't specify that and is a little vague on 
> whether that's true or not.
>

The best way to look at this is to consider how long a .pyc file is
valid. They're currently named something like
__pycache__/modulename.cpython-38.pyc which is a fairly clear
indication that the cached compiled module should be valid for any
CPython 3.8.x release. So yes, you should be safe within any given
release. The reason it's a CPython implementation detail is also
hinted at in the file name: if you were to use Jython or PyPy or
IronPython or MicroPython or any other implementation, you can't be
sure your bytecodes will work, even if the version number is the same.
(An interpreter can be compatible with Python 3.8 without having the
same bytecode as CPython 3.8.)

So for your purposes, it sounds like, yes, this WILL work. :)

ChrisA
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/ICZUG7FBLAVUTCGSAZMFDS2UDHMKY7MR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Question about bytecode stability

2020-04-05 Thread Jonathan Goble 
On Mon, Apr 6, 2020 at 12:02 AM Chris Angelico  wrote:

> On Mon, Apr 6, 2020 at 1:53 PM Jonathan Goble  wrote:
> >
> > I'm getting ready to get back into an old personal project with some new
> ideas. Before I do, I need to ask a question regarding the stability of
> CPython bytecode. Before you say it, I am well aware and fully understand
> that the opcodes may change in value and semantics between 3.x and 3.(x+1)
> without notice, and that's acceptable to me.
> >
> > My question is, are the opcodes guaranteed stable across the lifetime of
> a single 3.x release? In other words, are they guaranteed to not change
> values or semantics between 3.x.y and 3.x.(y+1)? Reading through the list
> of opcodes in the dis documentation, it seems that all changes have
> occurred in 3.x.0, so it seems the answer would be yes, but then the
> "CPython implementation detail" paragraph at the top doesn't specify that
> and is a little vague on whether that's true or not.
> >
>
> The best way to look at this is to consider how long a .pyc file is
> valid. They're currently named something like
> __pycache__/modulename.cpython-38.pyc which is a fairly clear
> indication that the cached compiled module should be valid for any
> CPython 3.8.x release. So yes, you should be safe within any given
> release. The reason it's a CPython implementation detail is also
> hinted at in the file name: if you were to use Jython or PyPy or
> IronPython or MicroPython or any other implementation, you can't be
> sure your bytecodes will work, even if the version number is the same.
> (An interpreter can be compatible with Python 3.8 without having the
> same bytecode as CPython 3.8.)
>
> So for your purposes, it sounds like, yes, this WILL work. :)
>
> ChrisA
>

Thanks, I never really looked closely at .pyc files, so I missed that. Good
to know that I only have to watch for changes between feature releases.

I knew that about "CPython implementation detail", that it wouldn't be
valid on non-CPython implementations, but didn't clearly state so in the
OP. Regardless, I appreciate the reminder.
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/O6MPYVBVDXDV3SNAYNNGVGQHDTMNHAQM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Question about bytecode stability

2020-04-05 Thread Brandt Bucher 
> The best way to look at this is to consider how long a .pyc file is valid. 
> They're currently named something like __pycache__/modulename.cpython-38.pyc 
> which is a fairly clear indication that the cached compiled module should be 
> valid for any CPython 3.8.x release.

Perhaps an even better indicator is the “magic number” that invalidates the 
cached bytecode. This can change several times during the alpha and beta 
phases, and a (mostly complete) record of changes can be found here:

https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap_external.py#L147-L284
___
Python-Dev mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/[email protected]/message/NKNIMQCET2OKNUY6MCYMBFYVGJY7ZN7L/
Code of Conduct: http://python.org/psf/codeofconduct/