Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
>
>
> https://refi64.com/posts/the-magic-of-rpython.html
>
> Note that this was written back when I used "like" and "really" in every
> sentence, and when I used to think that Python copied tuples (don't ask).
>
Thanks! And thanks for making the RPython language a bit more explicit for the
rest of us.
Reading your post reminded me that having nested lists of lists of ...
eventually strings (not all at the same nesting level) already made it
difficult. I don't remember if I stopped trying there or a bit more further
along.
asrp
> On Feb 3, 2018 at 10:05 AM, > wrote:
>
> > Are you aware of pypy?
> >
>
> Yes, but I do not know it well. It seems to be more focused on JIT
> compiling with many of its functions written for special cases and I
> did not find a way to make experimental changes easily.
>
> I even tried (briefly) to have the main loop of this interpreter run
> in RPython but didn't find enough information on which subset of
> Python does RPython handle.
>
>
> > Can you give an example of how you would do that? I don't mean the
> > mechanism used, I mean how would a developer implement a new syntactic
> > feature. Suppose I wanted to add a new clause to for...else, let's say:
> >
> > for ... :
> >block
> > otherwise:
> ># runs only if the for-loop was empty
> >
> > How would do I do that?
> >
>
> Here's one way to do that. Sorry for the lack of doc for the moment.
>
> To alter the language's grammar, you'd create a string with the new
> grammar, parse it and set the result to pyterp.parser. Then, depending
> on how the grammar was changed (which affects what AST is created),
> you'd change the corresponding function to handle the new semantics.
>
> I didn't quite understand what you mean by "was empty". However, I
> noticed that "else" blocks after for are ignored in the current
> implement. Here's an example session adding "else".
>
> If you tell me a bit more about the intended behaviour of "otherwise",
> I'd be happy to do an example with that clause.
>
> $ ipython -i test/python_repl.py
> p>> simport simple_ast
> p>> for i in [1, 2]:
> ... print i
> ...
> 1
> 2
> p>> for i in [1, 2, 3]:
> ... print i
> ... else:
> ... print 100
> ...
> 1
> 2
> 3
> p>> ^D
>
> In the above for loop, the else block is ignored! I press Control-D to
> exist the interpreter.
>
> In [1]: grammar = python_grammar.full_definition + python_grammar.extra
>
> In [2]: grammar += r"""
>...: for_stmt = "for" {exprlist} "in" {testlist} ":" {suite}
> {(SAME_INDENT "else" ":" {suite}) | void=pass_stmt}
>...: """
>
> In [3]: pyterp.parser = python.Interpreter(i3.parse("grammar", grammar))
>
> In [4]: pyterp.repl()
>
> Now edit for_stmt in simple_ast.py
>
> except StopIteration:
> +evaluate(else_block)
> return
>
> (If else_block was not already an (ignored) parameter, we'd have to
> change the `def for_stmt...` line too.) And run in the REPL we just
> restarted.
>
> p>> simple_ast.reload_module(simple_ast)
> p>> for i in [1, 2, 3]:
> ... print i
> ... else:
> ... print 100
> ...
> 1
> 2
> 3
> 100
> p>> for i in [1, 2, 3]:
> ... print i
> ... break
> ... else:
> ... print 100
> ...
> 1
> p>>
>
>
>
> Some notes:
>
> 1. I'm using the host Python interpreter to change the grammar here
> but that's not strictly necessary if we expose pyterp itself in the
> global scope.
> 2. I'm editing simple_ast.py and reloading because its less changes
> but redefining a new function and setting simple_ast.for_stmt (in
> other words, monkey-patching the change) should also work.
>
> If we wanted an "otherwise" clause on top of an "else" clause, we
> probably want to make some provision for the runtime to distinguish
> between the two so I'd edit the grammar with something like
>
> for_stmt = "for" {exprlist} "in" {testlist} ":" {suite}
> {((SAME_INDENT "else" ":" {suite}) | void=pass_stmt)=else_block
> ((SAME_INDENT "otherwise" ":" {suite}) |
> void=pass_stmt)=otherwise_block}
>
> (In this case, "otherwise" blocks have to come after the "else" block
> if both are present.)
>
> asrp
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
> Message-ID: <20180206034013.gz26...@ando.pearwood.info>
>
> On Sat, Feb 03, 2018 at 11:45:15AM +0100, asrp wrote:
>
> > > Can you give an example of how you would do that? I don't mean the
> > > mechanism used, I mean how would a developer implement a new syntactic
> > > feature. Suppose I wanted to add a new clause to for...else, let's say:
> > >
> > > for ... :
> > >block
> > > otherwise:
> > ># runs only if the for-loop was empty
> > >
> > > How would do I do that?
> [...]
> > If you tell me a bit more about the intended behaviour of "otherwise",
> > I'd be happy to do an example with that clause.
>
>
> Here's a faked session showing the sort of thing I am referring to.
> (Note that this is just an example, not a proposal for a new language
> feature.)
>
> for x in [1, 2, 3]:
> print(x)
> otherwise:
> print("nothing there")
>
>
> prints 1, 2, 3.
>
> for x in []:
> print(x)
> otherwise:
> print("nothing there")
>
> prints "nothing there". In other words, the otherwise block runs if, and
> only if, the loop iterable is empty and the for block does NOT run.
>
> Can you do something like that?
>
Oh, I see. Yes, definitely. This time I'll change for_stmt in lib/simple_ast.py
beforehand (but runtime reload also works).
def for_stmt(index_var, iterable, block, else_block, otherwise_block):
iterator = iter(evaluate(iterable))
try:
assignment(index_var, iterator.next())
except StopIteration:
evaluate(otherwise_block)
return
while_true:
__caller__['__continue__'] = __continue__
__caller__['__break__'] = __break__
evaluate(block)
try:
assignment(index_var, iterator.next())
except StopIteration:
evaluate(else_block)
return
Then start the interpreter
$ ipython -i test/python_repl.py
p>> simport simple_ast
p>> ^D
[...]
In [1]: grammar = python_grammar.full_definition + python_grammar.extra
In [2]: grammar += r"""
...: for_stmt = "for" {exprlist} "in" {testlist} ":" {suite} {((SAME_INDENT
"else" ":" {suite}) | void=pass_stmt) ((SAME_INDENT "otherwise" ":" {suite}) |
void=pass_stmt)}
...: """
In [3]: pyterp.parser = python.Interpreter(i3.parse("grammar", grammar))
In [4]: pyterp.repl()
p>> for x in [1, 2]:
... print(x)
... otherwise:
... print("Nothing there")
...
1
2
p>> for x in []:
... print(x)
... otherwise:
... print("Nothing there")
...
Nothing there
p>> for x in [1, 2]:
... print(x)
... else:
... print("Something there")
... otherwise:
... print("Nothing there")
...
1
2
Something there
I noticed since my last post that the else and otherwise blocks don't need to
be named if they are positional in for_stmt. The grammar change here reflects
that.
I've also posted an example with `until_stmt` and I talk a bit more about
debugging changes made there.
http://blog.asrpo.com/adding_new_statement
(Nothing here is a proposed language change, just demos.)
asrp
>
>
> --
> Steve
>
>
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
Oh and I forgot to mention that since everything is mostly done at run time, there could just be different modules for (slightly) different languages. I don't know if that'd be problematic for maintenance though. asrp > Sent: Saturday, February 03, 2018 at 3:17 PM > From: asrp <a...@email.com> > To: "Christian Tismer" <tis...@stackless.com> > Cc: python-dev@python.org > Subject: Re: [Python-Dev] A minimal Python interpreter written in Python for > experimenting with language changes > > Hi, > > (My other reply to Alex and Steven doesn't seems to appear, at least not on > pipermail so I hope these don't come in the wrong order.) > > > In the readme to python_terp you say: > > > > """ > > python_terp is intended to make language modification to Python easier > > to preview changes more quickly and is not intended for full CPython > > compatibility. However, a large subset of Python is already included. In > > particular, enough to run the first stage of its parser. > > """ > > > > This needs clarification. > > What do you mean by subset? A real subset or also things that > > are different and will stay different? > > To what extent are you planning to stay compatible, and where > > do you plan to deviate? > > > > The reason that I'm asking is that by compatible I mean the > > compatibility of PyPy. If you can reach that, and be it just > > by a subset, then it makes sense to speak of Python. > > > > Good questions! There are two part to what I meant by that passage. > > One is a reflection of the current state of the interpreter and its > capabilities. I put an example instead listing all differences because its > easier to update but admittedly much less precise. > > The other concerns the intention for the future. My mind is not completely > made but here's what I currently think it should/could be. > > First, fidelity would be determined by how the interpreter is used (if its > used). More (potentially) drastic changes in the beginning than later on. > Hopefully the deviations from Python (if any) for this reason does not grow > too much from this. But since its for testing language changes (to Python), I > don't think it would stray too much, if any. > > Second, I'd like to keep the whole thing small (of low complexity) as much as > possible so in some cases, the less featureful choice will be kept. For > example, in an earlier version, I had more functions parameters handling > (defaults, *args, **kwargs, etc) but remove it because just setting up the > initial local scope from the parameters passed and function definition took > more than 100 lines. (Maybe I just went about this the wrong way...) > > However, except for easier or simpler implementation of the interpreter or > some of the initial AST nodes, I don't intend to make any additions not in > the Python language. One example of this is simple_for and single_if > statements which I explain a bit in this old post: > > http://blog.asrpo.com/bootstrap_chicken_or_egg > > Although those are mainly intended to bootstrap the other statements like > if_stmt and for_stmt and not used again later. > > (Well, there might be some difference for class method lookup (late-binding > instead of early-binding) that's needed for edit-and-continue. But the > intention would be to list all of those differences.) > > Feel free to ask again if I missed some aspect of your questions here. > > asrp > > > Sent: Saturday, February 03, 2018 at 12:29 PM > > From: "Christian Tismer" <tis...@stackless.com> > > To: "asrp asrp" <a...@email.com>, python-dev@python.org > > Subject: Re: [Python-Dev] A minimal Python interpreter written in Python > > for experimenting with language changes > > > > Hi user with no real name yet, > > > > On 03.02.18 01:01, asrp asrp wrote: > > > Hello, > > > > > > I don't know if this is the right place to post this. Please redirect as > > > needed. > > > > > > I've made a small Python interpreter in Python with runtime AST node > > > semantics and edit-and-continue. I thought it could make prototyping > > > language changes more easily and visualize usage before writing them in C. > > > > > > Its here: https://github.com/asrp/python_terp > > > > > > So, for example, redefining the for_stmt function in the right scope > > > changes the behaviour of future for loops at runtime. > > > > > > Although from discussion I've read in PEPs, actual implementati
Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
> Are you aware of pypy?
>
Yes, but I do not know it well. It seems to be more focused on JIT compiling
with many of its functions written for special cases and I did not find a way
to make experimental changes easily.
I even tried (briefly) to have the main loop of this interpreter run in RPython
but didn't find enough information on which subset of Python does RPython
handle.
> Can you give an example of how you would do that? I don't mean the
> mechanism used, I mean how would a developer implement a new syntactic
> feature. Suppose I wanted to add a new clause to for...else, let's say:
>
> for ... :
>block
> otherwise:
># runs only if the for-loop was empty
>
> How would do I do that?
>
Here's one way to do that. Sorry for the lack of doc for the moment.
To alter the language's grammar, you'd create a string with the new grammar,
parse it and set the result to pyterp.parser. Then, depending on how the
grammar was changed (which affects what AST is created), you'd change the
corresponding function to handle the new semantics.
I didn't quite understand what you mean by "was empty". However, I noticed that
"else" blocks after for are ignored in the current implement. Here's an example
session adding "else".
If you tell me a bit more about the intended behaviour of "otherwise", I'd be
happy to do an example with that clause.
$ ipython -i test/python_repl.py
p>> simport simple_ast
p>> for i in [1, 2]:
... print i
...
1
2
p>> for i in [1, 2, 3]:
... print i
... else:
... print 100
...
1
2
3
p>> ^D
In the above for loop, the else block is ignored! I press Control-D to exist
the interpreter.
In [1]: grammar = python_grammar.full_definition + python_grammar.extra
In [2]: grammar += r"""
...: for_stmt = "for" {exprlist} "in" {testlist} ":" {suite}
{(SAME_INDENT "else" ":" {suite}) | void=pass_stmt}
...: """
In [3]: pyterp.parser = python.Interpreter(i3.parse("grammar", grammar))
In [4]: pyterp.repl()
Now edit for_stmt in simple_ast.py
except StopIteration:
+evaluate(else_block)
return
(If else_block was not already an (ignored) parameter, we'd have to change the
`def for_stmt...` line too.) And run in the REPL we just restarted.
p>> simple_ast.reload_module(simple_ast)
p>> for i in [1, 2, 3]:
... print i
... else:
... print 100
...
1
2
3
100
p>> for i in [1, 2, 3]:
... print i
... break
... else:
... print 100
...
1
p>>
Some notes:
1. I'm using the host Python interpreter to change the grammar here but that's
not strictly necessary if we expose pyterp itself in the global scope.
2. I'm editing simple_ast.py and reloading because its less changes but
redefining a new function and setting simple_ast.for_stmt (in other words,
monkey-patching the change) should also work.
If we wanted an "otherwise" clause on top of an "else" clause, we probably want
to make some provision for the runtime to distinguish between the two so I'd
edit the grammar with something like
for_stmt = "for" {exprlist} "in" {testlist} ":" {suite} {((SAME_INDENT
"else" ":" {suite}) | void=pass_stmt)=else_block ((SAME_INDENT "otherwise" ":"
{suite}) | void=pass_stmt)=otherwise_block}
(In this case, "otherwise" blocks have to come after the "else" block if both
are present.)
asrp
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
Hi, (My other reply to Alex and Steven doesn't seems to appear, at least not on pipermail so I hope these don't come in the wrong order.) > In the readme to python_terp you say: > > """ > python_terp is intended to make language modification to Python easier > to preview changes more quickly and is not intended for full CPython > compatibility. However, a large subset of Python is already included. In > particular, enough to run the first stage of its parser. > """ > > This needs clarification. > What do you mean by subset? A real subset or also things that > are different and will stay different? > To what extent are you planning to stay compatible, and where > do you plan to deviate? > > The reason that I'm asking is that by compatible I mean the > compatibility of PyPy. If you can reach that, and be it just > by a subset, then it makes sense to speak of Python. > Good questions! There are two part to what I meant by that passage. One is a reflection of the current state of the interpreter and its capabilities. I put an example instead listing all differences because its easier to update but admittedly much less precise. The other concerns the intention for the future. My mind is not completely made but here's what I currently think it should/could be. First, fidelity would be determined by how the interpreter is used (if its used). More (potentially) drastic changes in the beginning than later on. Hopefully the deviations from Python (if any) for this reason does not grow too much from this. But since its for testing language changes (to Python), I don't think it would stray too much, if any. Second, I'd like to keep the whole thing small (of low complexity) as much as possible so in some cases, the less featureful choice will be kept. For example, in an earlier version, I had more functions parameters handling (defaults, *args, **kwargs, etc) but remove it because just setting up the initial local scope from the parameters passed and function definition took more than 100 lines. (Maybe I just went about this the wrong way...) However, except for easier or simpler implementation of the interpreter or some of the initial AST nodes, I don't intend to make any additions not in the Python language. One example of this is simple_for and single_if statements which I explain a bit in this old post: http://blog.asrpo.com/bootstrap_chicken_or_egg Although those are mainly intended to bootstrap the other statements like if_stmt and for_stmt and not used again later. (Well, there might be some difference for class method lookup (late-binding instead of early-binding) that's needed for edit-and-continue. But the intention would be to list all of those differences.) Feel free to ask again if I missed some aspect of your questions here. asrp > Sent: Saturday, February 03, 2018 at 12:29 PM > From: "Christian Tismer" <tis...@stackless.com> > To: "asrp asrp" <a...@email.com>, python-dev@python.org > Subject: Re: [Python-Dev] A minimal Python interpreter written in Python for > experimenting with language changes > > Hi user with no real name yet, > > On 03.02.18 01:01, asrp asrp wrote: > > Hello, > > > > I don't know if this is the right place to post this. Please redirect as > > needed. > > > > I've made a small Python interpreter in Python with runtime AST node > > semantics and edit-and-continue. I thought it could make prototyping > > language changes more easily and visualize usage before writing them in C. > > > > Its here: https://github.com/asrp/python_terp > > > > So, for example, redefining the for_stmt function in the right scope > > changes the behaviour of future for loops at runtime. > > > > Although from discussion I've read in PEPs, actual implementation always > > look like a non-issue (which seems like magic to me) so maybe no-one here > > actually needs this. > > > > (I really needed edit-and-continue for one of my projects but of course, > > running it in this extra interpreter is much too slow.) > > > > asrp > > In the readme to python_terp you say: > > """ > python_terp is intended to make language modification to Python easier > to preview changes more quickly and is not intended for full CPython > compatibility. However, a large subset of Python is already included. In > particular, enough to run the first stage of its parser. > """ > > This needs clarification. > What do you mean by subset? A real subset or also things that > are different and will stay different? > To what extent are you planning to stay compatible, and where > do you plan to deviate? > > The reason that I'm aski
[Python-Dev] A minimal Python interpreter written in Python for experimenting with language changes
Hello, I don't know if this is the right place to post this. Please redirect as needed. I've made a small Python interpreter in Python with runtime AST node semantics and edit-and-continue. I thought it could make prototyping language changes more easily and visualize usage before writing them in C. Its here: https://github.com/asrp/python_terp So, for example, redefining the for_stmt function in the right scope changes the behaviour of future for loops at runtime. Although from discussion I've read in PEPs, actual implementation always look like a non-issue (which seems like magic to me) so maybe no-one here actually needs this. (I really needed edit-and-continue for one of my projects but of course, running it in this extra interpreter is much too slow.) asrp ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
