On Thu, Jul 8, 2021 at 6:25 PM Nick Coghlan <ncogh...@gmail.com> wrote:

> On Tue, 6 Jul 2021, 7:56 am Jim Baker, <jim.ba...@python.org> wrote:
>
>>
>>
>> On Mon, Jul 5, 2021, 2:40 PM Guido van Rossum <gu...@python.org> wrote:
>>
>>> FWIW, we could make f-strings properly nest  too, like you are proposing
>>> for backticks. It's just that we'd have to change the lexer. But it would
>>> not be any harder than would be for backticks (since it would be the same
>>> algorithm), nor would it be backward incompatible. So this is not an
>>> argument for backticks.
>>>
>>
>> Good point. At some point, I was probably thinking of backticks without a
>> tag, since JS supports this for their f-string like scenario. but if we
>> always require a tag - so long as it's not a prefix already in use (b, f,
>> r, fr, hopefully not forgetting as I type this email in a parking lot...) -
>> then it can be disambiguated using standard quotes.
>>
>
> There's a deferred PEP proposing a resolution to the f-string nesting
> limitations:
> https://www.python.org/dev/peps/pep-0536/#motivation
>
>
Thanks for pointing that PEP out - this looks quite reasonable for both
f-strings and for the tagged templates proposed here. I believe some
limitations on nesting expressions in f-strings with quote changing was
introduced in a relatively recent fix in 3.8 or 3.9, but I need to find the
specifics.


>>
>>> Separately, should there be a way to *delay* evaluation of the templated
>>> expressions (like we explored in our private little prototype last year)?
>>>
>>
>> I think so, but probably with an explicit marker on *each* deferred
>> expression. I'm in favor of Julia's expression quote, which generally needs
>> to be enclosed in parentheses, but possibly not needed in  expression
>> braces (at the risk of looking like a standalone format spec).
>> https://docs.julialang.org/en/v1/manual/metaprogramming/
>>
>> So this would like
>> x = 42
>> d = deferred_tag"Some expr: {:(x*2)}"
>>
>> All that is happening here is that this being wrapped in a lambda, which
>> captures any scope lexically as usual. Then per that experiment you
>> mentioned, it's possible to use that scope using fairly standard - or at
>> least portable to other Python implementations - metaprogramming,
>> including the deferred evaluation of the lambda.
>> (No frame walking required!)
>>
>> Other syntax could work for deferring.
>>
>
> It reminds me of the old simple implicit lambda proposal: https://www.
> python.org/dev/peps/pep-0312/
>
> The old proposal has more ambiguities to avoid now due to type hinting
> syntax but a parentheses-required version could work: "(:call_result)"
>
>
PEP 312 - suitably modified - could be quite useful for both deferring
evaluation in tagged templates and function calls (or other places where we
might want to put in a lambda). Ideally we can use minimum parens as well,
so it's

tag"{:x+1}"
f(:x+1, :y+2)

(but I haven't looked at ambiguities here).

A further idea that might make the approach in PEP 312 more powerful than
simply being an implicit lambda is if this was like Julia's quoted
expressions https://docs.julialang.org/en/v1/manual/metaprogramming/#Quoting
and
we recorded the *text body of the expression*. (Such recording would
presumably be done in any template scheme, but let's make this explicit
now.)

So let's call this implicit lambda with recorded text body a *quoted
expression*.

What's nice about doing such quoted expressions is that the lambda also
captures any lexical scope. So if I have an expression tag"{:x*y}", the
variables x and y are referenced appropriately. This means it would be
possible to do such things as rewrite complex expressions - eg an index
query on a Pandas data frame - while avoiding the use of dynamic scope.

I wrote a small piece of code to show how this can be exercised. The text
of the expression is simply an attribute on the lambda named "body", but we
might have a new type defined (eg types.QuotedExpression).

```
from functools import update_wrapper
from textwrap import dedent
from types import FunctionType


def rewrite(f, new_body):
    # Create a temporary outer function with arguments for the free
variables of
    # the original lambda wrapping an expression. This approach allows the
new
    # inner function, which has a new body, to compile with its dependency
on
    # free vars.
    #
    # When called, this new inner function code object can continue to
access
    # these variables from the cell vars in the closure - even without this
    # outer function, which we discard.
    #
    # This rewriting is generalizable to arbitrary functions, although the
    # syntax we are exploring is only for expressions.
    #
    # A similar idea - for monkeypatching an inner function - is explored in
    # https://stackoverflow.com/a/27550237 by Martijn Pieters' detailed
answer.
    #
    # Related ideas include https://en.wikipedia.org/wiki/Lambda_lifting and
    # lambda dropping where the free vars are lifted up/dropped from the
    # function parameters. This requires the (usual :) extra level of
    # indirection by another function.

    scoped = dedent(f"""
        def outer({", ".join(f.__code__.co_freevars)}):
             def inner():
                return {new_body}
        """)
    capture = {}
    exec(scoped, f.__globals__, capture)
    # NOTE: outer is co_consts[0], inner is co_consts[1] - this may not be
    # guaranteed by the compiler, so iterate over if necessary.
    inner_code = capture["outer"].__code__.co_consts[1]
    new_f = FunctionType(
        inner_code,
        f.__globals__,
        closure=f.__closure__)
    update_wrapper(new_f, f)
    new_f.body = new_body
    return new_f


def test_rewrite():
    x = 2
    y = 3
    # x, y and free vars in scopeit, and its own nested lambdas, assigned
    # to f and g below. We will also introduce an additional free var z
    # in the parameter of scopeit.
    def scopeit(z):
        f = lambda: x * y + z
        f.body = "x * y + z"
        print(f"{f()=}")

        # We can do an arbitrary manipulation on the above expression, so
        # long as we use variables that are in the original scope (or
symtab).
        # Note that adding new variables will look up globally (may or may
        # not exist, if not a NameError is raised).
        print(f.__code__.co_freevars)
        g = rewrite(f, "-(x * y + z)")
        print(f"{g()=}")
    scopeit(5)


if __name__ == "__main__":
    test_rewrite()
```

Similar ideas were explored in
https://github.com/jimbaker/fl-string-pep/issues

- Jim


On Thu, Jul 8, 2021 at 6:25 PM Nick Coghlan <ncogh...@gmail.com> wrote:

> On Tue, 6 Jul 2021, 7:56 am Jim Baker, <jim.ba...@python.org> wrote:
>
>>
>>
>> On Mon, Jul 5, 2021, 2:40 PM Guido van Rossum <gu...@python.org> wrote:
>>
>>> FWIW, we could make f-strings properly nest  too, like you are proposing
>>> for backticks. It's just that we'd have to change the lexer. But it would
>>> not be any harder than would be for backticks (since it would be the same
>>> algorithm), nor would it be backward incompatible. So this is not an
>>> argument for backticks.
>>>
>>
>> Good point. At some point, I was probably thinking of backticks without a
>> tag, since JS supports this for their f-string like scenario. but if we
>> always require a tag - so long as it's not a prefix already in use (b, f,
>> r, fr, hopefully not forgetting as I type this email in a parking lot...) -
>> then it can be disambiguated using standard quotes.
>>
>
> There's a deferred PEP proposing a resolution to the f-string nesting
> limitations:
> https://www.python.org/dev/peps/pep-0536/#motivation
>
>
>>
>>> Separately, should there be a way to *delay* evaluation of the templated
>>> expressions (like we explored in our private little prototype last year)?
>>>
>>
>> I think so, but probably with an explicit marker on *each* deferred
>> expression. I'm in favor of Julia's expression quote, which generally needs
>> to be enclosed in parentheses, but possibly not needed in  expression
>> braces (at the risk of looking like a standalone format spec).
>> https://docs.julialang.org/en/v1/manual/metaprogramming/
>>
>> So this would like
>> x = 42
>> d = deferred_tag"Some expr: {:(x*2)}"
>>
>> All that is happening here is that this being wrapped in a lambda, which
>> captures any scope lexically as usual. Then per that experiment you
>> mentioned, it's possible to use that scope using fairly standard - or at
>> least portable to other Python implementations - metaprogramming, including
>> the deferred evaluation of the lambda.
>> (No frame walking required!)
>>
>> Other syntax could work for deferring.
>>
>
> It reminds me of the old simple implicit lambda proposal:
> https://www.python.org/dev/peps/pep-0312/
>
> The old proposal has more ambiguities to avoid now due to type hinting
> syntax but a parentheses-required version could work: "(:call_result)"
>
> Cheers,
> Nick.
>
>>
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LWQVMYRZVC6C7K477V7I3L4O3QQIEWPJ/
Code of Conduct: http://python.org/psf/codeofconduct/

Reply via email to