On 2022-06-21 13:53, David Mertz, Ph.D. wrote:
Here is a very rough draft of an idea I've floated often, but not with
much specification. Take this as "ideas" with little firm commitment to
details from me. PRs, or issues, or whatever, can go to
https://github.com/DavidMertz/peps/blob/master/pep-9999.rst as well as
mentioning them in this thread.
After looking at this a bit more (with the newer revisions) and
following the discussion I think this proposal doesn't really achieve
what I would want from deferred evaluation. That may be because what I
want is unreasonable, but, well, such is life. :-)
First, it is not clear to me what the real point of this type of
deferred evaluation is. The PEP has a "motivation" section that makes a
link to Haskell and Dask, but as far as I can see it doesn't explicitly
say what is gained by introducing this new form of lazy evaluation into
Python.
In particular (as I think someone else mentioned on this thread),
Dask-style deferred computations are based on explicitly evaluating the
thunk, whereas this proposal would automatically evaluate it on
reference. I think that in practice this would make many Dask-style
usages unwieldy because you would have to keep repeating the `later`
keyword in order to gradually build up a complex deferred computation
over multiple statements. For such cases it is more natural to
explicitly evaluate the whole thing at the end, rather than explicitly
not evaluate it until then.
In theory there could be performance gains, as mentioned in the PEP.
But again I don't see a huge advantage to this in Python. It might make
sense in Haskell where laziness is built into the language at a
fundamental level. But in Python, where eager evaluation is the norm,
it again seems more natural to me to use "explicit laziness" (i.e.,
explicit rather than automatic evaluation). It seems rather unusual to
have cases where some variable or function argument might contain either
a computationally cheap expression or an expensive one; usually for
those types of applications you know where you might do something
expensive. And even if you don't, I see little downside to requiring an
explicit "eval this thunk" step at the end.
In contrast, what I would want out of deferred evaluation is precisely
the ability to evaluate the deferred expression in the *evaluating*
scope (not the definition scope) --- or in a custom provided namespace.
Whether this evaluation is implicit or explicit is less important to
me than the ability to control the scope in which it occurs. As others
mentioned in early posts on this thread, this could complicate things
too much to be feasible, but without it I don't really see the point.
The reason this is key for me is that I'm focused on a different set of
motivating use cases. What I'm interested in is "query-type" situations
where you want to pass an expression to some sort of "query engine",
which will evaluate the expression in a namespace representing the
dataset to be queried. One example would be SQL queries, where it would
be nice to be able to do things like:
my_sql_table.select(where=thunk (column1 == 2 and column2 > 5))
Likewise this would make pandas indexing less verbose, turning it from:
df[(df.column1 == 2) & (df.column2 > 5)]
to:
df[(column1 == 2) & (column2 > 3)]
or even potentially:
df[column1 == 2 and column2 > 3]
. . . because the evaluator would have control over the evaluation and
could provide a namespace in which `column1` and `column2` do not
evaluate directly to numpy-like arrays (for which `and` doesn't work),
but to some kind of combinable query object which converts the `and`
into something that will work with numpy-like elementwise comparison.
In other words, the point here is not performance gains or even
laziness, but simply the ability to use ordinary Python expression
syntax (not, say, a string) to create an unevaluated chunk which can be
passed to some other code which then gets to control its evaluation
scope, rather than having that scope locked to where it was defined.
Because of this, it is probably okay with me if explicit unwrapping of
the thunk is required. You know when you are writing a query handler
and so you know that what you want is an unevaluated query expression;
you don't need to have an argument whose value might either be an
unevaluated expression or a fully-evaluated result.
This would also mean that such deferred objects could handle the
late-bound default case, but the function would have to "commit" to
explicit evaluation of such defaults. Probably there could be a no-op
"unwrapping" operation that would work on non-deferred objects (so that
`unwrap([])` or whatever would just evaluate to the same regular list
you passed in), so you could still pass in a plain list a to an argument
whose default was `deferred []`, but the function would still have to
explicitly evaluate it in its body. Again, I think I'm okay with this,
partly because (as I mentioned in the other thread) I don't see PEP
671-style late-bound defaults as a particularly pressing need.
There are definitely some holes in my idea. For one thing, with
explicit evaluation required, it is much closer to a regular lambda.
The only real difference is that it would involve more flexible scope
control (rather than unalterably closing over the defining scope). For
another, because it is not lazy, it is closer to being achievable with
existing mechanisms, like requiring all "field" references in the query
to be specified as attributes on some base object (which is indeed most
SQL ORMs and pandas-like data structures do it currently). Other people
might not be as annoyed with these existing solutions as I am. :-)
There is also the question of whether it would unacceptably slow down
name references because functions would no longer know which variables
were local; I think I would be okay with saying that the thunk could not
mutate the enclosing namespace (so, e.g., walruses inside the thunk
would only affect an internal thunk namespace). The point here is for
the consumer to *evaluate* the thunk and get the result, not inline it
into the surrounding code.
My idea is much more half-baked than David's proto-PEP so this isn't
really worthy of being called an alternative proposal right now. But I
wanted to mention these ideas here to at least handwave about what to me
the gain would be from deferred evaluation, as I'm coming at it from a
somewhat different angle than the proto-PEP. I have a suspicion that
the response will be a combination of disgust and deafening silence but
that's life.
--
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is no
path, and leave a trail."
--author unknown
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/DMW5TI2WGHXYQQA334YT4VCA3PUZ237U/
Code of Conduct: http://python.org/psf/codeofconduct/
