[Python-ideas] Re: Generalized deferred computation in Python

2022-06-26 Thread Chris Angelico
On Sun, 26 Jun 2022 at 23:20, Jelle Zijlstra wrote: > > > > El sáb, 25 jun 2022 a las 13:44, Chris Angelico () escribió: >> >> On Sun, 26 Jun 2022 at 04:41, Brendan Barnwell wrote: >> > In contrast, what I would want out of deferred evaluation is >> > precisely >> > the ability to

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-26 Thread Jelle Zijlstra
El sáb, 25 jun 2022 a las 13:44, Chris Angelico () escribió: > On Sun, 26 Jun 2022 at 04:41, Brendan Barnwell > wrote: > > In contrast, what I would want out of deferred evaluation is > precisely > > the ability to evaluate the deferred expression in the *evaluating* > > scope (not the

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-26 Thread Piotr Duda
niedz., 26 cze 2022 o 08:17 Brendan Barnwell napisał(a): > In other words it really is very similar to what a lambda currently > is, > but with more fine-grained control over which variables are bound in > which namespaces (definition vs. eval). But lambda already have fine-grained

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-26 Thread Chris Angelico
On Sun, 26 Jun 2022 at 16:18, Brendan Barnwell wrote: > > On 2022-06-25 13:41, Chris Angelico wrote: > > On Sun, 26 Jun 2022 at 04:41, Brendan Barnwell > > wrote: > >> In contrast, what I would want out of deferred evaluation is > >> precisely > >> the ability to evaluate the deferred

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-26 Thread Brendan Barnwell
On 2022-06-25 13:41, Chris Angelico wrote: On Sun, 26 Jun 2022 at 04:41, Brendan Barnwell wrote: In contrast, what I would want out of deferred evaluation is precisely the ability to evaluate the deferred expression in the *evaluating* scope (not the definition scope) --- or in a

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-25 Thread Chris Angelico
On Sun, 26 Jun 2022 at 04:41, Brendan Barnwell wrote: > In contrast, what I would want out of deferred evaluation is precisely > the ability to evaluate the deferred expression in the *evaluating* > scope (not the definition scope) --- or in a custom provided namespace. > Whether this

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-25 Thread Brendan Barnwell
On 2022-06-21 13:53, David Mertz, Ph.D. wrote: Here is a very rough draft of an idea I've floated often, but not with much specification. Take this as "ideas" with little firm commitment to details from me. PRs, or issues, or whatever, can go to

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-25 Thread Stephen J. Turnbull
Chris Angelico writes: > So the only way around it would be to make the defer keyword somehow > magical when used in a function signature, which kinda defeats the > whole point about being able to reuse another mechanic to achieve > this. The defer keyword is already magical. Overloading it

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Chris Angelico
On Sat, 25 Jun 2022 at 11:56, Joao S. O. Bueno wrote: > > > > On Fri, Jun 24, 2022 at 10:05 PM Chris Angelico wrote: > > > > > Hmmm, I think possibly you're misunderstanding the nature of class > > slots, then. The most important part is that they are looked up on the > > *class*, not the

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Joao S. O. Bueno
On Fri, Jun 24, 2022 at 10:05 PM Chris Angelico wrote: > > Hmmm, I think possibly you're misunderstanding the nature of class > slots, then. The most important part is that they are looked up on the > *class*, not the instance; but there are some other quirks too: Sorry, no. I know how those

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Chris Angelico
On Sat, 25 Jun 2022 at 10:37, Joao S. O. Bueno wrote: > > > > On Fri, Jun 24, 2022 at 5:38 AM Chris Angelico wrote: > > > > On Fri, 24 Jun 2022 at 16:34, Joao S. O. Bueno > > wrote: > > > On Fri, Jun 24, 2022 at 1:06 AM Chris Angelico wrote: > > >> How much benefit would this be? You're

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Joao S. O. Bueno
On Fri, Jun 24, 2022 at 5:38 AM Chris Angelico wrote: > > On Fri, 24 Jun 2022 at 16:34, Joao S. O. Bueno wrote: > > On Fri, Jun 24, 2022 at 1:06 AM Chris Angelico wrote: > >> How much benefit would this be? You're proposing a syntactic construct > >> for something that isn't used all that

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread David Mertz, Ph.D.
On Fri, Jun 24, 2022 at 10:52 AM Stephen J. Turnbull < stephenjturnb...@gmail.com> wrote: > We (not sure how much help I'll be, but I'm in) need to deal with > Chris A's point that a pure memoizing object doesn't help with the > mutable defaults problem. That is with > > def

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Chris Angelico
On Sat, 25 Jun 2022 at 00:54, Stephen J. Turnbull wrote: > > David Mertz, Ph.D. writes: > > On Fri, Jun 24, 2022 at 3:50 AM Stephen J. Turnbull > > wrote: > > > > I'm suggesting modified semantics where deferreds can be a proxy > > > object, whose normal reaction to *any* operation

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Stephen J. Turnbull
David Mertz, Ph.D. writes: > On Fri, Jun 24, 2022 at 3:50 AM Stephen J. Turnbull > wrote: > > I'm suggesting modified semantics where deferreds can be a proxy > > object, whose normal reaction to *any* operation (possibly > > excepting name binding) is > > > > 1. check for a

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread David Mertz, Ph.D.
On Fri, Jun 24, 2022 at 3:50 AM Stephen J. Turnbull < stephenjturnb...@gmail.com> wrote: > That's true in David's proposed semantics, where the runtime does that > check. I'm suggesting modified semantics where deferreds can be a > proxy object, whose normal reaction to *any* operation (possibly

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Chris Angelico
On Fri, 24 Jun 2022 at 17:51, Stephen J. Turnbull wrote: > Then the question is, why do we need syntax? Well, there's the PEP > 671 rationales for deferring function argument defaults. Which the OP hasn't actually explained in a way that works yet. A vanilla argument default value, with any

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Chris Angelico
On Fri, 24 Jun 2022 at 16:34, Joao S. O. Bueno wrote: > On Fri, Jun 24, 2022 at 1:06 AM Chris Angelico wrote: >> How much benefit would this be? You're proposing a syntactic construct >> for something that isn't used all that often, so it needs to be a >> fairly dramatic improvement in the cases

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Stephen J. Turnbull
Barry writes: > > On 23 Jun 2022, at 08:27, Stephen J. Turnbull > > wrote: > Interest idea that ref does not auto evaluate in all cases. > I was wondering about what the compile/runtime can do it avoid the costs > of checking for an evaluation. I think the main thing to do is to put the

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Steve Jorgensen
Steve Jorgensen wrote: > I think I have an idea how to do something like what you're asking with less > magic, and I think an example implementation of this could actually be done > in pure Python code (though a more performant implementation would need > support at the C level). > What if a

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-24 Thread Joao S. O. Bueno
On Fri, Jun 24, 2022 at 1:06 AM Chris Angelico wrote: > On Fri, 24 Jun 2022 at 13:26, Joao S. O. Bueno > wrote: > > > > > > > > On Thu, Jun 23, 2022 at 2:53 AM Chris Angelico wrote: > >> > >> On Thu, 23 Jun 2022 at 11:35, Joao S. O. Bueno > wrote: > >> > > >> > Martin Di Paola wrote: > >> > >

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-23 Thread Chris Angelico
On Fri, 24 Jun 2022 at 13:26, Joao S. O. Bueno wrote: > > > > On Thu, Jun 23, 2022 at 2:53 AM Chris Angelico wrote: >> >> On Thu, 23 Jun 2022 at 11:35, Joao S. O. Bueno wrote: >> > >> > Martin Di Paola wrote: >> > > Three cases: Dask/PySpark, Django's ORM and selectq. All of them >> > >

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-23 Thread Joao S. O. Bueno
On Thu, Jun 23, 2022 at 2:53 AM Chris Angelico wrote: > On Thu, 23 Jun 2022 at 11:35, Joao S. O. Bueno > wrote: > > > > Martin Di Paola wrote: > > > Three cases: Dask/PySpark, Django's ORM and selectq. All of them > > > implement deferred expressions but all of them "compute" them in very > > >

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-23 Thread Barry
> On 23 Jun 2022, at 08:27, Stephen J. Turnbull > wrote: > > Barry Scott writes: > >> I can think of ways to implement evaluation-on-reference, but they >> all have the effect of making python slower. > > Probably. > >> The simple >> >>a = b >> >> will need to slow down so that the

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-23 Thread Stephen J. Turnbull
Barry Scott writes: > I can think of ways to implement evaluation-on-reference, but they > all have the effect of making python slower. Probably. > The simple > > a = b > > will need to slow down so that the object in b can checked to see > if it need evaluating. No, it doesn't.

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Chris Angelico
On Thu, 23 Jun 2022 at 11:35, Joao S. O. Bueno wrote: > > Martin Di Paola wrote: > > Three cases: Dask/PySpark, Django's ORM and selectq. All of them > > implement deferred expressions but all of them "compute" them in very > > specific ways (aka, they plan and execute the computation

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Ben Rudiak-Gould
On Wed, Jun 22, 2022 at 6:36 PM Joao S. O. Bueno wrote: > implement "all possible" > dunder methods, and proxy those to the underlying object, for a "future > type" that was > calculated off-process, and did not need any ".value()" or ".result()" > methods to be called. > Here's a package on

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Ben Rudiak-Gould
> > On Wed, Jun 22, 2022 at 02:30:14PM -0400, David Mertz, Ph.D. wrote: > >But basically, think about `x = (later expensive1() + later expensive2()) > / > >later expensive3()`. How can we make `x` itself be a zero argument > >lambda? [... see below ...] > x = lambda: (expensive1() +

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread David Mertz, Ph.D.
Thanks Rob, I recognize that I have so-far skirted the order-of-precedence concern. I believe I have used parens in my example everywhere there might be a question... But that's not a general description or rule. I have a bunch of issues that I know I need to flesh out, many coming as

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Martin Di Paola
On Wed, Jun 22, 2022 at 11:22:05PM +0100, Paul Moore wrote: Hang on, did the PEP change? The version I saw didn't have a compute() method, deferred objects were just evaluated when they were referenced. You are right, the PEP does not mention a compute() method but uses the that term. I just

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Joao S. O. Bueno
Martin Di Paola wrote: > Three cases: Dask/PySpark, Django's ORM and selectq. All of them > implement deferred expressions but all of them "compute" them in very > specific ways (aka, they plan and execute the computation differently). So - I've been hit with the "transparency execution of

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Chris Angelico
On Thu, 23 Jun 2022 at 10:44, Rob Cliffe via Python-ideas wrote: > > Thank you for your proposal David. At last we have a counter-proposal to > talk about. A few points: > > (1) (As I pointed out in an earlier post) There is a flaw in using the syntax > of an expression PRECEDED by a SOFT

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Rob Cliffe via Python-ideas
Thank you for your proposal David.  At last we have a counter-proposal to talk about.  A few points: (1) (As I pointed out in an earlier post) There is a flaw in using the syntax of an expression PRECEDED by a SOFT keyword:     x     =    later    -y With your proposal, x is assigned a

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Paul Moore
On Wed, 22 Jun 2022 at 22:52, Martin Di Paola wrote: > Perhaps this is the real focus to analyze. The PEP suggests that > x.compute() does something *fundamentally* different from just executing > the intermediate expressions. Hang on, did the PEP change? The version I saw didn't have a

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Martin Di Paola
On Wed, Jun 22, 2022 at 02:30:14PM -0400, David Mertz, Ph.D. wrote: The difference is in composition of operations. I can write a dozen zero-argument lambdas easily enough. But those are all isolated. But basically, think about `x = (later expensive1() + later expensive2()) / later

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Barry Scott
> On 22 Jun 2022, at 19:09, Paul Moore wrote: > > I suspect that you consider evaluation-on-reference as an important > feature of your proposal, but could you consider explicit evaluation > as an alternative? Or at the very least address in the PEP the fact > that this would close the door on

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Neil Girdhar
Could this idea be used to specify the default factory of a dataclass field? For example, @dataclass class X: x: list[int] = deferred [] instead of @dataclass class X: x: list[int] = field(default_factory=list) If so, it might be worth adding to your proposal as another common

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread David Mertz, Ph.D.
On Wed, Jun 22, 2022 at 2:17 PM Eric V. Smith wrote: > Every time I’ve looked at this, I come back to: other than the clunky > syntax, how is explicit evaluation different from a zero-argument lambda? The difference is in composition of operations. I can write a dozen zero-argument lambdas

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread David Mertz, Ph.D.
On Wed, Jun 22, 2022 at 10:47 AM Martin Di Paola wrote: > Hi David, I read the PEP and I think it would be useful to expand the > Motivation and Examples sections. > While indeed Dask uses lazy evaluation to build a complex computation > without executing it, I don't think that it is the whole

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Eric V. Smith via Python-ideas
> On Jun 22, 2022, at 2:12 PM, Paul Moore wrote: > > On Wed, 22 Jun 2022 at 18:35, David Mertz, Ph.D. > wrote: >> >> Hi Martin, >> >> Short answer: yes, I agree. >> Slightly longer: I would be eternally grateful if you wish to contribute to >> the PEP with any such expansion of the

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Paul Moore
On Wed, 22 Jun 2022 at 18:35, David Mertz, Ph.D. wrote: > > Hi Martin, > > Short answer: yes, I agree. > Slightly longer: I would be eternally grateful if you wish to contribute to > the PEP with any such expansion of the Motivation and Expansion. One concern I have, triggered by Martin's Dask,

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread David Mertz, Ph.D.
Hi Martin, Short answer: yes, I agree. Slightly longer: I would be eternally grateful if you wish to contribute to the PEP with any such expansion of the Motivation and Expansion. On Wed, Jun 22, 2022 at 10:47 AM Martin Di Paola wrote: > Hi David, I read the PEP and I think it would be useful

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Steve Jorgensen
I think I have an idea how to do something like what you're asking with less magic, and I think an example implementation of this could actually be done in pure Python code (though a more performant implementation would need support at the C level). What if a deferred object has 1 magic method

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-22 Thread Martin Di Paola
Hi David, I read the PEP and I think it would be useful to expand the Motivation and Examples sections. While indeed Dask uses lazy evaluation to build a complex computation without executing it, I don't think that it is the whole story. Dask takes this deferred complex computation and *plans*

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread Chris Angelico
On Wed, 22 Jun 2022 at 11:59, David Mertz, Ph.D. wrote: > > Thanks Carl and Chris. After reading your comments, and thinking some more > about it, I agree you are both correct that it only makes sense to have a > DeferredObject act like a closure in the scope of its creation. That really >

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread David Mertz, Ph.D.
Thanks Carl and Chris. After reading your comments, and thinking some more about it, I agree you are both correct that it only makes sense to have a DeferredObject act like a closure in the scope of its creation. That really should suffice for anything I could sensibly want; it's enough for

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread Carl Meyer via Python-ideas
On Tue, Jun 21, 2022 at 4:55 PM David Mertz, Ph.D. wrote: > I haven't gotten to writing that into the PEP yet, but I think the rule > has to be to take the scope of evaluation not the scope of definition. I > know it needs to be there, and I'm just thinking about helpful examples. > > Which is

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread Chris Angelico
On Wed, 22 Jun 2022 at 08:54, David Mertz, Ph.D. wrote: > > I haven't gotten to writing that into the PEP yet, but I think the rule has > to be to take the scope of evaluation not the scope of definition. I know it > needs to be there, and I'm just thinking about helpful examples. > > Which is

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread David Mertz, Ph.D.
I haven't gotten to writing that into the PEP yet, but I think the rule has to be to take the scope of evaluation not the scope of definition. I know it needs to be there, and I'm just thinking about helpful examples. Which is to say the semantics are more like `eval()` than like a lambda

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread Chris Angelico
On Wed, 22 Jun 2022 at 08:36, Carl Meyer wrote: > > > > On Tue, Jun 21, 2022 at 4:28 PM Chris Angelico wrote: >> >> On Wed, 22 Jun 2022 at 08:21, Carl Meyer via Python-ideas >> wrote: >> > >> > >> > >> > On Tue, Jun 21, 2022 at 4:10 PM David Mertz, Ph.D. >> > wrote: >> >> >> >> On Tue, Jun

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread Carl Meyer via Python-ideas
On Tue, Jun 21, 2022 at 4:28 PM Chris Angelico wrote: > On Wed, 22 Jun 2022 at 08:21, Carl Meyer via Python-ideas > wrote: > > > > > > > > On Tue, Jun 21, 2022 at 4:10 PM David Mertz, Ph.D. < > david.me...@gmail.com> wrote: > >> > >> On Tue, Jun 21, 2022 at 5:53 PM Brendan Barnwell > wrote: >

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread Chris Angelico
On Wed, 22 Jun 2022 at 08:21, Carl Meyer via Python-ideas wrote: > > > > On Tue, Jun 21, 2022 at 4:10 PM David Mertz, Ph.D. > wrote: >> >> On Tue, Jun 21, 2022 at 5:53 PM Brendan Barnwell >> wrote: >>> >>> > In the example, we assume that the built-in function `type()` is special >>> > in not

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread Carl Meyer via Python-ideas
On Tue, Jun 21, 2022 at 4:10 PM David Mertz, Ph.D. wrote: > On Tue, Jun 21, 2022 at 5:53 PM Brendan Barnwell > wrote: > >> > In the example, we assume that the built-in function `type()` is special >> > in not counting as a reference to the binding for purpose of realizing a >> > computation.

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread David Mertz, Ph.D.
On Tue, Jun 21, 2022 at 5:53 PM Brendan Barnwell wrote: > > In the example, we assume that the built-in function `type()` is special > > in not counting as a reference to the binding for purpose of realizing a > > computation. Alternately, some new special function like `isdeferred()` > might be

[Python-ideas] Re: Generalized deferred computation in Python

2022-06-21 Thread Brendan Barnwell
On 2022-06-21 13:53, David Mertz, Ph.D. wrote: In the example, we assume that the built-in function `type()` is special in not counting as a reference to the binding for purpose of realizing a computation. Alternately, some new special function like `isdeferred()` might be used to check for