On Wed, Jun 22, 2022 at 10:47 AM Martin Di Paola <martinp.dipa...@gmail.com>
wrote:
> Hi David, I read the PEP and I think it would be useful to expand the
> Motivation and Examples sections.
> While indeed Dask uses lazy evaluation to build a complex computation
> without executing it, I don't think that it is the whole story.
> Dask takes this deferred complex computation and *plans* how to execute it
> and then it *executes* it in non-obvious/direct ways.
>
Dask is very clever about execution planning. Ray is possibly even more
clever in that regard.
However, I think that that should be an explicit non-goal of the PEP.
DeferredObjects should create a DAG, yes. But I think Python itself should
not think about being *clever* in evaluating that DAG, nor in itself think
about parallelism. If my PEP were adopted, that would be something other
libraries like Dask or Django could build on top of with more elaborate
evaluation plans.
But just the DAG itself gets you more than just "wait until needed to do
the final computation." It allows for intermediate computation of nodes of
the DAG lower down than the final result. For example, imagine
dependencies like this (where all the computation steps are expensive):
A -> B
B -> Z
B -> Y
B -> X
A -> C
C -> X
X -> F
X -> G
C -> W
C -> V
A -> D
D -> V
D -> U
D -> T
Hopefully you either see my ASCII art in fixed font, or it's at least
intelligible. If I want to force evaluation of A, I need to do
everything. But if it turns out all I need within my program is C, then I
have to do computations C, X, F, G, W, V. Which is maybe still expensive,
but at least I don't worry about B, Z, Y, U, T, or A.
Yes, I should add something like this to the PEP.
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/4CN3PRNH4M7T5TF36IRDZ6GZMHQS26TW/
Code of Conduct: http://python.org/psf/codeofconduct/
