alamb commented on issue #21650:
URL: https://github.com/apache/datafusion/issues/21650#issuecomment-4262168029
> Let me ensure that I totally understand the issue (correct me @alamb )
Yes that is a good summary @xudong963
> State initialization happens in a new prepare_execution phase — before the
first execute() call, walk the plan tree and let each node register its shared
state in the TaskContext. This replaces the current pattern where state is
lazily initialized in execute().
That is an interesting idea -- another thing we could do would be create it
on demand -- something like
```rust
impl TaskContext {
fn shared_state(&self, id: SharedStateId, F: InitailizerFunc) -> ...
}
```
but maybe that would get too tricky 🤔
> We need to figure out a solution to keep it backward compatible, as
currently any DF based code expects that a pure .execute(...) call is enough to
launch a plan properly. e.g. keep a state in both plan and task context for
some time (until statefull plans are deprecated) and decide which one to use
based on the TaskContext attached value.
@askalt one thing we could do would be to change `reset_state` to clear out
the TaskContext's shared stae (and then deprecate reset_state). I think that
woudl be backwards compatible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
