ozankabak commented on PR #16196:
URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2940614830
> What do you mean built-in yielding? If it means we add some leaf nodes to
support built-in yielding support? Thanks!
Here is rough sketch of what I meant, let's discuss and optimize.
1. We add built-in yielding capabilities to `DataSourceExec` via a
constructor argument; i.e. it yields internally every now and then if created
with that constructor argument.
2. We add two `ExecutionPlan` APIs:
- `yields_cooperatively` (we can choose the name later, this is a
placeholder), which returns a boolean value. The default implementation simply
checks `EmissionType` of the operator and returns accordingly.
- `with_cooperative_yields`, which *optionally* returns a
cooperatively-yielding version of the operator if it exists. The default
implementation checks `yields_cooperatively` and returns self if true, and
`None` otherwise.
3. We override `with_cooperative_yields` for `DataSourceExec` and return the
yielding version.
4. If the rule decides that it needs the leaf to yield, it checks if a
yielding variant exists, and replaces it with a yielding variant. If it
doesn't, it adds a `YieldStreamExec`.
After our investigations, if it turns out that `InterleaveExec` or some
other operator can always misbehave, we can change the return value of
`yields_cooperatively` to signal that and tweak the rule to add intermediate
`YieldStreamExec` nodes too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
