pepijnve commented on code in PR #16398:
URL: https://github.com/apache/datafusion/pull/16398#discussion_r2152852677
##########
datafusion/physical-plan/src/execution_plan.rs:
##########
@@ -743,6 +733,38 @@ pub enum EmissionType {
Both,
}
+/// Represents whether an operator's `Stream` has been implemented to actively
cooperate with the
+/// Tokio scheduler or not.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum SchedulingType {
+ /// The stream generated by [`execute`](ExecutionPlan::execute) does not
actively participate in
+ /// cooperative scheduling. This means the implementation of the `Stream`
returned by
+ /// [`ExecutionPlan::execute`] does not contain explicit cooperative yield
points.
+ Blocking,
+ /// The stream generated by [`execute`](ExecutionPlan::execute) actively
participates in
+ /// cooperative scheduling by consuming task budget when it was able to
produce a
+ /// [`RecordBatch`]. Please refer to the [`coop`](crate::coop) module for
more details.
+ Cooperative,
+}
+
+/// Represents how an operator's `Stream` implementation generates
`RecordBatch`es.
+///
+/// Most operators in DataFusion generate `RecordBatch`es when asked to do so
by a call to
+/// `Stream::poll_next`. This is known as demand-driven or lazy evaluation.
+///
+/// Some operators like `Repartition` need to drive `RecordBatch` generation
themselves though. This
+/// is known as data-driven or eager evaluation.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum EvaluationType {
+ /// The stream generated by [`execute`](ExecutionPlan::execute) only
generates `RecordBatch`
+ /// instances when it is demanded by invoking `Stream::poll_next`.
Review Comment:
Examples added for both variants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
