zhuqi-lucas commented on code in PR #16196:
URL: https://github.com/apache/datafusion/pull/16196#discussion_r2128845315
##########
datafusion/common/src/config.rs:
##########
@@ -722,6 +722,19 @@ config_namespace! {
/// then the output will be coerced to a non-view.
/// Coerces `Utf8View` to `LargeUtf8`, and `BinaryView` to
`LargeBinary`.
pub expand_views_at_output: bool, default = false
+
+ /// When true, the optimizer will insert a Yield operator at the leaf
nodes of any pipeline
+ /// that contains a pipeline-breaking operator, allowing the Tokio
scheduler to switch to
+ /// other tasks while waiting.
+ /// Default: true (enabled).
+ pub enable_add_yield_for_pipeline_break: bool, default = true
+
+ /// Yield frequency in batches, it represents how many batches to
process before yielding
+ /// to the Tokio scheduler. The default value is 64, which means that
after processing
+ /// 64 batches, the execution will yield control back to the Tokio
scheduler.
+ /// This setting is only effective when
`enable_add_yield_for_pipeline_break` is set to true.
+ /// This value should be greater than 0.
+ pub yield_frequency_for_pipeline_break: usize, default = 64
Review Comment:
Addressed in latest RP, thanks!
##########
datafusion/physical-plan/src/execution_plan.rs:
##########
@@ -75,7 +75,19 @@ use futures::stream::{StreamExt, TryStreamExt};
/// [`execute`]: ExecutionPlan::execute
/// [`required_input_distribution`]: ExecutionPlan::required_input_distribution
/// [`required_input_ordering`]: ExecutionPlan::required_input_ordering
-pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync {
+/// The core trait for a physical execution plan node. Every operator
+/// implements this trait. We have extended it by adding two new methods
+/// for “cooperative yielding” support:
+///
+/// 1. `yields_cooperatively()` indicates whether this operator already
+/// supports async/yield behavior internally (default: false).
+///
+/// 2. `with_cooperative_yields(self: Arc<Self>)` returns an alternate
+/// plan node that has built-in yielding; if not available, returns None.
+///
+/// Because `with_cooperative_yields` moves `Arc<Self>` into `Arc<dyn
ExecutionPlan>`,
+/// we must ensure `Self: 'static`. Therefore, we add `+ 'static` here.
+pub trait ExecutionPlan: Debug + DisplayAs + Send + Sync + 'static {
Review Comment:
Addressed in latest RP, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
