zhuqi-lucas commented on code in PR #16196:
URL: https://github.com/apache/datafusion/pull/16196#discussion_r2128526165
##########
datafusion/common/src/config.rs:
##########
@@ -722,6 +722,19 @@ config_namespace! {
/// then the output will be coerced to a non-view.
/// Coerces `Utf8View` to `LargeUtf8`, and `BinaryView` to
`LargeBinary`.
pub expand_views_at_output: bool, default = false
+
+ /// When true, the optimizer will insert a Yield operator at the leaf
nodes of any pipeline
+ /// that contains a pipeline-breaking operator, allowing the Tokio
scheduler to switch to
+ /// other tasks while waiting.
+ /// Default: true (enabled).
+ pub enable_add_yield_for_pipeline_break: bool, default = true
+
+ /// Yield frequency in batches, it represents how many batches to
process before yielding
+ /// to the Tokio scheduler. The default value is 64, which means that
after processing
+ /// 64 batches, the execution will yield control back to the Tokio
scheduler.
+ /// This setting is only effective when
`enable_add_yield_for_pipeline_break` is set to true.
+ /// This value should be greater than 0.
+ pub yield_frequency_for_pipeline_break: usize, default = 64
Review Comment:
Great suggestion! @ozankabak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
