Dandandan commented on code in PR #12969:
URL: https://github.com/apache/datafusion/pull/12969#discussion_r1806044760
##########
datafusion/common/src/config.rs:
##########
@@ -338,6 +338,12 @@ config_namespace! {
/// if the source of statistics is accurate.
/// We plan to make this the default in the future.
pub use_row_number_estimates_to_optimize_partitioning: bool, default =
false
+
+ /// Should DataFusion enforce batch size in joins or not. By default,
+ /// DataFusion will not enforce batch size in joins. Enforcing batch
size
+ /// in joins can help to avoid out-of-memory errors when joining large
+ /// tables with a highly-selective join filter.
Review Comment:
In my profiling experience, `take` (for selecting the values for columns to
be filtered) is expensive as well if there is a filter (e.g. in nested loop
join). This is often more expensive than the filter operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
