alihan-synnada commented on code in PR #12969:
URL: https://github.com/apache/datafusion/pull/12969#discussion_r1805995016
##########
datafusion/common/src/config.rs:
##########
@@ -338,6 +338,12 @@ config_namespace! {
/// if the source of statistics is accurate.
/// We plan to make this the default in the future.
pub use_row_number_estimates_to_optimize_partitioning: bool, default =
false
+
+ /// Should DataFusion enforce batch size in joins or not. By default,
+ /// DataFusion will not enforce batch size in joins. Enforcing batch
size
+ /// in joins can help to avoid out-of-memory errors when joining large
+ /// tables with a highly-selective join filter.
Review Comment:
We profiled both approaches (generating multiple RecordBatches vs generating
one RecordBatch and then slicing it) and figured out that multiple calls to
`apply_join_filter_to_indices` and `adjust_indices_by_join_type` in the
incremental computation case slows down the process severely, especially with
smaller batch sizes
(https://github.com/apache/datafusion/pull/12634#issuecomment-2393215346).
Maybe we can try combining the two approaches such that we create multiple
RecordBatches to be sliced per probe side batch.
Also `adjust_indices_by_join_type` is optimized now so it might be worth
profiling again. The slowdown in `apply_join_filter_to_indices` comes from
`arrow::compute::filter` so I don't know if it can be optimized further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
