cxzl25 opened a new pull request, #1787: URL: https://github.com/apache/auron/pull/1787
# Which issue does this PR close? Closes #1763 # Rationale for this change Datafusion's BinaryExpr needs to meet a certain ratio before it can perform short-circuit calculation. There is an optimization in Auron that can perform short-circuit calculation. https://github.com/apache/datafusion/blob/33ac70dd6d634da040cc34abd414425b176a2b99/datafusion/physical-expr/src/expressions/binary.rs#L641-L644 ```rust /// Based on the results calculated from the left side of the short-circuit operation, /// if the proportion of `true` is less than 0.2 and the current operation is an `and`, /// the `RecordBatch` will be filtered in advance. const PRE_SELECTION_THRESHOLD: f32 = 0.2; ``` # What changes are included in this PR? # Are there any user-facing changes? Added the configuration `spark.auron.forceShortCircuitAndOr` The default is false. When setting true, the short-circuit operation is forced. # How was this patch tested? local test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
