rluvaton commented on code in PR #27471:
URL: https://github.com/apache/datafusion/pull/27471#discussion_r4598133814


##########
datafusion/physical-expr/src/expressions/binary.rs:
##########
@@ -7351,4 +7435,472 @@ mod tests {
             BooleanArray::from_iter(vec![Some(true), Some(true), Some(true), 
Some(true)]);
         assert_eq!(eq_result.into_array(4).unwrap().as_boolean(), &expected);
     }
+
+    /// Tests that the right side of `OR` is only evaluated on the rows
+    /// that need it, for every selectivity (not only those that hit the
+    /// [`PRE_SELECTION_THRESHOLD`] heuristics).
+    ///
+    /// The right side is a [`Probe`] that knows which rows it is allowed to 
see.
+    mod short_circuit {
+        use super::*;
+        use datafusion_common::exec_err;
+        use std::fmt::{Display, Formatter};
+        use std::hash::Hasher;
+        use std::ops::Range;
+        use std::sync::atomic::{AtomicUsize, Ordering::Relaxed};
+
+        const N: usize = 1000;
+
+        /// Fraction of rows that need the right side. Covers the extremes and
+        /// values around the threshold, so changing 
[`PRE_SELECTION_THRESHOLD`]
+        /// won't make the tests irrelevant.
+        const SELECTIVITIES: &[f32] = &[
+            0.0,
+            0.01,
+            0.1,
+            0.25,
+            0.5,
+            0.75,
+            0.9,
+            0.99,
+            1.0,
+            PRE_SELECTION_THRESHOLD / 2.0,
+            PRE_SELECTION_THRESHOLD,
+            PRE_SELECTION_THRESHOLD * 2.0,
+            1.0 - PRE_SELECTION_THRESHOLD / 2.0,
+            1.0 - PRE_SELECTION_THRESHOLD,
+            1.0 - PRE_SELECTION_THRESHOLD * 2.0,
+        ];
+
+        // -------------------------------------------------------------------
+        // 1. stateful RHS: evaluated exactly once on each needed row
+        //    (the probe panics if it sees any other row)
+        // -------------------------------------------------------------------
+
+        #[test]
+        fn or_stateful_rhs_only_evaluated_on_needed_rows() {
+            let batch = batch();
+            for &s in SELECTIVITIES {
+                // `col_1 >= s OR probe`: only rows where col_1 < s need the 
probe
+                let probe = Probe::new(Mode::Stateful, 0.0..s);

Review Comment:
   > The ten tests could be one table-driven test:
   > 
   > * `or_rhs_not_evaluated_when_lhs_all_true` (and its nested twin) is the `s 
= 0.0` entry of `SELECTIVITIES` here, since the probe panics on any unneeded 
row.
   
   the `or_rhs_not_evaluated_when_lhs_all_true` test is focused on that case 
and test `Infallible` and not `Stateful`
   
   I prefer to keep it for readability
   
   > * The `*_with_null_lhs` tests differ only in the batch, and the `expected` 
closure is repeated six times. A loop over `(mode, nested, batch)` with one 
`expected` helper also adds the missing nested plus NULL case.
   
   I did not do a loop on purpose since I want each test to focus on one thing 
and to be easy to read on its own, I will add the missing tests
   
   having it in a single test will just make the test harder to understand:
   
   <details>
   <summary>this is what it would look like:</summary>
   
   ```rust
   
   /// `col_1 >= s OR probe`, or nested `col_1 >= s OR (col_1 < s/2 OR probe)`.
   ///
   /// A stateful probe panics and a failing probe returns `Err` on any row
   /// the LHS already decided, so both must only see the undecided rows.
   #[test]
   fn or_rhs_only_evaluated_on_needed_rows() {
       for mode in [Mode::Stateful, Mode::FailOnUnneeded] {
           for nested in [false, true] {
               for (batch, nulls) in [(batch(), false), (batch_with_nulls(), 
true)] {
                   for null_output in [None, Some(false)] {
                       for &s in SELECTIVITIES {
                           let ctx = format!(
                               "mode={mode:?} nested={nested} nulls={nulls} \
                                null_output={null_output:?} selectivity={s}"
                           );
                           let needed = if nested { s / 2.0..s } else { 0.0..s 
};
                           let probe = Probe::with_null_output(
                               mode,
                               needed.clone(),
                               null_output,
                           );
                           let expr = or_probe(s, nested, Arc::clone(&probe) as 
_);
   
                           let result = evaluate(&expr, &batch)
                               .unwrap_or_else(|e| panic!("{ctx}: {e}"));
   
                           assert_eq!(result, expected(&batch, s, nested), 
"{ctx}");
                           if matches!(mode, Mode::Stateful) {
                               assert_eq!(
                                   probe.rows.load(Relaxed),
                                   needed_rows(&batch, needed),
                                   "{ctx}"
                               );
                           }
                       }
                   }
               }
           }
       }
   }
   ```
   
   </details>
   
   > * `SELECTIVITIES` contains 0.1 and 0.9 twice (`PRE_SELECTION_THRESHOLD / 
2.0` and `1.0 - PRE_SELECTION_THRESHOLD / 2.0`).
   
   yes, but this is just because `PRE_SELECTION_THRESHOLD` is `0.2`, if I 
remove that duplicate and the value change we are missing test cases, and 
because the tests are fast I'm ok with that duplication
   
   > * `RowCountRecorder` in this module already records rows per call. A 
volatile flag on it could replace most of `Probe` and its manual 
`PartialEq`/`Hash`/`Display`.
   
   `RowCountRecorder` only counts rows. These tests check which rows the `RHS` 
sees (panicking or returning `Err` on a decided row) and need a fixed output, 
so I'd keep Probe.
   
   
   > * The `FailOnUnneeded` cases are plain SQL (`x = 0 OR 1 / x > 1` at a few 
zero ratios, nested, NULL LHS), so 
   they fit in `select.slt` next to the existing query. Only the volatile 
row-count checks need Rust.
   
   If I move the tests I would lose testing around the selectivity since it is 
a rust const.
   the table generation need dedicated table per ratio
   
   > * Nit: `Mode::Stateful` and `or_stateful_*` are driven by 
`is_volatile_node`, so `Volatile` would match `right_is_volatile`.
   
   done
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to