voonhous commented on issue #20527:
URL: https://github.com/apache/hudi/issues/20527#issuecomment-6634697292

   This refactor also lets us handle order-reversing functions, such as 
negative factors (`x * -2`, `x / -2`) and `datediff(k, col)`, which master has 
to reject ([#20457](https://github.com/apache/hudi/pull/20457), 
[#20230](https://github.com/apache/hudi/pull/20230)).
   
   **Why master rejects them.** Each leaf case hard-codes which stat it reads: 
`GreaterThan` reads `max` and wraps it in `f`, which assumes the largest value 
of `f(col)` is `f(max)`. That only holds for increasing `f`. For `qty * -1 > 
-5` on a file with `qty` in [1, 10], it checks `10 * -1 > -5`, which is false, 
and prunes the file although `qty = 1` matches. Supporting negative factors on 
master would mean changing the matcher to return a direction alongside the 
column (all 27 cases) and swapping min and max in each of the ~17 leaf cases 
that pick a bound. That is too wide for a correctness fix, so the fixes reject 
instead.
   
   **Why the refactor can.** It keeps "what range can `f(col)` take" (Range) 
apart from "which end does the operator check" (Verdict):
   
   ```scala
   // RangeRule.scala, Step.apply
   case Increasing => (r.lo, r.hi)
   case Decreasing => (r.hi, r.lo)
   
   // Verdict.scala
   case Op.Gt => GreaterThan(r.hi, ...)
   ```
   
   `Range(lo, hi)` is the term's real range, and Verdict never needs to know 
whether `hi` came from `max` or `min`. Supporting negative factors is then a 
few cases in `ConstantOperandRule` that return `Decreasing`; no leaf case or 
interface changes.
   
   **Prototype.** A follow-up branch stacked on 
[#20524](https://github.com/apache/hudi/pull/20524) does this, with two 
exclusions where a swap alone is not enough:
   
   - Integral results: without ANSI mode, `Long.MinValue * -1` wraps to 
`Long.MinValue`. These wait for the overflow guard 
([#20460](https://github.com/apache/hudi/issues/20460)).
   - Float or double columns: a NaN max would become the low bound, and `NaN < 
c` is false. NaN in column stats is being fixed in 
[#18792](https://github.com/apache/hudi/pull/18792) (open).
   
   The bugs caught earlier stay caught:
   
   | Case | Test |
   | --- | --- |
   | `CAST(A AS DOUBLE) * -1.0 < 5.0` and `/ -2.0`, which pruned a matching 
file ([#20457](https://github.com/apache/hudi/pull/20457)) | the original file 
must still be kept; a new file without matches is now pruned |
   | `A + A * -2` (the column on both sides) | still rejected, test unchanged |
   | `datediff(k, col)` ([#20230](https://github.com/apache/hudi/pull/20230)) | 
now flipped; the unflipped bound would prune a matching file |
   | two flips, `* -1.0 * -2.0` | must compose back to increasing; a single 
swap would prune a matching file |
   | integral `A * -1` with `Long.MinValue` in the file | stays unpruned |
   | a flip combined with column references outside the chain | no range, file 
kept; disabling this check makes its test fail |
   
   All passed on Spark 3.5. Before opening the follow-up PR: re-run them on the 
final refactor, and run the [#20439](https://github.com/apache/hudi/pull/20439) 
oracle (which has a `negative_factor` family) against the branch.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to