voonhous commented on issue #20527: URL: https://github.com/apache/hudi/issues/20527#issuecomment-6634697292
This refactor also lets us handle order-reversing functions, such as negative factors (`x * -2`, `x / -2`) and `datediff(k, col)`, which master has to reject ([#20457](https://github.com/apache/hudi/pull/20457), [#20230](https://github.com/apache/hudi/pull/20230)). **Why master rejects them.** Each leaf case hard-codes which stat it reads: `GreaterThan` reads `max` and wraps it in `f`, which assumes the largest value of `f(col)` is `f(max)`. That only holds for increasing `f`. For `qty * -1 > -5` on a file with `qty` in [1, 10], it checks `10 * -1 > -5`, which is false, and prunes the file although `qty = 1` matches. Supporting negative factors on master would mean changing the matcher to return a direction alongside the column (all 27 cases) and swapping min and max in each of the ~17 leaf cases that pick a bound. That is too wide for a correctness fix, so the fixes reject instead. **Why the refactor can.** It keeps "what range can `f(col)` take" (Range) apart from "which end does the operator check" (Verdict): ```scala // RangeRule.scala, Step.apply case Increasing => (r.lo, r.hi) case Decreasing => (r.hi, r.lo) // Verdict.scala case Op.Gt => GreaterThan(r.hi, ...) ``` `Range(lo, hi)` is the term's real range, and Verdict never needs to know whether `hi` came from `max` or `min`. Supporting negative factors is then a few cases in `ConstantOperandRule` that return `Decreasing`; no leaf case or interface changes. **Prototype.** A follow-up branch stacked on [#20524](https://github.com/apache/hudi/pull/20524) does this, with two exclusions where a swap alone is not enough: - Integral results: without ANSI mode, `Long.MinValue * -1` wraps to `Long.MinValue`. These wait for the overflow guard ([#20460](https://github.com/apache/hudi/issues/20460)). - Float or double columns: a NaN max would become the low bound, and `NaN < c` is false. NaN in column stats is being fixed in [#18792](https://github.com/apache/hudi/pull/18792) (open). The bugs caught earlier stay caught: | Case | Test | | --- | --- | | `CAST(A AS DOUBLE) * -1.0 < 5.0` and `/ -2.0`, which pruned a matching file ([#20457](https://github.com/apache/hudi/pull/20457)) | the original file must still be kept; a new file without matches is now pruned | | `A + A * -2` (the column on both sides) | still rejected, test unchanged | | `datediff(k, col)` ([#20230](https://github.com/apache/hudi/pull/20230)) | now flipped; the unflipped bound would prune a matching file | | two flips, `* -1.0 * -2.0` | must compose back to increasing; a single swap would prune a matching file | | integral `A * -1` with `Long.MinValue` in the file | stays unpruned | | a flip combined with column references outside the chain | no range, file kept; disabling this check makes its test fail | All passed on Spark 3.5. Before opening the follow-up PR: re-run them on the final refactor, and run the [#20439](https://github.com/apache/hudi/pull/20439) oracle (which has a `negative_factor` family) against the branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
