voonhous commented on code in PR #20524: URL: https://github.com/apache/hudi/pull/20524#discussion_r4475443896
########## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/RangeRule.scala: ########## @@ -0,0 +1,124 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.hudi.dataskipping + +import org.apache.spark.sql.catalyst.expressions.{Expression, If, Literal} +import org.apache.spark.sql.hudi.dataskipping.rules.{CastRule, ConstantOperandRule, DateTimeFormatRule, MonotonicFunctionRule, PrefixRule, TimeZoneShiftRule} + +/** + * Stage 3 (Range), rules: how does a function of the child map the child's range? + * + * A [[Step]] says: `term` is a function of `child` with this [[Direction]], so the range of `term` + * is the range of `child` pushed through it. Invariant: a rule only emits a step when its guard + * holds for this `term`; a term no rule knows has no range (and the file is kept). + * + * Emitting a Step is a correctness claim: `term` is order-preserving in `child`. + * <ul> + * <li>[[Increasing]]: a <= b implies f(a) <= f(b)</li> + * <li>[[Decreasing]]: a <= b implies f(a) >= f(b)</li> + * </ul> + * The claim must hold for every value the file's range can hold (or wherever the step's + * `fileGuard` is true). If it is false, data skipping drops files that have matching rows. When + * unsure, return Nil: that only costs pruning, never correctness. + */ +sealed trait Direction + +/** + * Order-preserving (non-decreasing): a <= b implies f(a) <= f(b), so f(lo) is the low bound and + * f(hi) the high bound. Every step today is increasing. + */ +case object Increasing extends Direction + +/** + * Order-reversing (non-increasing): a <= b implies f(a) >= f(b), so f(hi) is the low bound and + * f(lo) the high bound. Unused today; negative factors would use it. + */ +case object Decreasing extends Direction + +/** + * One rule application. + * + * @param term the function application being ranged, for ex `x + 1` + * @param child the operand whose range is pushed through, for ex `x` + * @param direction whether `term` grows or shrinks with `child` + * @param fileGuard per-file condition under which the mapping holds (for ex, no overflow); outside it + * the bounds become NULL, i.e. unknown. `None` for every step today + * @param partial whether `term` can be NULL for a non-NULL `child` (for ex, `log` of a negative), in + * which case null / value counts of the child no longer describe `term`. No behavioral + * effect yet: IS NULL / IS NOT NULL only take bare columns + */ +case class Step(term: Expression, + child: Expression, + direction: Direction, + fileGuard: Option[Range => Expression], + partial: Boolean) { + + def apply(r: Range): Range = { + val (lo, hi) = direction match { + case Increasing => (r.lo, r.hi) + case Decreasing => (r.hi, r.lo) Review Comment: I've swapped the bounds for `Drecreasing, If(guard, bound, NULL)` on both bounds, and `partial` dropping the counts as suggested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
