voonhous commented on code in PR #20524: URL: https://github.com/apache/hudi/pull/20524#discussion_r4476242745
########## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/RangeRule.scala: ########## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.hudi.dataskipping + +import org.apache.spark.sql.catalyst.expressions.{Expression, If, Literal} +import org.apache.spark.sql.hudi.dataskipping.rules.{CastRule, ConstantOperandRule, DateTimeFormatRule, MonotonicFunctionRule, PrefixRule, TimeZoneShiftRule} + +/** + * Stage 3 (Range), rules: how does a function of the child map the child's range? + * + * A [[Step]] says: `term` is a function of `child` with this [[Direction]], so the range of `term` + * is the range of `child` pushed through it. Invariant: a rule only emits a step when its guard + * holds for this `term`; a term no rule knows has no range (and the file is kept). + * + * Emitting a Step is a correctness claim: `term` is order-preserving in `child`. + * <ul> + * <li>[[Increasing]]: a <= b implies f(a) <= f(b)</li> + * <li>[[Decreasing]]: a <= b implies f(a) >= f(b)</li> + * </ul> + * The claim must hold for every value the file's range can hold (or wherever the step's + * `fileGuard` is true). If it is false, data skipping drops files that have matching rows. When + * unsure, return Nil: that only costs pruning, never correctness. + */ +sealed trait Direction + +/** + * Order-preserving (non-decreasing): a <= b implies f(a) <= f(b), so f(lo) is the low bound and + * f(hi) the high bound. Every step today is increasing. + */ +case object Increasing extends Direction + +/** + * Order-reversing (non-increasing): a <= b implies f(a) >= f(b), so f(hi) is the low bound and + * f(lo) the high bound. Unused today; negative factors would use it. + * + * NOTE: A chain with a decreasing step derives no range when the term also references the column + * off the chain; see [[TermRange.derive]] + */ +case object Decreasing extends Direction + +/** + * One rule application. + * + * @param term the function application being ranged, for ex `x + 1` + * @param child the operand whose range is pushed through, for ex `x` + * @param direction whether `term` grows or shrinks with `child` + * @param fileGuard per-file condition under which the mapping holds (for ex, no overflow); outside it + * the bounds become NULL, i.e. unknown. `None` for every step today Review Comment: Yes, no guard should land before #19474. Enforced in 0849bb0b7e62: `TermRange.derive` gives no range for a chain with a `fileGuard` step, next to the `Decreasing` check, and the `keepOnNull` note says #19474 lifts it. `testDeriveGivesNoRangeForGuardedChain` pins it with a stub guarded rule. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
