wombatu-kun commented on code in PR #20524: URL: https://github.com/apache/hudi/pull/20524#discussion_r4475892875
########## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/RangeRule.scala: ########## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.hudi.dataskipping + +import org.apache.spark.sql.catalyst.expressions.{Expression, If, Literal} +import org.apache.spark.sql.hudi.dataskipping.rules.{CastRule, ConstantOperandRule, DateTimeFormatRule, MonotonicFunctionRule, PrefixRule, TimeZoneShiftRule} + +/** + * Stage 3 (Range), rules: how does a function of the child map the child's range? + * + * A [[Step]] says: `term` is a function of `child` with this [[Direction]], so the range of `term` + * is the range of `child` pushed through it. Invariant: a rule only emits a step when its guard + * holds for this `term`; a term no rule knows has no range (and the file is kept). + * + * Emitting a Step is a correctness claim: `term` is order-preserving in `child`. + * <ul> + * <li>[[Increasing]]: a <= b implies f(a) <= f(b)</li> + * <li>[[Decreasing]]: a <= b implies f(a) >= f(b)</li> + * </ul> + * The claim must hold for every value the file's range can hold (or wherever the step's + * `fileGuard` is true). If it is false, data skipping drops files that have matching rows. When + * unsure, return Nil: that only costs pruning, never correctness. + */ +sealed trait Direction + +/** + * Order-preserving (non-decreasing): a <= b implies f(a) <= f(b), so f(lo) is the low bound and + * f(hi) the high bound. Every step today is increasing. + */ +case object Increasing extends Direction + +/** + * Order-reversing (non-increasing): a <= b implies f(a) >= f(b), so f(hi) is the low bound and + * f(lo) the high bound. Unused today; negative factors would use it. + * + * NOTE: A chain with a decreasing step derives no range when the term also references the column + * off the chain; see [[TermRange.derive]] + */ +case object Decreasing extends Direction + +/** + * One rule application. + * + * @param term the function application being ranged, for ex `x + 1` + * @param child the operand whose range is pushed through, for ex `x` + * @param direction whether `term` grows or shrinks with `child` + * @param fileGuard per-file condition under which the mapping holds (for ex, no overflow); outside it + * the bounds become NULL, i.e. unknown. `None` for every step today Review Comment: Outside its `fileGuard` a step's bounds become NULL, but while `keepOnNull` is still the identity a NULL verdict prunes the file rather than keeping it, so a rule that adds a guard before #19474 lands would drop files with matching rows. Is the plan that no guard lands before #19474, and if so could the code enforce it, for ex by `TermRange.derive` deriving no range for a guarded chain until then, as it already does for a `Decreasing` chain with off-chain references? ########## hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/Leaf.scala: ########## @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.hudi.dataskipping + +import org.apache.spark.sql.catalyst.expressions.{AttributeReference, EqualNullSafe, EqualTo, Expression, GreaterThan, GreaterThanOrEqual, In, InSet, IsNotNull, IsNull, LessThan, LessThanOrEqual, Literal, Not, StartsWith, SubqueryExpression} +import org.apache.spark.unsafe.types.UTF8String + +/** + * Stage 2 (Leaf): is this predicate a single `term OP constant` the index can answer? + * + * Invariants: + * - The term references exactly one column and holds no subquery; the values reference no column. + * - The term is put on the left in exactly one place (`comparison`); nothing downstream flips. + * - IS NULL / IS NOT NULL only take a bare column. + * - The only NOTs understood here are NotEq, NotIn and NotStartsWith; any other NOT is not a leaf. + */ +sealed trait Op + +object Op { + case object Eq extends Op + case object NotEq extends Op + case object Lt extends Op + case object LtEq extends Op + case object Gt extends Op + case object GtEq extends Op + case object In extends Op + case object NotIn extends Op + case object StartsWith extends Op + case object NotStartsWith extends Op + case object IsNull extends Op + case object IsNotNull extends Op +} + +case class Leaf(term: Expression, op: Op, values: Seq[Expression]) + +object Leaf { + + def parse(e: Expression): Option[Leaf] = e match { + case EqualTo(left, right) => comparison(left, Op.Eq, right) + case Not(EqualTo(left, right)) => comparison(left, Op.NotEq, right) + + // "colA <=> null" is "colA IS NULL" + case EqualNullSafe(column: AttributeReference, Literal(null, _)) => Some(Leaf(column, Op.IsNull, Nil)) + + case LessThan(left, right) => comparison(left, Op.Lt, right) + case LessThanOrEqual(left, right) => comparison(left, Op.LtEq, right) + case GreaterThan(left, right) => comparison(left, Op.Gt, right) + case GreaterThanOrEqual(left, right) => comparison(left, Op.GtEq, right) + + case IsNull(column: AttributeReference) => Some(Leaf(column, Op.IsNull, Nil)) + case IsNotNull(column: AttributeReference) => Some(Leaf(column, Op.IsNotNull, Nil)) + + case In(term, list) if isTerm(term) && list.forall(isValueExpression) => Some(Leaf(term, Op.In, list)) + // NOTE: [[InSet]] is the optimized [[In]] over static values. [[Literal#apply]] does not accept + // [[UTF8String]], so strings are converted first + case InSet(term, hset) if isTerm(term) => Some(Leaf(term, Op.In, hset.toSeq.map(toLiteral))) Review Comment: `toLiteral` now runs on every `InSet` before any range check, and `Literal.apply` throws on struct and array values, so an IN list of more than 10 arrays (or structs) on such a column drops column stats pruning for the whole query, or fails it under `hoodie.fileIndex.dataSkippingFailureMode=strict`, where the base returned TRUE for the unindexed column. Could the literals be built only once `TermRange.derive` has found a range, as before, with a struct or array `InSet` in the golden inputs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
