This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/master/pr-20742-06dc47c2d4666d36c64fa39de4980a90bb64cde9 in repository https://gitbox.apache.org/repos/asf/hudi.git
commit 3e3a4955d33ca8dd6efa2216c76ea88056afa61f Author: Indian-boult <[email protected]> AuthorDate: Thu Nov 19 07:25:45 2026 +0000 fix(spark): stop data skipping through BitwiseOr and non-constant operands (#20742) * fix(spark): stop data skipping through BitwiseOr and non-constant operands Data skipping re-applies a filter's term to the min and max stats of each file or partition, which is only right when the term is order-preserving in the column. ConstantOperandRule claimed that for BitwiseOr, which is not monotonic even with a constant (1 | 2 = 3 > 2 | 2 = 2), and for DateAdd, DateSub, DateDiff and ShiftRight with any second operand. An operand that references the column is evaluated at the same bound, so date_add(d, dayofmonth(d)) maps 2024-01-31 to 2024-03-02 but 2024-02-01 to 2024-02-02. Files and partitions holding matching rows were pruned, and the query returned too few rows. - BitwiseOr is no longer matched - DateAdd and DateSub need a constant number of days, DateDiff a constant start date, and ShiftRight a constant shift amount - The golden index filters change only for these terms, each to TRUE Fixes #20741 * fix(spark): guard date arithmetic overflow and non-constant left length DateAdd, DateSub and DateDiff add or subtract a date's int number of days with no overflow check, even under ANSI, so date_add(d, 2147483647) over d in [1969-12-31, 1970-01-02] maps the max below the min and pruned the matching file. Their steps now carry a file guard like addFits, and a file whose dates may overflow is kept. PrefixRule now needs a constant length for left: left(s, length(s) - 1) maps 'ab' to 'a' but the larger 'b' to ''. The golden file changes only on the constant date_add, date_sub and datediff lines, now guarded, and appends the new date, shiftright(a, a) and left terms. * test(spark): share the data skipping toggle in TestDataSkippingQuery Pull the data skipping off/on toggle into checkWithAndWithoutDataSkipping and use it in the non-order-preserving test, and note which of its filters reach partition stats. --- .../spark/sql/hudi/dataskipping/RangeRule.scala | 2 +- .../spark/sql/hudi/dataskipping/TermRange.scala | 8 +- .../dataskipping/rules/ConstantOperandRule.scala | 74 ++++--- .../sql/hudi/dataskipping/rules/PrefixRule.scala | 5 +- .../data-skipping/golden-index-filters.txt | 217 +++++++++++++-------- .../org/apache/hudi/DataSkippingGoldenInputs.scala | 25 ++- .../org/apache/hudi/TestDataSkippingUtils.scala | 113 ++++++++++- .../hudi/TestHoodieCatalystExpressionUtils.scala | 52 ++++- .../sql/hudi/feature/TestDataSkippingQuery.scala | 63 ++++++ 9 files changed, 428 insertions(+), 131 deletions(-) diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/RangeRule.scala b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/RangeRule.scala index 57a2798099f1..439a72340fae 100644 --- a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/RangeRule.scala +++ b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/RangeRule.scala @@ -136,7 +136,7 @@ object Step { * A family of functions sharing one guard. Not sealed: families live one per file under `rules`. * * `steps` returns every step whose guard holds, in operand order; usually zero or one, two - * for operators accepted on either side (for ex, `a | 1` and `1 | a`). The first whose child derives + * for operators accepted on either side (for ex, `a + 1` and `1 + a`). The first whose child derives * a range wins, as in a Scala match falling through to its next case. */ trait RangeRule { diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TermRange.scala b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TermRange.scala index 6ccf75ff78c8..bfe1371fd9ca 100644 --- a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TermRange.scala +++ b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TermRange.scala @@ -115,7 +115,7 @@ object TermRange { /** * The source first; otherwise the first rule step whose child derives a range. Several steps - * only arise for operators accepted on either side (for ex, `BitwiseOr`), tried in today's order. + * only arise for operators accepted on either side (for ex, `Add`), tried in today's order. */ private def viaRules(term: Expression, source: RangeSource, rules: Seq[RangeRule]): Option[Chain] = source.rangeOf(term).map(r => Chain(r, hasDecreasingStep = false)).orElse( @@ -143,9 +143,11 @@ object TermRange { /** * Re-binds EVERY reference to the column, including ones in operands off the rule chain (for ex, - * the days operand of `date_add(d, dayofmonth(d))`), matching the pre-stage translator. A [[Step]] + * the length operand of `left(s, length(s) - 1)`), matching the pre-stage translator. A [[Step]] * only rebuilds its chain slot, so the leftover references are re-bound to the same bound here. - * Such terms are not order-preserving in general; kept as is so the output stays identical. + * Such terms are not order-preserving in general (that one maps 'ab' to 'a', but the larger 'b' to + * ''), so the rules need such an operand to be constant, unless the term ignores it (for ex, the + * format of `to_unix_timestamp` on a timestamp). Kept as is so the output stays identical. * * No range when the source holds no stats for the bare column (an expression index holds only its * `f(col)`): the bound would otherwise keep a column the stats table does not have. diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala index 09b42c047f6a..3a194767e168 100644 --- a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala +++ b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala @@ -17,15 +17,15 @@ package org.apache.spark.sql.hudi.dataskipping.rules -import org.apache.spark.sql.catalyst.expressions.{Add, And, BitwiseOr, DateAdd, DateDiff, DateSub, Divide, Expression, GreaterThanOrEqual, LessThanOrEqual, Literal, Multiply, ShiftLeft, ShiftRight} +import org.apache.spark.sql.catalyst.expressions.{Add, And, DateAdd, DateDiff, DateSub, Divide, Expression, GreaterThanOrEqual, LessThanOrEqual, Literal, Multiply, ShiftLeft, ShiftRight} import org.apache.spark.sql.hudi.dataskipping.{Increasing, RangeRule, Step, TermRange} -import org.apache.spark.sql.types.{ByteType, DataType, Decimal, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType} +import org.apache.spark.sql.types.{ByteType, DataType, DateType, Decimal, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType} import scala.util.Try /** * Binary operators with the column on one side and a constant on the other: - * `Add`, `Multiply`, `Divide`, `DateAdd`, `DateSub`, `DateDiff`, `ShiftLeft`, `ShiftRight`, `BitwiseOr`. + * `Add`, `Multiply`, `Divide`, `DateAdd`, `DateSub`, `DateDiff`, `ShiftLeft`, `ShiftRight`. * * Shared guard: the other operand is constant. Today that is enforced per function: * - `Add`: float / double result and a finite constant, or decimal or integral result and a constant @@ -35,29 +35,39 @@ import scala.util.Try * - `Divide`: the divisor is a positive numeric literal (column as dividend only) * - Decimal `Add`, `Multiply` and `Divide` steps are `partial`: an overflowing bound is NULL and the * file is kept - * - `ShiftLeft`: the shift amount is constant - * - `DateAdd`, `DateSub`, `DateDiff` (end only), `ShiftRight`, `BitwiseOr`: ANY second operand is - * accepted today, constant or not + * - `ShiftLeft`, `ShiftRight`: the shift amount is constant + * - `DateAdd`, `DateSub`: the number of days is constant + * - `DateDiff` (end only): the start date is constant * - * NOTE: A column-referencing other operand could reverse the ordering (for ex, A + A * -2 = -A), and a + * NOTE: A column-referencing other operand could reverse the ordering (for ex, A + A * -2 = -A, and + * date_add(d, dayofmonth(d)) maps 2024-01-31 to 2024-03-02 but 2024-02-01 to 2024-02-02), and a * non-deterministic one is re-evaluated against the min/max bounds (for ex, A + rand()) * + * NOTE: `BitwiseOr` is not matched: it is not monotonic, even with a constant (for ex, 1 < 2 but + * 1 | 2 = 3 > 2 | 2 = 2), #20449 + * * NOTE: Integral `Add`, `Multiply` and `ShiftLeft` wrap around on overflow, so the max can map below - * the min (for ex, Long.MaxValue * 2 = -2). `Add` and `Multiply` throw instead in ANSI mode, while - * `ShiftLeft` wraps around in either mode. Their steps carry a file guard that the file's range - * does not overflow (see [[IntegralLimits]]). Float / double results round monotonically and - * saturate to infinity instead. Decimal results round HALF_UP, which is monotonic, and overflow - * into NULL (or throw in ANSI mode): their steps are `partial`, so an overflowing bound is NULL - * and the file is kept. If neither the min nor the max overflows, no value in between does (#20460) + * the min (for ex, Long.MaxValue * 2 = -2). So do `DateAdd`, `DateSub` and `DateDiff`, which add or + * subtract a date's int number of days (for ex, date_add(1970-01-02, 2147483647) is before + * date_add(1969-12-31, 2147483647)). `Add` and `Multiply` throw instead in ANSI mode, while + * `ShiftLeft` and the date functions wrap around in either mode. Their steps carry a file guard + * that the file's range does not overflow (see [[IntegralLimits]]). Float / double results round + * monotonically and saturate to infinity instead. Decimal results round HALF_UP, which is + * monotonic, and overflow into NULL (or throw in ANSI mode): their steps are `partial`, so an + * overflowing bound is NULL and the file is kept. If neither the min nor the max overflows, no + * value in between does (#20460) */ object ConstantOperandRule extends RangeRule { override def steps(term: Expression): Seq[Step] = term match { - case DateAdd(start, _) => Seq(Step(term, start, Increasing, fileGuard = None, partial = false)) - case DateSub(start, _) => Seq(Step(term, start, Increasing, fileGuard = None, partial = false)) + case DateAdd(start, days) => + integralConstant(days).filter(Days.holds).map(c => guardedStep(term, start, Days.addFits(c))).toSeq + case DateSub(start, days) => + integralConstant(days).filter(Days.holds).map(c => guardedStep(term, start, Days.subtractFits(c))).toSeq // NOTE: DateDiff(end, start) is end - start, so it reverses the ordering of its start date, - // which hence must not be matched - case DateDiff(end, _) => Seq(Step(term, end, Increasing, fileGuard = None, partial = false)) + // which hence must not be matched. A constant date evaluates to its number of days + case DateDiff(end, start) => + integralConstant(start).filter(Days.holds).map(c => guardedStep(term, end, Days.subtractFits(c))).toSeq case Add(left, right, _) if hasFloatingPointResult(term) => Seq( @@ -101,18 +111,12 @@ object ConstantOperandRule extends RangeRule { case Divide(dividend, divisor, _) if isPositiveNumericLiteral(divisor) => Seq(Step(term, dividend, Increasing, fileGuard = None, partial = hasDecimalResult(term))) - // NOTE: BitwiseOr is not monotonic (for ex, 1 < 2 but 1 | 2 = 3 > 2 | 2 = 2), #20449 - case BitwiseOr(left, right) => - Seq( - Step(term, left, Increasing, fileGuard = None, partial = false), - Step(term, right, Increasing, fileGuard = None, partial = false)) - // NOTE: Spark only shifts int and bigint values, by an int amount case ShiftLeft(value, amount) => IntegralLimits.of(term).filter(t => t.dataType == IntegerType || t.dataType == LongType).toSeq.flatMap { t => integralConstant(amount).map(n => guardedStep(term, value, t.shiftLeftFits(n))) } - case ShiftRight(value, _) => Seq(Step(term, value, Increasing, fileGuard = None, partial = false)) + case ShiftRight(value, amount) if amount.foldable => Seq(Step(term, value, Increasing, fileGuard = None, partial = false)) case _ => Nil } @@ -162,11 +166,12 @@ object ConstantOperandRule extends RangeRule { } /** - * The limits of an integral result type. Past them, integral arithmetic wraps around, except that - * `Add` and `Multiply` throw instead in ANSI mode. Each guard holds when no value of the file's range - * goes past them: the step is increasing, so checking the min and the max covers every value in - * between. A file that may overflow gets a NULL bound and is kept, and a file that holds the guard - * never evaluates an overflowing bound, so ANSI `Add` and `Multiply` do not throw. + * The limits of an integral result type, or of a date's int number of days ([[Days]]). Past them, + * integral arithmetic wraps around, except that `Add` and `Multiply` throw instead in ANSI mode. + * Each guard holds when no value of the file's range goes past them: the step is increasing, so + * checking the min and the max covers every value in between. A file that may overflow gets a NULL + * bound and is kept, and a file that holds the guard never evaluates an overflowing bound, so ANSI + * `Add` and `Multiply` do not throw. */ private case class IntegralLimits(dataType: DataType, min: Long, max: Long, bits: Int) { @@ -180,6 +185,9 @@ object ConstantOperandRule extends RangeRule { def addFits(c: Long)(r: TermRange): Expression = if (c >= 0) LessThanOrEqual(r.hi, literal(max - c)) else GreaterThanOrEqual(r.lo, literal(min - c)) + /** `x - c` does not overflow: it is x + (-c), and -c is exact as a Long (for ex, -Int.MinValue). */ + def subtractFits(c: Long)(r: TermRange): Expression = addFits(-c)(r) + /** * `x * c` does not overflow, for c > 0: min / c <= x <= max / c. Long division truncates towards * zero, so `min / c` rounds up and `max / c` rounds down, as needed. @@ -206,9 +214,17 @@ object ConstantOperandRule extends RangeRule { case ShortType => Literal(v.toShort, ShortType) case IntegerType => Literal(v.toInt, IntegerType) case LongType => Literal(v, LongType) + case DateType => Literal(v.toInt, DateType) } } + /** + * A date is its int number of days since the epoch. `DateAdd` and `DateSub` add or subtract the days + * as an int, and `DateDiff` subtracts the start date's, with no overflow check even in ANSI mode. The + * guard reads the range of the date operand, so its limits are dates. + */ + private val Days = IntegralLimits(DateType, Int.MinValue, Int.MaxValue, 32) + private object IntegralLimits { // NOTE: Expression indexes skip the resolution check, and some unresolved terms throw on `dataType` def of(term: Expression): Option[IntegralLimits] = Try(term.dataType).toOption.collect { diff --git a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/PrefixRule.scala b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/PrefixRule.scala index b67ccae23eb9..56cdafc17e8d 100644 --- a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/PrefixRule.scala +++ b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/PrefixRule.scala @@ -23,7 +23,8 @@ import org.apache.spark.sql.hudi.dataskipping.{Increasing, RangeRule, Step} /** * String prefixes: `Left`. * - * Shared guard: none; any length operand is accepted today. + * Shared guard: the length is constant. A length that references the column could reverse the + * ordering (for ex, left(s, length(s) - 1) maps 'ab' to 'a', but the larger 'b' to ''). * * NOTE: Lower/Upper are not matched: case folding does not preserve the binary ordering of strings * (for ex, "Zebra" < "apple", but lower("Zebra") = "zebra" > "apple") @@ -34,7 +35,7 @@ object PrefixRule extends RangeRule { override def steps(term: Expression): Seq[Step] = term match { // Left API change: Improve RuntimeReplaceable // https://issues.apache.org/jira/browse/SPARK-38240 - case org.apache.spark.sql.catalyst.expressions.Left(str, _) => + case org.apache.spark.sql.catalyst.expressions.Left(str, len) if len.foldable => Seq(Step(term, str, Increasing, fileGuard = None, partial = false)) case _ => Nil diff --git a/hudi-spark-datasource/hudi-spark/src/test/resources/data-skipping/golden-index-filters.txt b/hudi-spark-datasource/hudi-spark/src/test/resources/data-skipping/golden-index-filters.txt index 3f3ff8950463..36a8f263f187 100644 --- a/hudi-spark-datasource/hudi-spark/src/test/resources/data-skipping/golden-index-filters.txt +++ b/hudi-spark-datasource/hudi-spark/src/test/resources/data-skipping/golden-index-filters.txt @@ -299,40 +299,40 @@ cs (x / x) IN (0.0:double, null:double) true:boolean cs NOT (x / x) IN (0.0:double) true:boolean cs startswith((x / x), 'ab') true:boolean cs NOT startswith((x / x), 'ab') true:boolean -cs ((a | 1:int) = 0:int) (((a_minValue | 1:int) <= 0:int) AND ((a_maxValue | 1:int) >= 0:int)) -cs (0:int < (a | 1:int)) ((a_maxValue | 1:int) > 0:int) -cs ((a | 1:int) >= 0:int) ((a_maxValue | 1:int) >= 0:int) -cs NOT ((a | 1:int) = 0:int) NOT (((a_minValue | 1:int) = 0:int) AND ((a_maxValue | 1:int) = 0:int)) -cs ((a | 1:int) = cast(0:int as int)) (((a_minValue | 1:int) <= cast(0:int as int)) AND ((a_maxValue | 1:int) >= cast(0:int as int))) -cs (a | 1:int) IN (0:int, null:int) ((((a_minValue | 1:int) <= 0:int) AND ((a_maxValue | 1:int) >= 0:int)) OR (((a_minValue | 1:int) <= null:int) AND ((a_maxValue | 1:int) >= null:int))) -cs NOT (a | 1:int) IN (0:int) NOT (((a_minValue | 1:int) = 0:int) AND ((a_maxValue | 1:int) = 0:int)) +cs ((a | 1:int) = 0:int) true:boolean +cs (0:int < (a | 1:int)) true:boolean +cs ((a | 1:int) >= 0:int) true:boolean +cs NOT ((a | 1:int) = 0:int) true:boolean +cs ((a | 1:int) = cast(0:int as int)) true:boolean +cs (a | 1:int) IN (0:int, null:int) true:boolean +cs NOT (a | 1:int) IN (0:int) true:boolean cs startswith((a | 1:int), 'ab') true:boolean cs NOT startswith((a | 1:int), 'ab') true:boolean -cs ((1:int | a) = 0:int) (((1:int | a_minValue) <= 0:int) AND ((1:int | a_maxValue) >= 0:int)) -cs (0:int < (1:int | a)) ((1:int | a_maxValue) > 0:int) -cs ((1:int | a) >= 0:int) ((1:int | a_maxValue) >= 0:int) -cs NOT ((1:int | a) = 0:int) NOT (((1:int | a_minValue) = 0:int) AND ((1:int | a_maxValue) = 0:int)) -cs ((1:int | a) = cast(0:int as int)) (((1:int | a_minValue) <= cast(0:int as int)) AND ((1:int | a_maxValue) >= cast(0:int as int))) -cs (1:int | a) IN (0:int, null:int) ((((1:int | a_minValue) <= 0:int) AND ((1:int | a_maxValue) >= 0:int)) OR (((1:int | a_minValue) <= null:int) AND ((1:int | a_maxValue) >= null:int))) -cs NOT (1:int | a) IN (0:int) NOT (((1:int | a_minValue) = 0:int) AND ((1:int | a_maxValue) = 0:int)) +cs ((1:int | a) = 0:int) true:boolean +cs (0:int < (1:int | a)) true:boolean +cs ((1:int | a) >= 0:int) true:boolean +cs NOT ((1:int | a) = 0:int) true:boolean +cs ((1:int | a) = cast(0:int as int)) true:boolean +cs (1:int | a) IN (0:int, null:int) true:boolean +cs NOT (1:int | a) IN (0:int) true:boolean cs startswith((1:int | a), 'ab') true:boolean cs NOT startswith((1:int | a), 'ab') true:boolean -cs ((abs(a) | a) = 0:int) (((abs(a_minValue) | a_minValue) <= 0:int) AND ((abs(a_maxValue) | a_maxValue) >= 0:int)) -cs (0:int < (abs(a) | a)) ((abs(a_maxValue) | a_maxValue) > 0:int) -cs ((abs(a) | a) >= 0:int) ((abs(a_maxValue) | a_maxValue) >= 0:int) -cs NOT ((abs(a) | a) = 0:int) NOT (((abs(a_minValue) | a_minValue) = 0:int) AND ((abs(a_maxValue) | a_maxValue) = 0:int)) -cs ((abs(a) | a) = cast(0:int as int)) (((abs(a_minValue) | a_minValue) <= cast(0:int as int)) AND ((abs(a_maxValue) | a_maxValue) >= cast(0:int as int))) -cs (abs(a) | a) IN (0:int, null:int) ((((abs(a_minValue) | a_minValue) <= 0:int) AND ((abs(a_maxValue) | a_maxValue) >= 0:int)) OR (((abs(a_minValue) | a_minValue) <= null:int) AND ((abs(a_maxValue) | a_maxValue) >= null:int))) -cs NOT (abs(a) | a) IN (0:int) NOT (((abs(a_minValue) | a_minValue) = 0:int) AND ((abs(a_maxValue) | a_maxValue) = 0:int)) +cs ((abs(a) | a) = 0:int) true:boolean +cs (0:int < (abs(a) | a)) true:boolean +cs ((abs(a) | a) >= 0:int) true:boolean +cs NOT ((abs(a) | a) = 0:int) true:boolean +cs ((abs(a) | a) = cast(0:int as int)) true:boolean +cs (abs(a) | a) IN (0:int, null:int) true:boolean +cs NOT (abs(a) | a) IN (0:int) true:boolean cs startswith((abs(a) | a), 'ab') true:boolean cs NOT startswith((abs(a) | a), 'ab') true:boolean -cs ((a | a) = 0:int) (((a_minValue | a_minValue) <= 0:int) AND ((a_maxValue | a_maxValue) >= 0:int)) -cs (0:int < (a | a)) ((a_maxValue | a_maxValue) > 0:int) -cs ((a | a) >= 0:int) ((a_maxValue | a_maxValue) >= 0:int) -cs NOT ((a | a) = 0:int) NOT (((a_minValue | a_minValue) = 0:int) AND ((a_maxValue | a_maxValue) = 0:int)) -cs ((a | a) = cast(0:int as int)) (((a_minValue | a_minValue) <= cast(0:int as int)) AND ((a_maxValue | a_maxValue) >= cast(0:int as int))) -cs (a | a) IN (0:int, null:int) ((((a_minValue | a_minValue) <= 0:int) AND ((a_maxValue | a_maxValue) >= 0:int)) OR (((a_minValue | a_minValue) <= null:int) AND ((a_maxValue | a_maxValue) >= null:int))) -cs NOT (a | a) IN (0:int) NOT (((a_minValue | a_minValue) = 0:int) AND ((a_maxValue | a_maxValue) = 0:int)) +cs ((a | a) = 0:int) true:boolean +cs (0:int < (a | a)) true:boolean +cs ((a | a) >= 0:int) true:boolean +cs NOT ((a | a) = 0:int) true:boolean +cs ((a | a) = cast(0:int as int)) true:boolean +cs (a | a) IN (0:int, null:int) true:boolean +cs NOT (a | a) IN (0:int) true:boolean cs startswith((a | a), 'ab') true:boolean cs NOT startswith((a | a), 'ab') true:boolean cs (shiftleft(a, 1:int) = 0:int) coalesce(((if(((a_minValue >= -1073741824:int) AND (a_maxValue <= 1073741823:int)), shiftleft(a_minValue, 1:int), null:int) <= 0:int) AND (if(((a_minValue >= -1073741824:int) AND (a_maxValue <= 1073741823:int)), shiftleft(a_maxValue, 1:int), null:int) >= 0:int)), true:boolean) @@ -578,22 +578,22 @@ cs cast(f(a) as bigint) IN (0:bigint, null:bigint) true:boolean cs NOT cast(f(a) as bigint) IN (0:bigint) true:boolean cs startswith(cast(f(a) as bigint), 'ab') true:boolean cs NOT startswith(cast(f(a) as bigint), 'ab') true:boolean -cs (date_add(d, 1:int) = 0:date) ((date_add(d_minValue, 1:int) <= 0:date) AND (date_add(d_maxValue, 1:int) >= 0:date)) -cs (0:date < date_add(d, 1:int)) (date_add(d_maxValue, 1:int) > 0:date) -cs (date_add(d, 1:int) >= 0:date) (date_add(d_maxValue, 1:int) >= 0:date) -cs NOT (date_add(d, 1:int) = 0:date) NOT ((date_add(d_minValue, 1:int) = 0:date) AND (date_add(d_maxValue, 1:int) = 0:date)) -cs (date_add(d, 1:int) = cast(0:date as date)) ((date_add(d_minValue, 1:int) <= cast(0:date as date)) AND (date_add(d_maxValue, 1:int) >= cast(0:date as date))) -cs date_add(d, 1:int) IN (0:date, null:date) (((date_add(d_minValue, 1:int) <= 0:date) AND (date_add(d_maxValue, 1:int) >= 0:date)) OR ((date_add(d_minValue, 1:int) <= null:date) AND (date_add(d_maxValue, 1:int) >= null:date))) -cs NOT date_add(d, 1:int) IN (0:date) NOT ((date_add(d_minValue, 1:int) = 0:date) AND (date_add(d_maxValue, 1:int) = 0:date)) +cs (date_add(d, 1:int) = 0:date) coalesce(((if((d_maxValue <= 2147483646:date), date_add(d_minValue, 1:int), null:date) <= 0:date) AND (if((d_maxValue <= 2147483646:date), date_add(d_maxValue, 1:int), null:date) >= 0:date)), true:boolean) +cs (0:date < date_add(d, 1:int)) coalesce((if((d_maxValue <= 2147483646:date), date_add(d_maxValue, 1:int), null:date) > 0:date), true:boolean) +cs (date_add(d, 1:int) >= 0:date) coalesce((if((d_maxValue <= 2147483646:date), date_add(d_maxValue, 1:int), null:date) >= 0:date), true:boolean) +cs NOT (date_add(d, 1:int) = 0:date) coalesce(NOT ((if((d_maxValue <= 2147483646:date), date_add(d_minValue, 1:int), null:date) = 0:date) AND (if((d_maxValue <= 2147483646:date), date_add(d_maxValue, 1:int), null:date) = 0:date)), true:boolean) +cs (date_add(d, 1:int) = cast(0:date as date)) coalesce(((if((d_maxValue <= 2147483646:date), date_add(d_minValue, 1:int), null:date) <= cast(0:date as date)) AND (if((d_maxValue <= 2147483646:date), date_add(d_maxValue, 1:int), null:date) >= cast(0:date as date))), true:boolean) +cs date_add(d, 1:int) IN (0:date, null:date) coalesce((((if((d_maxValue <= 2147483646:date), date_add(d_minValue, 1:int), null:date) <= 0:date) AND (if((d_maxValue <= 2147483646:date), date_add(d_maxValue, 1:int), null:date) >= 0:date)) OR ((if((d_maxValue <= 2147483646:date), date_add(d_minValue, 1:int), null:date) <= null:date) AND (if((d_maxValue <= 2147483646:date), date_add(d_maxValue, 1:int), null:date) >= null:date))), true:boolean) +cs NOT date_add(d, 1:int) IN (0:date) coalesce(NOT ((if((d_maxValue <= 2147483646:date), date_add(d_minValue, 1:int), null:date) = 0:date) AND (if((d_maxValue <= 2147483646:date), date_add(d_maxValue, 1:int), null:date) = 0:date)), true:boolean) cs startswith(date_add(d, 1:int), 'ab') true:boolean cs NOT startswith(date_add(d, 1:int), 'ab') true:boolean -cs (date_add(d, dayofmonth(d)) = 0:date) ((date_add(d_minValue, dayofmonth(d_minValue)) <= 0:date) AND (date_add(d_maxValue, dayofmonth(d_maxValue)) >= 0:date)) -cs (0:date < date_add(d, dayofmonth(d))) (date_add(d_maxValue, dayofmonth(d_maxValue)) > 0:date) -cs (date_add(d, dayofmonth(d)) >= 0:date) (date_add(d_maxValue, dayofmonth(d_maxValue)) >= 0:date) -cs NOT (date_add(d, dayofmonth(d)) = 0:date) NOT ((date_add(d_minValue, dayofmonth(d_minValue)) = 0:date) AND (date_add(d_maxValue, dayofmonth(d_maxValue)) = 0:date)) -cs (date_add(d, dayofmonth(d)) = cast(0:date as date)) ((date_add(d_minValue, dayofmonth(d_minValue)) <= cast(0:date as date)) AND (date_add(d_maxValue, dayofmonth(d_maxValue)) >= cast(0:date as date))) -cs date_add(d, dayofmonth(d)) IN (0:date, null:date) (((date_add(d_minValue, dayofmonth(d_minValue)) <= 0:date) AND (date_add(d_maxValue, dayofmonth(d_maxValue)) >= 0:date)) OR ((date_add(d_minValue, dayofmonth(d_minValue)) <= null:date) AND (date_add(d_maxValue, dayofmonth(d_maxValue)) >= null:date))) -cs NOT date_add(d, dayofmonth(d)) IN (0:date) NOT ((date_add(d_minValue, dayofmonth(d_minValue)) = 0:date) AND (date_add(d_maxValue, dayofmonth(d_maxValue)) = 0:date)) +cs (date_add(d, dayofmonth(d)) = 0:date) true:boolean +cs (0:date < date_add(d, dayofmonth(d))) true:boolean +cs (date_add(d, dayofmonth(d)) >= 0:date) true:boolean +cs NOT (date_add(d, dayofmonth(d)) = 0:date) true:boolean +cs (date_add(d, dayofmonth(d)) = cast(0:date as date)) true:boolean +cs date_add(d, dayofmonth(d)) IN (0:date, null:date) true:boolean +cs NOT date_add(d, dayofmonth(d)) IN (0:date) true:boolean cs startswith(date_add(d, dayofmonth(d)), 'ab') true:boolean cs NOT startswith(date_add(d, dayofmonth(d)), 'ab') true:boolean cs (date_add(d, a) = 0:date) true:boolean @@ -605,31 +605,31 @@ cs date_add(d, a) IN (0:date, null:date) true:boolean cs NOT date_add(d, a) IN (0:date) true:boolean cs startswith(date_add(d, a), 'ab') true:boolean cs NOT startswith(date_add(d, a), 'ab') true:boolean -cs (date_sub(d, 3:int) = 0:date) ((date_sub(d_minValue, 3:int) <= 0:date) AND (date_sub(d_maxValue, 3:int) >= 0:date)) -cs (0:date < date_sub(d, 3:int)) (date_sub(d_maxValue, 3:int) > 0:date) -cs (date_sub(d, 3:int) >= 0:date) (date_sub(d_maxValue, 3:int) >= 0:date) -cs NOT (date_sub(d, 3:int) = 0:date) NOT ((date_sub(d_minValue, 3:int) = 0:date) AND (date_sub(d_maxValue, 3:int) = 0:date)) -cs (date_sub(d, 3:int) = cast(0:date as date)) ((date_sub(d_minValue, 3:int) <= cast(0:date as date)) AND (date_sub(d_maxValue, 3:int) >= cast(0:date as date))) -cs date_sub(d, 3:int) IN (0:date, null:date) (((date_sub(d_minValue, 3:int) <= 0:date) AND (date_sub(d_maxValue, 3:int) >= 0:date)) OR ((date_sub(d_minValue, 3:int) <= null:date) AND (date_sub(d_maxValue, 3:int) >= null:date))) -cs NOT date_sub(d, 3:int) IN (0:date) NOT ((date_sub(d_minValue, 3:int) = 0:date) AND (date_sub(d_maxValue, 3:int) = 0:date)) +cs (date_sub(d, 3:int) = 0:date) coalesce(((if((d_minValue >= -2147483645:date), date_sub(d_minValue, 3:int), null:date) <= 0:date) AND (if((d_minValue >= -2147483645:date), date_sub(d_maxValue, 3:int), null:date) >= 0:date)), true:boolean) +cs (0:date < date_sub(d, 3:int)) coalesce((if((d_minValue >= -2147483645:date), date_sub(d_maxValue, 3:int), null:date) > 0:date), true:boolean) +cs (date_sub(d, 3:int) >= 0:date) coalesce((if((d_minValue >= -2147483645:date), date_sub(d_maxValue, 3:int), null:date) >= 0:date), true:boolean) +cs NOT (date_sub(d, 3:int) = 0:date) coalesce(NOT ((if((d_minValue >= -2147483645:date), date_sub(d_minValue, 3:int), null:date) = 0:date) AND (if((d_minValue >= -2147483645:date), date_sub(d_maxValue, 3:int), null:date) = 0:date)), true:boolean) +cs (date_sub(d, 3:int) = cast(0:date as date)) coalesce(((if((d_minValue >= -2147483645:date), date_sub(d_minValue, 3:int), null:date) <= cast(0:date as date)) AND (if((d_minValue >= -2147483645:date), date_sub(d_maxValue, 3:int), null:date) >= cast(0:date as date))), true:boolean) +cs date_sub(d, 3:int) IN (0:date, null:date) coalesce((((if((d_minValue >= -2147483645:date), date_sub(d_minValue, 3:int), null:date) <= 0:date) AND (if((d_minValue >= -2147483645:date), date_sub(d_maxValue, 3:int), null:date) >= 0:date)) OR ((if((d_minValue >= -2147483645:date), date_sub(d_minValue, 3:int), null:date) <= null:date) AND (if((d_minValue >= -2147483645:date), date_sub(d_maxValue, 3:int), null:date) >= null:date))), true:boolean) +cs NOT date_sub(d, 3:int) IN (0:date) coalesce(NOT ((if((d_minValue >= -2147483645:date), date_sub(d_minValue, 3:int), null:date) = 0:date) AND (if((d_minValue >= -2147483645:date), date_sub(d_maxValue, 3:int), null:date) = 0:date)), true:boolean) cs startswith(date_sub(d, 3:int), 'ab') true:boolean cs NOT startswith(date_sub(d, 3:int), 'ab') true:boolean -cs (date_sub(d, dayofmonth(d)) = 0:date) ((date_sub(d_minValue, dayofmonth(d_minValue)) <= 0:date) AND (date_sub(d_maxValue, dayofmonth(d_maxValue)) >= 0:date)) -cs (0:date < date_sub(d, dayofmonth(d))) (date_sub(d_maxValue, dayofmonth(d_maxValue)) > 0:date) -cs (date_sub(d, dayofmonth(d)) >= 0:date) (date_sub(d_maxValue, dayofmonth(d_maxValue)) >= 0:date) -cs NOT (date_sub(d, dayofmonth(d)) = 0:date) NOT ((date_sub(d_minValue, dayofmonth(d_minValue)) = 0:date) AND (date_sub(d_maxValue, dayofmonth(d_maxValue)) = 0:date)) -cs (date_sub(d, dayofmonth(d)) = cast(0:date as date)) ((date_sub(d_minValue, dayofmonth(d_minValue)) <= cast(0:date as date)) AND (date_sub(d_maxValue, dayofmonth(d_maxValue)) >= cast(0:date as date))) -cs date_sub(d, dayofmonth(d)) IN (0:date, null:date) (((date_sub(d_minValue, dayofmonth(d_minValue)) <= 0:date) AND (date_sub(d_maxValue, dayofmonth(d_maxValue)) >= 0:date)) OR ((date_sub(d_minValue, dayofmonth(d_minValue)) <= null:date) AND (date_sub(d_maxValue, dayofmonth(d_maxValue)) >= null:date))) -cs NOT date_sub(d, dayofmonth(d)) IN (0:date) NOT ((date_sub(d_minValue, dayofmonth(d_minValue)) = 0:date) AND (date_sub(d_maxValue, dayofmonth(d_maxValue)) = 0:date)) +cs (date_sub(d, dayofmonth(d)) = 0:date) true:boolean +cs (0:date < date_sub(d, dayofmonth(d))) true:boolean +cs (date_sub(d, dayofmonth(d)) >= 0:date) true:boolean +cs NOT (date_sub(d, dayofmonth(d)) = 0:date) true:boolean +cs (date_sub(d, dayofmonth(d)) = cast(0:date as date)) true:boolean +cs date_sub(d, dayofmonth(d)) IN (0:date, null:date) true:boolean +cs NOT date_sub(d, dayofmonth(d)) IN (0:date) true:boolean cs startswith(date_sub(d, dayofmonth(d)), 'ab') true:boolean cs NOT startswith(date_sub(d, dayofmonth(d)), 'ab') true:boolean -cs (datediff(d, 19723:date) = 0:int) ((datediff(d_minValue, 19723:date) <= 0:int) AND (datediff(d_maxValue, 19723:date) >= 0:int)) -cs (0:int < datediff(d, 19723:date)) (datediff(d_maxValue, 19723:date) > 0:int) -cs (datediff(d, 19723:date) >= 0:int) (datediff(d_maxValue, 19723:date) >= 0:int) -cs NOT (datediff(d, 19723:date) = 0:int) NOT ((datediff(d_minValue, 19723:date) = 0:int) AND (datediff(d_maxValue, 19723:date) = 0:int)) -cs (datediff(d, 19723:date) = cast(0:int as int)) ((datediff(d_minValue, 19723:date) <= cast(0:int as int)) AND (datediff(d_maxValue, 19723:date) >= cast(0:int as int))) -cs datediff(d, 19723:date) IN (0:int, null:int) (((datediff(d_minValue, 19723:date) <= 0:int) AND (datediff(d_maxValue, 19723:date) >= 0:int)) OR ((datediff(d_minValue, 19723:date) <= null:int) AND (datediff(d_maxValue, 19723:date) >= null:int))) -cs NOT datediff(d, 19723:date) IN (0:int) NOT ((datediff(d_minValue, 19723:date) = 0:int) AND (datediff(d_maxValue, 19723:date) = 0:int)) +cs (datediff(d, 19723:date) = 0:int) coalesce(((if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) <= 0:int) AND (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) >= 0:int)), true:boolean) +cs (0:int < datediff(d, 19723:date)) coalesce((if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) > 0:int), true:boolean) +cs (datediff(d, 19723:date) >= 0:int) coalesce((if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) >= 0:int), true:boolean) +cs NOT (datediff(d, 19723:date) = 0:int) coalesce(NOT ((if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) = 0:int) AND (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) = 0:int)), true:boolean) +cs (datediff(d, 19723:date) = cast(0:int as int)) coalesce(((if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) <= cast(0:int as int)) AND (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) >= cast(0:int as int))), true:boolean) +cs datediff(d, 19723:date) IN (0:int, null:int) coalesce((((if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) <= 0:int) AND (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) >= 0:int)) OR ((if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) <= null:int) AND (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) >= null:int))), true:boolean) +cs NOT datediff(d, 19723:date) IN (0:int) coalesce(NOT ((if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) = 0:int) AND (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) = 0:int)), true:boolean) cs startswith(datediff(d, 19723:date), 'ab') true:boolean cs NOT startswith(datediff(d, 19723:date), 'ab') true:boolean cs (datediff(19723:date, d) = 0:int) true:boolean @@ -641,22 +641,22 @@ cs datediff(19723:date, d) IN (0:int, null:int) true:boolean cs NOT datediff(19723:date, d) IN (0:int) true:boolean cs startswith(datediff(19723:date, d), 'ab') true:boolean cs NOT startswith(datediff(19723:date, d), 'ab') true:boolean -cs (datediff(d, d) = 0:int) ((datediff(d_minValue, d_minValue) <= 0:int) AND (datediff(d_maxValue, d_maxValue) >= 0:int)) -cs (0:int < datediff(d, d)) (datediff(d_maxValue, d_maxValue) > 0:int) -cs (datediff(d, d) >= 0:int) (datediff(d_maxValue, d_maxValue) >= 0:int) -cs NOT (datediff(d, d) = 0:int) NOT ((datediff(d_minValue, d_minValue) = 0:int) AND (datediff(d_maxValue, d_maxValue) = 0:int)) -cs (datediff(d, d) = cast(0:int as int)) ((datediff(d_minValue, d_minValue) <= cast(0:int as int)) AND (datediff(d_maxValue, d_maxValue) >= cast(0:int as int))) -cs datediff(d, d) IN (0:int, null:int) (((datediff(d_minValue, d_minValue) <= 0:int) AND (datediff(d_maxValue, d_maxValue) >= 0:int)) OR ((datediff(d_minValue, d_minValue) <= null:int) AND (datediff(d_maxValue, d_maxValue) >= null:int))) -cs NOT datediff(d, d) IN (0:int) NOT ((datediff(d_minValue, d_minValue) = 0:int) AND (datediff(d_maxValue, d_maxValue) = 0:int)) +cs (datediff(d, d) = 0:int) true:boolean +cs (0:int < datediff(d, d)) true:boolean +cs (datediff(d, d) >= 0:int) true:boolean +cs NOT (datediff(d, d) = 0:int) true:boolean +cs (datediff(d, d) = cast(0:int as int)) true:boolean +cs datediff(d, d) IN (0:int, null:int) true:boolean +cs NOT datediff(d, d) IN (0:int) true:boolean cs startswith(datediff(d, d), 'ab') true:boolean cs NOT startswith(datediff(d, d), 'ab') true:boolean -cs (date_add(cast(ts as date), 1:int) = 0:date) ((date_add(cast(ts_minValue as date), 1:int) <= 0:date) AND (date_add(cast(ts_maxValue as date), 1:int) >= 0:date)) -cs (0:date < date_add(cast(ts as date), 1:int)) (date_add(cast(ts_maxValue as date), 1:int) > 0:date) -cs (date_add(cast(ts as date), 1:int) >= 0:date) (date_add(cast(ts_maxValue as date), 1:int) >= 0:date) -cs NOT (date_add(cast(ts as date), 1:int) = 0:date) NOT ((date_add(cast(ts_minValue as date), 1:int) = 0:date) AND (date_add(cast(ts_maxValue as date), 1:int) = 0:date)) -cs (date_add(cast(ts as date), 1:int) = cast(0:date as date)) ((date_add(cast(ts_minValue as date), 1:int) <= cast(0:date as date)) AND (date_add(cast(ts_maxValue as date), 1:int) >= cast(0:date as date))) -cs date_add(cast(ts as date), 1:int) IN (0:date, null:date) (((date_add(cast(ts_minValue as date), 1:int) <= 0:date) AND (date_add(cast(ts_maxValue as date), 1:int) >= 0:date)) OR ((date_add(cast(ts_minValue as date), 1:int) <= null:date) AND (date_add(cast(ts_maxValue as date), 1:int) >= null:date))) -cs NOT date_add(cast(ts as date), 1:int) IN (0:date) NOT ((date_add(cast(ts_minValue as date), 1:int) = 0:date) AND (date_add(cast(ts_maxValue as date), 1:int) = 0:date)) +cs (date_add(cast(ts as date), 1:int) = 0:date) coalesce(((if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_minValue as date), 1:int), null:date) <= 0:date) AND (if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_maxValue as date), 1:int), null:date) >= 0:date)), true:boolean) +cs (0:date < date_add(cast(ts as date), 1:int)) coalesce((if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_maxValue as date), 1:int), null:date) > 0:date), true:boolean) +cs (date_add(cast(ts as date), 1:int) >= 0:date) coalesce((if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_maxValue as date), 1:int), null:date) >= 0:date), true:boolean) +cs NOT (date_add(cast(ts as date), 1:int) = 0:date) coalesce(NOT ((if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_minValue as date), 1:int), null:date) = 0:date) AND (if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_maxValue as date), 1:int), null:date) = 0:date)), true:boolean) +cs (date_add(cast(ts as date), 1:int) = cast(0:date as date)) coalesce(((if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_minValue as date), 1:int), null:date) <= cast(0:date as date)) AND (if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_maxValue as date), 1:int), null:date) >= cast(0:date as date))), true:boolean) +cs date_add(cast(ts as date), 1:int) IN (0:date, null:date) coalesce((((if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_minValue as date), 1:int), null:date) <= 0:date) AND (if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_maxValue as date), 1:int), null:date) >= 0:date)) OR ((if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_minValue as date), 1:int), null:date) <= null:date) AND (if((tryeval(cast(ts_max [...] +cs NOT date_add(cast(ts as date), 1:int) IN (0:date) coalesce(NOT ((if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_minValue as date), 1:int), null:date) = 0:date) AND (if((tryeval(cast(ts_maxValue as date)) <= 2147483646:date), date_add(cast(ts_maxValue as date), 1:int), null:date) = 0:date)), true:boolean) cs startswith(date_add(cast(ts as date), 1:int), 'ab') true:boolean cs NOT startswith(date_add(cast(ts as date), 1:int), 'ab') true:boolean cs (date_format(ts, 'yyyy-MM-dd') = '') coalesce(((if(((ts_minValue >= -62135510400000000:timestamp) AND (ts_maxValue < 253402214400000000:timestamp)), date_format(ts_minValue, 'yyyy-MM-dd'), null:string) <= '') AND (if(((ts_minValue >= -62135510400000000:timestamp) AND (ts_maxValue < 253402214400000000:timestamp)), date_format(ts_maxValue, 'yyyy-MM-dd'), null:string) >= '')), true:boolean) @@ -821,13 +821,13 @@ cs (cast(a as bigint) + 1:bigint) IN (0:bigint, null:bigint) coalesce((((if((try cs NOT (cast(a as bigint) + 1:bigint) IN (0:bigint) coalesce(NOT ((if((tryeval(cast(a_maxValue as bigint)) <= 9223372036854775806:bigint), (cast(a_minValue as bigint) + 1:bigint), null:bigint) = 0:bigint) AND (if((tryeval(cast(a_maxValue as bigint)) <= 9223372036854775806:bigint), (cast(a_maxValue as bigint) + 1:bigint), null:bigint) = 0:bigint)), true:boolean) cs startswith((cast(a as bigint) + 1:bigint), 'ab') true:boolean cs NOT startswith((cast(a as bigint) + 1:bigint), 'ab') true:boolean -cs ((datediff(d, 19723:date) + 1:int) = 0:int) coalesce(((if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_minValue, 19723:date) + 1:int), null:int) <= 0:int) AND (if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_maxValue, 19723:date) + 1:int), null:int) >= 0:int)), true:boolean) -cs (0:int < (datediff(d, 19723:date) + 1:int)) coalesce((if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_maxValue, 19723:date) + 1:int), null:int) > 0:int), true:boolean) -cs ((datediff(d, 19723:date) + 1:int) >= 0:int) coalesce((if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_maxValue, 19723:date) + 1:int), null:int) >= 0:int), true:boolean) -cs NOT ((datediff(d, 19723:date) + 1:int) = 0:int) coalesce(NOT ((if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_minValue, 19723:date) + 1:int), null:int) = 0:int) AND (if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_maxValue, 19723:date) + 1:int), null:int) = 0:int)), true:boolean) -cs ((datediff(d, 19723:date) + 1:int) = cast(0:int as int)) coalesce(((if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_minValue, 19723:date) + 1:int), null:int) <= cast(0:int as int)) AND (if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_maxValue, 19723:date) + 1:int), null:int) >= cast(0:int as int))), true:boolean) -cs (datediff(d, 19723:date) + 1:int) IN (0:int, null:int) coalesce((((if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_minValue, 19723:date) + 1:int), null:int) <= 0:int) AND (if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_maxValue, 19723:date) + 1:int), null:int) >= 0:int)) OR ((if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_minValue, 19723:date) + 1:int), null:int) <= null:int) AND (if((tryeval(d [...] -cs NOT (datediff(d, 19723:date) + 1:int) IN (0:int) coalesce(NOT ((if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_minValue, 19723:date) + 1:int), null:int) = 0:int) AND (if((tryeval(datediff(d_maxValue, 19723:date)) <= 2147483646:int), (datediff(d_maxValue, 19723:date) + 1:int), null:int) = 0:int)), true:boolean) +cs ((datediff(d, 19723:date) + 1:int) = 0:int) coalesce(((if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) + 1:int), null:int) <= 0:int) AND (if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) + 1:int [...] +cs (0:int < (datediff(d, 19723:date) + 1:int)) coalesce((if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) + 1:int), null:int) > 0:int), true:boolean) +cs ((datediff(d, 19723:date) + 1:int) >= 0:int) coalesce((if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) + 1:int), null:int) >= 0:int), true:boolean) +cs NOT ((datediff(d, 19723:date) + 1:int) = 0:int) coalesce(NOT ((if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) + 1:int), null:int) = 0:int) AND (if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) [...] +cs ((datediff(d, 19723:date) + 1:int) = cast(0:int as int)) coalesce(((if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) + 1:int), null:int) <= cast(0:int as int)) AND (if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 1972 [...] +cs (datediff(d, 19723:date) + 1:int) IN (0:int, null:int) coalesce((((if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) + 1:int), null:int) <= 0:int) AND (if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null: [...] +cs NOT (datediff(d, 19723:date) + 1:int) IN (0:int) coalesce(NOT ((if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_minValue, 19723:date), null:int) + 1:int), null:int) = 0:int) AND (if((tryeval(if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int)) <= 2147483646:int), (if((d_minValue >= -2147463925:date), datediff(d_maxValue, 19723:date), null:int) [...] cs startswith((datediff(d, 19723:date) + 1:int), 'ab') true:boolean cs NOT startswith((datediff(d, 19723:date) + 1:int), 'ab') true:boolean cs (concat(s, s) = '') true:boolean @@ -897,7 +897,7 @@ cs shiftleft(a, 1:int) IN (0:int, null:int) coalesce((((if(((a_minValue >= -1073 cs NOT (cast(a as bigint) = 0:bigint) NOT ((cast(a_minValue as bigint) = 0:bigint) AND (cast(a_maxValue as bigint) = 0:bigint)) cs (NOT (substring(s, 1:int, 2:int) IN ('', null:string) OR (0:bigint < cast(a as bigint))) OR NOT (NOT (datediff(19723:date, d) IN (0:int, null:int) AND startswith(l, 'ab')) AND NOT ((a + b) IN (0:int, null:int) OR NOT (x / -2.0:double) IN (0.0:double)))) true:boolean cs ('' < substring(s, 1:int, 2:int)) true:boolean -cs (datediff(d, d) >= 0:int) (datediff(d_maxValue, d_maxValue) >= 0:int) +cs (datediff(d, d) >= 0:int) true:boolean cs NOT (NOT (NOT (((a * -2:int) = 0:int) OR NOT date_sub(d, dayofmonth(d)) IN (0:date)) AND NOT NOT (left(s, 2:int) = '')) OR (NOT (startswith((a * 2:int), 'ab') AND NOT ((a + a) = 0:int)) AND NOT (NOT (cast(s as int) = 0:int) OR (0:int < (a | a))))) true:boolean cs ((abs(a) | a) IN (0:int, null:int) OR ((NOT (shiftleft(a, 1:int) = 0:int) OR NOT startswith((a | 1:int), 'ab')) OR (NOT startswith((a + b), 'ab') OR NOT startswith(to_date(s), 'ab')))) true:boolean cs upper(s) IN ('', null:string) true:boolean @@ -922,7 +922,7 @@ cs (cast(a as smallint) = 0:smallint) true:boolean cs NOT ((NOT NOT startswith(left(s, 2:int), 'ab') OR NOT startswith(cast(a as bigint), 'ab')) AND NOT ((NOT startswith((dec * 1.5:decimal(2,1)), 'ab') AND ((x / x) >= 0.0:double)) AND NOT startswith(cast(d as timestamp), 'ab'))) true:boolean cs (cast(dec as double) = cast(0.0:double as double)) ((cast(dec_minValue as double) <= cast(0.0:double as double)) AND (cast(dec_maxValue as double) >= cast(0.0:double as double))) cs (NOT NOT (startswith(cast(a as smallint), 'ab') OR NOT (cast(cast(a as bigint) as double) = 0.0:double)) AND (0:int < datediff(d, d))) true:boolean -cs (datediff(d, d) >= 0:int) (datediff(d_maxValue, d_maxValue) >= 0:int) +cs (datediff(d, d) >= 0:int) true:boolean cs (((date_add(cast(ts as date), 1:int) >= 0:date) AND (to_unix_timestamp(s, 'yyyy-MM-dd') = cast(0:bigint as bigint))) OR NOT NOT ((a + b) = cast(0:int as int))) true:boolean cs NOT ((NOT (d = 0:date) OR NOT to_unix_timestamp(s, 'yyyy-MM-dd') IN (0:bigint)) AND (NOT startswith(date_format(ts, cast(ts as string)), 'ab') OR NOT (startswith((a + a), 'ab') OR ((a * a) = cast(0:int as int))))) true:boolean cs startswith((2:int * a), 'ab') true:boolean @@ -1076,3 +1076,48 @@ cs (dec / 2:decimal(10,2)) IN (0E-13:decimal(23,13), null:decimal(23,13)) coales cs NOT (dec / 2:decimal(10,2)) IN (0E-13:decimal(23,13)) coalesce(NOT ((tryeval((dec_minValue / 2:decimal(10,2))) = 0E-13:decimal(23,13)) AND (tryeval((dec_maxValue / 2:decimal(10,2))) = 0E-13:decimal(23,13))), true:boolean) cs startswith((dec / 2:decimal(10,2)), 'ab') true:boolean cs NOT startswith((dec / 2:decimal(10,2)), 'ab') true:boolean +cs (date_add(d, 2147483647:int) = 0:date) coalesce(((if((d_maxValue <= 0:date), date_add(d_minValue, 2147483647:int), null:date) <= 0:date) AND (if((d_maxValue <= 0:date), date_add(d_maxValue, 2147483647:int), null:date) >= 0:date)), true:boolean) +cs (0:date < date_add(d, 2147483647:int)) coalesce((if((d_maxValue <= 0:date), date_add(d_maxValue, 2147483647:int), null:date) > 0:date), true:boolean) +cs (date_add(d, 2147483647:int) >= 0:date) coalesce((if((d_maxValue <= 0:date), date_add(d_maxValue, 2147483647:int), null:date) >= 0:date), true:boolean) +cs NOT (date_add(d, 2147483647:int) = 0:date) coalesce(NOT ((if((d_maxValue <= 0:date), date_add(d_minValue, 2147483647:int), null:date) = 0:date) AND (if((d_maxValue <= 0:date), date_add(d_maxValue, 2147483647:int), null:date) = 0:date)), true:boolean) +cs (date_add(d, 2147483647:int) = cast(0:date as date)) coalesce(((if((d_maxValue <= 0:date), date_add(d_minValue, 2147483647:int), null:date) <= cast(0:date as date)) AND (if((d_maxValue <= 0:date), date_add(d_maxValue, 2147483647:int), null:date) >= cast(0:date as date))), true:boolean) +cs date_add(d, 2147483647:int) IN (0:date, null:date) coalesce((((if((d_maxValue <= 0:date), date_add(d_minValue, 2147483647:int), null:date) <= 0:date) AND (if((d_maxValue <= 0:date), date_add(d_maxValue, 2147483647:int), null:date) >= 0:date)) OR ((if((d_maxValue <= 0:date), date_add(d_minValue, 2147483647:int), null:date) <= null:date) AND (if((d_maxValue <= 0:date), date_add(d_maxValue, 2147483647:int), null:date) >= null:date))), true:boolean) +cs NOT date_add(d, 2147483647:int) IN (0:date) coalesce(NOT ((if((d_maxValue <= 0:date), date_add(d_minValue, 2147483647:int), null:date) = 0:date) AND (if((d_maxValue <= 0:date), date_add(d_maxValue, 2147483647:int), null:date) = 0:date)), true:boolean) +cs startswith(date_add(d, 2147483647:int), 'ab') true:boolean +cs NOT startswith(date_add(d, 2147483647:int), 'ab') true:boolean +cs (date_add(d, -2147483648:int) = 0:date) coalesce(((if((d_minValue >= 0:date), date_add(d_minValue, -2147483648:int), null:date) <= 0:date) AND (if((d_minValue >= 0:date), date_add(d_maxValue, -2147483648:int), null:date) >= 0:date)), true:boolean) +cs (0:date < date_add(d, -2147483648:int)) coalesce((if((d_minValue >= 0:date), date_add(d_maxValue, -2147483648:int), null:date) > 0:date), true:boolean) +cs (date_add(d, -2147483648:int) >= 0:date) coalesce((if((d_minValue >= 0:date), date_add(d_maxValue, -2147483648:int), null:date) >= 0:date), true:boolean) +cs NOT (date_add(d, -2147483648:int) = 0:date) coalesce(NOT ((if((d_minValue >= 0:date), date_add(d_minValue, -2147483648:int), null:date) = 0:date) AND (if((d_minValue >= 0:date), date_add(d_maxValue, -2147483648:int), null:date) = 0:date)), true:boolean) +cs (date_add(d, -2147483648:int) = cast(0:date as date)) coalesce(((if((d_minValue >= 0:date), date_add(d_minValue, -2147483648:int), null:date) <= cast(0:date as date)) AND (if((d_minValue >= 0:date), date_add(d_maxValue, -2147483648:int), null:date) >= cast(0:date as date))), true:boolean) +cs date_add(d, -2147483648:int) IN (0:date, null:date) coalesce((((if((d_minValue >= 0:date), date_add(d_minValue, -2147483648:int), null:date) <= 0:date) AND (if((d_minValue >= 0:date), date_add(d_maxValue, -2147483648:int), null:date) >= 0:date)) OR ((if((d_minValue >= 0:date), date_add(d_minValue, -2147483648:int), null:date) <= null:date) AND (if((d_minValue >= 0:date), date_add(d_maxValue, -2147483648:int), null:date) >= null:date))), true:boolean) +cs NOT date_add(d, -2147483648:int) IN (0:date) coalesce(NOT ((if((d_minValue >= 0:date), date_add(d_minValue, -2147483648:int), null:date) = 0:date) AND (if((d_minValue >= 0:date), date_add(d_maxValue, -2147483648:int), null:date) = 0:date)), true:boolean) +cs startswith(date_add(d, -2147483648:int), 'ab') true:boolean +cs NOT startswith(date_add(d, -2147483648:int), 'ab') true:boolean +cs (date_sub(d, -2147483648:int) = 0:date) coalesce(((if((d_maxValue <= -1:date), date_sub(d_minValue, -2147483648:int), null:date) <= 0:date) AND (if((d_maxValue <= -1:date), date_sub(d_maxValue, -2147483648:int), null:date) >= 0:date)), true:boolean) +cs (0:date < date_sub(d, -2147483648:int)) coalesce((if((d_maxValue <= -1:date), date_sub(d_maxValue, -2147483648:int), null:date) > 0:date), true:boolean) +cs (date_sub(d, -2147483648:int) >= 0:date) coalesce((if((d_maxValue <= -1:date), date_sub(d_maxValue, -2147483648:int), null:date) >= 0:date), true:boolean) +cs NOT (date_sub(d, -2147483648:int) = 0:date) coalesce(NOT ((if((d_maxValue <= -1:date), date_sub(d_minValue, -2147483648:int), null:date) = 0:date) AND (if((d_maxValue <= -1:date), date_sub(d_maxValue, -2147483648:int), null:date) = 0:date)), true:boolean) +cs (date_sub(d, -2147483648:int) = cast(0:date as date)) coalesce(((if((d_maxValue <= -1:date), date_sub(d_minValue, -2147483648:int), null:date) <= cast(0:date as date)) AND (if((d_maxValue <= -1:date), date_sub(d_maxValue, -2147483648:int), null:date) >= cast(0:date as date))), true:boolean) +cs date_sub(d, -2147483648:int) IN (0:date, null:date) coalesce((((if((d_maxValue <= -1:date), date_sub(d_minValue, -2147483648:int), null:date) <= 0:date) AND (if((d_maxValue <= -1:date), date_sub(d_maxValue, -2147483648:int), null:date) >= 0:date)) OR ((if((d_maxValue <= -1:date), date_sub(d_minValue, -2147483648:int), null:date) <= null:date) AND (if((d_maxValue <= -1:date), date_sub(d_maxValue, -2147483648:int), null:date) >= null:date))), true:boolean) +cs NOT date_sub(d, -2147483648:int) IN (0:date) coalesce(NOT ((if((d_maxValue <= -1:date), date_sub(d_minValue, -2147483648:int), null:date) = 0:date) AND (if((d_maxValue <= -1:date), date_sub(d_maxValue, -2147483648:int), null:date) = 0:date)), true:boolean) +cs startswith(date_sub(d, -2147483648:int), 'ab') true:boolean +cs NOT startswith(date_sub(d, -2147483648:int), 'ab') true:boolean +cs (shiftright(a, a) = 0:int) true:boolean +cs (0:int < shiftright(a, a)) true:boolean +cs (shiftright(a, a) >= 0:int) true:boolean +cs NOT (shiftright(a, a) = 0:int) true:boolean +cs (shiftright(a, a) = cast(0:int as int)) true:boolean +cs shiftright(a, a) IN (0:int, null:int) true:boolean +cs NOT shiftright(a, a) IN (0:int) true:boolean +cs startswith(shiftright(a, a), 'ab') true:boolean +cs NOT startswith(shiftright(a, a), 'ab') true:boolean +cs (left(s, (length(s) - 1:int)) = '') true:boolean +cs ('' < left(s, (length(s) - 1:int))) true:boolean +cs (left(s, (length(s) - 1:int)) >= '') true:boolean +cs NOT (left(s, (length(s) - 1:int)) = '') true:boolean +cs (left(s, (length(s) - 1:int)) = cast('' as string)) true:boolean +cs left(s, (length(s) - 1:int)) IN ('', null:string) true:boolean +cs NOT left(s, (length(s) - 1:int)) IN ('') true:boolean +cs startswith(left(s, (length(s) - 1:int)), 'ab') true:boolean +cs NOT startswith(left(s, (length(s) - 1:int)), 'ab') true:boolean diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/DataSkippingGoldenInputs.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/DataSkippingGoldenInputs.scala index aeb5240e2176..d5709d3899e8 100644 --- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/DataSkippingGoldenInputs.scala +++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/DataSkippingGoldenInputs.scala @@ -19,7 +19,7 @@ package org.apache.hudi import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.analysis.{UnresolvedAttribute, UnresolvedFunction} -import org.apache.spark.sql.catalyst.expressions.{Abs, Add, And, AttributeReference, BinaryOperator, BitwiseOr, Cast, Coalesce, Concat, DateAdd, DateDiff, DateFormatClass, DateSub, DayOfMonth, Divide, EqualNullSafe, EqualTo, Exp, Expm1, Expression, FromUnixTime, FromUTCTimestamp, GreaterThanOrEqual, In, InSet, IsNotNull, IsNull, LessThan, Literal, Log, Log10, Log1p, Log2, Lower, Multiply, Not, Or, ParseToDate, ParseToTimestamp, ShiftLeft, ShiftRight, StartsWith, Substring, TimeZoneAwareE [...] +import org.apache.spark.sql.catalyst.expressions.{Abs, Add, And, AttributeReference, BinaryOperator, BitwiseOr, Cast, Coalesce, Concat, DateAdd, DateDiff, DateFormatClass, DateSub, DayOfMonth, Divide, EqualNullSafe, EqualTo, Exp, Expm1, Expression, FromUnixTime, FromUTCTimestamp, GreaterThanOrEqual, In, InSet, IsNotNull, IsNull, Length, LessThan, Literal, Log, Log10, Log1p, Log2, Lower, Multiply, Not, Or, ParseToDate, ParseToTimestamp, ShiftLeft, ShiftRight, StartsWith, Substring, Subtra [...] import org.apache.spark.sql.catalyst.util.ArrayData import org.apache.spark.sql.types.{ArrayType, BooleanType, ByteType, DataType, DateType, Decimal, DecimalType, DoubleType, IntegerType, LongType, ShortType, StringType, StructField, StructType, TimestampType} @@ -126,14 +126,27 @@ object DataSkippingGoldenInputs { * * An int shifts by the low 5 bits of the amount, which 33 pins (a 6-bit mask would shift by 33). * The smallint and tinyint terms take a literal of their own type, as an int literal would widen - * the result to int. Decimal results overflow into NULL instead, so their ranges are partial. + * the result to int. Decimal results overflow into NULL instead, so their ranges are partial. A date + * adds or subtracts its int number of days: the extreme day counts pin each side of its guard, and + * date_sub by Int.MinValue a constant whose negation is not an int. */ private val overflowTerms: Seq[Expression] = Seq( ShiftLeft(a, Literal(33)), Add(sm, Literal(1.toShort)), Multiply(tn, Literal(3.toByte)), Add(dec, Literal(Decimal(1), DecimalType(10, 2))), - Divide(dec, Literal(Decimal(2), DecimalType(10, 2)))) + Divide(dec, Literal(Decimal(2), DecimalType(10, 2))), + DateAdd(d, Literal(Int.MaxValue)), DateAdd(d, Literal(Int.MinValue)), DateSub(d, Literal(Int.MinValue))) + + /** + * Terms whose other operand references the column (#20741), added after the golden file was first + * generated, kept out of [[terms]] for the same reason as [[dateTimeTerms]]. Neither has a range: an + * int shifts by the low 5 bits of the amount (32 >> 32 = 32 > 33 >> 33 = 16), and left(s, length(s) - 1) + * maps 'ab' to 'a', but the larger 'b' to ''. + */ + private val nonConstantOperandTerms: Seq[Expression] = Seq( + ShiftRight(a, a), + new org.apache.spark.sql.catalyst.expressions.Left(s, Subtract(Length(s), Literal(1)))) /** * Sets a time zone wherever the analyzer would, so that resolution does not depend on the Spark @@ -166,7 +179,8 @@ object DataSkippingGoldenInputs { /** * Every input, in golden file order: the leaves, a seeded sample of AND / OR / NOT trees, the - * struct and array leaves, the leaves of [[dateTimeTerms]], then the leaves of [[overflowTerms]]. + * struct and array leaves, the leaves of [[dateTimeTerms]], of [[overflowTerms]], then of + * [[nonConstantOperandTerms]]. */ def inputs: Seq[Expression] = { val allLeaves = terms.map(withTimeZone).flatMap(leaves).map(withTimeZone) @@ -185,7 +199,8 @@ object DataSkippingGoldenInputs { } allLeaves ++ (1 to 60).map(_ => tree(3)) ++ nestedLeaves ++ dateTimeTerms.map(withTimeZone).flatMap(leaves).map(withTimeZone) ++ - overflowTerms.map(withTimeZone).flatMap(leaves).map(withTimeZone) + overflowTerms.map(withTimeZone).flatMap(leaves).map(withTimeZone) ++ + nonConstantOperandTerms.map(withTimeZone).flatMap(leaves).map(withTimeZone) } /** diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSkippingUtils.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSkippingUtils.scala index 7d2926afbde7..37bea9ac87a1 100644 --- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSkippingUtils.scala +++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestDataSkippingUtils.scala @@ -565,6 +565,20 @@ object TestDataSkippingUtils { private val NyFallBackEdt0130 = new Timestamp(1793511000000L) // 2026-11-01 05:30Z private val NyFallBackEst0110 = new Timestamp(1793513400000L) // 2026-11-01 06:10Z + // The last day of a month and the first day of the next: dayofmonth drops from 31 to 1 + private val fileAcrossMonthEnd = IndexRow("file_1", valueCount = 2, + C_minValue = new Timestamp(1706702400000L), // 2024-01-31 12:00Z + C_maxValue = new Timestamp(1706788800000L), // 2024-02-01 12:00Z + C_nullCount = 0) + + // A file whose CAST(C AS DATE) spans the given days since the epoch in UTC (for ex, -1 is 1969-12-31), + // at noon so that the date does not depend on the time of day + private def fileOfDays(fileName: String, minDay: Int, maxDay: Int): IndexRow = + IndexRow(fileName, valueCount = 2, + C_minValue = new Timestamp((minDay * 24L + 12) * 3600000L), + C_maxValue = new Timestamp((maxDay * 24L + 12) * 3600000L), + C_nullCount = 0) + def dottedColumnStatsPredicates(): java.util.stream.Stream[Arguments] = { val cases = Seq( ("A >= 0 AND `profile.name` = 'alice'", Set("missing", "matching"), false), @@ -1467,7 +1481,74 @@ object TestDataSkippingUtils { C_maxValue = new Timestamp(1793494800000L), // 2026-11-01 01:00Z C_nullCount = 0) ), - Seq("file_1")) + Seq("file_1")), + // BitwiseOr is not order-preserving, even with a constant: 1 | 2 = 3, but the larger 2 | 2 = 2. + // Re-applying it over min/max stats would wrongly prune file_1 even though its row 1 matches; must + // fall back to always-true, keeping the file + arguments( + "(A | 2) = 3", + Seq(IndexRow("file_1", valueCount = 2, A_minValue = 1, A_maxValue = 2, A_nullCount = 0)), + Seq("file_1")), + arguments( + "(2 | A) = 3", + Seq(IndexRow("file_1", valueCount = 2, A_minValue = 1, A_maxValue = 2, A_nullCount = 0)), + Seq("file_1")), + // A bigint shifts by the low 6 bits of the amount, so shiftright(A, A) maps 64 to 64, but the larger + // 65 to 32. Re-applying it over min/max stats would wrongly prune file_1 even though its row 64 + // matches; must fall back to always-true, keeping the file + arguments( + "shiftright(A, A) = 64", + Seq(IndexRow("file_1", valueCount = 2, A_minValue = 64, A_maxValue = 65, A_nullCount = 0)), + Seq("file_1")), + // date_add(d, dayofmonth(d)) maps 2024-01-31 to 2024-03-02, but the later 2024-02-01 to 2024-02-02. + // Re-applying it over min/max stats would wrongly prune file_1 even though its 2024-01-31 row matches; + // must fall back to always-true, keeping the file. Same for date_sub and datediff with an operand + // that depends on the column + arguments( + "date_add(CAST(C AS DATE), dayofmonth(C)) = DATE'2024-03-02'", + Seq(fileAcrossMonthEnd), + Seq("file_1")), + arguments( + "date_sub(CAST(C AS DATE), -dayofmonth(C)) = DATE'2024-03-02'", + Seq(fileAcrossMonthEnd), + Seq("file_1")), + arguments( + "datediff(CAST(C AS DATE), trunc(CAST(C AS DATE), 'MM')) = 30", + Seq(fileAcrossMonthEnd), + Seq("file_1")), + // With a constant operand, they still translate and prune + arguments( + "shiftright(A, 1) > 10", + Seq( + IndexRow("file_1", valueCount = 2, A_minValue = 1, A_maxValue = 20, A_nullCount = 0), + IndexRow("file_2", valueCount = 2, A_minValue = 30, A_maxValue = 40, A_nullCount = 0) + ), + Seq("file_2")), + arguments( + "datediff(CAST(C AS DATE), DATE'2024-01-01') = 30", + Seq( + fileAcrossMonthEnd, + IndexRow("file_2", valueCount = 2, + C_minValue = new Timestamp(1709294400000L), // 2024-03-01 12:00Z + C_maxValue = new Timestamp(1709380800000L), // 2024-03-02 12:00Z + C_nullCount = 0) + ), + Seq("file_1")), + // left(B, length(B) - 1) maps 'ab' to 'a', but the larger 'b' to ''. Re-applying it over min/max + // stats would wrongly prune file_1 even though its 'ab' row matches; must fall back to always-true, + // keeping the file. Only OptimizeIn runs here, so left is not rewritten into substring (#19479) + arguments( + "left(B, length(B) - 1) = 'a'", + Seq(IndexRow("file_1", valueCount = 2, B_minValue = "ab", B_maxValue = "b", B_nullCount = 0)), + Seq("file_1")), + // With a constant length, it still translates and prunes + arguments( + "left(B, 1) = 'c'", + Seq( + IndexRow("file_1", valueCount = 2, B_minValue = "ab", B_maxValue = "b", B_nullCount = 0), + IndexRow("file_2", valueCount = 2, B_minValue = "ca", B_maxValue = "cb", B_nullCount = 0) + ), + Seq("file_2")) ) } @@ -1568,7 +1649,35 @@ object TestDataSkippingUtils { Seq( IndexRow("file_1", valueCount = 2, A_minValue = 1, A_maxValue = Long.MaxValue, A_nullCount = 0) ), - Seq("file_1")) + Seq("file_1")), + // date_add, date_sub and datediff add or subtract a date's int number of days without an overflow + // check, even in ANSI mode. date_add(d, 2147483647) maps 1969-12-31 (day -1) to day 2147483646, but + // 1970-01-02 (day 1) wraps around to day -2147483648, before 2024-01-01: re-applying it over min/max + // stats would wrongly prune file_2, whose 1970-01-02 row matches. At the limit (file_1, up to + // 1970-01-01) no day wraps around and the file prunes; one past it (file_2) may, and is kept + arguments( + "date_add(CAST(C AS DATE), 2147483647) < DATE'2024-01-01'", + Seq(fileOfDays("file_1", -1, 0), fileOfDays("file_2", -1, 1)), + Seq("file_2")), + // A negative number of days wraps around the other way: 1969-12-30 (day -2) maps to day 2147483647 + arguments( + "date_add(CAST(C AS DATE), -2147483647) > DATE'2024-01-01'", + Seq(fileOfDays("file_1", -1, 0), fileOfDays("file_2", -2, 0)), + Seq("file_2")), + arguments( + "date_sub(CAST(C AS DATE), 2147483647) > DATE'2024-01-01'", + Seq(fileOfDays("file_1", -1, 0), fileOfDays("file_2", -2, 0)), + Seq("file_2")), + arguments( + "date_sub(CAST(C AS DATE), -2147483647) < DATE'2024-01-01'", + Seq(fileOfDays("file_1", -1, 0), fileOfDays("file_2", -1, 1)), + Seq("file_2")), + // datediff(d, start) is d - start: from a start date of day -2147483647, 1970-01-02 (day 1) wraps + // around to -2147483648 days + arguments( + "datediff(CAST(C AS DATE), date_sub(DATE'1970-01-01', 2147483647)) < 0", + Seq(fileOfDays("file_1", -1, 0), fileOfDays("file_2", -1, 1)), + Seq("file_2")) ) } } diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieCatalystExpressionUtils.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieCatalystExpressionUtils.scala index 93569202e0ce..afaac4dcac52 100644 --- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieCatalystExpressionUtils.scala +++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieCatalystExpressionUtils.scala @@ -17,7 +17,7 @@ package org.apache.hudi -import org.apache.spark.sql.catalyst.expressions.{Add, AttributeReference, BitwiseOr, Cast, DateAdd, DateDiff, DateFormatClass, DateSub, Divide, Exp, Expression, FromUnixTime, FromUTCTimestamp, GetTimestamp, Literal, Log, Lower, Multiply, ParseToDate, ParseToTimestamp, Rand, ShiftLeft, Sqrt, ToUnixTimestamp, ToUTCTimestamp, UnixTimestamp, Upper} +import org.apache.spark.sql.catalyst.expressions.{Add, AttributeReference, BitwiseOr, Cast, DateAdd, DateDiff, DateFormatClass, DateSub, DayOfMonth, Divide, Exp, Expression, FromUnixTime, FromUTCTimestamp, GetTimestamp, Length, Literal, Log, Lower, Multiply, ParseToDate, ParseToTimestamp, Rand, ShiftLeft, ShiftRight, Sqrt, Subtract, ToUnixTimestamp, ToUTCTimestamp, TruncDate, UnaryMinus, UnixTimestamp, Upper} import org.apache.spark.sql.hudi.dataskipping.{ColumnStatsSource, TermRange} import org.apache.spark.sql.types.{BinaryType, BooleanType, ByteType, DataType, DateType, Decimal, DecimalType, DoubleType, FloatType, IntegerType, LongType, ShortType, StringType, TimestampType} import org.junit.jupiter.api.Assertions.{assertEquals, assertTrue} @@ -73,8 +73,15 @@ class TestHoodieCatalystExpressionUtils extends SparkAdapterSupport { assertEquals(Some(dblAttr), matched(Multiply(dblAttr, Literal(2.0d)))) assertEquals(Some(dblAttr), matched(Multiply(Literal(2.0d), dblAttr))) assertEquals(Some(intAttr), matched(Divide(intAttr, Literal(2)))) - assertEquals(Some(intAttr), matched(BitwiseOr(intAttr, Literal(1)))) - assertEquals(Some(intAttr), matched(BitwiseOr(Literal(1), intAttr))) + assertEquals(Some(longAttr), matched(ShiftRight(longAttr, Literal(2)))) + } + + @Test + def testBitwiseOrDoesNotPreserveOrdering(): Unit = { + // 1 < 2, but 1 | 2 = 3 > 2 | 2 = 2, whichever side the constant is on. + assertEquals(None, matched(BitwiseOr(intAttr, Literal(2)))) + assertEquals(None, matched(BitwiseOr(Literal(2), intAttr))) + assertEquals(None, matched(BitwiseOr(intAttr, intAttr))) } @Test @@ -106,6 +113,14 @@ class TestHoodieCatalystExpressionUtils extends SparkAdapterSupport { assertEquals(Some(false), guarded(Add(fltAttr, Literal(1.0f)), fltAttr)) assertEquals(Some(false), guarded(Multiply(Literal(2.0f), fltAttr), fltAttr)) assertEquals(Some(false), guarded(Add(Cast(longAttr, DoubleType), Literal(1.0d)), longAttr)) + // A date adds or subtracts its int number of days, which wraps around in either mode, so the range + // is guarded too, with int limits for a smallint number of days as well. + assertEquals(Some(true), guarded(DateAdd(dateAttr, Literal(1)), dateAttr)) + assertEquals(Some(true), guarded(DateAdd(dateAttr, Literal(1.toShort)), dateAttr)) + assertEquals(Some(true), guarded(DateSub(dateAttr, Literal(Int.MinValue)), dateAttr)) + assertEquals(Some(true), guarded(DateDiff(dateAttr, Literal(java.sql.Date.valueOf("2022-03-07"))), dateAttr)) + // An unresolved number of days outside an int has no limits within it. + assertEquals(None, guarded(DateAdd(dateAttr, Literal(Long.MaxValue)), dateAttr)) } @Test @@ -166,6 +181,37 @@ class TestHoodieCatalystExpressionUtils extends SparkAdapterSupport { assertEquals(Some(dateAttr), matched(DateDiff(dateAttr, Literal(java.sql.Date.valueOf("2022-03-07"))))) } + @Test + def testDateArithmeticAndShiftRightRequireConstantOtherOperand(): Unit = { + // date_add(d, dayofmonth(d)) maps 2024-01-31 to 2024-03-02, past the 2024-02-02 of the later + // 2024-02-01, so an operand referencing the column must not match. + assertEquals(None, matched(DateAdd(dateAttr, DayOfMonth(dateAttr)))) + assertEquals(None, matched(DateSub(dateAttr, UnaryMinus(DayOfMonth(dateAttr))))) + assertEquals(None, matched(DateDiff(dateAttr, TruncDate(dateAttr, Literal("MM"))))) + // A bigint shifts by the low 6 bits of the amount: shiftright(64, 64) = 64 > shiftright(65, 65) = 32. + assertEquals(None, matched(ShiftRight(longAttr, longAttr))) + // Data skipping re-evaluates the operand against the min/max bounds, so it must be a constant. + assertEquals(None, matched(DateAdd(dateAttr, Cast(Rand(Literal(0L)), IntegerType)))) + assertEquals(None, matched(ShiftRight(longAttr, Cast(Rand(Literal(0L)), IntegerType)))) + // A foldable operand is accepted, as the optimizer would fold it into a literal. + assertEquals(Some(dateAttr), matched(DateAdd(dateAttr, Add(Literal(1), Literal(2))))) + assertEquals(Some(dateAttr), matched(DateSub(dateAttr, Add(Literal(1), Literal(2))))) + assertEquals(Some(longAttr), matched(ShiftRight(longAttr, Add(Literal(1), Literal(2))))) + } + + @Test + def testLeftRequiresConstantLength(): Unit = { + def left(str: Expression, len: Expression): Expression = new org.apache.spark.sql.catalyst.expressions.Left(str, len) + // left(s, length(s) - 1) maps 'ab' to 'a', but the larger 'b' to '', so a length referencing the + // column must not match. + assertEquals(None, matched(left(strAttr, Subtract(Length(strAttr), Literal(1))))) + // Data skipping re-evaluates the length against the min/max bounds, so it must be a constant. + assertEquals(None, matched(left(strAttr, Cast(Rand(Literal(0L)), IntegerType)))) + // A foldable length is accepted, as the optimizer would fold it into a literal. + assertEquals(Some(strAttr), matched(left(strAttr, Literal(2)))) + assertEquals(Some(strAttr), matched(left(strAttr, Add(Literal(1), Literal(1))))) + } + @Test def testUtcTimestampConversionsRequireFixedOffsetZone(): Unit = { // A zone with DST runs local time backwards across a fall-back and shifts the spring-forward gap. diff --git a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala index bcef6459e144..a94978f19dc4 100644 --- a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala +++ b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala @@ -39,6 +39,19 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase { initQueryIndexConf() } + /** + * Runs `check` with data skipping off, as the caller's session conf has it, then again with data + * skipping on in the session conf and the table properties. + */ + private def checkWithAndWithoutDataSkipping(tableName: String)(check: => Unit): Unit = { + check + withSQLConf("hoodie.enable.data.skipping" -> "true") { + spark.sql(s"alter table $tableName set tblproperties ('hoodie.enable.data.skipping' = 'true')") + spark.catalog.refreshTable(tableName) + check + } + } + test("Test the data skipping query involves conditions " + "that cover both columns supported by column stats and those that are not supported.") { Seq("cow", "mor").foreach { tableType => @@ -361,6 +374,56 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase { } } + test("Data skipping keeps files for terms that are not order-preserving in the column") { + // With partition stats on, PartitionStatsIndexSupport.filterNonSqlExpressions drops the date_add, + // date_sub and shiftright(a, a) filters before translation (it matches DateAdd, DateSub and the cast + // of the shift amount to int), so only the two | filters and datediff exercise partition stats + Seq(false, true).foreach { partitionStats => + withTempDir { tmp => + val tableName = generateTableName + withSQLConf( + "hoodie.metadata.enable" -> "true", + "hoodie.metadata.index.column.stats.enable" -> "true", + "hoodie.metadata.index.partition.stats.enable" -> partitionStats.toString, + "hoodie.metadata.index.column.stats.column.list" -> "a,d", + "hoodie.enable.data.skipping" -> "false", + "hoodie.fileIndex.dataSkippingFailureMode" -> "strict" + ) { + spark.sql( + s""" + |create table $tableName (id int, a bigint, d date, ts long, dt string) using hudi + | tblproperties (primaryKey = 'id', orderingFields = 'ts', type = 'cow') + | partitioned by (dt) + | location '${tmp.getCanonicalPath}' + |""".stripMargin) + // One file per partition, so that each file's min and max come from two rows + spark.sql(s"insert into $tableName values (1, 1, date'2024-01-31', 1, 'par1'), " + + "(2, 2, date'2024-02-01', 1, 'par1')") + spark.sql(s"insert into $tableName values (3, 64, date'2024-05-10', 1, 'par2'), " + + "(4, 65, date'2024-05-11', 1, 'par2')") + assert(spark.sql(s"select input_file_name() from $tableName").collect().map(_.getString(0)).toSet.size == 2) + // Each term maps the min of its file above the max: 1 | 2 = 3 > 2 | 2 = 2, a bigint shifts by + // the low 6 bits of the amount (64 >> 64 = 64 > 65 >> 65 = 32), and dayofmonth drops from 31 to + // 1 at the end of January + val queries = Seq( + "(a | 2) = 3" -> 1, + "(2 | a) = 3" -> 1, + "shiftright(a, a) = 64" -> 3, + "date_add(d, dayofmonth(d)) = date'2024-03-02'" -> 1, + "date_sub(d, -dayofmonth(d)) = date'2024-03-02'" -> 1, + "datediff(d, trunc(d, 'MM')) = 30" -> 1) + checkWithAndWithoutDataSkipping(tableName) { + queries.foreach { case (filter, id) => + withClue(s"$filter: ") { + checkAnswer(s"select id from $tableName where $filter")(Seq(id)) + } + } + } + } + } + } + } + test("Partition statistics survive MOR delete-only inline log appends") { withTempDir { tmp => val tableName = generateTableName
