Indian-boult commented on code in PR #20742:
URL: https://github.com/apache/hudi/pull/20742#discussion_r4623822006


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala:
##########
@@ -53,11 +57,11 @@ import scala.util.Try
 object ConstantOperandRule extends RangeRule {
 
   override def steps(term: Expression): Seq[Step] = term match {
-    case DateAdd(start, _) => Seq(Step(term, start, Increasing, fileGuard = 
None, partial = false))
-    case DateSub(start, _) => Seq(Step(term, start, Increasing, fileGuard = 
None, partial = false))
+    case DateAdd(start, days) if days.foldable => Seq(Step(term, start, 
Increasing, fileGuard = None, partial = false))

Review Comment:
   Added a guard like `addFits` in 66f799ae5. `DateAdd` and `DateSub` with a 
constant day count now emit a `guardedStep` over int day limits (`Days`, with 
`subtractFits(c) = addFits(-c)`). `DateDiff` gets the same guard, since its 
eval and codegen are a bare int subtract too, from Spark 3.3 to 4.1. The 
overflow NOTE lists all three.
   
   Tests: your repro, plus a file at the limit, both signs of the day count and 
a `datediff` case in `testIntegralOverflowSource` (ANSI on and off). There are 
also guarded cases in `testOverflowingArithmeticIsGuarded` and three appended 
golden terms.
   



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TermRange.scala:
##########
@@ -143,7 +143,7 @@ object TermRange {
 
   /**
    * Re-binds EVERY reference to the column, including ones in operands off 
the rule chain (for ex,
-   * the days operand of `date_add(d, dayofmonth(d))`), matching the pre-stage 
translator. A [[Step]]
+   * the length operand of `left(s, length(s))`), matching the pre-stage 
translator. A [[Step]]

Review Comment:
   Added `if len.foldable` to the `Left` arm and changed the doc example to 
`left(s, length(s) - 1)` in 66f799ae5. Tests: `testLeftRequiresConstantLength`, 
plus two cases in `testNonOrderPreservingTransformationsSource`: `left(B, 
length(B) - 1) = 'a'` keeps the file, and `left(B, 1) = 'c'` still prunes. That 
source only runs OptimizeIn, so `Left` reaches the rule.
   



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala:
##########
@@ -361,6 +361,57 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase 
{
     }
   }
 
+  test("Data skipping keeps files for terms that are not order-preserving in 
the column") {
+    Seq(false, true).foreach { partitionStats =>
+      withTempDir { tmp =>
+        val tableName = generateTableName
+        withSQLConf(
+          "hoodie.metadata.enable" -> "true",
+          "hoodie.metadata.index.column.stats.enable" -> "true",
+          "hoodie.metadata.index.partition.stats.enable" -> 
partitionStats.toString,
+          "hoodie.metadata.index.column.stats.column.list" -> "a,d",
+          "hoodie.enable.data.skipping" -> "false",
+          "hoodie.fileIndex.dataSkippingFailureMode" -> "strict"
+        ) {
+          spark.sql(
+            s"""
+               |create table $tableName (id int, a bigint, d date, ts long, dt 
string) using hudi
+               | tblproperties (primaryKey = 'id', orderingFields = 'ts', type 
= 'cow')
+               | partitioned by (dt)
+               | location '${tmp.getCanonicalPath}'
+               |""".stripMargin)
+          // One file per partition, so that each file's min and max come from 
two rows
+          spark.sql(s"insert into $tableName values (1, 1, date'2024-01-31', 
1, 'par1'), " +
+            "(2, 2, date'2024-02-01', 1, 'par1')")
+          spark.sql(s"insert into $tableName values (3, 64, date'2024-05-10', 
1, 'par2'), " +
+            "(4, 65, date'2024-05-11', 1, 'par2')")
+          assert(spark.sql(s"select input_file_name() from 
$tableName").collect().map(_.getString(0)).toSet.size == 2)
+          // Each term maps the min of its file above the max: 1 | 2 = 3 > 2 | 
2 = 2, a bigint shifts by
+          // the low 6 bits of the amount (64 >> 64 = 64 > 65 >> 65 = 32), and 
dayofmonth drops from 31 to
+          // 1 at the end of January
+          val queries = Seq(
+            "(a | 2) = 3" -> 1,
+            "(2 | a) = 3" -> 1,
+            "shiftright(a, a) = 64" -> 3,
+            "date_add(d, dayofmonth(d)) = date'2024-03-02'" -> 1,
+            "date_sub(d, -dayofmonth(d)) = date'2024-03-02'" -> 1,
+            "datediff(d, trunc(d, 'MM')) = 30" -> 1)
+          def checkQueries(): Unit = queries.foreach { case (filter, id) =>
+            withClue(s"$filter: ") {
+              checkAnswer(s"select id from $tableName where $filter")(Seq(id))
+            }
+          }
+          checkQueries()
+          withSQLConf("hoodie.enable.data.skipping" -> "true") {
+            spark.sql(s"alter table $tableName set tblproperties 
('hoodie.enable.data.skipping' = 'true')")

Review Comment:
   Added `checkWithAndWithoutDataSkipping(tableName)(check)` in ad1dc6e3e and 
used it in the new test. I left the four older copies alone for now.
   



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala:
##########
@@ -101,18 +105,12 @@ object ConstantOperandRule extends RangeRule {
     case Divide(dividend, divisor, _) if isPositiveNumericLiteral(divisor) =>
       Seq(Step(term, dividend, Increasing, fileGuard = None, partial = 
hasDecimalResult(term)))
 
-    // NOTE: BitwiseOr is not monotonic (for ex, 1 < 2 but 1 | 2 = 3 > 2 | 2 = 
2), #20449
-    case BitwiseOr(left, right) =>
-      Seq(
-        Step(term, left, Increasing, fileGuard = None, partial = false),
-        Step(term, right, Increasing, fileGuard = None, partial = false))
-
     // NOTE: Spark only shifts int and bigint values, by an int amount
     case ShiftLeft(value, amount) =>
       IntegralLimits.of(term).filter(t => t.dataType == IntegerType || 
t.dataType == LongType).toSeq.flatMap { t =>
         integralConstant(amount).map(n => guardedStep(term, value, 
t.shiftLeftFits(n)))
       }
-    case ShiftRight(value, _) => Seq(Step(term, value, Increasing, fileGuard = 
None, partial = false))
+    case ShiftRight(value, amount) if amount.foldable => Seq(Step(term, value, 
Increasing, fileGuard = None, partial = false))

Review Comment:
   Appended `ShiftRight(a, a)` and `left(s, length(s) - 1)` as a new 
`nonConstantOperandTerms` golden list in 66f799ae5: 18 new lines, all 
`true:boolean`. The date overflow terms went into `overflowTerms`. No other 
existing golden line changed, apart from the constant date lines from the other 
thread.
   



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala:
##########
@@ -361,6 +361,57 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase 
{
     }
   }
 
+  test("Data skipping keeps files for terms that are not order-preserving in 
the column") {
+    Seq(false, true).foreach { partitionStats =>

Review Comment:
   Confirmed. `filterNonSqlExpressions` matches `DateAdd`, `DateSub` and any 
`UnaryExpression`, and that includes the `cast(a as int)` the analyzer adds to 
the bigint shift amount. `datediff`/`trunc` and the folded `|` filters have 
none of these. I added a comment above the loop saying which filters reach 
partition stats, in ad1dc6e3e.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to