voonhous commented on code in PR #20742:
URL: https://github.com/apache/hudi/pull/20742#discussion_r4623621015


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/TermRange.scala:
##########
@@ -143,7 +143,7 @@ object TermRange {
 
   /**
    * Re-binds EVERY reference to the column, including ones in operands off 
the rule chain (for ex,
-   * the days operand of `date_add(d, dayofmonth(d))`), matching the pre-stage 
translator. A [[Step]]
+   * the length operand of `left(s, length(s))`), matching the pre-stage 
translator. A [[Step]]

Review Comment:
   **minor:** After this PR, `PrefixRule`'s `Left(str, _)` is the only arm 
still taking a column-dependent operand, and `left(s, length(s))` is the 
identity, so it does not show the hazard; `left(s, length(s) - 1)` does ('ab' 
maps to 'a', the larger 'b' to ''). It is dead post-optimizer today but goes 
live once #19479 matches `Substring`. Not blocking. Could we add `if 
len.foldable` to that arm here and use the reversing form as the example?



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala:
##########
@@ -53,11 +57,11 @@ import scala.util.Try
 object ConstantOperandRule extends RangeRule {
 
   override def steps(term: Expression): Seq[Step] = term match {
-    case DateAdd(start, _) => Seq(Step(term, start, Increasing, fileGuard = 
None, partial = false))
-    case DateSub(start, _) => Seq(Step(term, start, Increasing, fileGuard = 
None, partial = false))
+    case DateAdd(start, days) if days.foldable => Seq(Step(term, start, 
Increasing, fileGuard = None, partial = false))

Review Comment:
   **minor:** Not introduced here, but `date_add`/`date_sub` with a constant 
can still wrap: Spark's `nullSafeEval` is a bare int add/subtract in 3.5 and 
4.x, even under ANSI, so `date_add(d, 2147483647) < DATE'2024-01-01'` over d in 
[1969-12-31, 1970-01-02] prunes the matching file (reproduced locally in 
`testNonOrderPreservingTransformationsSource`). Contrived, so not blocking. 
Could we add a file guard like #20532's `addFits` to these two arms, or at 
least list them in the overflow NOTE and track it under #20449?



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/rules/ConstantOperandRule.scala:
##########
@@ -101,18 +105,12 @@ object ConstantOperandRule extends RangeRule {
     case Divide(dividend, divisor, _) if isPositiveNumericLiteral(divisor) =>
       Seq(Step(term, dividend, Increasing, fileGuard = None, partial = 
hasDecimalResult(term)))
 
-    // NOTE: BitwiseOr is not monotonic (for ex, 1 < 2 but 1 | 2 = 3 > 2 | 2 = 
2), #20449
-    case BitwiseOr(left, right) =>
-      Seq(
-        Step(term, left, Increasing, fileGuard = None, partial = false),
-        Step(term, right, Increasing, fileGuard = None, partial = false))
-
     // NOTE: Spark only shifts int and bigint values, by an int amount
     case ShiftLeft(value, amount) =>
       IntegralLimits.of(term).filter(t => t.dataType == IntegerType || 
t.dataType == LongType).toSeq.flatMap { t =>
         integralConstant(amount).map(n => guardedStep(term, value, 
t.shiftLeftFits(n)))
       }
-    case ShiftRight(value, _) => Seq(Step(term, value, Increasing, fileGuard = 
None, partial = false))
+    case ShiftRight(value, amount) if amount.foldable => Seq(Step(term, value, 
Increasing, fileGuard = None, partial = false))

Review Comment:
   **nit:** The golden inputs (`DataSkippingGoldenInputs.scala:80`) have 
`ShiftLeft(a, a)` and `ShiftRight(Literal(8), a)` but no `ShiftRight(a, a)`, so 
none of the 51 changed golden lines pins this new guard; only the matcher and 
SQL tests do. Feel free to ignore, but could we add `ShiftRight(a, a)` to an 
appended golden list so the golden file records it too?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala:
##########
@@ -361,6 +361,57 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase 
{
     }
   }
 
+  test("Data skipping keeps files for terms that are not order-preserving in 
the column") {
+    Seq(false, true).foreach { partitionStats =>
+      withTempDir { tmp =>
+        val tableName = generateTableName
+        withSQLConf(
+          "hoodie.metadata.enable" -> "true",
+          "hoodie.metadata.index.column.stats.enable" -> "true",
+          "hoodie.metadata.index.partition.stats.enable" -> 
partitionStats.toString,
+          "hoodie.metadata.index.column.stats.column.list" -> "a,d",
+          "hoodie.enable.data.skipping" -> "false",
+          "hoodie.fileIndex.dataSkippingFailureMode" -> "strict"
+        ) {
+          spark.sql(
+            s"""
+               |create table $tableName (id int, a bigint, d date, ts long, dt 
string) using hudi
+               | tblproperties (primaryKey = 'id', orderingFields = 'ts', type 
= 'cow')
+               | partitioned by (dt)
+               | location '${tmp.getCanonicalPath}'
+               |""".stripMargin)
+          // One file per partition, so that each file's min and max come from 
two rows
+          spark.sql(s"insert into $tableName values (1, 1, date'2024-01-31', 
1, 'par1'), " +
+            "(2, 2, date'2024-02-01', 1, 'par1')")
+          spark.sql(s"insert into $tableName values (3, 64, date'2024-05-10', 
1, 'par2'), " +
+            "(4, 65, date'2024-05-11', 1, 'par2')")
+          assert(spark.sql(s"select input_file_name() from 
$tableName").collect().map(_.getString(0)).toSet.size == 2)
+          // Each term maps the min of its file above the max: 1 | 2 = 3 > 2 | 
2 = 2, a bigint shifts by
+          // the low 6 bits of the amount (64 >> 64 = 64 > 65 >> 65 = 32), and 
dayofmonth drops from 31 to
+          // 1 at the end of January
+          val queries = Seq(
+            "(a | 2) = 3" -> 1,
+            "(2 | a) = 3" -> 1,
+            "shiftright(a, a) = 64" -> 3,
+            "date_add(d, dayofmonth(d)) = date'2024-03-02'" -> 1,
+            "date_sub(d, -dayofmonth(d)) = date'2024-03-02'" -> 1,
+            "datediff(d, trunc(d, 'MM')) = 30" -> 1)
+          def checkQueries(): Unit = queries.foreach { case (filter, id) =>
+            withClue(s"$filter: ") {
+              checkAnswer(s"select id from $tableName where $filter")(Seq(id))
+            }
+          }
+          checkQueries()
+          withSQLConf("hoodie.enable.data.skipping" -> "true") {
+            spark.sql(s"alter table $tableName set tblproperties 
('hoodie.enable.data.skipping' = 'true')")

Review Comment:
   **nit:** This is the fifth copy of the skipping off/on toggle in this class 
(also :190, :246, :305, :353). Feel free to ignore, but could we pull it into a 
private `checkWithAndWithoutDataSkipping(tableName)(check)` helper and use it 
here? Moving the older copies can wait.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/feature/TestDataSkippingQuery.scala:
##########
@@ -361,6 +361,57 @@ class TestDataSkippingQuery extends HoodieSparkSqlTestBase 
{
     }
   }
 
+  test("Data skipping keeps files for terms that are not order-preserving in 
the column") {
+    Seq(false, true).foreach { partitionStats =>

Review Comment:
   **nit:** With partition stats on, 
`PartitionStatsIndexSupport.filterNonSqlExpressions` drops the `date_add`, 
`date_sub` and `shiftright(a, a)` filters before translation (it matches 
`DateAdd`, `DateSub` and the `Cast` inside `shiftright`), so only the two `|` 
filters and `datediff` reach partition stats here. Feel free to ignore, but 
could a comment above this loop say which filters exercise that path?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to