This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.1 by this push:
     new 9762e41a34f2 [SPARK-60423][SQL] Fix Hive ORC filter pushdown on 
timestamp literals with sub-millisecond digits
9762e41a34f2 is described below

commit 9762e41a34f2f2f53f266b2bf123dc83775d30a3
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 21 10:46:40 2026 -0700

    [SPARK-60423][SQL] Fix Hive ORC filter pushdown on timestamp literals with 
sub-millisecond digits
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix Hive ORC filter pushdown (`spark.sql.orc.impl=hive`) on 
`TIMESTAMP` columns by not pushing down a top-level filter that has a timestamp 
literal with sub-millisecond digits, e.g. `ts = TIMESTAMP '2020-01-01 
00:00:00.123456'`. The literal can be inside `AND`, `OR`, or `NOT`, and can be 
a `java.sql.Timestamp` or a `java.time.Instant` 
(`spark.sql.datetime.java8API.enabled=true`). Spark still evaluates these 
filters after the scan, and the other filters are pushed down  [...]
    
    ### Why are the changes needed?
    
    Spark writes the ORC `SearchArgument` into `sarg.pushdown` with a plain 
`Kryo` (`OrcFileFormat.toKryo`), and Hive's 
`OrcInputFormat.createReaderFromFile`, which creates the record reader, reads 
it back with a plain `Kryo` (`ConvertAstToSearchArg.create`). Kryo's default 
serializer of `java.util.Date` keeps only `getTime()`, so a 
`java.sql.Timestamp` literal loses its sub-millisecond digits, e.g. `2020-01-01 
00:00:00.123456` becomes `2020-01-01 00:00:00.123`.
    
    Since ORC 1.6.4 (ORC-611), ORC timestamp statistics have nanosecond 
precision, so ORC compares the truncated literal with the nanosecond minimums 
and maximums.
    - `=`, `<=>`, `IN`, `<`, and `<=` skip a stripe or row group whose minimum 
is between the truncated literal and the literal, although it can hold matching 
rows.
    - `!=` and `NOT IN` skip a stripe or row group whose values are all equal 
to the truncated literal, because ORC evaluates `=` to `YES` for it.
    - `>` and `>=` stay correct, because the truncated literal only makes them 
less selective. To keep it simple, this PR does not push them down either.
    
    For example, the following query returns 0 instead of 1. The result is 
correct with `spark.sql.orc.filterPushdown=false`.
    
    ```scala
    spark.conf.set("spark.sql.orc.impl", "hive")
    spark.sql("SELECT TIMESTAMP '2020-01-01 00:00:00.123456' AS 
ts").write.orc(path)
    spark.read.orc(path).where("ts = TIMESTAMP '2020-01-01 
00:00:00.123456'").count()
    ```
    
    This affects `TIMESTAMP` columns since Apache Spark 3.2.0 
([v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0), 2021-10-06), 
the first release with ORC 1.6 (SPARK-33295). The example above returns 0 with 
Apache Spark 4.0.4, 4.1.3, 4.2.0, and 4.3.0-rc1.
    
    The native ORC reader (`spark.sql.orc.impl=native`) has the same Kryo 
issue, which is handled separately. Hive's 
`OrcInputFormat.createReaderFromFile` gets the `SearchArgument` only from the 
Hadoop configuration, so this PR stops pushing down such filters instead, like 
SPARK-60396 does for nanosecond timestamp columns.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. With the Hive ORC reader, predicates that compare a `TIMESTAMP` column 
with a literal with sub-millisecond digits return correct results. Previously, 
they could skip matching rows silently since Apache Spark 3.2.0. Such 
predicates no longer skip stripes or row groups.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case in `HiveOrcSourceSuite`. It 
checks `=`, `<=>`, `IN`, `<`, `<=`, `>`, `>=`, `!=`, `NOT IN`, and `OR` with a 
literal with sub-millisecond digits, for both `java.sql.Timestamp` and 
`java.time.Instant` literals. It also checks that a pushed down `=` with a 
millisecond literal still skips the other files. Without the fix, it fails.
    
    ```
    [info] - SPARK-60423: Hive ORC filter pushdown on timestamps with 
sub-millisecond digits *** FAILED *** (738 milliseconds)
    [info]   Results do not match for query:
    ...
    [info]   'Filter ('ts = 2020-01-01 00:00:00.123456)
    ...
    [info]   !== Correct Answer - 1 ==       == Spark Answer - 0 ==
    [info]   !struct<ts:timestamp>           struct<>
    [info]   ![2020-01-01 00:00:00.123456] (CheckAnswerHelper.scala:61)
    ```
    
    The related test suites also pass.
    
    ```
    $ build/sbt -Phive "hive/testOnly org.apache.spark.sql.hive.orc.*"
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59673 from dongjoon-hyun/SPARK-60423.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 000f342f7378111ccb6160059e29d6631713a5ae)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../apache/spark/sql/hive/orc/OrcFileFormat.scala  | 32 ++++++++++++++++-
 .../spark/sql/hive/orc/HiveOrcSourceSuite.scala    | 42 ++++++++++++++++++++++
 2 files changed, 73 insertions(+), 1 deletion(-)

diff --git 
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala 
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
index ba37e5c176de..a16cbca0f3de 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
@@ -17,6 +17,8 @@
 
 package org.apache.spark.sql.hive.orc
 
+import java.sql.Timestamp
+import java.time.Instant
 import java.util.{Base64, Properties}
 
 import scala.jdk.CollectionConverters._
@@ -43,6 +45,7 @@ import org.apache.spark.internal.Logging
 import org.apache.spark.sql.SparkSession
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.DateTimeConstants.NANOS_PER_MILLIS
 import org.apache.spark.sql.execution.datasources._
 import org.apache.spark.sql.execution.datasources.orc.{OrcFilters, OrcOptions, 
OrcUtils}
 import org.apache.spark.sql.hive.{HiveInspectors, HiveShim}
@@ -139,7 +142,8 @@ case class OrcFileFormat() extends FileFormat
 
     if (getSqlConf(sparkSession).orcFilterPushDown) {
       // Sets pushed predicates
-      OrcFilters.createFilter(requiredSchema, filters).foreach { f =>
+      val pushedFilters = filters.filterNot(hasSubMillisTimestamp)
+      OrcFilters.createFilter(requiredSchema, pushedFilters).foreach { f =>
         hadoopConf.set(OrcFileFormat.SARG_PUSHDOWN, toKryo(f))
         hadoopConf.setBoolean("hive.optimize.index.filter", true)
       }
@@ -211,6 +215,32 @@ case class OrcFileFormat() extends FileFormat
     case _ => false
   }
 
+  // Whether the filter has a TIMESTAMP value with a sub-millisecond part. 
Such a top-level filter
+  // is not pushed down, because the Kryo round trip of the search argument 
(see `toKryo`)
+  // truncates the value to milliseconds, while ORC compares it with 
nanosecond statistics.
+  private def hasSubMillisTimestamp(filter: Filter): Boolean = {
+    import org.apache.spark.sql.sources._
+
+    def isSubMillis(value: Any): Boolean = value match {
+      case t: Timestamp => t.getNanos % NANOS_PER_MILLIS != 0
+      case i: Instant => i.getNano % NANOS_PER_MILLIS != 0
+      case _ => false
+    }
+    filter match {
+      case And(left, right) => hasSubMillisTimestamp(left) || 
hasSubMillisTimestamp(right)
+      case Or(left, right) => hasSubMillisTimestamp(left) || 
hasSubMillisTimestamp(right)
+      case Not(child) => hasSubMillisTimestamp(child)
+      case EqualTo(_, value) => isSubMillis(value)
+      case EqualNullSafe(_, value) => isSubMillis(value)
+      case LessThan(_, value) => isSubMillis(value)
+      case LessThanOrEqual(_, value) => isSubMillis(value)
+      case GreaterThan(_, value) => isSubMillis(value)
+      case GreaterThanOrEqual(_, value) => isSubMillis(value)
+      case In(_, values) => values.exists(isSubMillis)
+      case _ => false
+    }
+  }
+
   // HIVE-11253 moved `toKryo` from `SearchArgument` to `storage-api` module.
   // This is copied from Hive 1.2's SearchArgumentImpl.toKryo().
   private def toKryo(sarg: SearchArgument): String = {
diff --git 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
index c1084dd4ee7f..50164ee61518 100644
--- 
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
+++ 
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
@@ -24,6 +24,7 @@ import org.apache.spark.sql.TestingUDT.{IntervalData, 
IntervalUDT}
 import org.apache.spark.sql.execution.datasources.orc.OrcSuite
 import org.apache.spark.sql.hive.HiveUtils
 import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
 import org.apache.spark.util.Utils
 
@@ -346,6 +347,47 @@ class HiveOrcSourceSuite extends OrcSuite with 
TestHiveSingleton {
     }
   }
 
+  test("SPARK-60423: Hive ORC filter pushdown on timestamps with 
sub-millisecond digits") {
+    // Each value is written to its own file, so a pushed down predicate that 
skips a file with a
+    // matching row changes the result. The Kryo round trip of the search 
argument truncates
+    // `00:00:00.123456` to `00:00:00.123`, the first value.
+    def ts(time: String): String = s"TIMESTAMP'2020-01-01 $time'"
+    val value = ts("00:00:00.123456")
+    withSQLConf(
+        SQLConf.ORC_IMPLEMENTATION.key -> "hive",
+        SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+      val input = sql(s"SELECT ${ts("00:00:00.123")} AS ts UNION ALL SELECT 
$value " +
+        s"UNION ALL SELECT ${ts("00:00:01")}")
+      withTempPath { dir =>
+        val path = dir.getCanonicalPath
+        input.coalesce(1).write.option("maxRecordsPerFile", 1).orc(path)
+        // The filter value is a java.sql.Timestamp or a java.time.Instant.
+        Seq(false, true).foreach { java8Api =>
+          withSQLConf(SQLConf.DATETIME_JAVA8API_ENABLED.key -> 
java8Api.toString) {
+            val df = spark.read.schema(input.schema).orc(path)
+            Seq(
+              s"ts = $value",
+              s"ts <=> $value",
+              s"ts IN ($value, ${ts("00:00:01")})",
+              s"ts < $value",
+              s"ts <= $value",
+              s"ts > $value",
+              s"ts >= $value",
+              s"ts != $value",
+              s"ts NOT IN ($value, ${ts("00:00:01")})",
+              s"ts = $value OR ts > ${ts("00:00:01")}"
+            ).foreach { condition =>
+              checkAnswer(df.where(condition), input.where(condition))
+            }
+            // A predicate with a millisecond value is still pushed down and 
skips the other files.
+            val condition = s"ts = ${ts("00:00:00.123")}"
+            checkAnswer(stripSparkFilter(df.where(condition)), 
input.where(condition))
+          }
+        }
+      }
+    }
+  }
+
   test("SPARK-31580: Read a file written before ORC-569") {
     // Test ORC file came from ORC-621
     val df = 
readResourceOrcFile("test-data/TestStringDictionary.testRowIndex.orc")


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to