This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new c3799e1e1f90 [SPARK-60423][SQL] Fix Hive ORC filter pushdown on
timestamp literals with sub-millisecond digits
c3799e1e1f90 is described below
commit c3799e1e1f90ed4d8f2726c68260ccada9e043e0
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 21 10:46:40 2026 -0700
[SPARK-60423][SQL] Fix Hive ORC filter pushdown on timestamp literals with
sub-millisecond digits
### What changes were proposed in this pull request?
This PR aims to fix Hive ORC filter pushdown (`spark.sql.orc.impl=hive`) on
`TIMESTAMP` columns by not pushing down a top-level filter that has a timestamp
literal with sub-millisecond digits, e.g. `ts = TIMESTAMP '2020-01-01
00:00:00.123456'`. The literal can be inside `AND`, `OR`, or `NOT`, and can be
a `java.sql.Timestamp` or a `java.time.Instant`
(`spark.sql.datetime.java8API.enabled=true`). Spark still evaluates these
filters after the scan, and the other filters are pushed down [...]
### Why are the changes needed?
Spark writes the ORC `SearchArgument` into `sarg.pushdown` with a plain
`Kryo` (`OrcFileFormat.toKryo`), and Hive's
`OrcInputFormat.createReaderFromFile`, which creates the record reader, reads
it back with a plain `Kryo` (`ConvertAstToSearchArg.create`). Kryo's default
serializer of `java.util.Date` keeps only `getTime()`, so a
`java.sql.Timestamp` literal loses its sub-millisecond digits, e.g. `2020-01-01
00:00:00.123456` becomes `2020-01-01 00:00:00.123`.
Since ORC 1.6.4 (ORC-611), ORC timestamp statistics have nanosecond
precision, so ORC compares the truncated literal with the nanosecond minimums
and maximums.
- `=`, `<=>`, `IN`, `<`, and `<=` skip a stripe or row group whose minimum
is between the truncated literal and the literal, although it can hold matching
rows.
- `!=` and `NOT IN` skip a stripe or row group whose values are all equal
to the truncated literal, because ORC evaluates `=` to `YES` for it.
- `>` and `>=` stay correct, because the truncated literal only makes them
less selective. To keep it simple, this PR does not push them down either.
For example, the following query returns 0 instead of 1. The result is
correct with `spark.sql.orc.filterPushdown=false`.
```scala
spark.conf.set("spark.sql.orc.impl", "hive")
spark.sql("SELECT TIMESTAMP '2020-01-01 00:00:00.123456' AS
ts").write.orc(path)
spark.read.orc(path).where("ts = TIMESTAMP '2020-01-01
00:00:00.123456'").count()
```
This affects `TIMESTAMP` columns since Apache Spark 3.2.0
([v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0), 2021-10-06),
the first release with ORC 1.6 (SPARK-33295). The example above returns 0 with
Apache Spark 4.0.4, 4.1.3, 4.2.0, and 4.3.0-rc1.
The native ORC reader (`spark.sql.orc.impl=native`) has the same Kryo
issue, which is handled separately. Hive's
`OrcInputFormat.createReaderFromFile` gets the `SearchArgument` only from the
Hadoop configuration, so this PR stops pushing down such filters instead, like
SPARK-60396 does for nanosecond timestamp columns.
### Does this PR introduce _any_ user-facing change?
Yes. With the Hive ORC reader, predicates that compare a `TIMESTAMP` column
with a literal with sub-millisecond digits return correct results. Previously,
they could skip matching rows silently since Apache Spark 3.2.0. Such
predicates no longer skip stripes or row groups.
### How was this patch tested?
Pass the CIs with the newly added test case in `HiveOrcSourceSuite`. It
checks `=`, `<=>`, `IN`, `<`, `<=`, `>`, `>=`, `!=`, `NOT IN`, and `OR` with a
literal with sub-millisecond digits, for both `java.sql.Timestamp` and
`java.time.Instant` literals. It also checks that a pushed down `=` with a
millisecond literal still skips the other files. Without the fix, it fails.
```
[info] - SPARK-60423: Hive ORC filter pushdown on timestamps with
sub-millisecond digits *** FAILED *** (738 milliseconds)
[info] Results do not match for query:
...
[info] 'Filter ('ts = 2020-01-01 00:00:00.123456)
...
[info] !== Correct Answer - 1 == == Spark Answer - 0 ==
[info] !struct<ts:timestamp> struct<>
[info] ![2020-01-01 00:00:00.123456] (CheckAnswerHelper.scala:61)
```
The related test suites also pass.
```
$ build/sbt -Phive "hive/testOnly org.apache.spark.sql.hive.orc.*"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59673 from dongjoon-hyun/SPARK-60423.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 000f342f7378111ccb6160059e29d6631713a5ae)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../apache/spark/sql/hive/orc/OrcFileFormat.scala | 32 ++++++++++++++++-
.../spark/sql/hive/orc/HiveOrcSourceSuite.scala | 42 ++++++++++++++++++++++
2 files changed, 73 insertions(+), 1 deletion(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
index 4e1567aac5f2..be211fd65ebf 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala
@@ -17,6 +17,8 @@
package org.apache.spark.sql.hive.orc
+import java.sql.Timestamp
+import java.time.Instant
import java.util.Properties
import scala.jdk.CollectionConverters._
@@ -44,6 +46,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.util.DateTimeConstants.NANOS_PER_MILLIS
import org.apache.spark.sql.execution.datasources._
import org.apache.spark.sql.execution.datasources.orc.{OrcFilters, OrcOptions,
OrcUtils}
import org.apache.spark.sql.hive.{HiveInspectors, HiveShim}
@@ -136,7 +139,8 @@ class OrcFileFormat extends FileFormat with
DataSourceRegister with Serializable
if (sparkSession.sessionState.conf.orcFilterPushDown) {
// Sets pushed predicates
- OrcFilters.createFilter(requiredSchema, filters).foreach { f =>
+ val pushedFilters = filters.filterNot(hasSubMillisTimestamp)
+ OrcFilters.createFilter(requiredSchema, pushedFilters).foreach { f =>
hadoopConf.set(OrcFileFormat.SARG_PUSHDOWN, toKryo(f))
hadoopConf.setBoolean("hive.optimize.index.filter", true)
}
@@ -208,6 +212,32 @@ class OrcFileFormat extends FileFormat with
DataSourceRegister with Serializable
case _ => false
}
+ // Whether the filter has a TIMESTAMP value with a sub-millisecond part.
Such a top-level filter
+ // is not pushed down, because the Kryo round trip of the search argument
(see `toKryo`)
+ // truncates the value to milliseconds, while ORC compares it with
nanosecond statistics.
+ private def hasSubMillisTimestamp(filter: Filter): Boolean = {
+ import org.apache.spark.sql.sources._
+
+ def isSubMillis(value: Any): Boolean = value match {
+ case t: Timestamp => t.getNanos % NANOS_PER_MILLIS != 0
+ case i: Instant => i.getNano % NANOS_PER_MILLIS != 0
+ case _ => false
+ }
+ filter match {
+ case And(left, right) => hasSubMillisTimestamp(left) ||
hasSubMillisTimestamp(right)
+ case Or(left, right) => hasSubMillisTimestamp(left) ||
hasSubMillisTimestamp(right)
+ case Not(child) => hasSubMillisTimestamp(child)
+ case EqualTo(_, value) => isSubMillis(value)
+ case EqualNullSafe(_, value) => isSubMillis(value)
+ case LessThan(_, value) => isSubMillis(value)
+ case LessThanOrEqual(_, value) => isSubMillis(value)
+ case GreaterThan(_, value) => isSubMillis(value)
+ case GreaterThanOrEqual(_, value) => isSubMillis(value)
+ case In(_, values) => values.exists(isSubMillis)
+ case _ => false
+ }
+ }
+
// HIVE-11253 moved `toKryo` from `SearchArgument` to `storage-api` module.
// This is copied from Hive 1.2's SearchArgumentImpl.toKryo().
private def toKryo(sarg: SearchArgument): String = {
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
index c1084dd4ee7f..50164ee61518 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala
@@ -24,6 +24,7 @@ import org.apache.spark.sql.TestingUDT.{IntervalData,
IntervalUDT}
import org.apache.spark.sql.execution.datasources.orc.OrcSuite
import org.apache.spark.sql.hive.HiveUtils
import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.util.Utils
@@ -346,6 +347,47 @@ class HiveOrcSourceSuite extends OrcSuite with
TestHiveSingleton {
}
}
+ test("SPARK-60423: Hive ORC filter pushdown on timestamps with
sub-millisecond digits") {
+ // Each value is written to its own file, so a pushed down predicate that
skips a file with a
+ // matching row changes the result. The Kryo round trip of the search
argument truncates
+ // `00:00:00.123456` to `00:00:00.123`, the first value.
+ def ts(time: String): String = s"TIMESTAMP'2020-01-01 $time'"
+ val value = ts("00:00:00.123456")
+ withSQLConf(
+ SQLConf.ORC_IMPLEMENTATION.key -> "hive",
+ SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ val input = sql(s"SELECT ${ts("00:00:00.123")} AS ts UNION ALL SELECT
$value " +
+ s"UNION ALL SELECT ${ts("00:00:01")}")
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ input.coalesce(1).write.option("maxRecordsPerFile", 1).orc(path)
+ // The filter value is a java.sql.Timestamp or a java.time.Instant.
+ Seq(false, true).foreach { java8Api =>
+ withSQLConf(SQLConf.DATETIME_JAVA8API_ENABLED.key ->
java8Api.toString) {
+ val df = spark.read.schema(input.schema).orc(path)
+ Seq(
+ s"ts = $value",
+ s"ts <=> $value",
+ s"ts IN ($value, ${ts("00:00:01")})",
+ s"ts < $value",
+ s"ts <= $value",
+ s"ts > $value",
+ s"ts >= $value",
+ s"ts != $value",
+ s"ts NOT IN ($value, ${ts("00:00:01")})",
+ s"ts = $value OR ts > ${ts("00:00:01")}"
+ ).foreach { condition =>
+ checkAnswer(df.where(condition), input.where(condition))
+ }
+ // A predicate with a millisecond value is still pushed down and
skips the other files.
+ val condition = s"ts = ${ts("00:00:00.123")}"
+ checkAnswer(stripSparkFilter(df.where(condition)),
input.where(condition))
+ }
+ }
+ }
+ }
+ }
+
test("SPARK-31580: Read a file written before ORC-569") {
// Test ORC file came from ORC-621
val df =
readResourceOrcFile("test-data/TestStringDictionary.testRowIndex.orc")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]