[
https://issues.apache.org/jira/browse/SPARK-60152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yaoxuan Wu updated SPARK-60152:
-------------------------------
Labels: correctness (was: )
> ORC filter pushdown drops rows with a TIMESTAMP before 1582-10-15
> -----------------------------------------------------------------
>
> Key: SPARK-60152
> URL: https://issues.apache.org/jira/browse/SPARK-60152
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.8, 4.2.0, 4.1.1, 5.0.0
> Reporter: Yaoxuan Wu
> Priority: Major
> Labels: correctness
>
> {code:java}
> import tempfile
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.config("spark.sql.session.timeZone",
> "UTC").getOrCreate()
> path = tempfile.mkdtemp()
> spark.sql("SELECT TIMESTAMP'0001-01-02 00:00:00' AS
> t").write.mode("overwrite").orc(path)
> spark.read.orc(path).createOrReplaceTempView("o")spark.sql("SELECT cast(t AS
> STRING) FROM o").collect()
> # [Row(t='0001-01-02 00:00:00')]: the value round-tripsspark.sql("SELECT
> count(*) FROM o WHERE t = TIMESTAMP'0001-01-02 00:00:00'").first()[0]
> # expected: 1
> # Spark 4.2.0: 0
> spark.sql("SELECT count(*) FROM o WHERE t <= TIMESTAMP'0001-01-02
> 00:00:00'").first()[0]
> # expected: 1
> # Spark 4.2.0: 0
> spark.sql("SELECT count(*) FROM o WHERE t >= TIMESTAMP'0001-01-02
> 00:00:00'").first()[0]
> # 1 {code}
> With {{spark.sql.orc.filterPushdown=false}} all three return 1. Same on
> 3.5.8, 4.0.0, 4.1.1, 4.2.0 and master, with the JVM time zone UTC or
> America/Los_Angeles, and with the vectorized reader on or off.
> ||Value||Predicates that return 0 rows||
> |TIMESTAMP '0001-01-02 00:00:00'|{{{}={}}}, {{<=}}|
> |TIMESTAMP '1500-01-01 00:00:00'|{{{}={}}}, {{>=}}|
> |TIMESTAMP '1582-10-14 23:00:00'|{{=}}|
> |TIMESTAMP '1582-10-16 00:00:00', DATE '0001-01-02', TIMESTAMP_NTZ
> '0001-01-02 00:00:00'|none|
> *Cause* (checked with the ORC Java API through py4j):
> * Spark writes TIMESTAMP values rebased to the hybrid Julian/Gregorian
> calendar ({{{}OrcSerializer{}}} uses {{{}DateTimeUtils.toJavaTimestamp{}}}),
> and builds the pushed literal the same way
> ({{{}OrcFilters.castLiteralValue{}}}). The literal's {{getTime()}} equals the
> file statistics' {{{}getMinimum().getTime(){}}}.
> * ORC's row-group pruning, {{{}RecordReaderImpl.evaluatePredicateProto{}}},
> reads the statistics with {{{}ColumnStatisticsImpl.deserialize(null,
> statsProto, writerUsedProlepticGregorian, true){}}}. The {{true}} converts
> min/max to the proleptic Gregorian calendar (2 days later in year 1, 9 days
> earlier in 1500). The hybrid literal is then outside [min, max].
> * Reading the file with ORC directly and the same SearchArgument: Spark's
> hybrid literal reads 0 rows, a proleptic literal ({{{}new Timestamp(micros /
> 1000){}}}) reads 1 row. ORC 1.9.8 and 2.3.0 behave the same.
> * DATE works because ORC compares dates as local dates. TIMESTAMP_NTZ is
> pushed as a long.
> A fix on the Spark side could push TIMESTAMP literals without the hybrid
> rebase, or skip pushdown for literals before 1582-10-15.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
