Aleksandar Trajković created SPARK-61543:
--------------------------------------------

             Summary: JDBC pushdown renders java.time.Instant filter literals 
in the session time zone, so PostgreSQL returns wrong rows when 
spark.sql.session.timeZone differs from the JVM time zone
                 Key: SPARK-61543
                 URL: https://issues.apache.org/jira/browse/SPARK-61543
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.4.0
            Reporter: Aleksandar Trajković


JDBC pushdown renders java.time.Instant filter literals in the session time 
zone, so PostgreSQL returns wrong rows when spark.sql.session.timeZone differs 
from the JVM time zone

When spark.sql.datetime.java8API.enabled=true (the Thrift server always enables 
it), TIMESTAMP literals in pushed-down filters reach JdbcDialect.compileValue 
as java.time.Instant. compileValue formats them as a zoneless string in 
spark.sql.session.timeZone. With java8API=false the literal is a 
java.sql.Timestamp, formatted by Timestamp.toString in the JVM default time 
zone.

PostgreSQL reads a zoneless literal compared with a timestamptz column in its 
session TimeZone, which pgjdbc sets to the JVM default time zone at connect. 
When the Spark session time zone differs from the JVM time zone, the pushed 
bounds shift by the offset between the two zones. The query returns different 
rows than the same query with pushdown disabled.

Repro (JVM zone UTC): a timestamptz table with 8 rows across the UTC/Chicago 
day boundary. Then SET spark.sql.datetime.java8API.enabled=true; SET TIME ZONE 
'America/Chicago'; SELECT id FROM pg.public.events WHERE ts >= DATE 
'2026-06-26' AND ts < DATE '2026-06-27'.
Pushed: ("ts" >= '2026-06-26 00:00:00') AND ("ts" < '2026-06-27 00:00:00'). 
Actual: ids 1–4 (the UTC day). Expected: ids 3–7 (the Chicago day), which is 
what Spark returns with pushDownPredicate=false or java8API=false.

Other dialects that use the base compileValue for Instant may be affected as 
well.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to