[ 
https://issues.apache.org/jira/browse/SPARK-61543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61543:
-----------------------------------
    Labels: pull-request-available  (was: )

> JDBC pushdown renders java.time.Instant filter literals in the session time 
> zone, so PostgreSQL returns wrong rows when spark.sql.session.timeZone 
> differs from the JVM time zone
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61543
>                 URL: https://issues.apache.org/jira/browse/SPARK-61543
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.4.0
>            Reporter: Aleksandar Trajković
>            Priority: Major
>              Labels: pull-request-available
>
> JDBC pushdown renders java.time.Instant filter literals in the session time 
> zone, so PostgreSQL returns wrong rows when spark.sql.session.timeZone 
> differs from the JVM time zone
> When spark.sql.datetime.java8API.enabled=true (the Thrift server always 
> enables it), TIMESTAMP literals in pushed-down filters reach 
> JdbcDialect.compileValue as java.time.Instant. compileValue formats them as a 
> zoneless string in spark.sql.session.timeZone. With java8API=false the 
> literal is a java.sql.Timestamp, formatted by Timestamp.toString in the JVM 
> default time zone.
> PostgreSQL reads a zoneless literal compared with a timestamptz column in its 
> session TimeZone, which pgjdbc sets to the JVM default time zone at connect. 
> When the Spark session time zone differs from the JVM time zone, the pushed 
> bounds shift by the offset between the two zones. The query returns different 
> rows than the same query with pushdown disabled.
> Repro (JVM zone UTC): a timestamptz table with 8 rows across the UTC/Chicago 
> day boundary. Then SET spark.sql.datetime.java8API.enabled=true; SET TIME 
> ZONE 'America/Chicago'; SELECT id FROM pg.public.events WHERE ts >= DATE 
> '2026-06-26' AND ts < DATE '2026-06-27'.
> Pushed: ("ts" >= '2026-06-26 00:00:00') AND ("ts" < '2026-06-27 00:00:00'). 
> Actual: ids 1–4 (the UTC day). Expected: ids 3–7 (the Chicago day), which is 
> what Spark returns with pushDownPredicate=false or java8API=false.
> Other dialects that use the base compileValue for Instant may be affected as 
> well.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to