[ 
https://issues.apache.org/jira/browse/SPARK-61739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61739:
-----------------------------------
    Labels: correctness pull-request-available  (was: correctness)

> Filter pushdown with a DATE or TIMESTAMP literal between 1582-10-05 and 
> 1582-10-14 skips matching rows
> ------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61739
>                 URL: https://issues.apache.org/jira/browse/SPARK-61739
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.9, 5.0.0, 4.2.1, 4.0.5, 4.1.4
>            Reporter: Metin Hatipogullari
>            Priority: Major
>              Labels: correctness, pull-request-available
>
> Those days exist in Spark's proleptic Gregorian calendar but not in 
> java.sql.Date's hybrid calendar. DataSourceStrategy converts the literal with 
> convertToScala, so the pushed filter carries 1582-10-15 (DATE) or a time ten 
> days later (TIMESTAMP), and row groups are skipped. Default configuration.
> {code:python}
> import datetime
> for d in ["1582-10-04", "1582-10-15", "1582-10-16"]:   # one file each
>     spark.createDataFrame([(datetime.date.fromisoformat(d),)], "d 
> DATE").coalesce(1).write.mode("append").parquet("/tmp/gap")
> df = spark.read.parquet("/tmp/gap")
> df.where("d > DATE'1582-10-10'").count()    # 1, expected 2
> df.where("d <> DATE'1582-10-10'").count()   # 2, expected 3
> {code}
> PushedFilters shows GreaterThan(d,1582-10-15). TIMESTAMP_MICROS Parquet 
> columns are affected as well. ORC DATE columns behave the same, also with 
> spark.sql.datetime.java8API.enabled=true (OrcFilters converts LocalDate back 
> to java.sql.Date). For ORC TIMESTAMP, SPARK-60152 (apache/spark#60293) 
> already stops pushing these literals, but only inside OrcFilters, so the 
> Parquet path and ORC DATE are still affected.
> When the file itself holds a gap value (any writer using the proleptic 
> calendar: Spark in CORRECTED mode, pyarrow, delta-rs), equality fails too:
> {code:python}
> import datetime, pyarrow as pa, pyarrow.parquet as pq
> pq.write_table(pa.table({"d": pa.array([datetime.date(1582, 10, 10)], 
> pa.date32())}), "/tmp/gap2/a.parquet")
> spark.read.parquet("/tmp/gap2").where("d = DATE'1582-10-10'").count()   # 0, 
> expected 1
> {code}
> The same happens through Delta tables (spark.read.format("delta")), because 
> the scan uses the same Parquet pushdown; it does not depend on Delta's data 
> skipping.
> Reproduced on 3.5.9, 4.0.5, 4.1.4, 4.2.1 and master. I have a patch that 
> skips pushdown for literals in the gap, with tests, and can open a PR.
> Found while comparing the same files with pushdown on and off.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to