[
https://issues.apache.org/jira/browse/SPARK-61739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61739:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> Filter pushdown with a DATE or TIMESTAMP literal between 1582-10-05 and
> 1582-10-14 skips matching rows
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61739
> URL: https://issues.apache.org/jira/browse/SPARK-61739
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 5.0.0, 4.2.1, 4.0.5, 4.1.4
> Reporter: Metin Hatipogullari
> Priority: Major
> Labels: correctness, pull-request-available
>
> Those days exist in Spark's proleptic Gregorian calendar but not in
> java.sql.Date's hybrid calendar. DataSourceStrategy converts the literal with
> convertToScala, so the pushed filter carries 1582-10-15 (DATE) or a time ten
> days later (TIMESTAMP), and row groups are skipped. Default configuration.
> {code:python}
> import datetime
> for d in ["1582-10-04", "1582-10-15", "1582-10-16"]: # one file each
> spark.createDataFrame([(datetime.date.fromisoformat(d),)], "d
> DATE").coalesce(1).write.mode("append").parquet("/tmp/gap")
> df = spark.read.parquet("/tmp/gap")
> df.where("d > DATE'1582-10-10'").count() # 1, expected 2
> df.where("d <> DATE'1582-10-10'").count() # 2, expected 3
> {code}
> PushedFilters shows GreaterThan(d,1582-10-15). TIMESTAMP_MICROS Parquet
> columns are affected as well. ORC DATE columns behave the same, also with
> spark.sql.datetime.java8API.enabled=true (OrcFilters converts LocalDate back
> to java.sql.Date). For ORC TIMESTAMP, SPARK-60152 (apache/spark#60293)
> already stops pushing these literals, but only inside OrcFilters, so the
> Parquet path and ORC DATE are still affected.
> When the file itself holds a gap value (any writer using the proleptic
> calendar: Spark in CORRECTED mode, pyarrow, delta-rs), equality fails too:
> {code:python}
> import datetime, pyarrow as pa, pyarrow.parquet as pq
> pq.write_table(pa.table({"d": pa.array([datetime.date(1582, 10, 10)],
> pa.date32())}), "/tmp/gap2/a.parquet")
> spark.read.parquet("/tmp/gap2").where("d = DATE'1582-10-10'").count() # 0,
> expected 1
> {code}
> The same happens through Delta tables (spark.read.format("delta")), because
> the scan uses the same Parquet pushdown; it does not depend on Delta's data
> skipping.
> Reproduced on 3.5.9, 4.0.5, 4.1.4, 4.2.1 and master. I have a patch that
> skips pushdown for literals in the gap, with tests, and can open a PR.
> Found while comparing the same files with pushdown on and off.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]