[ 
https://issues.apache.org/jira/browse/SPARK-61738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61738:
-----------------------------------
    Labels: correctness pull-request-available  (was: correctness)

> Parquet filter pushdown on TIMESTAMP_MILLIS columns truncates sub-millisecond 
> literals and skips matching rows
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61738
>                 URL: https://issues.apache.org/jira/browse/SPARK-61738
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.5.9, 5.0.0, 4.2.1, 4.0.5, 4.1.4
>            Reporter: Metin Hatipogullari
>            Priority: Major
>              Labels: correctness, pull-request-available
>
> ParquetFilters.timestampToMillis floors the literal to whole milliseconds. 
> That is safe for =, >, >=, <=, because Spark re-evaluates the filter after 
> the scan, but not for < and <> (and NOT IN, which becomes notEq). Row groups 
> whose min, or whose only value, equals the floored literal are skipped. 
> Default configuration.
> {code:python}
> spark.conf.set("spark.sql.session.timeZone", "UTC")
> spark.conf.set("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS")
> for t in ["1900-01-01 00:00:00", "2020-01-01 00:00:00"]:   # one file each
>     spark.sql(f"SELECT TIMESTAMP'{t}' AS 
> t").coalesce(1).write.mode("append").parquet("/tmp/z1")
> df = spark.read.parquet("/tmp/z1")
> df.where("t < TIMESTAMP'1900-01-01 00:00:00.000001'").count()    # 0, 
> expected 1
> df.where("t <> TIMESTAMP'2020-01-01 00:00:00.000001'").count()   # 1, 
> expected 2
> {code}
> Correct with spark.sql.parquet.filterPushdown=false, with TIMESTAMP_MICROS 
> files, and with literals on the millisecond grid. Same result with the V2 
> source and with the vectorized reader off.
> Reproduced on 3.5.9, 4.0.5, 4.1.4, 4.2.1 and master (b2fc1bef1b).
> The ORC equivalent was SPARK-60326 / SPARK-60436. For Parquet, 
> valueCanMakeFilterOn already refuses sub-microsecond bounds; refusing 
> literals that are not on the millisecond grid for ParquetTimestampMillisType 
> would fix it (or round lt up and skip notEq for them).
> Found while comparing the same files with pushdown on and off.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to