dongjoon-hyun opened a new pull request, #59606:
URL: https://github.com/apache/spark/pull/59606
### What changes were proposed in this pull request?
This PR aims to fix the Parquet filter pushdown on a `TIME(p)` column (`p <
6`) when the file has values with more fractional digits than `p`.
Since SPARK-57551, the Parquet reader truncates a `TIME` value to the read
precision. However, Parquet evaluates a pushed down predicate against the
stored values, so the predicate can skip a row whose read value matches. With
this PR, only `IsNull` and `IsNotNull` are pushed down for such a column.
- `ParquetFilterOps.acceptsReadType` is added (default `true`). The ops of
`TIME(MICROS)` returns `false` for `TIME(0)` to `TIME(5)`.
- `ParquetFilters` gets the read schema (`requiredSchema` in V1,
`readDataSchema` in V2) and checks `acceptsReadType` before it pushes down a
predicate with a value on a Types Framework column.
`TIME(6)` to `TIME(9)` are not affected because the reader keeps the micros
of a `TIME(MICROS)` file as they are, and there is no filter pushdown on
`TIME(NANOS)`.
### Why are the changes needed?
The query results are wrong. For example, `12:00:00.1005` is stored in a
`TIME(MICROS)` file and read as `12:00:00.100` with `TIME(3)`.
```
$ bin/spark-shell -c spark.sql.timeType.enabled=true
scala> sql("SELECT TIME '12:00:00.1005' AS t UNION ALL SELECT TIME
'12:00:01'").coalesce(1).write.parquet("/tmp/time")
scala> val df = spark.read.schema("t TIME(3)").parquet("/tmp/time")
scala> df.collect()
val res1: Array[org.apache.spark.sql.Row] = Array([12:00:00.100], [12:00:01])
scala> df.where("t = CAST('12:00:00.1' AS TIME(3))").collect()
val res2: Array[org.apache.spark.sql.Row] = Array()
```
The pushed down `EqualTo(t,12:00:00.100)` is compared with the stored
`12:00:00.100500`. `<=>`, `<=` and `IN` return no rows in the same way, with
both V1 and V2 and with both the vectorized and the row-based readers. The
expected result is `Array([12:00:00.100])`, which is the result with
`spark.sql.parquet.filterPushdown=false`.
With SPARK-60355, this is reachable from `TIME` literals too, e.g. `t = TIME
'12:00:00.1'`, because the cast moves from the column to the literal.
### Does this PR introduce _any_ user-facing change?
Yes, this fixes the wrong query results. The bug exists only in the
unreleased branches (`master`, `branch-4.x` and `branch-4.3`), including
[v4.3.0-rc1](https://github.com/apache/spark/releases/tag/v4.3.0-rc1)
(2026-08-31) and
[v4.3.0-rc2](https://github.com/apache/spark/releases/tag/v4.3.0-rc2)
(2026-10-12). The `TIME` type is disabled by default
(`spark.sql.timeType.enabled`).
The predicates with a value on `TIME(0)` to `TIME(5)` columns are not pushed
down to Parquet anymore.
### How was this patch tested?
Pass the CIs with the newly added test cases in `ParquetV1FilterSuite` and
`ParquetV2FilterSuite`. The end-to-end test case fails without this fix.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
