dongjoon-hyun commented on PR #59571:
URL: https://github.com/apache/spark/pull/59571#issuecomment-6269039612
> Is this mixed-precision pruning tradeoff acceptable for the precision
correction?
Thank you, @cloud-fan. I checked it with a `TIME(6)` Parquet table. Before
this PR, `WHERE t = :p` with `p = expr("TIME '12:00:00.100000000'")` was
substituted as `TIME '12:00:00.1'` (`TIME(6)`), so `EqualTo(t,12:00:00.100)`
was pushed down. With this PR, the condition becomes `CAST(t AS TIME(9)) = TIME
'12:00:00.100000000'` and only `IsNotNull(t)` is pushed down, as you described.
However, this is the same plan as when the same literal is written in SQL
directly (`WHERE t = TIME '12:00:00.100000000'`), or when a `TIME(9)` argument
has no trailing zeros (`TIME '12:00:00.123456789'`), with or without this PR.
So the previous pushdown depended on the trailing zeros of the argument value,
as a side effect of the narrowed type. The query results are the same in all
cases. I believe the precision correction is the right trade-off. The pruning
can be restored for all these cases by unwrapping the `TIME` precision-widening
cast in `UnwrapCastInBinaryComparison`, which currently handles only numeric
and date/timestamp casts. Let me handle it in a new JIRA issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
