MaxGekk commented on code in PR #59641:
URL: https://github.com/apache/spark/pull/59641#discussion_r4328432129
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala:
##########
@@ -2557,6 +2561,86 @@ abstract class ParquetFilterSuite extends ParquetTest
with SharedSparkSession {
}
}
+ test("SPARK-60380: filter pushdown - nanosecond timestamp read with a lower
precision") {
Review Comment:
You are right, I missed that `ParquetFilterSuite` runs as
`ParquetV1FilterSuite` and `ParquetV2FilterSuite`. Sorry for the noise, the V2
path is covered.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimestampNanosParquetOps.scala:
##########
@@ -320,6 +320,16 @@ private[ops] object TimestampNanosParquetOps {
private def epochNanosInRange(encode: => JLong): Boolean =
try { encode; true } catch { case _: ArithmeticException => false }
+ // Below nanosecond precision, the reader truncates the stored
epoch-nanoseconds to the read
+ // precision (see nanosConverter and TimestampNanosVectorUpdater), while a
pushed down predicate
+ // compares the stored epoch-nanoseconds. E.g. a stored 00:00:00.123456789
is read as
+ // 00:00:00.1234567 for TIMESTAMP_NTZ(7), so a pushed down `ts =
00:00:00.1234567` would skip it.
+ private def acceptsNanosReadType(readType: DataType): Boolean = readType
match {
Review Comment:
Good point about the inversion: `NOT (c > X)` is pushed as `not(gt(X))`,
which parquet-mr rewrites to `ltEq(X)`, so `>` and `!=` are only safe when they
are not negated, and an operator-aware gate would have to know the context.
Keeping this PR minimal for `branch-4.3` is fine with me. Could you link the
new JIRA for the bound rewrite from the PR description once it is filed, so the
deferred row-group skipping for precisions 7 and 8 (and `TIME(0-5)`) is tracked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
