voonhous commented on code in PR #20526:
URL: https://github.com/apache/hudi/pull/20526#discussion_r4479703676
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/dataskipping/StatsFilterCompiler.scala:
##########
@@ -64,10 +64,12 @@ object StatsFilterCompiler {
case leaf => compileLeaf(leaf, source)
}
- // Stages 2 to 4, for one leaf
+ // Stages 2 to 4, for one leaf. A verdict over a date rendering also keeps
the files spanning a local date
+ // reversal ([[LocalDateReversalGuard]])
private def compileLeaf(e: Expression, source: RangeSource): Expression =
Leaf.parse(e)
.flatMap(l => TermRange.derive(l.term, source).map(r => Verdict(l, r)))
+ .flatMap(LocalDateReversalGuard(e, source, _))
Review Comment:
This is blocked by #19474.
`LocalDateReversalGuard` runs between `Verdict` and `keepOnNull`, which adds
a fifth stage to the four from #20527 (Shape, Leaf, Range, Verdict). It also
re-scans the raw leaf for `Cast` / `DateFormatClass` / `FromUnixTime`, so a
rule arm added later that renders a local date would skip the guard silently.
The NOTE is right that a `fileGuard` ends pruning on master today. #19474
lifts that: a guarded chain still derives a range, and `keepOnNull` keeps the
file when a guarded bound is NULL. Once it lands, could we move this into the
Range stage as a `fileGuard` on the rendering arms and drop the extra stage?
Rough shape on top of #19474 (names are placeholders):
```scala
// RangeRule.scala: combine per-file guards; None stays "unguarded"
type Guard = TermRange => Expression
def andGuards(a: Option[Guard], b: Option[Guard]): Option[Guard] = (a, b)
match {
case (Some(x), Some(y)) => Some(r => And(x(r), y(r)))
case _ => a.orElse(b)
}
// rules/DateTimeOrdering.scala (#19474), next to isFixedOffsetZone
/** Instants at which the zone's local date runs backwards; None if its
current rules keep doing it. */
def localDateReversals(timeZoneId: Option[String]): Option[Seq[Instant]] =
... // today's localDateReversals, cached
val micros: Instant => Literal = i =>
Literal(DateTimeUtils.instantToMicros(i), TimestampType)
/** File guard: [lo, hi] spans none of the instants. None when there are
none, so the step stays unguarded. */
def noLocalDateReversal(instants: Seq[Instant], toLiteral: Instant =>
Literal): Option[Guard] =
if (instants.isEmpty) None
else Some(r => instants.map(toLiteral)
.map(t => Not(And(LessThan(r.lo, t), GreaterThanOrEqual(r.hi, t))))
.reduce[Expression](And))
// rules/CastRule.scala: ahead of the generic arm, which accepts
(TimestampType, DateType) unguarded today
case Cast(child, DateType, timeZoneId, _) if hasDataType(child) &&
child.dataType == TimestampType =>
localDateReversals(timeZoneId).toSeq.map { instants =>
Step(term, child, Increasing, fileGuard = noLocalDateReversal(instants,
micros), partial = false)
}
// rules/DateTimeFormatRule.scala: next to the existing formattableInstants
guard
case DateFormatClass(timestamp, format, timeZoneId)
if dataTypeOf(timestamp).contains(TimestampType) &&
DateTimeOrdering.formatPreservesOrder(format) =>
localDateReversals(timeZoneId).toSeq.map { instants =>
Step(term, timestamp, Increasing,
fileGuard = andGuards(Some(formattableInstants),
noLocalDateReversal(instants, micros)), partial = false)
}
// FromUnixTime: the same, with epoch seconds and formattableSeconds
```
With this, `RuntimeReplaceable` handling, `rebind` and `TryEval` come from
the existing stages; #19474 already wraps partial bounds in `TryEval`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
