Max Gekk created SPARK-57748:
--------------------------------
Summary: Use a dedicated tree-pattern bit for the TIME ->
TIMESTAMP_NTZ cast rewrite in ComputeCurrentTime
Key: SPARK-57748
URL: https://issues.apache.org/jira/browse/SPARK-57748
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3.0
Reporter: Max Gekk
h2. Summary
Replace the broad {{containsPattern(CAST)}} pruning condition in the
{{ComputeCurrentTime}}
optimizer rule with a dedicated tree-pattern bit, so the rule only descends
into plans that
actually contain a cast to the {{TIMESTAMP_NTZ}} family instead of any plan
that contains any cast.
h2. Background
SPARK-57618 added {{CAST(TIME(p) AS TIMESTAMP_NTZ(q))}}, whose date fields come
from
{{CURRENT_DATE}}. To keep the value query-stable, {{ComputeCurrentTime}}
rewrites such casts into a
date+time builder anchored on the same current-date literal as
{{current_date()}}.
For the rule to visit those casts, its pruning predicate was widened to:
{code:scala}
bits.containsPattern(CURRENT_LIKE) || bits.containsPattern(CAST)
{code}
Tagging the {{Cast}} with {{CURRENT_LIKE}} was rejected earlier because that
pattern has shared
semantics (e.g. inline-table validation in {{EvaluateUnresolvedInlineTable}}
treats {{CURRENT_LIKE}}
expressions as safe to defer, which would let unrelated non-foldable NTZ-target
casts such as
{{CAST(rand() AS TIMESTAMP_NTZ)}} bypass validation).
The {{CAST}} fallback is correct but defeats pruning: casts are present in
almost every query, so
{{ComputeCurrentTime}} now traverses the full expression tree of essentially
every plan even though
the {{TIME -> TIMESTAMP_NTZ}} rewrite fires rarely.
h2. Proposal
Introduce a dedicated {{TreePattern}} (e.g. {{CAST_TO_TIMESTAMP_NTZ}}) and:
* Tag it in {{Cast.nodePatternsInternal}} keyed on the *target* type only (the
{{TIMESTAMP_NTZ}} /
{{TIMESTAMP_NTZ(p)}} families), never on {{child.dataType}}.
* Prune {{ComputeCurrentTime}} on {{CURRENT_LIKE || CAST_TO_TIMESTAMP_NTZ}}
instead of {{CAST}}.
The node-level {{Cast.isTimeToTimestampNTZ}} guard stays, so only {{TIME ->
TIMESTAMP_NTZ}} casts are
actually rewritten.
h2. Constraints / notes
* The tag must be keyed on the target type, not the source:
{{nodePatternsInternal}} is computed
eagerly at {{Cast}} construction, before the child is resolved, and reading
{{child.dataType}} there
can throw even when {{child.resolved}} is true (e.g. an {{OuterReference}}
wrapping an unresolved
attribute - the {{makeSQLTableFunctionPlan}} / {{sql-udf.sql}} crash seen
during SPARK-57618). The
target type is always safe to read.
* This still slightly over-tags (all NTZ-target casts, not strictly {{TIME
->}}), but the bit is
dedicated with no other consumers, so it cannot leak into inline-table
validation, streaming
{{CURRENT_LIKE}} handling, or {{ReplaceCurrentLike}}. Full source precision is
not safely achievable
at construction time.
h2. Testing
* Keep the existing inline-table regression test ({{CAST(rand() AS
TIMESTAMP_NTZ)}} still rejected).
* Add a {{ComputeCurrentTimeSuite}} assertion that a plan whose only casts are
unrelated
(e.g. {{string -> int}}) is left untouched, while {{TIME -> TIMESTAMP_NTZ}} is
still rewritten to a
date literal consistent with {{current_date()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
