[
https://issues.apache.org/jira/browse/SPARK-58423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan reassigned SPARK-58423:
-----------------------------------
Assignee: Nikolina Vraneš
> Raise MISSING_ATTRIBUTES for missing input after single-pass hidden-output
> insertion
> ------------------------------------------------------------------------------------
>
> Key: SPARK-58423
> URL: https://issues.apache.org/jira/browse/SPARK-58423
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Nikolina Vraneš
> Assignee: Nikolina Vraneš
> Priority: Major
> Labels: pull-request-available
>
> A clause like ORDER BY / WHERE / HAVING / QUALIFY / DISTRIBUTE BY can
> reference a column that is not in its operator's output. The single-pass
> resolver resolves such a column from hidden output and appends it to the
> Project / Aggregate / Window below, in
> ResolvesNameByHiddenOutput.expandOperatorsOutputList, so the clause can
> reference it.
> That append never checked whether the child actually produces the column.
> When the child drops it, the operator ends up referencing an attribute its
> child does not output. This is an invalid query that the fixed-point analyzer
> rejects with a clean MISSING_ATTRIBUTES error. Single-pass instead hit a
> failed assert in the phase-2 ResolutionValidator and reported INTERNAL_ERROR.
> The single-pass missingInput check in Resolver.validateOperatorResolution
> runs once per operator when that operator is resolved. The operator that
> becomes invalid here is a Project / Aggregate / Window further down the tree,
> which is checked (and passes) when it is resolved; the later append that
> makes it invalid does not re-run the check.
> The fix runs the same missing-input check right after
> expandOperatorsOutputList builds the operator, raising MISSING_ATTRIBUTES. A
> shared QueryCompilationErrors.missingAttributesError builds the error for
> both the single-pass call sites and the fixed-point CheckAnalysis. This only
> affects invalid queries that both analyzers already reject; no valid query
> changes result.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
