vranes opened a new pull request, #57637: URL: https://github.com/apache/spark/pull/57637
### What changes were proposed in this pull request? A clause like `ORDER BY` / `WHERE` / `HAVING` / `QUALIFY` / `DISTRIBUTE BY` can reference a column that is not in its operator's output. The single-pass resolver resolves such a column from hidden output and appends it to the `Project` / `Aggregate` / `Window` below, in `ResolvesNameByHiddenOutput.expandOperatorsOutputList`, so the clause can reference it. That append never checked whether the child actually produces the column. When the child drops it (for example `PIVOT` / `UNPIVOT`), the operator ends up referencing an attribute its child does not output. This is an invalid query that the fixed-point analyzer rejects with a clean `MISSING_ATTRIBUTES` error. Single-pass instead hit a failed `assert` in the phase-2 `ResolutionValidator` and reported `INTERNAL_ERROR`. This PR makes single-pass reject the query with the same clean `MISSING_ATTRIBUTES` error. **Why the existing check did not catch it.** Single-pass already has a `missingInput` check in `Resolver.validateOperatorResolution`. It runs once per operator, when that operator is resolved. The operator that becomes invalid here is a `Project` / `Aggregate` / `Window` further down the tree; it is checked (and passes) when it is resolved. Later, while resolving an operator above it (for example the `Sort`), `expandOperatorsOutputList` appends the hidden-output column to it, which is what makes it reference a column its child does not produce. That append does not re-run the check on the modified operator. **The fix.** Run the same missing-input check right after `expandOperatorsOutputList` builds the operator, and raise `MISSING_ATTRIBUTES` there. Placing it after the `match` covers `Project`, `Aggregate`, and `Window` with one check. The error is built by a shared `QueryCompilationErrors.missingAttributesError`, used by the single-pass call sites and by the fixed-point `CheckAnalysis`. Attribute ordering stays each caller's responsibility, so neither analyzer's existing message changes. The new check runs during resolution, before the phase-2 rewrite rules (`PruneMetadataColumns`, `CleanupAliases`, `PullOutNondeterministic`). None of those removes a reference the check would throw on: `CleanupAliases` only strips alias wrappers, `PullOutNondeterministic` adds a reference and its producer together, and `PruneMetadataColumns` only removes `qualifiedAccessOnly` metadata columns (always produced by the child at check time, since `expandOperatorsOutputList` widens the child before appending). The column that is actually missing in this bug is an ordinary data column, which no rewrite removes. ### Why are the changes needed? An invalid query that the fixed-point analyzer rejects cleanly with `MISSING_ATTRIBUTES` surfaced as `INTERNAL_ERROR` under the single-pass resolver. This aligns the two analyzers on the clean error. Only invalid queries that both analyzers already reject are affected; no valid query changes result. ### Does this PR introduce _any_ user-facing change? Yes, for invalid queries analyzed by the single-pass resolver: the reported error changes from `INTERNAL_ERROR` to `MISSING_ATTRIBUTES`, matching the fixed-point analyzer. Valid queries are unaffected. ### How was this patch tested? New tests assert the clean `MISSING_ATTRIBUTES` error (not `INTERNAL_ERROR`) for a sort on a column dropped or re-output by `PIVOT` / `UNPIVOT` / `BIN BY`, and confirm a valid query does not trip the new check: - `DataFramePivotSuite`, `DatasetUnpivotSuite`, `BinBySuite`: the fixed-point analyzer rejects a sort on a dropped column. For `PIVOT` a bare `ORDER BY t.v` reports `UNRESOLVED_COLUMN` (the column is never appended to a lower operator), while a DataFrame `sort` carrying the dropped column's id reaches hidden-output insertion and reports `MISSING_ATTRIBUTES`; both cases are pinned. The single-pass resolver does not yet support `PIVOT` / `UNPIVOT` / `BIN BY`, so these run on the fixed-point path. - `DataFrameAnalyzerTestGapsSuite`: a chained `orderBy` that resolves hidden columns through several nested `Project`s under the single-pass resolver, confirming the new check runs and does not fire on a valid query. - Existing `AnalysisErrorSuite` `MISSING_ATTRIBUTES` coverage confirms the shared error helper preserves the message. - Compiled and ran with Scala 2.13. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Anthropic) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
