sadikovi opened a new pull request, #59687: URL: https://github.com/apache/spark/pull/59687
### What changes were proposed in this pull request? Backport for branch-4.x. Original PR https://github.com/apache/spark/pull/59310. Spark Connect represents DataFrame subqueries such as `col.isin(df)`, `df.scalar()`, `df.exists()`, and table arguments as `UnresolvedPlanId` expressions (`UnresolvedInSubqueryPlanId`, `UnresolvedScalarSubqueryPlanId`, `UnresolvedExistsPlanId`, `UnresolvedTableArgPlanId`). These expressions point to the subquery plan by plan id. Only `SparkConnectPlanner.transformWithRelations` replaces them with the actual plan, using the enclosing `WithRelations` node. No analyzer rule resolves them. This PR: - Adds a new sub-condition `CONNECT_INVALID_PLAN.SUBQUERY_PLAN_NOT_ATTACHED`. It inherits SQLSTATE `56K00` from `CONNECT_INVALID_PLAN`, the same classification `transformWithRelations` already uses for a missing plan id inside a `WithRelations`. - Adds an `UnresolvedPlanId` case to the expression-level checks in `CheckAnalysis.checkAnalysis0`. The case throws the new `AnalysisException`, which carries the plan id and the expression's origin. - Adds `QueryCompilationErrors.subqueryPlanNotAttached`. ### Why are the changes needed? If the client never attaches the referenced plan (for example, when the root relation is not wrapped in a `WithRelations` that contains it), the `UnresolvedPlanId` survives analysis. CheckAnalysis has no specific case for it, so it reaches the generic unresolved-operator check and fails with an internal error: [INTERNAL_ERROR] Found the unresolved operator: ... SQLSTATE: XX000 This means the Connect client sent an invalid plan. It can come from a non-conforming third-party client, or from the Scala Connect client itself: the typed `Dataset.select[U1](TypedColumn)` and `Dataset.joinWith` do not collect subquery expressions, so they send plan ids without `WithRelations`. It is not a Spark server bug, so it should be reported as an invalid Spark Connect plan rather than `INTERNAL_ERROR`. ### Does this PR introduce _any_ user-facing change? Yes, but only for queries that already fail. A Spark Connect DataFrame subquery whose plan id is never attached used to fail with `[INTERNAL_ERROR] ... SQLSTATE: XX000`. It now fails with an `AnalysisException`: [CONNECT_INVALID_PLAN.SUBQUERY_PLAN_NOT_ATTACHED] The Spark Connect plan is invalid. The subquery plan referenced by plan id 42 is not attached to the request. SQLSTATE: 56K00 Valid queries are not affected. ### How was this patch tested? Added new tests to `SparkConnectProtoSuite`. Ran locally: - `build/sbt "connect/testOnly *SparkConnectProtoSuite -- -z SPARK-60096"`: passed - `build/sbt "core/testOnly *SparkThrowableSuite"`: 38 passed (validates `error-conditions.json`) ### Was this patch authored or co-authored using generative AI tooling? Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
