anshulbaliga7 opened a new pull request, #59226: URL: https://github.com/apache/spark/pull/59226
### What changes were proposed in this pull request? `CreateSQLFunctionCommand` now captures the temporary variables that the function body reads only inside IDENTIFIER clauses and treats them like any other temporary variable reference: a persistent function is rejected with `INVALID_TEMP_OBJ_REFERENCE`, and a temporary function records them in its `referredTempVariableNames` property. The names are recorded by `ResolveIdentifierClause` in `AnalysisContext.referredTempVariableNamesUnderIdentifier` (SPARK-59323), but the command analyzed the body with `Analyzer.execute`, which opens a fresh `AnalysisContext` and discards the accumulator on exit. A new `Analyzer.executeWithReferredTempVariablesUnderIdentifier` runs the same fixed-point analysis in a context it owns and returns the recorded names alongside the analyzed plan. It is the sibling of `executeAndCheckReferredTempVariablesUnderIdentifier`, minus `checkAnalysis`, because the function command inspects the analyzed body (return type inference, aggregate/window checks) before checking it. Both analysis sites (scalar and table function) use it, and the names are unioned into `verifyTemporaryObjectsNotExists` and `generateFunctionProperties`. Bodies without IDENTIFIER clauses are unaffected. ### Why are the changes needed? A persistent SQL function could capture a session-scoped dependency and then fail in another session: ```sql DECLARE OR REPLACE VARIABLE col STRING DEFAULT 'a'; CREATE FUNCTION f(x INT) RETURNS INT RETURN (SELECT IDENTIFIER(col) FROM VALUES (1) AS T(a)); -- succeeds; in a new session: SELECT f(1); -- [NOT_A_CONSTANT_STRING.NOT_CONSTANT] The expression col used for the routine or clause -- IDENTIFIER must be a constant STRING which is NOT NULL. ``` A direct reference (`RETURN x + length(col)`) was already rejected, because `ViewHelper.collectTemporaryVariables` sees the `VariableReference` in the analyzed plan. Under an IDENTIFIER clause the reference is gone by then: the placeholder is replaced by the expression or plan built from the evaluated name. SPARK-59323 (#58606) fixed the same gap for CREATE VIEW, ALTER VIEW, CACHE TABLE AS SELECT and metric views; its review noted the SQL function path as pre-existing and to be tracked separately. ### Does this PR introduce _any_ user-facing change? Yes. `CREATE FUNCTION` (persistent) whose body reads a temporary variable only through an IDENTIFIER clause now fails with `INVALID_TEMP_OBJ_REFERENCE` at creation, matching a direct variable reference, instead of succeeding and failing later when the function is called from a session where the variable does not exist. Temporary functions are unchanged in behavior; their stored `referredTempVariableNames` now also lists such variables. ### How was this patch tested? New cases in `sql-udf.sql` (regenerated through `SQLQueryTestSuite`), in the existing "permanent functions with temp objects" section: - a direct temporary variable reference, as the baseline rejection; - a variable read through IDENTIFIER in a scalar function body and in a table function body, both rejected with `INVALID_TEMP_OBJ_REFERENCE` naming the variable (both were accepted before this change); - a temporary function reading the variable through IDENTIFIER is created and evaluates. ``` SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z sql-udf.sql" build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z sql-udf.sql -z identifier-clause.sql" "sql/testOnly org.apache.spark.sql.execution.SQLFunctionSuite" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude claude-opus-4-8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
