srielau commented on code in PR #56040: URL: https://github.com/apache/spark/pull/56040#discussion_r3288245471
########## docs/sql-ref-syntax-ddl-create-sql-function.md: ########## @@ -124,6 +140,15 @@ characteristic - [Aggregate functions](sql-ref-functions-builtin.md#aggregate-functions) - [Window functions](sql-ref-functions-builtin.md#analytic-window-functions) - [Ranking functions](sql-ref-functions-builtin.md#ranking-window-functions) + + A persistent SQL UDF cannot reference temporary views, temporary functions, or session + variables. + + The SQL Path in effect at `CREATE FUNCTION` time is captured into the function's metadata; the + body resolves against that frozen path on every invocation, not the invoker's current path. + `current_schema()` and `current_path()` inside the body still return the invoker's context. + Use [DESCRIBE FUNCTION EXTENDED](sql-ref-syntax-aux-describe-function.html) to inspect the + captured path. See [SET PATH](sql-ref-syntax-aux-conf-mgmt-set-path.html). - Row producing functions such as `explode` Review Comment: Fixed in d04ce7afb90 — moved the frozen-path paragraphs after the disallowed-expressions list ends. Verified the rendered `<ul>` now closes with `Row producing functions such as explode` as the last bullet. ########## docs/sql-ref-syntax-aux-conf-mgmt-set-path.md: ########## @@ -0,0 +1,238 @@ +--- +layout: global +title: SET PATH +displayTitle: SET PATH +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +### Description + +`SET PATH` changes the **SQL Path** of the current session. + +The SQL Path is an ordered list of catalog-qualified schema names that Spark walks when +resolving unqualified references to functions, tables, views, and session variables in queries +and DML (`SELECT`, `INSERT`, `UPDATE`, `DELETE`, `MERGE`). The first match wins. DDL +(`CREATE TABLE`, `CREATE VIEW`, `CREATE FUNCTION`, `DROP`, `ALTER`, ...) resolves unqualified +object names against `current_catalog.current_schema`, not the path; so `CREATE TABLE t` always +creates `t` in the current schema regardless of the path. + +The path can include two virtual namespaces in the `system` catalog: + +- `system.builtin` — built-in functions, including those injected by + `SparkSessionExtensions`. +- `system.session` — temporary views, temporary functions, and session variables in the + current session. + +`SET PATH` is controlled by `spark.sql.path.enabled`. When it is `false` (the default), +`SET PATH` raises `UNSUPPORTED_FEATURE.SET_PATH_WHEN_DISABLED`. Unqualified resolution and +[`current_path()`](sql-ref-function-current-path.html) still use the default path. + +The initial value of `PATH` in a session is `DEFAULT_PATH`. `DEFAULT_PATH` is either the value of +`spark.sql.defaultPath`, or, when that configuration is empty, a built-in value composed of +`system.builtin`, `system.session`, and the current schema. To override, set +`spark.sql.defaultPath`. See the [`DEFAULT_PATH` parameter](#parameters) for the exact derivation +rules. + +The effect of `SET PATH` is scoped to the current session and is lost when the session ends. To +re-apply the current default path mid-session, run `SET PATH = DEFAULT_PATH`. (This stores a +snapshot of `DEFAULT_PATH` at the moment of the statement; later changes to +`spark.sql.defaultPath` are not picked up automatically.) Cloned sessions inherit the parent's +path at clone time; later changes in the child do not propagate back. + +Persistent views and SQL UDFs capture the path at `CREATE` time into the object's metadata. +Each invocation resolves the body against that frozen path, not the invoker's current path; +`current_schema()` and `current_path()` inside the body still return the invoker's context. + +The leading names `session` and `builtin` have special meaning in 2-part references; see +[Reserved system names](sql-ref-identifier.html#reserved-system-names). + +### Syntax + +```sql +SET PATH = path_element [ , ... ] + +path_element + { DEFAULT_PATH | + SYSTEM_PATH | + PATH | + CURRENT_SCHEMA | + CURRENT_DATABASE | + catalog_name . schema_name } +``` + +### Parameters + +* **`DEFAULT_PATH`** + + Expands to the session's default path. The default path has two layers: + + 1. If `spark.sql.defaultPath` is set to a non-empty value, that value is parsed using the same + grammar as `SET PATH` (with one restriction: the `PATH` keyword is not allowed inside the + conf value, since it would be self-referential). + + The conf value is validated for syntax at the time it is set; an invalid value is rejected. + Static duplicates inside the conf are tolerated (unlike interactive `SET PATH`, which + rejects them) so a later `USE SCHEMA` cannot turn a previously valid default into a runtime + error. A `DEFAULT_PATH` token inside the conf value resolves to the spark-built-in default + below to avoid a cycle, rather than recursing. + + 2. If `spark.sql.defaultPath` is empty (the factory setting), the spark-built-in default + applies: `system.builtin`, `system.session`, and the current schema + (`current_catalog.current_schema`), in that order. + + To change the default path, set `spark.sql.defaultPath` via any of the usual mechanisms + (`SET spark.sql.defaultPath = ...` at runtime, `--conf` on `spark-submit`, `SparkConf`, or + `spark-defaults.conf`); clear it with `RESET spark.sql.defaultPath` to return to the + spark-built-in default. + +* **`SYSTEM_PATH`** + + Expands to the two system namespaces, `system.builtin` and `system.session`. + +* **`PATH`** + + Expands to the **current** value of the SQL Path. Useful for appending entries without + re-typing them, for example `SET PATH = PATH, spark_catalog.analytics`. + `PATH` is not allowed in the value of `spark.sql.defaultPath` (it would create a cycle). + +* **`CURRENT_SCHEMA`** / **`CURRENT_DATABASE`** + + A virtual marker that resolves to the catalog-qualified current schema + (`current_catalog.current_schema`) every time the path is consulted. This means subsequent + `USE SCHEMA` statements are picked up without re-issuing `SET PATH`. + `CURRENT_DATABASE` is a synonym for `CURRENT_SCHEMA`. + +* **`schema_name`** + + An explicit catalog-qualified schema reference (`catalog.schema`). Both parts are required. Review Comment: Good catch — applied your suggested wording in d04ce7afb90, and updated the grammar block to `catalog_name . namespace [ . namespace ... ]` so the two are consistent. Agreed that the `INVALID_SQL_PATH_SCHEMA_REFERENCE` message itself advertises multi-level support; committing to two parts in the doc was both inaccurate and inconsistent with what users would see in the error. ########## docs/sql-migration-guide.md: ########## @@ -31,6 +31,9 @@ license: | - Since Spark 4.2, Spark enables order-independent checksums for shuffle outputs by default to detect data inconsistencies during indeterminate shuffle stage retries. If a checksum mismatch is detected, Spark rolls back and re-executes all succeeding stages that depend on the shuffle output. If rolling back is not possible for some succeeding stages, the job will fail. To restore the previous behavior, set `spark.sql.shuffle.orderIndependentChecksum.enabled` and `spark.sql.shuffle.orderIndependentChecksum.enableFullRetryOnMismatch` to `false`. - Since Spark 4.2, support for Derby JDBC datasource is deprecated. - Since Spark 4.2, a new default method `mergeWith` has been added to the `CustomTaskMetric` interface. The default implementation sums the two metric values, which is correct for count-type metrics. Data source connector implementations that report non-additive metrics (e.g., maximum, average, compression ratio, or gauge values) must override `mergeWith` to provide correct merge semantics. +- Since Spark 4.2, the virtual `system` catalog hosts the new `system.builtin` and `system.session` namespaces. `system.builtin` exposes built-in functions and functions injected through `SparkSessionExtensions`; `system.session` exposes temporary views, temporary functions, and session variables created in the current session. As a result, 2-part references like `builtin.func()` and `session.func()` now follow a mini-path that tries the system namespace first and the current catalog second, so a persistent schema named `builtin` or `session` is no longer reached by `builtin.func()` / `session.func()` when the system namespace contains an object of the same name. To restore the previous behavior (current catalog first), set `spark.sql.legacy.persistentCatalogFirst` to `true`. Persistent schemas with these names are still allowed but should be reached with an explicit catalog prefix (for example, `spark_catalog.session.x`). See [Reserved system names](sql-ref-identifier.html#reserved -system-names). +- Since Spark 4.2, `CREATE TEMPORARY VIEW`, `CREATE TEMPORARY FUNCTION`, and the corresponding `DROP` statements accept the `session` and `system.session` qualifiers on the object name (in addition to the previously supported unqualified form); for example, `CREATE TEMPORARY VIEW system.session.v AS ...` and `DROP TEMPORARY FUNCTION session.f` are now valid. Any other qualifier on a temporary object is rejected with `INVALID_TEMP_OBJ_QUALIFIER`. +- Spark 4.2 introduces the SQL standard `PATH` feature: the `SET PATH` statement, the `current_path()` function, the path-based resolution of unqualified routines / tables / views, and the configurations `spark.sql.path.enabled` (default `false`) and `spark.sql.defaultPath`. The feature is opt-in; when `spark.sql.path.enabled` is `false`, unqualified resolution falls back to a fixed default path and `SET PATH` is rejected with `UNSUPPORTED_FEATURE.SET_PATH_WHEN_DISABLED`. See [SET PATH](sql-ref-syntax-aux-conf-mgmt-set-path.html) and [Name Resolution](sql-ref-name-resolution.html). Review Comment: Both fixed in d04ce7afb90 — added session variables to the PATH-consumer list and re-opened the bullet with "Since Spark 4.2," to match the other entries in the section. ########## docs/sql-ref-function-current-path.md: ########## @@ -0,0 +1,85 @@ +--- +layout: global +title: current_path function +displayTitle: current_path function +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +Returns the effective SQL Path for the current session as a comma-separated string of +qualified namespace names. See [`SET PATH`](sql-ref-syntax-aux-conf-mgmt-set-path.html) for a +description of what the path is, how to enable it, and how to change it, and +[Name Resolution](sql-ref-name-resolution.html) for how the path drives unqualified name +resolution. + +### Syntax + +```sql +current_path() +``` + +### Arguments + +This function takes no arguments. The parentheses may be omitted. + +### Returns + +A non-nullable `STRING`. Each path entry is written as a dotted name with backticks added only +where required by Spark's identifier rules. Entries are separated by a single comma. + +When the path contains the virtual `CURRENT_SCHEMA` marker, the marker is materialized as the +catalog-qualified current schema (`current_catalog.current_schema`) each time +`current_path()` is evaluated, so subsequent `USE SCHEMA` statements are reflected without +re-issuing `SET PATH`. + +### Examples + +```sql +> SELECT current_path(); + system.builtin,system.session,spark_catalog.default + +-- ANSI no-parens form returns the same value. +> SELECT CURRENT_PATH; + system.builtin,system.session,spark_catalog.default + +-- The output reflects the latest SET PATH. +> SET PATH = spark_catalog.default, system.builtin; +> SELECT current_path(); + spark_catalog.default,system.builtin + +-- CURRENT_SCHEMA on the path is re-evaluated on every call. +> SET PATH = CURRENT_SCHEMA, system.builtin; +> USE spark_catalog.finance; +> SELECT current_path(); + spark_catalog.finance,system.builtin +> USE spark_catalog.default; +> SELECT current_path(); + spark_catalog.default,system.builtin + +-- Inside a persisted view or SQL function body, current_path() returns the invoker's path, Review Comment: Fixed in d04ce7afb90. ########## docs/sql-ref-name-resolution.md: ########## @@ -256,37 +259,54 @@ This restriction also applies to parameter references in SQL functions. frm.a lat.b func.c ``` -## Table and view resolution - -An identifier in table-reference can be any one of the following: +## Object name resolution -- Persistent table or view -- Common table expression (CTE) -- [Temporary view](sql-ref-syntax-ddl-create-view.html) +Tables, views, and functions follow the same resolution rule. It depends on how many parts the +identifier has. -Resolution of an identifier depends on whether it is qualified: +### Fully qualified (3 parts) — `catalog.schema.object` -- **Qualified** +The reference is unique and is looked up in `catalog.schema`. `system.builtin.object` identifies +a built-in function; `system.session.object` identifies a temporary view, function, or session +variable. - If the identifier is fully qualified with three parts: `catalog.schema.relation`, it is unique. +### Partially qualified (2 parts) — `schema.object` - If the identifier consists of two parts: `schema.relation`, it is further qualified with the result of `SELECT current_catalog()` to make it unique. +The identifier is qualified with `current_catalog` — producing +`current_catalog.schema.object` — unless the leading part is `session` (or `builtin`, for +functions). In that case Spark uses the +[mini-path](sql-ref-identifier.html#reserved-system-names) to choose the implicit catalog, Review Comment: Fixed in d04ce7afb90 by introducing the term "mini-path" in the prose of `Reserved system names`, so a reader following the link from `name-resolution.md` now lands on a definition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
