[
https://issues.apache.org/jira/browse/SPARK-60806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha S updated SPARK-60806:
------------------------------
Description:
CREATE FUNCTION decides whether a permanent SQL function references a temporary
function by scanning the unresolved body for called names that match a
temporary function, not by checking which function each call actually resolves
to. So it rejects valid functions and accepts invalid ones:
- Over-rejection: a temporary function that only shares a name with a built-in
blocks a body calling that built-in.
- Under-rejection: the scan only matches plain scalar calls, so a body that
calls a temporary table function, or a temporary function named through
IDENTIFIER(...), is accepted. The permanent function then only works in the
session that created it.
*Steps to reproduce (default settings; verified on master):*
{code:java}
-- Over-rejection: the built-in owns abs, yet the body is rejected.
CREATE TEMPORARY FUNCTION abs(x INT) RETURNS INT RETURN 42;
SELECT abs(-1); -- 1
(the built-in)
CREATE FUNCTION p_abs() RETURNS INT RETURN abs(-1); --
INVALID_TEMP_OBJ_REFERENCE
DROP TEMPORARY FUNCTION abs;
-- Under-rejection: permanent functions that depend on temporary functions are
accepted.
CREATE TEMPORARY FUNCTION temp_tvf() RETURNS TABLE (a INT) RETURN SELECT 1;
CREATE FUNCTION p_tvf() RETURNS TABLE (a INT) RETURN SELECT * FROM temp_tvf();
-- succeeds
CREATE TEMPORARY FUNCTION temp_fn() RETURNS INT RETURN 1;
CREATE FUNCTION p_id() RETURNS INT RETURN IDENTIFIER('temp_fn')(); --
succeeds
SELECT * FROM p_tvf(); -- 1
(this session only)
SELECT p_id(); -- 1
(this session only)
-- In any other session (e.g. spark.newSession()):
SELECT * FROM p_tvf(); --
UNRESOLVABLE_TABLE_VALUED_FUNCTION
SELECT p_id(); --
UNRESOLVED_ROUTINE
Expected behavior:
- p_abs is created and returns 1, because the built-in owns the call.
- p_tvf and p_id fail with INVALID_TEMP_OBJ_REFERENCE.
{code}
{*}Proposed fix{*}: as CREATE VIEW already does, reject only the temporary
functions that analysis of the body actually binds
(AnalysisContext.referredTempFunctionNames), and report them in sorted order so
the error is deterministic.
*Compatibility:* rejecting p_tvf and p_id is a behavior change for CREATE
FUNCTION statements that succeed today. It needs a migration-guide entry, and
possibly a legacy config to restore the old behavior.
was:
CREATE FUNCTION decides whether a permanent SQL function references a temporary
function by scanning the unresolved body for called names that match a
temporary function, not by checking which function each call actually resolves
to. So it rejects valid functions and accepts invalid ones:
- Over-rejection: a temporary function that only shares a name with a built-in
blocks a body calling that built-in.
- Under-rejection: the scan only matches plain scalar calls, so a body that
calls a temporary table function, or a temporary function named through
IDENTIFIER(...), is accepted. The permanent function then only works in the
session that created it.
*Steps to reproduce (default settings; verified on master):*
{code:java}
-- Over-rejection: the built-in owns abs, yet the body is rejected.
CREATE TEMPORARY FUNCTION abs(x INT) RETURNS INT RETURN 42;
SELECT abs(-1); -- 1
(the built-in)
CREATE FUNCTION p_abs() RETURNS INT RETURN abs(-1); --
INVALID_TEMP_OBJ_REFERENCE
DROP TEMPORARY FUNCTION abs;
-- Under-rejection: permanent functions that depend on temporary functions are
accepted.
CREATE TEMPORARY FUNCTION temp_tvf() RETURNS TABLE (a INT) RETURN SELECT 1;
CREATE FUNCTION p_tvf() RETURNS TABLE (a INT) RETURN SELECT * FROM temp_tvf();
-- succeeds
CREATE TEMPORARY FUNCTION temp_fn() RETURNS INT RETURN 1;
CREATE FUNCTION p_id() RETURNS INT RETURN IDENTIFIER('temp_fn')(); --
succeeds
SELECT * FROM p_tvf(); -- 1
(this session only)
SELECT p_id(); -- 1
(this session only)
-- In any other session (e.g. spark.newSession()):
SELECT * FROM p_tvf(); --
UNRESOLVABLE_TABLE_VALUED_FUNCTION
SELECT p_id(); --
UNRESOLVED_ROUTINE
Expected behavior:
- p_abs is created and returns 1, because the built-in owns the call.
- p_tvf and p_id fail with INVALID_TEMP_OBJ_REFERENCE.
{code}
{*}Proposed fix{*}: as CREATE VIEW already does, reject only the temporary
functions that analysis of the body actually binds
(AnalysisContext.referredTempFunctionNames), and report them in sorted order so
the error is deterministic.
*Compatibility:* rejecting p_tvf and p_id is a behavior change for CREATE
FUNCTION statements that succeed today. It needs a migration-guide entry, and
possibly a legacy config to restore the old behavior.
> CREATE FUNCTION should reject only the temporary functions a permanent
> function's body actually calls
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-60806
> URL: https://issues.apache.org/jira/browse/SPARK-60806
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Ganesha S
> Priority: Major
>
> CREATE FUNCTION decides whether a permanent SQL function references a
> temporary function by scanning the unresolved body for called names that
> match a temporary function, not by checking which function each call actually
> resolves to. So it rejects valid functions and accepts invalid ones:
> - Over-rejection: a temporary function that only shares a name with a
> built-in blocks a body calling that built-in.
> - Under-rejection: the scan only matches plain scalar calls, so a body that
> calls a temporary table function, or a temporary function named through
> IDENTIFIER(...), is accepted. The permanent function then only works in the
> session that created it.
> *Steps to reproduce (default settings; verified on master):*
> {code:java}
> -- Over-rejection: the built-in owns abs, yet the body is rejected.
> CREATE TEMPORARY FUNCTION abs(x INT) RETURNS INT RETURN 42;
> SELECT abs(-1); -- 1
> (the built-in)
> CREATE FUNCTION p_abs() RETURNS INT RETURN abs(-1); --
> INVALID_TEMP_OBJ_REFERENCE
> DROP TEMPORARY FUNCTION abs;
> -- Under-rejection: permanent functions that depend on temporary functions
> are accepted.
> CREATE TEMPORARY FUNCTION temp_tvf() RETURNS TABLE (a INT) RETURN SELECT 1;
> CREATE FUNCTION p_tvf() RETURNS TABLE (a INT) RETURN SELECT * FROM
> temp_tvf(); -- succeeds
> CREATE TEMPORARY FUNCTION temp_fn() RETURNS INT RETURN 1;
> CREATE FUNCTION p_id() RETURNS INT RETURN IDENTIFIER('temp_fn')(); --
> succeeds
> SELECT * FROM p_tvf(); -- 1
> (this session only)
> SELECT p_id(); -- 1
> (this session only)
> -- In any other session (e.g. spark.newSession()):
> SELECT * FROM p_tvf(); --
> UNRESOLVABLE_TABLE_VALUED_FUNCTION
> SELECT p_id(); --
> UNRESOLVED_ROUTINE
> Expected behavior:
> - p_abs is created and returns 1, because the built-in owns the call.
> - p_tvf and p_id fail with INVALID_TEMP_OBJ_REFERENCE.
> {code}
>
> {*}Proposed fix{*}: as CREATE VIEW already does, reject only the temporary
> functions that analysis of the body actually binds
> (AnalysisContext.referredTempFunctionNames), and report them in sorted order
> so the error is deterministic.
> *Compatibility:* rejecting p_tvf and p_id is a behavior change for CREATE
> FUNCTION statements that succeed today. It needs a migration-guide entry, and
> possibly a legacy config to restore the old behavior.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
