Holden Karau created SPARK-59069:
------------------------------------
Summary: Transpile Python UDFs with explicit positional-only
parameters
Key: SPARK-59069
URL: https://issues.apache.org/jira/browse/SPARK-59069
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.4.0
Reporter: Holden Karau
The transpiler refuses a UDF with positional-only parameters outright:
```python
def f(a, b, /):
return a + b
```
falls back to interpreted Python, alongside defaults, `*args`, `**kwargs` and
keyword-only parameters. For those four the refusal is necessary: the transpiled
expression references its inputs positionally through `_udf_param_N`, and a
call site
that omits a defaulted argument, or passes one by keyword only, leaves a
placeholder
pointing at a position nothing bound.
A positional-only parameter has no such gap. It is *always* bound by position
-- that is
what positional-only means -- so the placeholder scheme fits it exactly. The
refusal was
over-broad, and it came about only because `_get_parameter_list` read
`node.args.args` and never `node.args.posonlyargs`, so the parameter list would
simply
have been short by however many positional-only parameters the function
declared. Refusing
was the safe response to that, not a statement about the feature.
Two parts:
**1. Lower them.** Read `posonlyargs + args` everywhere the positional
parameter list is
derived -- `_get_parameter_list`, `_param_category_combos` (so each parameter's
input-type category lands in the right slot), and the located-versus-held lambda
parameter comparison in `_get_function_from_ast`. A shared `_positional_args`
helper
keeps the concatenation in one place. Drop `posonlyargs` from the refusal list;
a
*defaulted* positional-only parameter still hits the `defaults` check, as it
must.
**2. Do not resolve a positional-only kwarg to a slot.**
`UserDefinedFunction.__call__`
rewrites user kwargs to positional order, so that a transpiled `_udf_param_N`
expression
sees plain column references rather than a `NamedArgumentExpression` (which
breaks nested
calls such as `isnotnull`). Once part 1 makes these UDFs candidates, that
rewrite would
happily resolve `f(a=col)` for a positional-only `a` -- silently turning a call
**Python
itself rejects** into a valid one:
```python
def f(a, /):
return a + 1
f(a=1) # TypeError: f() got some positional-only arguments passed as
keyword arguments
udf_f(a=col) # would have succeeded, with transpilation on
```
So track which public parameters are positional-only and exclude them from the
rewrite.
Left unresolved, the kwarg reaches the JVM as a `NamedArgumentExpression`,
which drops the
transpiled expression and routes to the interpreted path -- where the worker's
own keyword
call raises the same `TypeError` Python would. The fallback does the right
thing; it just
has to be allowed to happen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
