Yicong Huang created SPARK-54598:
------------------------------------
Summary: Refactor UDF fetching logic out from invocation
Key: SPARK-54598
URL: https://issues.apache.org/jira/browse/SPARK-54598
Project: Spark
Issue Type: Task
Components: PySpark
Affects Versions: 4.2.0
Reporter: Yicong Huang
Currently we always fetch UDFs (function and its arguments)
Single UDF:
```
arg_offsets, f = read_single_udf(
pickleSer, infile, eval_type, runner_conf, udf_index=0, profiler=profiler
)
parsed_offsets=extract_key_value_indexes(arg_offsets)
```
Multi UDFs:
```
udfs = []
for i in range(num_udfs):
udfs.append(
read_single_udf(
pickleSer, infile, eval_type, runner_conf, udf_index=i,
profiler=profiler
)
)
```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
