GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/20137
[SPARK-22939] [PySpark] Support Spark UDF in registerFunction [WIP]
## What changes were proposed in this pull request?
```Python
import random
from pyspark.sql.functions import udf
from pyspark.sql.types import IntegerType, StringType
random_udf = udf(lambda: int(random.random() * 100),
IntegerType()).asNondeterministic()
spark.catalog.registerFunction("random_udf", random_udf, StringType())
spark.sql("SELECT random_udf()").collect()
```
We will get the following error.
```
Py4JError: An error occurred while calling o29.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
at py4j.Gateway.invoke(Gateway.java:274)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:745)
```
This PR is to support it.
## How was this patch tested?
WIP
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark registerFunction
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20137.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20137
----
commit 8216b6bb52082883fc9b212cd9ab21227f2b8491
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-02T15:28:19Z
wip
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
