GitHub user gatorsmile reopened a pull request:
https://github.com/apache/spark/pull/20171
[SPARK-22978] [PySpark] Register Vectorized UDFs for SQL Statement [WIP]
## What changes were proposed in this pull request?
Register Vectorized UDFs for SQL Statement. For example,
```Python
>>> import random
>>> from pyspark.sql.types import IntegerType
>>> from pyspark.sql.functions import pandas_udf
>>> random_pandas_udf = pandas_udf(
... lambda x: random.randint(0, 100) + x, IntegerType())
... .asNondeterministic() # doctest: +SKIP
>>> _ = spark.catalog.registerFunction(
... "random_pandas_udf", random_pandas_udf, IntegerType()) # doctest:
+SKIP
>>> spark.sql("SELECT random_pandas_udf(2)").collect() # doctest: +SKIP
[Row(random_pandas_udf(2)=84)]
```
## How was this patch tested?
Added test cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark supportVectorizedUDF
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20171.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20171
----
commit 5e0c8e1f08b4bb0716b8f21a8393acea361981dc
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-06T08:06:36Z
fix
commit f41e74e9ba87900dbd66cec12fbb579755d4290b
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-06T08:20:41Z
Merge remote-tracking branch 'upstream/master' into supportVectorizedUDF
commit 3983bcbbd7a1279b899e72d07822afa5d4ef7749
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-06T08:36:37Z
rename
commit fe8dcbe47c7800e771420b416f6e5352c745e85c
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-06T09:09:44Z
fix
commit 3411dccc6ab57f3135a99d2b2b535fd5c135cbc7
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-06T10:00:05Z
import
commit b801e7094c7c730adb53a25e13363973930a0b42
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-06T11:41:33Z
fix
commit 3c08f3d6b7ec58735260de687bb74b104e6f7009
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-07T00:41:22Z
fix
commit 423c832b89f8bef5f4812d24c67921644f169f15
Author: gatorsmile <gatorsmile@...>
Date: 2018-01-13T16:34:39Z
fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
