Github user ptkool commented on the issue:
https://github.com/apache/spark/pull/18906
@HyukjinKwon @holdenk There is still an issue with the use of
`SparkSession.udf.register` that needs to be resolved.
For instance, the following will not work as expected:
```python
spark.udf.register("func", lambda x: x * 4, IntegerType()).asNonNullable()`
```
As well, the following is not currently possible:
```python
f = udf(lambda x: x * 4, IntegerType()).asNonNullable())
f = spark.udf.register(f)
```
The same applies to the use of `asNondeterministic` in the above examples.
One other thing I noticed with the implementation of `asNondeterministic`
in Python class `UserDefinedFunction` is that it returns an instance of
`UserDefinedFunction`, not a function object.
```text
>>> udf_random_col = udf(lambda: int(100 * random.random()), IntegerType())
>>> udf_random_col
<function <lambda> at 0x10fad28c0>
>>> udf_random_col = udf_random_col.asNondeterministic()
>>> udf_random_col
<pyspark.sql.udf.UserDefinedFunction object at 0x10fac3e50>
>>>
```
Is this correct?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
