Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/16533
@rxin
We can do something like this:
```python
class UserDefinedFunction(object):
"""
User defined function in Python
.. versionadded:: 1.3
"""
def __init__(self, func, returnType, name=None):
self.func = func
self.returnType = returnType
self._judf = self._create_judf(name)
def _jdt(self, spark):
if isinstance(self.returnType, DataType):
return
spark._jsparksession.parsedatatype(self.returntype.json())
else:
jvm = spark.sparkContext._jvm
return
jvm.org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parseDataType(self.returnType)
def _create_judf(self, name):
from pyspark.sql import SparkSession
sc = SparkContext.getOrCreate()
wrapped_func = _wrap_function(sc, self.func, self.returnType)
spark = SparkSession.builder.getOrCreate()
jdt = self._jdt(spark)
if name is None:
f = self.func
name = f.__name__ if hasattr(f, '__name__') else
f.__class__.__name__
judf =
sc._jvm.org.apache.spark.sql.execution.python.UserDefinedPythonFunction(
name, wrapped_func, jdt)
return judf
def __call__(self, *cols):
sc = SparkContext._active_spark_context
jc = self._judf.apply(_to_seq(sc, cols, _to_java_column))
return Column(jc)
```
and then easily support:
```python
@udf("integer")
def f(x):
return x
```
but I am not sure if this is the right place to handle this. For example it
will raise some questions what should be the type of
`UserDefinedFunction.returnType`. `DataType` or the thing we passed (`DataType`
or `str`)? I am happy to work on this but I would prefer to add it as a
separate task to
[SPARK-19159](https://issues.apache.org/jira/browse/SPARK-19159). What do you
think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
