Almaz Murzabekov created SPARK-37539:
----------------------------------------
Summary: Create Spark interface for creating UDFs
Key: SPARK-37539
URL: https://issues.apache.org/jira/browse/SPARK-37539
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.2.0
Reporter: Almaz Murzabekov
Currently, based on [spark
documentation|https://spark.apache.org/docs/3.2.0/sql-ref-syntax-ddl-create-function.html],
if you need to create a custom UDF with specific logic implemented in (let's
say) Scala, you have to extend your class from:
# UDF or UDAF from org.apache.hadoop.hive.ql.exec package
# AbstractGenericUDAFResolver, GenericUDF, or GenericUDTF from
org.apache.hadoop.hive.ql.exec package
# UserDefinedAggregateFunction in org.apache.spark.sql.expressions (which is
deprecated)
The third option is deprecated, and (I'm not sure) this option is used only for
agg user functions. So, for plain UDF we have to follow these first two
options. That means we MUST add a dependency for the hive-exec. Maybe, we can
bring spark native interface for creating UDFs.
WDYT guys?
Thanks in advance
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
