[
https://issues.apache.org/jira/browse/SPARK-4867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560434#comment-14560434
]
Reynold Xin commented on SPARK-4867:
------------------------------------
That's a good idea. I created SPARK-4867 for that. Can you submit a pull
request for it?
It'd also be good to look into what expressions cannot be constructed this way.
Ideally all functions should just go through function registry without being
hardcoded into the parsers.
> UDF clean up
> ------------
>
> Key: SPARK-4867
> URL: https://issues.apache.org/jira/browse/SPARK-4867
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Michael Armbrust
> Priority: Blocker
>
> Right now our support and internal implementation of many functions has a few
> issues. Specifically:
> - UDFS don't know their input types and thus don't do type coercion.
> - We hard code a bunch of built in functions into the parser. This is bad
> because in SQL it creates new reserved words for things that aren't actually
> keywords. Also it means that for each function we need to add support to
> both SQLContext and HiveContext separately.
> For this JIRA I propose we do the following:
> - Change the interfaces for registerFunction and ScalaUdf to include types
> for the input arguments as well as the output type.
> - Add a rule to analysis that does type coercion for UDFs.
> - Add a parse rule for functions to SQLParser.
> - Rewrite all the UDFs that are currently hacked into the various parsers
> using this new functionality.
> Depending on how big this refactoring becomes we could split parts 1&2 from
> part 3 above.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
