[
https://issues.apache.org/jira/browse/SPARK-33721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-33721:
------------------------------------
Assignee: Apache Spark
> Support to use Hive build-in functions by configuration
> -------------------------------------------------------
>
> Key: SPARK-33721
> URL: https://issues.apache.org/jira/browse/SPARK-33721
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 2.4.3, 3.2.0
> Reporter: chenliang
> Assignee: Apache Spark
> Priority: Major
>
> Hive and Spark SQL engines have many differences in built-in functions.The
> differences between several functions are shown below：
> ||*build-in functions*||SQL|| result of Hive SQL ||result of Spark SQL||
> |unix_timestamp|{{select}} {{unix_timestamp(concat(}}{{'2020-06-01'}}{{,
> }}{{' 24:00:00'}}{{));}}|1591027200| NULL|
> |to_date|{{select}} {{to_date(}}{{'0000-00-00'}}{{);}}|0002-11-30| NULL|
> |datediff|{{select }}{{datediff(}}{{CURRENT_DATE}}{{,
> }}{{'0000-00-00'}}{{);}}|737986| NULL|
> |collect_set|{{select}}{{c1}}{{,c2}}{{,concat_ws(}}{{'##'}}{{,
> collect_set(c3)) c3_set }}{{from}}{{bigdata_offline.test_collect_set
> }}{{group }}{{by }}{{c1, c2;}}
> {{bigdata_offline.test_collect_set contains data:}}
> {{(1, 1, }}{{'1'}}{{)，}}{{(1, 1, }}{{'2'}}{{)}}{{,}}
> {{(1, 1, }}{{'3'}}{{)}}{{,}}{{(1, 1, }}{{'4'}}{{)}}{{,}}
> {{(1, 1, }}{{'5'}}{{)}}|{{c1 c2 c3_set}}
> {{1 1 2##3##4##5##1}}|{{c1 c2 c3_set}}
> {{1 1 3##1##2##5##4}}|
> There is no conclusion on which engine is more accurate. Users prefer to be
> able to make choices according to their real production environment.
> I think we should do some improvement for this.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
