[ https://issues.apache.org/jira/browse/SPARK-16275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400854#comment-15400854 ]
Xiao Li commented on SPARK-16275: --------------------------------- https://github.com/apache/hive/blob/15bdce43db4624a63be1f648e46d1f2baa1c67de/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java#L638-L748 This is the hash function of Hive. The implementation sounds ok, but I might need to check it with [~cloud_fan]. Not all the data types (e.g. Union) are supported. It is highly related to the data types. I am not exactly sure whether we have the same value ranges for each data type. To make sure they always generate the same result. The test cases might be a lot. > Implement all the Hive fallback functions > ----------------------------------------- > > Key: SPARK-16275 > URL: https://issues.apache.org/jira/browse/SPARK-16275 > Project: Spark > Issue Type: New Feature > Components: SQL > Reporter: Reynold Xin > > As of Spark 2.0, Spark falls back to Hive for only the following built-in > functions: > {code} > "elt", "hash", "java_method", "histogram_numeric", > "map_keys", "map_values", > "parse_url", "percentile", "percentile_approx", "reflect", "sentences", > "stack", "str_to_map", > "xpath", "xpath_boolean", "xpath_double", "xpath_float", "xpath_int", > "xpath_long", > "xpath_number", "xpath_short", "xpath_string", > // table generating function > "inline", "posexplode" > {code} > The goal of the ticket is to implement all of these in Spark so we don't need > to fall back into Hive's UDFs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org