[
https://issues.apache.org/jira/browse/SPARK-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Yadav updated SPARK-18832:
---------------------------------
Environment:
HDP: 2.5
Spark: 2.0.0
Description:
Spark Thriftserver is unable to run a HiveUDTF.
It throws the error that it is unable to find the functions although the
function registration succeeds and the funtions does show up in the list output
by {{show functions}}.
I am using a Hive UDTF, registering it using a jar placed on my local machine.
Calling it using the following commands:
//Registering the functions, this command succeeds.
{{CREATE FUNCTION SampleUDTF AS
'com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF' USING JAR
'/root/spark_files/experiments-1.2.jar';}}
//Thriftserver is able to look up the functuion, on this command:
{{DESCRIBE FUNCTION SampleUDTF;}}
{quote}
Output:
+-----------------------------------------------------------+--+
| function_desc |
+-----------------------------------------------------------+--+
| Function: default.SampleUDTF |
| Class: com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF |
| Usage: N/A. |
+-----------------------------------------------------------+--+
{quote}
// Calling the function:
{{SELECT SampleUDTF('Paris');}}
bq. Output of the above command: Error: org.apache.spark.sql.AnalysisException:
Undefined function: 'SampleUDTF'. This function is neither a registered
temporary function nor a permanent function registered in the database
'default'.; line 1 pos 7 (state=,code=0)
I have also tried with using a non-local (on hdfs) jar, but I get the same
error.
My environment: HDP 2.4 with spark 2.00
was:
On calling a registered UDF in metastore from spark-sql CLI, it gives a generic
error:
Error in query: Undefined function: 'Sample_UDF'. This function is neither a
registered temporary function nor a permanent function registered in the
database 'default'.
The functions is registered and it shoes up in the list output by 'show
functions'.
I am using a Hive UDTF, registering it using the statement: create function
Sample_UDF as 'com.udf.Sample_UDF' using JAR
'/local/jar/path/containing/the/class';
and I am calling the functions from spark-sql CLI as: SELECT
Sample_UDF("input_1", "input_2" )
Summary: Spark SQL: thiriftserver unable to run a registered Hive UDTF
(was: Spark SQL: Incorrect error message on calling registered UDF.)
> Spark SQL: thiriftserver unable to run a registered Hive UDTF
> -------------------------------------------------------------
>
> Key: SPARK-18832
> URL: https://issues.apache.org/jira/browse/SPARK-18832
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0, 2.0.1, 2.0.2
> Environment: HDP: 2.5
> Spark: 2.0.0
> Reporter: Lokesh Yadav
>
> Spark Thriftserver is unable to run a HiveUDTF.
> It throws the error that it is unable to find the functions although the
> function registration succeeds and the funtions does show up in the list
> output by {{show functions}}.
> I am using a Hive UDTF, registering it using a jar placed on my local
> machine. Calling it using the following commands:
> //Registering the functions, this command succeeds.
> {{CREATE FUNCTION SampleUDTF AS
> 'com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF' USING JAR
> '/root/spark_files/experiments-1.2.jar';}}
> //Thriftserver is able to look up the functuion, on this command:
> {{DESCRIBE FUNCTION SampleUDTF;}}
> {quote}
> Output:
> +-----------------------------------------------------------+--+
> | function_desc |
> +-----------------------------------------------------------+--+
> | Function: default.SampleUDTF |
> | Class: com.fuzzylogix.experiments.udf.hiveUDF.SampleUDTF |
> | Usage: N/A. |
> +-----------------------------------------------------------+--+
> {quote}
> // Calling the function:
> {{SELECT SampleUDTF('Paris');}}
> bq. Output of the above command: Error:
> org.apache.spark.sql.AnalysisException: Undefined function: 'SampleUDTF'.
> This function is neither a registered temporary function nor a permanent
> function registered in the database 'default'.; line 1 pos 7 (state=,code=0)
> I have also tried with using a non-local (on hdfs) jar, but I get the same
> error.
> My environment: HDP 2.4 with spark 2.00
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
