Hello,
I am using SparkSQL along with ThriftServer so that we can access using Hive
queries.
With Spark 1.3.1, I can register UDF function. But, Spark 1.4.0 doesn't work
for that. The jar of the udf is same.
Below is logs:
I appreciate any advice.
== With Spark 1.4
Beeline version 1.4.0 by Apache Hive
0: jdbc:hive2://localhost:10000> add jar
hdfs:///user/hive/lib/dw-udf-2015.06.06-SNAPSHOT.jar;
0: jdbc:hive2://localhost:10000> create temporary function parse_trace as
'com. mycom.dataengine.udf.GenericUDFParseTraceAnnotation';
15/07/14 23:49:43 DEBUG transport.TSaslTransport: writing data length: 206
15/07/14 23:49:43 DEBUG transport.TSaslTransport: CLIENT: reading data
length: 201
Error: org.apache.spark.sql.execution.QueryExecutionException: FAILED:
Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.FunctionTask (state=,code=0)
== With Spark 1.3.1:
Beeline version 1.3.1 by Apache Hive
0: jdbc:hive2://localhost:10001> add jar
hdfs:///user/hive/lib/dw-udf-2015.06.06-SNAPSHOT.jar;
+---------+
| Result |
+---------+
+---------+
No rows selected (1.313 seconds)
0: jdbc:hive2://localhost:10001> create temporary function parse_trace as
'com. mycom.dataengine.udf.GenericUDFParseTraceAnnotation';
+---------+
| result |
+---------+
+---------+
No rows selected (0.999 seconds)
=== The logs of ThriftServer of Spark 1.4.0
15/07/14 23:49:43 INFO SparkExecuteStatementOperation: Running query 'create
temporary function parse_trace as
'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation''
15/07/14 23:49:43 INFO ParseDriver: Parsing command: create temporary
function parse_trace as
'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation'
15/07/14 23:49:43 INFO ParseDriver: Parse Completed
15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=Driver.run
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=TimeToSubmit
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO Driver: Concurrency mode is disabled, not creating a
lock manager
15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=compile
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=parse
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO ParseDriver: Parsing command: create temporary
function parse_trace as
'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation'
15/07/14 23:49:43 INFO ParseDriver: Parse Completed
15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=parse
start=1436917783106 end=1436917783106 duration=0
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=semanticAnalyze
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO HiveMetaStore: 2: get_database: default
15/07/14 23:49:43 INFO audit: ugi=anonymous ip=unknown-ip-addr
cmd=get_database: default
15/07/14 23:49:43 INFO HiveMetaStore: 2: Opening raw store with
implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
15/07/14 23:49:43 INFO ObjectStore: ObjectStore, initialize called
15/07/14 23:49:43 INFO MetaStoreDirectSql: MySQL check failed, assuming we
are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64),
after : "".
15/07/14 23:49:43 INFO Query: Reading in results for query
"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is
closing
15/07/14 23:49:43 INFO ObjectStore: Initialized ObjectStore
15/07/14 23:49:43 INFO FunctionSemanticAnalyzer: analyze done
15/07/14 23:49:43 INFO Driver: Semantic Analysis Completed
15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=semanticAnalyze
start=1436917783106 end=1436917783114 duration=8
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO Driver: Returning Hive schema:
Schema(fieldSchemas:null, properties:null)
15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=compile
start=1436917783106 end=1436917783114 duration=8
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=Driver.execute
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO Driver: Starting command: create temporary function
parse_trace as 'com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation'
15/07/14 23:49:43 INFO PerfLogger: </PERFLOG method=TimeToSubmit
start=1436917783105 end=1436917783115 duration=10
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=runTasks
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 INFO PerfLogger: <PERFLOG method=task.FUNCTION.Stage-0
from=org.apache.hadoop.hive.ql.Driver>
15/07/14 23:49:43 ERROR Task: FAILED: Class
com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation not found
15/07/14 23:49:43 INFO FunctionTask: create function:
java.lang.ClassNotFoundException:
com.quixey.dataengine.udf.GenericUDFParseTraceAnnotation
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/SparkSQL-1-4-can-t-accept-registration-of-UDF-tp23840.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
