[
https://issues.apache.org/jira/browse/SPARK-7851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567997#comment-14567997
]
Cheolsoo Park commented on SPARK-7851:
--------------------------------------
[~yhuai], thanks for checking.
1) I am setting properties in spark-defaults.conf.
2) I rebuilt my jars including all the commits upto RC3 and still see the error.
{code}
SET spark.sql.hive.convertMetastoreParquet=false
15/06/01 20:58:07 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
15/06/01 20:58:07 INFO metastore: Trying to connect to metastore with URI
thrift://admin.gateway.dataeng.netflix.net:11002
15/06/01 20:58:07 INFO metastore: Connected to metastore.
SET spark.sql.hive.metastore.version=0.12.0
SET spark.sql.hive.env=prod
SET spark.sql.hive.verifyPartitionPath=false
SET spark.sql.hive.metastore.jars=/home/cheolsoop/hive-0.12.0-bin/lib/*
SET spark.sql.franklin.connection.timeout=1800000
SET spark.sql.hive.version=0.13.1
SET spark.sql.hive.version=0.13.1
15/06/01 20:58:08 WARN RetryingMetaStoreClient: MetaStoreClient lost
connection. Attempting to reconnect.
org.apache.thrift.TApplicationException: Invalid method name: 'get_functions'
at
org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_functions(ThriftHiveMetastore.java:2886)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_functions(ThriftHiveMetastore.java:2872)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunctions(HiveMetaStoreClient.java:1727)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
at com.sun.proxy.$Proxy12.getFunctions(Unknown Source)
at org.apache.hadoop.hive.ql.metadata.Hive.getFunctions(Hive.java:2670)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionNames(FunctionRegistry.java:674)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionNames(FunctionRegistry.java:662)
at
org.apache.hadoop.hive.cli.CliDriver.getCommandCompletor(CliDriver.java:540)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:175)
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:664)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:169)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:111)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
15/06/01 20:58:09 INFO metastore: Trying to connect to metastore with URI
thrift://admin.gateway.dataeng.netflix.net:11002
15/06/01 20:58:09 INFO metastore: Connected to metastore.
15/06/01 20:58:09 ERROR FunctionRegistry:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.thrift.TApplicationException: Invalid method name: 'get_functions'
spark-sql>
{code}
Btw, I should clarify that I see this error only with {{spark-sql}}, and I can
execute queries in the same shell despite the error at start-up.
3) {{set javax.jdo.option.ConnectionURL}} shows a temp dir as follows-
{code}
spark-sql> set javax.jdo.option.ConnectionURL;
javax.jdo.option.ConnectionURL=jdbc:derby:;databaseName=/home/cheolsoop/spark/tmp/spark-41754d12-8140-4388-963c-6dbaf859ad95/metastore;create=true
{code}
> SparkSQL cli built against Hive 0.13 throws exception when using with Hive
> 0.12 HCat
> ------------------------------------------------------------------------------------
>
> Key: SPARK-7851
> URL: https://issues.apache.org/jira/browse/SPARK-7851
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Cheolsoo Park
> Priority: Minor
>
> I built Spark with {{Hive 0.13}} and set the following properties-
> {code}
> spark.sql.hive.metastore.version=0.12.0
> spark.sql.hive.metastore.jars=path_to_hive_0.12_jars
> {code}
> But when the SparkSQL CLI starts up, I get the following error-
> {code}
> 15/05/24 05:03:29 WARN RetryingMetaStoreClient: MetaStoreClient lost
> connection. Attempting to reconnect.
> org.apache.thrift.TApplicationException: Invalid method name: 'get_functions'
> at
> org.apache.thrift.TApplicationException.read(TApplicationException.java:108)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:71)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_functions(ThriftHiveMetastore.java:2886)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_functions(ThriftHiveMetastore.java:2872)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFunctions(HiveMetaStoreClient.java:1727)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:89)
> at com.sun.proxy.$Proxy12.getFunctions(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.getFunctions(Hive.java:2670)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionNames(FunctionRegistry.java:674)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.getFunctionNames(FunctionRegistry.java:662)
> at
> org.apache.hadoop.hive.cli.CliDriver.getCommandCompletor(CliDriver.java:540)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:175)
> at
> org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
> {code}
> What's happening is that when SparkSQL Cli starts up, it tries to fetch
> permanent udfs from Hive metastore (due to HIVE-6330, which was introduced in
> Hive 0.13). But then, it ends up invoking an incompatible thrift function
> that doesn't exist in Hive 0.12.
> To work around this error, I have to comment out the following line of code-
> https://goo.gl/wcfnH1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
