leishuiyu created SPARK-31191:
---------------------------------
Summary: Spark SQL and hive metastore are incompatible
Key: SPARK-31191
URL: https://issues.apache.org/jira/browse/SPARK-31191
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0
Environment:
the spark version 2.3.0
the hive version 2.3.3
Reporter: leishuiyu
Fix For: 2.3.0
#
h3. When I execute bin/spark-sql, an exception occurs
{code:java}
Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClientCaused by:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) at
org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) ...
12 moreCaused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
... 18 moreCaused by: MetaException(message:Hive Schema version 1.2.0 does not
match metastore's schema version 2.3.0 Metastore is not upgraded or corrupt) at
org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6679)
at
org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
at com.sun.proxy.$Proxy6.verifySchema(Unknown Source) at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66)
at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72)
at
org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
... 23 more
{code}
h3. 2.Find the reason
query the source code, in spark jars directory have
hive-metastore-1.2.1.spark2.jar
the 1.2.1 version match 1.2.0 ,so generate the exception
{code:java}
//代码占位符
private static final Map<String, String> EQUIVALENT_VERSIONS =
ImmutableMap.of("0.13.1", "0.13.0",
"1.0.0", "0.14.0",
"1.0.1", "1.0.0",
"1.1.1", "1.1.0",
"1.2.1", "1.2.0"
);
{code}
h3. 3.Is there any solution to this problem
can edit hive-site.xml hive.metastore.schema.verification set true,but new
problems may arise
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
