Hi, I am able to reproduce this error only when using spark 1.6.0 and hive 1.6.0. The hive-site.xml is in the classpath but somehow spark rejects the classpath search for hive-site.xml and start using the default metastore Derby.
16/03/09 10:37:52 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 16/03/09 10:37:52 INFO ObjectStore: Initialized ObjectStore 16/03/09 10:37:52 WARN Hive: Failed to access metastore. This class should not accessed in runtime. org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236) at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174) at org.apache.hadoop.hive.ql.metadata.Hive.<clinit>(Hive.java:166) at org.apache.hadoop.hive.ql.session.SessionState.start( SessionState.java:503) at org.apache.spark.sql.hive.client.ClientWrapper.<init>( ClientWrapper.scala:194) On Wed, Mar 9, 2016 at 9:00 AM, Dave Maughan <davidamaug...@gmail.com> wrote: > Hi, > > We're having a similar issue. We have a standalone cluster running 1.5.2 > with Hive working fine having dropped hive-site.xml into the conf folder. > We've just updated to 1.6.0, using the same configuration. Now when > starting a spark-shell we get the following: > > java.lang.RuntimeException: java.lang.RuntimeException: Unable to > instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreCli > ent > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) > at > org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194) > at > org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238) > at > org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218) > at > org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208) > at > org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:462) > at > org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:461) > at > org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40) > at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:330) > at > org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90) > at > org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > at > org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028) > at $iwC$$iwC.<init>(<console>:15) > at $iwC.<init>(<console>:24) > at <init>(<console>:26) > > On stepping though the code and enabling debug it shows that > hive.metastore.uris is not set: > > DEBUG ClientWrapper: Hive Config: hive.metastore.uris= > > ..So it looks like it's not finding hive-site.xml? Weirdly, if I remove > hive-site.xml the exception does not occur which implies that it WAS on the > classpath... > > Dave > > > > On Tue, 9 Feb 2016 at 22:26 Koert Kuipers <ko...@tresata.com> wrote: > >> i do not have phoenix, but i wonder if its something related. will check >> my classpaths >> >> On Tue, Feb 9, 2016 at 5:00 PM, Benjamin Kim <bbuil...@gmail.com> wrote: >> >>> I got the same problem when I added the Phoenix plugin jar in the driver >>> and executor extra classpaths. Do you have those set too? >>> >> >>> On Feb 9, 2016, at 1:12 PM, Koert Kuipers <ko...@tresata.com> wrote: >>> >>> yes its not using derby i think: i can see the tables in my actual hive >>> metastore. >>> >>> i was using a symlink to /etc/hive/conf/hive-site.xml for my >>> hive-site.xml which has a lot more stuff than just hive.metastore.uris >>> >>> let me try your approach >>> >>> >>> >>> On Tue, Feb 9, 2016 at 3:57 PM, Alexandr Dzhagriev <dzh...@gmail.com> >>> wrote: >>> >>>> I'm using spark 1.6.0, hive 1.2.1 and there is just one property in the >>>> hive-site.xml hive.metastore.uris Works for me. Can you check in the >>>> logs, that when the HiveContext is created it connects to the correct uri >>>> and doesn't use derby. >>>> >>>> Cheers, Alex. >>>> >>>> On Tue, Feb 9, 2016 at 9:39 PM, Koert Kuipers <ko...@tresata.com> >>>> wrote: >>>> >>>>> hey thanks. hive-site is on classpath in conf directory >>>>> >>>>> i currently got it to work by changing this hive setting in >>>>> hive-site.xml: >>>>> hive.metastore.schema.verification=true >>>>> to >>>>> hive.metastore.schema.verification=false >>>>> >>>>> this feels like a hack, because schema verification is a good thing i >>>>> would assume? >>>>> >>>>> On Tue, Feb 9, 2016 at 3:25 PM, Alexandr Dzhagriev <dzh...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Koert, >>>>>> >>>>>> As far as I can see you are using derby: >>>>>> >>>>>> Using direct SQL, underlying DB is DERBY >>>>>> >>>>>> not mysql, which is used for the metastore. That means, spark >>>>>> couldn't find hive-site.xml on your classpath. Can you check that, >>>>>> please? >>>>>> >>>>>> Thanks, Alex. >>>>>> >>>>>> On Tue, Feb 9, 2016 at 8:58 PM, Koert Kuipers <ko...@tresata.com> >>>>>> wrote: >>>>>> >>>>>>> has anyone successfully connected to hive metastore using spark >>>>>>> 1.6.0? i am having no luck. worked fine with spark 1.5.1 for me. i am on >>>>>>> cdh 5.5 and launching spark with yarn. >>>>>>> >>>>>>> this is what i see in logs: >>>>>>> 16/02/09 14:49:12 INFO hive.metastore: Trying to connect to >>>>>>> metastore with URI thrift://metastore.mycompany.com:9083 >>>>>>> 16/02/09 14:49:12 INFO hive.metastore: Connected to metastore. >>>>>>> >>>>>>> and then a little later: >>>>>>> >>>>>>> 16/02/09 14:49:34 INFO hive.HiveContext: Initializing execution >>>>>>> hive, version 1.2.1 >>>>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Inspected Hadoop >>>>>>> version: 2.6.0-cdh5.4.4 >>>>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Loaded >>>>>>> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version >>>>>>> 2.6.0-cdh5.4.4 >>>>>>> 16/02/09 14:49:34 WARN conf.HiveConf: HiveConf of name >>>>>>> hive.server2.enable.impersonation does not exist >>>>>>> 16/02/09 14:49:35 INFO metastore.HiveMetaStore: 0: Opening raw store >>>>>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >>>>>>> 16/02/09 14:49:35 INFO metastore.ObjectStore: ObjectStore, >>>>>>> initialize called >>>>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property >>>>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored >>>>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property >>>>>>> datanucleus.cache.level2 unknown - will be ignored >>>>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified but >>>>>>> not present in CLASSPATH (or one of dependencies) >>>>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified but >>>>>>> not present in CLASSPATH (or one of dependencies) >>>>>>> 16/02/09 14:49:37 WARN conf.HiveConf: HiveConf of name >>>>>>> hive.server2.enable.impersonation does not exist >>>>>>> 16/02/09 14:49:37 INFO metastore.ObjectStore: Setting MetaStore >>>>>>> object pin classes with >>>>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >>>>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class >>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class >>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class >>>>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class >>>>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>>>> "embedded-only" so does not have its own datastore table. >>>>>>> 16/02/09 14:49:40 INFO metastore.MetaStoreDirectSql: Using direct >>>>>>> SQL, underlying DB is DERBY >>>>>>> 16/02/09 14:49:40 INFO metastore.ObjectStore: Initialized ObjectStore >>>>>>> java.lang.RuntimeException: java.lang.RuntimeException: Unable to >>>>>>> instantiate >>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient >>>>>>> at >>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) >>>>>>> at >>>>>>> org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194) >>>>>>> at >>>>>>> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238) >>>>>>> at >>>>>>> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218) >>>>>>> at >>>>>>> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208) >>>>>>> at >>>>>>> org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:440) >>>>>>> at >>>>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272) >>>>>>> at >>>>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:271) >>>>>>> at scala.collection.Iterator$class.foreach(Iterator.scala:742) >>>>>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) >>>>>>> at >>>>>>> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) >>>>>>> at scala.collection.AbstractIterable.foreach(Iterable.scala:54) >>>>>>> at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:271) >>>>>>> at >>>>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:97) >>>>>>> at >>>>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101) >>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>>>> Method) >>>>>>> at >>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>>>> at >>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>>>> at org.apache.spark.repl.Main$.createSQLContext(Main.scala:89) >>>>>>> ... 47 elided >>>>>>> Caused by: java.lang.RuntimeException: Unable to instantiate >>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) >>>>>>> at >>>>>>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) >>>>>>> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) >>>>>>> at >>>>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) >>>>>>> ... 66 more >>>>>>> Caused by: java.lang.reflect.InvocationTargetException: >>>>>>> org.apache.hadoop.hive.metastore.api.MetaException: Version information >>>>>>> not >>>>>>> found in metastore. >>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>>>> Method) >>>>>>> at >>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>>>> at >>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) >>>>>>> ... 72 more >>>>>>> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: >>>>>>> Version information not found in metastore. >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6664) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>> at >>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>>>> at >>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) >>>>>>> at com.sun.proxy.$Proxy26.verifySchema(Unknown Source) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762) >>>>>>> at >>>>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199) >>>>>>> at >>>>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) >>>>>>> ... 77 more >>>>>>> >>>>>>> i seem to get this exact same error no matter what i set >>>>>>> spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars to. i >>>>>>> tried building different spark 1.6.0 jars (with hadoop provided, with >>>>>>> hadoop included), but no effect. >>>>>>> >>>>>>> any ideas? >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>>
