i do not have phoenix, but i wonder if its something related. will check my classpaths
On Tue, Feb 9, 2016 at 5:00 PM, Benjamin Kim <bbuil...@gmail.com> wrote: > I got the same problem when I added the Phoenix plugin jar in the driver > and executor extra classpaths. Do you have those set too? > > On Feb 9, 2016, at 1:12 PM, Koert Kuipers <ko...@tresata.com> wrote: > > yes its not using derby i think: i can see the tables in my actual hive > metastore. > > i was using a symlink to /etc/hive/conf/hive-site.xml for my hive-site.xml > which has a lot more stuff than just hive.metastore.uris > > let me try your approach > > > > On Tue, Feb 9, 2016 at 3:57 PM, Alexandr Dzhagriev <dzh...@gmail.com> > wrote: > >> I'm using spark 1.6.0, hive 1.2.1 and there is just one property in the >> hive-site.xml hive.metastore.uris Works for me. Can you check in the >> logs, that when the HiveContext is created it connects to the correct uri >> and doesn't use derby. >> >> Cheers, Alex. >> >> On Tue, Feb 9, 2016 at 9:39 PM, Koert Kuipers <ko...@tresata.com> wrote: >> >>> hey thanks. hive-site is on classpath in conf directory >>> >>> i currently got it to work by changing this hive setting in >>> hive-site.xml: >>> hive.metastore.schema.verification=true >>> to >>> hive.metastore.schema.verification=false >>> >>> this feels like a hack, because schema verification is a good thing i >>> would assume? >>> >>> On Tue, Feb 9, 2016 at 3:25 PM, Alexandr Dzhagriev <dzh...@gmail.com> >>> wrote: >>> >>>> Hi Koert, >>>> >>>> As far as I can see you are using derby: >>>> >>>> Using direct SQL, underlying DB is DERBY >>>> >>>> not mysql, which is used for the metastore. That means, spark couldn't >>>> find hive-site.xml on your classpath. Can you check that, please? >>>> >>>> Thanks, Alex. >>>> >>>> On Tue, Feb 9, 2016 at 8:58 PM, Koert Kuipers <ko...@tresata.com> >>>> wrote: >>>> >>>>> has anyone successfully connected to hive metastore using spark 1.6.0? >>>>> i am having no luck. worked fine with spark 1.5.1 for me. i am on cdh 5.5 >>>>> and launching spark with yarn. >>>>> >>>>> this is what i see in logs: >>>>> 16/02/09 14:49:12 INFO hive.metastore: Trying to connect to metastore >>>>> with URI thrift://metastore.mycompany.com:9083 >>>>> 16/02/09 14:49:12 INFO hive.metastore: Connected to metastore. >>>>> >>>>> and then a little later: >>>>> >>>>> 16/02/09 14:49:34 INFO hive.HiveContext: Initializing execution hive, >>>>> version 1.2.1 >>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Inspected Hadoop version: >>>>> 2.6.0-cdh5.4.4 >>>>> 16/02/09 14:49:34 INFO client.ClientWrapper: Loaded >>>>> org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version >>>>> 2.6.0-cdh5.4.4 >>>>> 16/02/09 14:49:34 WARN conf.HiveConf: HiveConf of name >>>>> hive.server2.enable.impersonation does not exist >>>>> 16/02/09 14:49:35 INFO metastore.HiveMetaStore: 0: Opening raw store >>>>> with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore >>>>> 16/02/09 14:49:35 INFO metastore.ObjectStore: ObjectStore, initialize >>>>> called >>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property >>>>> hive.metastore.integral.jdo.pushdown unknown - will be ignored >>>>> 16/02/09 14:49:35 INFO DataNucleus.Persistence: Property >>>>> datanucleus.cache.level2 unknown - will be ignored >>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified but >>>>> not present in CLASSPATH (or one of dependencies) >>>>> 16/02/09 14:49:35 WARN DataNucleus.Connection: BoneCP specified but >>>>> not present in CLASSPATH (or one of dependencies) >>>>> 16/02/09 14:49:37 WARN conf.HiveConf: HiveConf of name >>>>> hive.server2.enable.impersonation does not exist >>>>> 16/02/09 14:49:37 INFO metastore.ObjectStore: Setting MetaStore object >>>>> pin classes with >>>>> hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" >>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class >>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>> "embedded-only" so does not have its own datastore table. >>>>> 16/02/09 14:49:38 INFO DataNucleus.Datastore: The class >>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>> "embedded-only" so does not have its own datastore table. >>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class >>>>> "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as >>>>> "embedded-only" so does not have its own datastore table. >>>>> 16/02/09 14:49:40 INFO DataNucleus.Datastore: The class >>>>> "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as >>>>> "embedded-only" so does not have its own datastore table. >>>>> 16/02/09 14:49:40 INFO metastore.MetaStoreDirectSql: Using direct SQL, >>>>> underlying DB is DERBY >>>>> 16/02/09 14:49:40 INFO metastore.ObjectStore: Initialized ObjectStore >>>>> java.lang.RuntimeException: java.lang.RuntimeException: Unable to >>>>> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient >>>>> at >>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522) >>>>> at >>>>> org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194) >>>>> at >>>>> org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238) >>>>> at >>>>> org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:218) >>>>> at >>>>> org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:208) >>>>> at >>>>> org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:440) >>>>> at >>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:272) >>>>> at >>>>> org.apache.spark.sql.SQLContext$$anonfun$4.apply(SQLContext.scala:271) >>>>> at scala.collection.Iterator$class.foreach(Iterator.scala:742) >>>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) >>>>> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) >>>>> at scala.collection.AbstractIterable.foreach(Iterable.scala:54) >>>>> at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:271) >>>>> at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:97) >>>>> at >>>>> org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101) >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>> Method) >>>>> at >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>> at >>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>> at org.apache.spark.repl.Main$.createSQLContext(Main.scala:89) >>>>> ... 47 elided >>>>> Caused by: java.lang.RuntimeException: Unable to instantiate >>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient >>>>> at >>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523) >>>>> at >>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86) >>>>> at >>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132) >>>>> at >>>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) >>>>> at >>>>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005) >>>>> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024) >>>>> at >>>>> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503) >>>>> ... 66 more >>>>> Caused by: java.lang.reflect.InvocationTargetException: >>>>> org.apache.hadoop.hive.metastore.api.MetaException: Version information >>>>> not >>>>> found in metastore. >>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>>>> Method) >>>>> at >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>>>> at >>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>>>> at >>>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521) >>>>> ... 72 more >>>>> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Version >>>>> information not found in metastore. >>>>> at >>>>> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6664) >>>>> at >>>>> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6645) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>> at >>>>> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114) >>>>> at com.sun.proxy.$Proxy26.verifySchema(Unknown Source) >>>>> at >>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:572) >>>>> at >>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:620) >>>>> at >>>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:461) >>>>> at >>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:66) >>>>> at >>>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:72) >>>>> at >>>>> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762) >>>>> at >>>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:199) >>>>> at >>>>> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74) >>>>> ... 77 more >>>>> >>>>> i seem to get this exact same error no matter what i set >>>>> spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars to. i >>>>> tried building different spark 1.6.0 jars (with hadoop provided, with >>>>> hadoop included), but no effect. >>>>> >>>>> any ideas? >>>>> >>>>> >>>> >>> >> > >
