finally got it work out, mimicked how spark added datanucleus jars in compute-classpath.sh, and added the db2jcc*.jar in the classpath, it works now.
Thanks! On Tue, Jun 17, 2014 at 10:50 AM, Jenny Zhao <linlin200...@gmail.com> wrote: > Thanks Michael! > > as I run it using spark-shell, so I added both jars through > bin/spark-shell --jars options. I noticed if I don't pass these jars, it > complains it couldn't find the driver, if I pass them through --jars > options, it complains there is no suitable driver. > > Regards. > > > On Tue, Jun 17, 2014 at 2:43 AM, Michael Armbrust <mich...@databricks.com> > wrote: > >> First a clarification: Spark SQL does not talk to HiveServer2, as that >> JDBC interface is for retrieving results from queries that are executed >> using Hive. Instead Spark SQL will execute queries itself by directly >> accessing your data using Spark. >> >> Spark SQL's Hive module can use JDBC to connect to an external metastore, >> in your case DB2. This is only used to retrieve the metadata (i.e., column >> names and types, HDFS locations for data) >> >> Looking at your exception I still see "java.sql.SQLException: No >> suitable driver", so my guess would be that the DB2 JDBC drivers are not >> being correctly included. How are you trying to add them to the classpath? >> >> Michael >> >> >> On Tue, Jun 17, 2014 at 1:29 AM, Jenny Zhao <linlin200...@gmail.com> >> wrote: >> >>> >>> Hi, >>> >>> my hive configuration use db2 as it's metastore database, I have built >>> spark with the extra step sbt/sbt assembly/assembly to include the >>> dependency jars. and copied HIVE_HOME/conf/hive-site.xml under spark/conf. >>> when I ran : >>> >>> hql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") >>> >>> got following exception, pasted portion of the stack trace here, looking >>> at the stack, this made me wondering if Spark supports remote metastore >>> configuration, it seems spark doesn't talk to hiveserver2 directly? the >>> driver jars: db2jcc-10.5.jar, db2jcc_license_cisuz-10.5.jar both are >>> included in the classpath, otherwise, it will complain it couldn't find the >>> driver. >>> >>> Appreciate any help to resolve it. >>> >>> Thanks! >>> >>> Caused by: java.sql.SQLException: Unable to open a test connection to >>> the given database. JDBC url = jdbc:db2://localhost:50001/BIDB, username = >>> catalog. Terminating connection pool. Original Exception: ------ >>> java.sql.SQLException: No suitable driver >>> at java.sql.DriverManager.getConnection(DriverManager.java:422) >>> at java.sql.DriverManager.getConnection(DriverManager.java:374) >>> at >>> com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:254) >>> at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:305) >>> at >>> com.jolbox.bonecp.BoneCPDataSource.maybeInit(BoneCPDataSource.java:150) >>> at >>> com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:112) >>> at >>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:479) >>> at >>> org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:304) >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39) >>> at >>> java.lang.reflect.Constructor.newInstance(Constructor.java:527) >>> at >>> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631) >>> at >>> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301) >>> at >>> org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1069) >>> at >>> org.datanucleus.NucleusContext.initialise(NucleusContext.java:359) >>> at >>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:768) >>> at >>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326) >>> at >>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) >>> at java.lang.reflect.Method.invoke(Method.java:611) >>> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965) >>> at >>> java.security.AccessController.doPrivileged(AccessController.java:277) >>> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960) >>> at >>> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166) >>> at >>> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808) >>> at >>> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701) >>> at >>> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:275) >>> at >>> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:304) >>> at >>> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:234) >>> at >>> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:209) >>> at >>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) >>> at >>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) >>> at >>> org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:64) >>> at >>> org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286) >>> at >>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54) >>> at >>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060) >>> at >>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121) >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39) >>> at >>> java.lang.reflect.Constructor.newInstance(Constructor.java:527) >>> at >>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210) >>> at >>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62) >>> at >>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) >>> at >>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372) >>> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383) >>> at >>> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139) >>> at >>> org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:542) >>> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:493) >>> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342) >>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977) >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888) >>> at >>> org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:186) >>> at >>> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:160) >>> at >>> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:250) >>> at >>> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:247) >>> at >>> org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:85) >>> at >>> org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:90) >>> >>> >> >