finally got it work out, mimicked how spark added datanucleus jars in
compute-classpath.sh, and added the db2jcc*.jar in the classpath, it works
now.
Thanks!
On Tue, Jun 17, 2014 at 10:50 AM, Jenny Zhao <linlin200...@gmail.com> wrote:
> Thanks Michael!
>
> as I run it using spark-shell, so I added both jars through
> bin/spark-shell --jars options. I noticed if I don't pass these jars, it
> complains it couldn't find the driver, if I pass them through --jars
> options, it complains there is no suitable driver.
>
> Regards.
>
>
> On Tue, Jun 17, 2014 at 2:43 AM, Michael Armbrust <mich...@databricks.com>
> wrote:
>
>> First a clarification: Spark SQL does not talk to HiveServer2, as that
>> JDBC interface is for retrieving results from queries that are executed
>> using Hive. Instead Spark SQL will execute queries itself by directly
>> accessing your data using Spark.
>>
>> Spark SQL's Hive module can use JDBC to connect to an external metastore,
>> in your case DB2. This is only used to retrieve the metadata (i.e., column
>> names and types, HDFS locations for data)
>>
>> Looking at your exception I still see "java.sql.SQLException: No
>> suitable driver", so my guess would be that the DB2 JDBC drivers are not
>> being correctly included. How are you trying to add them to the classpath?
>>
>> Michael
>>
>>
>> On Tue, Jun 17, 2014 at 1:29 AM, Jenny Zhao <linlin200...@gmail.com>
>> wrote:
>>
>>>
>>> Hi,
>>>
>>> my hive configuration use db2 as it's metastore database, I have built
>>> spark with the extra step sbt/sbt assembly/assembly to include the
>>> dependency jars. and copied HIVE_HOME/conf/hive-site.xml under spark/conf.
>>> when I ran :
>>>
>>> hql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
>>>
>>> got following exception, pasted portion of the stack trace here, looking
>>> at the stack, this made me wondering if Spark supports remote metastore
>>> configuration, it seems spark doesn't talk to hiveserver2 directly? the
>>> driver jars: db2jcc-10.5.jar, db2jcc_license_cisuz-10.5.jar both are
>>> included in the classpath, otherwise, it will complain it couldn't find the
>>> driver.
>>>
>>> Appreciate any help to resolve it.
>>>
>>> Thanks!
>>>
>>> Caused by: java.sql.SQLException: Unable to open a test connection to
>>> the given database. JDBC url = jdbc:db2://localhost:50001/BIDB, username =
>>> catalog. Terminating connection pool. Original Exception: ------
>>> java.sql.SQLException: No suitable driver
>>> at java.sql.DriverManager.getConnection(DriverManager.java:422)
>>> at java.sql.DriverManager.getConnection(DriverManager.java:374)
>>> at
>>> com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:254)
>>> at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:305)
>>> at
>>> com.jolbox.bonecp.BoneCPDataSource.maybeInit(BoneCPDataSource.java:150)
>>> at
>>> com.jolbox.bonecp.BoneCPDataSource.getConnection(BoneCPDataSource.java:112)
>>> at
>>> org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl.getConnection(ConnectionFactoryImpl.java:479)
>>> at
>>> org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:304)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
>>> at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
>>> at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:527)
>>> at
>>> org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:631)
>>> at
>>> org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
>>> at
>>> org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:1069)
>>> at
>>> org.datanucleus.NucleusContext.initialise(NucleusContext.java:359)
>>> at
>>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:768)
>>> at
>>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:326)
>>> at
>>> org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:195)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
>>> at java.lang.reflect.Method.invoke(Method.java:611)
>>> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
>>> at
>>> java.security.AccessController.doPrivileged(AccessController.java:277)
>>> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
>>> at
>>> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
>>> at
>>> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
>>> at
>>> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
>>> at
>>> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:275)
>>> at
>>> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:304)
>>> at
>>> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:234)
>>> at
>>> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:209)
>>> at
>>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
>>> at
>>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
>>> at
>>> org.apache.hadoop.hive.metastore.RetryingRawStore.<init>(RetryingRawStore.java:64)
>>> at
>>> org.apache.hadoop.hive.metastore.RetryingRawStore.getProxy(RetryingRawStore.java:73)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:415)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:402)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:441)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:326)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:286)
>>> at
>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
>>> at
>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4060)
>>> at
>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:121)
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> Method)
>>> at
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:56)
>>> at
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:39)
>>> at
>>> java.lang.reflect.Constructor.newInstance(Constructor.java:527)
>>> at
>>> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1210)
>>> at
>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
>>> at
>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
>>> at
>>> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2372)
>>> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2383)
>>> at
>>> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1139)
>>> at
>>> org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:542)
>>> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:493)
>>> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342)
>>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
>>> at
>>> org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:186)
>>> at
>>> org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:160)
>>> at
>>> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd$lzycompute(HiveContext.scala:250)
>>> at
>>> org.apache.spark.sql.hive.HiveContext$QueryExecution.toRdd(HiveContext.scala:247)
>>> at
>>> org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:85)
>>> at
>>> org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:90)
>>>
>>>
>>
>
