Ok, just for posterity, I seemed to have gotten this working now. It seems
that specifying it in jpox.properties does not work. That is strange since
when I run it with verbose output, it says that it found jpox.properties but
I guess it doesn't use the values set there. hive-default.xml will work, but
since I like to keep the defaults there, putting it in hive-site.xml will
also work. Below is the exact hive-site.xml I used to get this to work for
branch-0.2/hadoop 18.3. Again, this is probably just an oversight, but
thought it wouldn't hurt to post in case it helps anyone.
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive_metastore?createDatabaseIfNotExist=true</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>Driver class name for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive_user</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive_pass</value>
<description>JDBC connect string for a JDBC metastore</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>this is local store</description>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
<description>default location for Hive tables</description>
</property>
</configuration>
On Tue, Feb 24, 2009 at 2:42 AM, Ryan Shih <[email protected]> wrote:
> Nope, all the jpox stuff is specified in jpox.properties -- all basically
> out of the box stuff. I had some issues getting the auxilliary jar path to
> work when putting it in hive-site and even directly in hive-default (I had
> to set the export explicitly into the hive-config) so there could be some
> problems there in loading in these things. Still, some ppl seem to have
> gotten this working, or at least gotten further, so it's likely I'm just
> overlooking something.
>
> I attempted the metastore/build.xml but that didn't work either. In the end
> though, I'll need this to be possible from the install directory. I'm using
> Hadoop 18.3 with the 0.2 branch build if that is helpful.
>
>
> Prasad Chakka wrote:
>
> Are there any jpox related properties in hive-default.xml? In DEBUG mode,
> all the parameters that get passed to JPOX get logged. Could you send me
> those?
>
> As I said, alternatively you can try to create the schema using ant target
> in metastore/build.xml.
>
> Prasad
>
> ------------------------------
> *From: *Ryan Shih <[email protected]>
> *Reply-To: *<[email protected]>
> *Date: *Mon, 23 Feb 2009 18:25:19 -0800
> *To: *<[email protected]>
> *Subject: *Re: mysql metastore problems
>
> Thanks Prasad. That is the website I got my parameters from, specifically
> for the local metastore. Still don't seem to have any luck at the moment
> even setting the conf params and doing another build install. I'll probably
> futz around a little more and then start looking into the code base. If
> anyone's been sucessful at getting this going, would love to hear how you
> made that happen.
>
> Ryan
>
> On Mon, Feb 23, 2009 at 5:37 PM, Prasad Chakka <[email protected]>
> wrote:
>
> Can you use the properties from here?
> http://wiki.apache.org/hadoop/Hive/AdminManual/MetastoreAdmin
> Default JPOX options with only changes from above should work since unit
> tests do create the database and schema.
>
> Also there is an ant target to create schema in metastore/build.xml that
> you can try to create the schema.
>
> Prasad
>
>
> *From: *Ryan Shih <[email protected] <http://[email protected]> >
> *Reply-To: *<[email protected] <
> http://[email protected]> >
> *Date: *Mon, 23 Feb 2009 17:31:22 -0800
> *To: *<[email protected] <http://[email protected]> >
> *Subject: *Fwd: mysql metastore problems
>
>
>
>
> ---------- Forwarded message ----------
> From: *Ryan Shih* <[email protected] <http://[email protected]> >
> Date: Mon, Feb 23, 2009 at 5:29 PM
> Subject: mysql metastore problems
> To: [email protected] <http://[email protected]>
>
>
> Hi, I'm having some problems setting up the metastore using mysql. I've
> browsed the message archives, but don't see anything that helps. My
> configuration files, look like:
> **<<hive-site.xml>>**
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <configuration>
> <property>
> <name>hive.metastore.local</name>
> <value>true</value>
> <description>this is local store</description>
> </property>
> <property>
> <name>hive.metastore.warehouse.dir</name>
> <value>/user/hive/warehouse</value>
> <description>default location for Hive tables</description>
> </property>
> <property>
> <name>hive.aux.jars.path</name>
> <value>/home/ryan/hive/branch-0.2/install/custom/</value>
> <description>where custom serdes live </description>
> </property>
> </configuration>
>
>
> **<<jpox.properties>>**
>
> javax.jdo.PersistenceManagerFactoryClass=org.jpox.PersistenceManagerFactoryImpl
> org.jpox.validateTables=false
> org.jpox.validateColumns=false
> org.jpox.validateConstraints=false
> org.jpox.storeManagerType=rdbms
> org.jpox.autoCreateSchema=true
> org.jpox.autoStartMechanismMode=checked
> org.jpox.transactionIsolation=read_committed
> javax.jdo.option.DetachAllOnCommit=true
> javax.jdo.option.NontransactionalRead=true
> javax.jdo.option.ConnectionDriverName=com.mysql.jdbc.Driver
>
> javax.jdo.option.ConnectionURL=jdbc:mysql://localhost/hive_metastore?createDatabaseIfNotExist=true
> javax.jdo.option.ConnectionUserName=hive_user
> javax.jdo.option.ConnectionPassword=hive_pass
> org.jpox.cache.level2=true
> org.jpox.cache.level2.type=SOFT
>
> And then with just the default hive-default.xml file. I haven't worked with
> JPOX tables before, but I'm under the impression that this will
> automatically get created from the autoCreateSchema flag. Has anyone had any
> luck with this?
>
> I'm getting the following error:
> hive> r...@dali:~/hive/branch-0.2/install/bin$ hive
> Hive history file=/tmp/ryan/hive_job_log_ryan_200902231722_1088538316.txt
> hive> create table test_table (id INT, name STRING);
> FAILED: Error in metadata: javax.jdo.JDODataStoreException: Error adding
> class org.apache.hadoop.hive.metastore.model.MDatabase to list of
> persistence-managed classes : Table/View 'JPOX_TABLES' does not exist.
> java.sql.SQLSyntaxErrorException: Table/View 'JPOX_TABLES' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(Unknown
> Source)
> at org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown
> Source)
> at
> org.jpox.store.rdbms.SQLController.getStatementForQuery(SQLController.java:324)
> at
> org.jpox.store.rdbms.SQLController.getStatementForQuery(SQLController.java:263)
> at
> org.jpox.store.rdbms.table.SchemaTable.hasClass(SchemaTable.java:280)
> at
> org.jpox.store.rdbms.table.SchemaTable.addClass(SchemaTable.java:222)
> at
> org.jpox.store.rdbms.SchemaAutoStarter.addClass(SchemaAutoStarter.java:255)
> at
> org.jpox.store.AbstractStoreManager.registerStoreData(AbstractStoreManager.java:363)
> at org.jpox.store.rdbms.RDBMSManager.access$3000(RDBMSManager.java:171)
> at
> org.jpox.store.rdbms.RDBMSManager$ClassAdder.addClassTable(RDBMSManager.java:3001)
> at
> org.jpox.store.rdbms.RDBMSManager$ClassAdder.addClassTables(RDBMSManager.java:2804)
> at
> org.jpox.store.rdbms.RDBMSManager$ClassAdder.addClassTablesAndValidate(RDBMSManager.java:3098)
> at
> org.jpox.store.rdbms.RDBMSManager$ClassAdder.run(RDBMSManager.java:2729)
> at
> org.jpox.store.rdbms.RDBMSManager$MgmtTransaction.execute(RDBMSManager.java:2609)
> at org.jpox.store.rdbms.RDBMSManager.addClasses(RDBMSManager.java:825)
> at
> org.jpox.store.AbstractStoreManager.addClass(AbstractStoreManager.java:624)
> at
> org.jpox.store.mapped.MappedStoreManager.getDatastoreClass(MappedStoreManager.java:343)
> at
> org.jpox.store.rdbms.RDBMSManager.getPropertiesForGenerator(RDBMSManager.java:1630)
> at
> org.jpox.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:945)
> at org.jpox.ObjectManagerImpl.newObjectId(ObjectManagerImpl.java:2473)
> at
> org.jpox.state.JDOStateManagerImpl.setIdentity(JDOStateManagerImpl.java:792)
> at
> org.jpox.state.JDOStateManagerImpl.initialiseForPersistentNew(JDOStateManagerImpl.java:467)
> at
> org.jpox.state.StateManagerFactory.newStateManagerForPersistentNew(StateManagerFactory.java:151)
> at
> org.jpox.ObjectManagerImpl.persistObjectInternal(ObjectManagerImpl.java:1230)
> at
> org.jpox.ObjectManagerImpl.persistObject(ObjectManagerImpl.java:1077)
> at
> org.jpox.jdo.AbstractPersistenceManager.jdoMakePersistent(AbstractPersistenceManager.java:666)
> at
> org.jpox.jdo.AbstractPersistenceManager.makePersistent(AbstractPersistenceManager.java:691)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.createDatabase(ObjectStore.java:272)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:145)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:115)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:100)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:73)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:785)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:798)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:254)
> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:740)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:97)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:238)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:174)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:306)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
> Caused by: java.sql.SQLException: Table/View 'JPOX_TABLES' does not exist.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
> ... 61 more
> Caused by: ERROR 42X05: Table/View 'JPOX_TABLES' does not exist.
> at org.apache.derby.iapi.error.StandardException.newException(Unknown
> Source)
> at
> org.apache.derby.impl.sql.compile.FromBaseTable.bindTableDescriptor(Unknown
> Source)
> at
> org.apache.derby.impl.sql.compile.FromBaseTable.bindNonVTITables(Unknown
> Source)
> at org.apache.derby.impl.sql.compile.FromList.bindTables(Unknown
> Source)
> at
> org.apache.derby.impl.sql.compile.SelectNode.bindNonVTITables(Unknown
> Source)
> at
> org.apache.derby.impl.sql.compile.DMLStatementNode.bindTables(Unknown
> Source)
> at org.apache.derby.impl.sql.compile.DMLStatementNode.bind(Unknown
> Source)
> at org.apache.derby.impl.sql.compile.CursorNode.bindStatement(Unknown
> Source)
> at org.apache.derby.impl.sql.GenericStatement.prepMinion(Unknown
> Source)
> at org.apache.derby.impl.sql.GenericStatement.prepare(Unknown Source)
> at
> org.apache.derby.impl.sql.conn.GenericLanguageConnectionContext.prepareInternalStatement(Unknown
> Source)
> ... 55 more
>
> NestedThrowables:
> java.sql.SQLSyntaxErrorException: Table/View 'JPOX_TABLES' does not exist.
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> Time taken: 1.883 seconds
>
>
>
>
>
>
>
>
>
>
>
