We had a huge outpouring of wiki love in the Docathon last week :)
In the comig weeks we will keep enhancing it to make it really top
class.
Ashish
From: Prasad Chakka
To: [email protected]
Sent: Sat Jan 24 15:11:10 2009
Subject: Re: Still can't get metastore working with mysql
If you had just checked the wiki here
http://wiki.apache.org/hadoop/Hive/AdminManual/MetastoreAdmin#head-baa6450ac33f1324e55dcb0d52344a440f14d75e
But was only created last week :)
From: Josh Ferguson <[email protected]>
Reply-To: <[email protected]>
Date: Sat, 24 Jan 2009 15:08:08 -0800
To: <[email protected]>
Subject: Re: Still can't get metastore working with mysql
That's why I had tried to do initially but the errors I was seeing
made me think my configuration was incorrect so I was changing the
configuration stuff. This all should be solved in the future with
some wiki love. My guess is most people are going to want to use
mysql as their metadata store and not derby. Derby is fine
developing on your local machine but not much else. Hadoop has a
good write up on their wiki about local, psuedo-distributed, and
fully distributed setups. Hive could probably benefit from something
similar as making the shift from just prototyping on your local
machine to a production deployment can be a difficult step.
Josh F.
On Jan 24, 2009, at 3:00 PM, Prasad Chakka wrote:
Well, if you had just used the defaults that are in the hive source
and changed just the JDBC params, that is what will happen. The
parameters that you are using comes from internal Facebook settings
and they are tuned for production. Checking for correctness of
schema every time hive connects to metastore is inefficient in
production cluster.
From: Josh Ferguson <[email protected]>
Reply-To: <[email protected]>
Date: Sat, 24 Jan 2009 14:55:21 -0800
To: <[email protected]>
Subject: Re: Still can't get metastore working with mysql
Joydeep Sen Sarma sent them out in the list last month when I was
first trying to get this working. I tried to modify them
appropriately.
This whole "lazy" creation of schemas and tables seems like a
really bad way to do things. My configuration shouldn't be different
for something that is up and running and something that needs to be
created. I should just be able to define my connection parameters
and then have a command that migrates my database into the proper
state for use. Trying to figure out what magical set of esoteric
configuration parameters will tell this black box library to create
my schema is really frustrating.
Josh F.
On Jan 24, 2009, at 2:26 PM, Prasad Chakka wrote:
Try setting fixedDataStore to false.
BTB, where did you get the initial values from the config you
posted? These values are for the store that is already running in
production and you don’t expect any changes to the schema.
From: Josh Ferguson <[email protected]>
Reply-To: <[email protected]>
Date: Sat, 24 Jan 2009 14:14:14 -0800
To: <[email protected]>
Subject: Re: Still can't get metastore working with mysql
That got me a little further, now I'm getting this error:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDODataStoreException Required table missing : "`SERDES`"
in Catalog "" Schema "". JPOX requires this table to perform its
persistence operations. Either your MetaData is incorrect, or you
need to enable "org.jpox.autoCreateTables")
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
In my hive-default.xml I turned on autoCreateTables and
autoCreateSchema and it's still throwing this error back to me? Is
there something else I need to do?
Josh Ferguson
On Jan 24, 2009, at 1:28 PM, Prasad Chakka wrote:
Down load the jar from here http://dev.mysql.com/downloads/connector/j/5.1.html
From: Prasad Chakka <[email protected]>
Reply-To: <[email protected]>
Date: Sat, 24 Jan 2009 13:13:35 -0800
To: <[email protected]>
Subject: Re: Still can't get metastore working with mysql
JPOX/Hive doesn’t yet download the relevant client libraries
based on the class name :) Drop mysql client jar in hive/lib. Also
try setting org.jpox.autoCreateSchema to true.
From: Josh Ferguson <[email protected]>
Reply-To: <[email protected]>
Date: Sat, 24 Jan 2009 13:06:26 -0800
To: <[email protected]>
Subject: Re: Still can't get metastore working with mysql
So I'm not really sure what the state of the database should be,
I have a database called hive and the user has permissions to access
it. I'm not sure which mysql client libraries need to be in the
classpath, I assumed it was all rolled into the jpox stuff. Here is
the complete error:
09/01/24 16:02:16 DEBUG exec.DDLTask:
org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got
exception: javax.jdo.JDOFatalInternalException Error creating
transactional connection factory)
at
org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
446)
at org.apache.hadoop.hive.ql.metadata.Hive.getAllTables(Hive.java:
433)
at org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:
330)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:127)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:212)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:
174)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:268)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun
.reflect
.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun
.reflect
.DelegatingMethodAccessorImpl
.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
at
org
.apache
.hadoop
.hive
.metastore
.MetaStoreUtils.logAndThrowMetaException(MetaStoreUtils.java:543)
at
org
.apache
.hadoop
.hive
.metastore.HiveMetaStoreClient.getTables(HiveMetaStoreClient.java:451)
at
org.apache.hadoop.hive.ql.metadata.Hive.getTablesByPattern(Hive.java:
444)
... 16 more
On Jan 24, 2009, at 12:58 PM, Prasad Chakka wrote:
There could be many reasons for it. Could you post the stacktrace
for the JDOFatal?
Some reasons could be
That mysql client libs are in the classpath.
That there is a database called ‘hive’ in mysql
That there is schema already created in that database (otherwise
set autoCreateSchema to true) etc...
Check documentation for the many jpox options
From: Josh Ferguson <[email protected]>
Reply-To: <[email protected]>
Date: Sat, 24 Jan 2009 12:48:27 -0800
To: <[email protected]>
Subject: Still can't get metastore working with mysql
Ok I still can't manage to get the metastore working with mysql
after
all this time trying. The error is:
FAILED: Error in metadata: MetaException(message:Got exception:
javax.jdo.JDOFatalInternalException Error creating transactional
connection factory)
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
The relevant config is:
<!-- Hadoop Setup -->
<property>
<name>hadoop.bin.path</name>
<value>/usr/local/hadoop/bin/hadoop</value>
<description>Path to hadoop binary. Assumes that by default
we are
executing from hive</description>
</property>
<property>
<name>hadoop.config.dir</name>
<value>/usr/local/hadoop/conf</value>
<description>Path to hadoop configuration. Again assumes that
by
default we are executing from hive/</description>
</property>
<!-- Hive Execution Parameters -->
<property>
<name>hive.exec.scratchdir</name>
<value>/tmp/hive-${user.name}</value>
<description>Scratch space for Hive jobs</description>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>******</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>******</value>
</property>
<property>
<name>org.jpox.autoCreateSchema</name>
<value>false</value>
</property>
<property>
<name>org.jpox.fixedDatastore</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.usefilestore</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.checkForDefaultDb</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.rawstore.impl</name>
<value>org.apache.hadoop.hive.metastore.ObjectStore</value>
<description>Name of the class that implements
org.apache.hadoop.hive.metastore.rawstore interface. This class is
used to store and retrieval of raw metadata objects such as table,
database</description>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
<description>controls whether to connect to remove metastore
server
or open a new metastore server in Hive Client JVM</description>
</property>
