I see from your listing that your jar contains the contents of many, if not
all, of the Apache and logging jars that are also in the hadoop/lib and
hive/lib directories, including the core hadoop and hive jars themselves.
plus some Google, JSON, and other libraries. Most likely, this is causing
issues, besides creating an unnecessarily large jar file of your code.
You should remove ALL the contents of other jars from your jar and use ADD
JAR only on the unique jars, like the Joda time jar you're using. I suspect
that will reduce or eliminate the problems, even though that it can be
tedious. I suspect you only really need to add a few extra jars, though.
The root-cause exception :
Caused by: java.lang.NullPointerException
at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(
NonManagedPluginRegistry.java:443)
at org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(
NonManagedPluginRegistry.java:355)
...
involves the $HIVE_HOME/lib/datanucleus-core-X.Y.Z.jar. I can only guess
that loading multiple copies of other jar contents is tripping it up
somehow.
Good luck,
dean
On Wed, Jan 23, 2013 at 11:44 AM, Ehsan Haq <ehsan....@klarna.com> wrote:
> I tried to rename my serde jar to zzzz.jar so that it would be loaded in
> the last, but still the same behaviour. when i run "show tables;" on the
> terminal, I get this exception in the logs. Also the list of the classes in
> the serde jar files is added in the attachment
>
> 2013-01-23 18:29:59,852 ERROR exec.Task
> (SessionState.java:printError(380)) - FAILED: Error in metadata:
> javax.jdo.JDOFatalInternalException: Unexpected exception caught.
> NestedThrowables:
> java.lang.reflect.InvocationTargetException
> org.apache.hadoop.hive.ql.metadata.HiveException:
> javax.jdo.JDOFatalInternalException: Unexpected exception caught.
> NestedThrowables:
> java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1099)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:1084)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.showTables(DDLTask.java:1957)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:306)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception
> caught.
> NestedThrowables:
> java.lang.reflect.InvocationTargetException
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1186)
> at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:803)
> at
> javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:698)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:246)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:275)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:208)
> at
> org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:183)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:407)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.executeWithRetry(HiveMetaStore.java:359)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:504)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:266)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:228)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:114)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2110)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2120)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:1095)
> ... 18 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at javax.jdo.JDOHelper$16.run(JDOHelper.java:1958)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.jdo.JDOHelper.invoke(JDOHelper.java:1953)
> at
> javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1159)
> ... 35 more
> Caused by: java.lang.NullPointerException
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:443)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerBundle(NonManagedPluginRegistry.java:355)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensions(NonManagedPluginRegistry.java:215)
> at
> org.datanucleus.plugin.NonManagedPluginRegistry.registerExtensionPoints(NonManagedPluginRegistry.java:156)
> at
> org.datanucleus.plugin.PluginManager.registerExtensionPoints(PluginManager.java:82)
> at org.datanucleus.OMFContext.<init>(OMFContext.java:156)
> at org.datanucleus.OMFContext.<init>(OMFContext.java:137)
> at
> org.datanucleus.ObjectManagerFactoryImpl.initialiseOMFContext(ObjectManagerFactoryImpl.java:132)
> at
> org.datanucleus.jdo.JDOPersistenceManagerFactory.initialiseProperties(JDOPersistenceManagerFactory.java:363)
> at
> org.datanucleus.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:307)
> at
> org.datanucleus.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:255)
> at
> org.datanucleus.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:182)
> ... 43 more
>
>
>
> On Wed, Jan 23, 2013 at 3:05 PM, Ehsan Haq <ehsan....@klarna.com> wrote:
>
>> Thanks dean, I knew about the .hiverc script, will try the other
>> alternative of renaming jar and get back. Thanks
>>
>> /Ehsan
>>
>>
>> On Wed, Jan 23, 2013 at 2:57 PM, Dean Wampler <
>> dean.wamp...@thinkbiganalytics.com> wrote:
>>
>>> Is there anything in the logs about problems loading the jar, etc.?
>>>
>>> The jar files in $HVE_HOME are added to the CLASSPATH in alphabetical
>>> order. As an experiment, rename your jar with a name that will go last,
>>> something like zzz.jar, and see what happens when you start Hive. If it
>>> seems to be working normally, it may be that some file in your jar file has
>>> a name that's the same as a file in one of Hive's jar files and your's gets
>>> read instead. An XML config file, for example.
>>>
>>> If this experiment appears to work, run "jar tf name-of-your-file.jar"
>>> and post the listing here, if you don't mind.
>>>
>>> Note that an alternative to dropping jar files in Hive's lib directory
>>> is to put the "add jar ..." command in your $HOME/.hiverc file, which Hive
>>> will read on startup (Hive v0.7.0 and later). What Hive version are you
>>> using, by the way? I also put lots of custom configuration setting
>>> commands, e.g., set hive.exec.mode.local.auto=true; (to encourage local
>>> mode execution, when possible).
>>>
>>> dean
>>>
>>>
>>> On Wed, Jan 23, 2013 at 6:41 AM, Ehsan Haq <ehsan....@klarna.com> wrote:
>>>
>>>> Hi,
>>>> I have writen a custom serde and bundle it in a jar file, which is
>>>> working fine, when I add the jar using the CLI command add jar. However
>>>> when I put the jar in the hive/lib folder so that I dont have to explicitly
>>>> add the jar, it looks like other jars were failed to load. The outcome is
>>>> that the meta data is also not accessible due to that. The jar works just
>>>> fine if put it somewhere else and add it via add jar.
>>>> Any idea what might be wrong?
>>>>
>>>> /Ehsan
>>>>
>>>
>>>
>>>
>>> --
>>> *Dean Wampler, Ph.D.*
>>> thinkbiganalytics.com
>>> +1-312-339-1330
>>>
>>>
>>
>>
>> --
>> *Muhammad Ehsan ul Haque*
>> Klarna AB
>> Norra Stationsgatan 61
>> SE-113 43 Stockholm
>>
>> Tel: +46 (0)8- 120 120 00
>> Fax: +46 (0)8- 120 120 99
>> Web: www.klarna.com
>>
>
>
>
> --
> *Muhammad Ehsan ul Haque*
> Klarna AB
> Norra Stationsgatan 61
> SE-113 43 Stockholm
>
> Tel: +46 (0)8- 120 120 00
> Fax: +46 (0)8- 120 120 99
> Web: www.klarna.com
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
