[
https://issues.apache.org/jira/browse/HIVE-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13888781#comment-13888781
]
Nigel Savage commented on HIVE-6336:
------------------------------------
Thanks andy for your response
you are correct re datanucleus-api-jdo exclusions, thanks for the link to the
pom, the reason the exclusions are there is because i used the
org.apache.hive:hive-metastore:0.12.0 pom as a guide, I assumed that this pom
would be correct, that is if the exclusions are there then you need them seeing
we use this class in our code and it is the main database that is jdo in terms
of transactions
http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
basically I am getting the exceptions outlined and I am trying to resolve the
issue so I am trying diferent things, basically trying to harmonize the pom so
it uses the same datanucleus as the dependencies like the
org.apache.hive:hive-metastore:0.12.0
I notice you have a datanucleus email so if possible would you be willing to
comment on what versions of datanucleus we should be using ?
for hive 12 and hadoop 2.2.0
<datanucleus.version>3.2.2</datanucleus.version> for
<artifactId>datanucleus-core</artifactId>
<datanucleus-rdbms.version>3.2.1</datanucleus-rdbms.version>
<artifactId>datanucleus-rdbms</artifactId>
<datanucleus.jdo.version>3.2.1</datanucleus.jdo.version>
<artifactId>datanucleus-api-jdo</artifactId>
> Issue is hive 12 datanucleus incompatability with
> org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-6336
> URL: https://issues.apache.org/jira/browse/HIVE-6336
> Project: Hive
> Issue Type: Wish
> Components: HiveServer2
> Affects Versions: 0.12.0
> Environment: Hadoop 2.2 local derby Meatastore embedded
> Reporter: Nigel Savage
> Priority: Blocker
> Labels: HADOOP
>
> There is an with hive 12 datanucleus incompatability which seems to have
> invompatibility with org.apache.hadoop.hive.contrib.serde2.RegexSerDe
> The main question:
> *IF hive 0.12.0 and datanucleus are compatabile, then what is the version of
> datanucleus I should be using with Hive 12 and Hadoop 2.2?*
> The error which Im getting (this blocks me from properly running hive queries
> invoked from the "test" phase of a maven project)
> *To reproduce*
> I have hadoop and hive running as a pseudo cluster local mode and derby as
> the metastore
> I have the following environment variables
> {noformat}
> HADOOP_HOME=/home/ubu/hadoop
> JAVA_HOME=/usr/lib/jvm/java-7-oracle
> {noformat}
> I have the RegexSerDe declared in the hive-site.xml
> {noformat}
> <property>
> <name>hive.aux.jars.path</name>
> <value>file:///home/ubu/hadoop/lib/hive-contrib-0.12.0.jar </value>
> <description>This JAR file available to all users for
> alljobs</description>
> </property>
> {noformat}
> If I run with
> {noformat}
> <datanucleus.version>3.0.2</datanucleus.version>
> {noformat}
> I get the following 1 exception only
> 'java.lang.ClassNotFoundException...org.datanucleus.store.types.backed.Ma'
> HOWEVER, If I run with
> {noformat}
> <datanucleus.version>3.2.0-release</datanucleus.version>
> {noformat}
> I get the following 1 exception exception only
> java.lang.ClassNotFoundException:
> org/apache/hadoop/hive/contrib/serde2/RegexSerDe
> EXPLANATION
> The RegexSerDe class is picked up at run time but the datanucleus Map class
> is not available, I have checked in the datanucleus-core 3.0.2 jar and it is
> missing, Upgrading to the first datanucleus above 3.0.2 that includes the
> Map class throws the ClassNotFoundException for RegexSerDe.
> The earlier *3.0.2* datanucleus, code fails with the missing Map class but
> the RegexSerDe class is found, then when I upgrade to the
> 3.2.0-release the Map class is found but for some unkown reason the code/Hive
> no longer finds the RegexSerDe class
> I started using the same datanucleus dependencies found in this hive pom
> http://maven-repository.com/artifact/org.apache.hive/hive-metastore/0.12.0/pom
> below are the dependencies my latest attempts to get a functioning pom
> {noformat}
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-server</artifactId>
> <version>0.96.0-hadoop2</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-client</artifactId>
> <version>0.96.0-hadoop2</version>
> </dependency>
> <!-- misc -->
> <dependency>
> <groupId>org.apache.commons</groupId>
> <artifactId>commons-lang3</artifactId>
> <version>3.1</version>
> </dependency>
> <dependency>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> <version>${guava.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.derby</groupId>
> <artifactId>derby</artifactId>
> <version>${derby.version}</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-core</artifactId>
> <version>${datanucleus.version}</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-rdbms</artifactId>
> <version>${datanucleus-rdbms.version}</version>
> </dependency>
> <dependency>
> <groupId>javax.jdo</groupId>
> <artifactId>jdo-api</artifactId>
> <version>3.0.1</version>
> </dependency>
> <dependency>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-api-jdo</artifactId>
> <version>${datanucleus.jdo.version}</version>
> <exclusions>
> <exclusion>
> <groupId>javax.jdo</groupId>
> <artifactId>jdo2-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> </exclusion>
> <exclusion>
> <groupId>log4j</groupId>
> <artifactId>log4j</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <!-- hadoop -->
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${hadoop.version}</version>
> </dependency>
> <!-- hive -->
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-common</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-serde</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-exec</artifactId>
> <version>${hive.version}</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-jdbc</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-contrib</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-metastore</artifactId>
> <version>${hive.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.hive</groupId>
> <artifactId>hive-cli</artifactId>
> <version>${hive.version}</version>
> <exclusions>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-core</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-api-jdo</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.datanucleus</groupId>
> <artifactId>datanucleus-rdbms</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> </exclusion>
> <exclusion>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-log4j12</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>com.jolbox</groupId>
> <artifactId>bonecp</artifactId>
> <version>${bonecp.version}</version>
> </dependency>
> <!-- logging -->
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-api</artifactId>
> <version>${slf4j.version}</version>
> </dependency>
> <!-- SL4J Binding provided at runtime -->
> <dependency>
> <groupId>log4j</groupId>
> <artifactId>log4j</artifactId>
> <version>1.2.12</version>
> <scope>provided</scope>
> </dependency>
> <dependency>
> <groupId>org.slf4j</groupId>
> <artifactId>slf4j-log4j12</artifactId>
> <version>${slf4j.version}</version>
> <scope>provided</scope>
> </dependency>
> <!-- Unit test artifacts -->
> <dependency>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> <version>4.11</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.hamcrest</groupId>
> <artifactId>hamcrest-all</artifactId>
> <version>1.3</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.mrunit</groupId>
> <artifactId>mrunit</artifactId>
> <version>1.0.0</version>
> <classifier>hadoop2</classifier>
> </dependency>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
