Which hadoop module does HBase pull in? In general if you specify the hadoop module in your pom, it will omit the one specified in HBase's pom.
Typically you do not need get the pom in HBase to your project, just put all hadoop modules in your pom with the version you want. Please see this topic on stackoverflow, it is mainly a maven usage problem... https://stackoverflow.com/questions/7175398/maven-dependency-resolution-conflicted Thanks. Benoit Sigoure <tsuna...@gmail.com> 于2022年1月4日周二 17:33写道： > > Our pom.xml is pulling in Hadoop 3 jars, but HBase is transitively pulling > some Hadoop 2 jars, and the combination of both leads to strange runtime > errors like ClassNotFoundException. If we manually fix the classpath to > only references Hadoop 3 jars, everything works fine, but Maven and Eclipse > etc keep on mixing Hadoop 3 and Hadoop 2 jars, which leads to various > problems down the road. > > We already build HBase with -Dhadoop.profile=3.0 but that's not the > question. We want to build *our* MapReduce code and have Maven only pull in > Hadoop 3 dependencies. > We tried to copy the <profile> from HBase's pom.xml into ours — admittedly > mostly just cargo-culting it, since we're not really knowledgeable about > Maven in general, hence this question. > > How is a project dependent on hbase-mapreduce supposed to specify in its > pom.xml that it only wants Hadoop 3 dependencies pulled in? > > On Tue, Jan 4, 2022 at 5:14 AM 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > > > I think hadoop 2.10.0 client can communicate with hadoop 3.3.1 cluster? > > > > If you want to build hbase-2.4.9 by yourself so you can depend on > > hadoop 3.3.1, please download the src tarball, try this command > > > > mvn -DskipTests -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.1 > > clean install && mvn -DskipTests -Dhadoop.profile=3.0 > > -Dhadoop-three.version=3.3.1 package assembly:single > > > > Thanks. > > > > Giuseppe Valente <gvale...@arista.com.invalid> 于2022年1月4日周二 09:48写道： > > > > > > Hi, > > > > > > I have a project that has a dependency on hbase-mapreduce (2.4.9), which > > in turn depends on hbase-hadoop2-compat. The latter doesn’t have any > > version specified for hadoop-mapreduce-client-core and ends up pulling > > 2.10.0, which conflicts with the Hadoop version I’m running against > > (3.3.1). What’s the correct way to force hadoop-mapreduce to work with > > Hadoop 3.3.1? I tried using profiles but I couldn’t get it to work, I > > wonder if it has anything to do with this change < > > https://issues.apache.org/jira/browse/HBASE-24309>: > > > > > > commit dc2146069cd5ef6120f0a9d1918acba0f74ac5c3 > > > Author: Duo Zhang <zhang...@apache.org> > > > Date: Wed May 13 17:59:21 2020 +0800 > > > > > > HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for > > … (#1697) > > > > > > Signed-off-by: stack <st...@apache.org> > > > > > > diff --git a/hbase-hadoop2-compat/pom.xml b/hbase-hadoop2-compat/pom.xml > > > index 236b52ce65..96472fb8f3 100644 > > > --- a/hbase-hadoop2-compat/pom.xml > > > +++ b/hbase-hadoop2-compat/pom.xml > > > @@ -118,14 +118,12 @@ limitations under the License. > > > <dependency> > > > <groupId>org.apache.hbase</groupId> > > > <artifactId>hbase-hadoop-compat</artifactId> > > > - <version>${project.version}</version> > > > <type>test-jar</type> > > > <scope>test</scope> > > > </dependency> > > > <dependency> > > > <groupId>org.apache.hadoop</groupId> > > > <artifactId>hadoop-mapreduce-client-core</artifactId> > > > - <version>${hadoop.version}</version> > > > <exclusions> > > > <exclusion> > > > <groupId>com.google.guava</groupId> > > > > > > Thanks, > > > Giuseppe > > > > > -- > Benoit "tsuna" Sigoure
