Hi, thanks for the help. As I specified above we’re pulling in hbase-mapreduce, the reason is we’re overriding some classes (HFileInputFormat et al). This results in the following:
hbase-mapreduce (2.4.9) -> hbase-hadoop2-compat -> hadoop-mapreduce-client-core (2.10.0) I tried adding the following to my pom file and it seems to have worked. + <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core --> + <dependency> + <groupId>org.apache.hadoop</groupId> + <artifactId>hadoop-mapreduce-client-core</artifactId> + <version>3.3.1</version> + </dependency> Thanks, Giuseppe > On Jan 4, 2022, at 3:13 AM, 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > > Which hadoop module does HBase pull in? In general if you specify the > hadoop module in your pom, it will omit the one specified in HBase's > pom. > > Typically you do not need get the pom in HBase to your project, just > put all hadoop modules in your pom with the version you want. > > Please see this topic on stackoverflow, it is mainly a maven usage problem... > > https://stackoverflow.com/questions/7175398/maven-dependency-resolution-conflicted > > Thanks. > > Benoit Sigoure <tsuna...@gmail.com> 于2022年1月4日周二 17:33写道： >> >> Our pom.xml is pulling in Hadoop 3 jars, but HBase is transitively pulling >> some Hadoop 2 jars, and the combination of both leads to strange runtime >> errors like ClassNotFoundException. If we manually fix the classpath to >> only references Hadoop 3 jars, everything works fine, but Maven and Eclipse >> etc keep on mixing Hadoop 3 and Hadoop 2 jars, which leads to various >> problems down the road. >> >> We already build HBase with -Dhadoop.profile=3.0 but that's not the >> question. We want to build *our* MapReduce code and have Maven only pull in >> Hadoop 3 dependencies. >> We tried to copy the <profile> from HBase's pom.xml into ours — admittedly >> mostly just cargo-culting it, since we're not really knowledgeable about >> Maven in general, hence this question. >> >> How is a project dependent on hbase-mapreduce supposed to specify in its >> pom.xml that it only wants Hadoop 3 dependencies pulled in? >> >> On Tue, Jan 4, 2022 at 5:14 AM 张铎(Duo Zhang) <palomino...@gmail.com> wrote: >> >>> I think hadoop 2.10.0 client can communicate with hadoop 3.3.1 cluster? >>> >>> If you want to build hbase-2.4.9 by yourself so you can depend on >>> hadoop 3.3.1, please download the src tarball, try this command >>> >>> mvn -DskipTests -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.1 >>> clean install && mvn -DskipTests -Dhadoop.profile=3.0 >>> -Dhadoop-three.version=3.3.1 package assembly:single >>> >>> Thanks. >>> >>> Giuseppe Valente <gvale...@arista.com.invalid> 于2022年1月4日周二 09:48写道： >>>> >>>> Hi, >>>> >>>> I have a project that has a dependency on hbase-mapreduce (2.4.9), which >>> in turn depends on hbase-hadoop2-compat. The latter doesn’t have any >>> version specified for hadoop-mapreduce-client-core and ends up pulling >>> 2.10.0, which conflicts with the Hadoop version I’m running against >>> (3.3.1). What’s the correct way to force hadoop-mapreduce to work with >>> Hadoop 3.3.1? I tried using profiles but I couldn’t get it to work, I >>> wonder if it has anything to do with this change < >>> https://issues.apache.org/jira/browse/HBASE-24309>: >>>> >>>> commit dc2146069cd5ef6120f0a9d1918acba0f74ac5c3 >>>> Author: Duo Zhang <zhang...@apache.org> >>>> Date: Wed May 13 17:59:21 2020 +0800 >>>> >>>> HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for >>> … (#1697) >>>> >>>> Signed-off-by: stack <st...@apache.org> >>>> >>>> diff --git a/hbase-hadoop2-compat/pom.xml b/hbase-hadoop2-compat/pom.xml >>>> index 236b52ce65..96472fb8f3 100644 >>>> --- a/hbase-hadoop2-compat/pom.xml >>>> +++ b/hbase-hadoop2-compat/pom.xml >>>> @@ -118,14 +118,12 @@ limitations under the License. >>>> <dependency> >>>> <groupId>org.apache.hbase</groupId> >>>> <artifactId>hbase-hadoop-compat</artifactId> >>>> - <version>${project.version}</version> >>>> <type>test-jar</type> >>>> <scope>test</scope> >>>> </dependency> >>>> <dependency> >>>> <groupId>org.apache.hadoop</groupId> >>>> <artifactId>hadoop-mapreduce-client-core</artifactId> >>>> - <version>${hadoop.version}</version> >>>> <exclusions> >>>> <exclusion> >>>> <groupId>com.google.guava</groupId> >>>> >>>> Thanks, >>>> Giuseppe >>> >> >> >> -- >> Benoit "tsuna" Sigoure
