RE: Issues building 1.9-snapshot and Hadoop 3.1.3

[Arvind Shyamsundar]

Thanks, Keith for all your inputs. FYI this cluster was deployed via. Muchos 
and that accumulo-site template has:

      $HADOOP_PREFIX/share/hadoop/common/[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/yarn/lib/jersey.*.jar

I will try modifying this and get back.

Thanks again!

-----Original Message-----
From: Keith Turner <ke...@deenlo.com> 
Sent: Thursday, November 21, 2019 9:59 AM
To: Accumulo Dev List <dev@accumulo.apache.org>
Subject: Re: Issues building 1.9-snapshot and Hadoop 3.1.3

Can you check that your accumulo-site.xml only adds 
$HADOOP_PREFIX/share/hadoop/client/[^.].*.jar for hadoop deps for the setting 
general.classpaths?  Not completely sure, but I think this will use the hadoop 
shaded jars.

Do not want the non-shaded hadoop jars like 
$HADOOP_PREFIX/share/hadoop/common/[^.].*.jar on the path.

On Wed, Nov 20, 2019 at 10:51 PM Arvind Shyamsundar 
<arvin...@microsoft.com.invalid> wrote:
>
> Hello!
> Per this 
> issue(https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Faccumulo%2Fissues%2F569&amp;data=02%7C01%7Carvindsh%40microsoft.com%7C87c861e6940f4e6a80fe08d76eac7beb%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637099559408252616&amp;sdata=wzAizSdxYMZHHkAtqXBvq5TUwj77sTovcr2%2BuZ1Zcnw%3D&amp;reserved=0)
>  building 1.9.x with Hadoop 3 support needs hadoop.profile=3. So I checked 
> out current 1.9 branch and built with -Dhadoop.profile=3. When I deployed 
> this "custom" Accumulo build with Hadoop 3.1.3, accumulo init failed:
>
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
>         at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
>         at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
>         at 
> org.apache.hadoop.conf.Configuration.setInt(Configuration.java:1515)
>         at 
> org.apache.hadoop.hdfs.server.namenode.ha.AbstractNNFailoverProxyProvider.<init>(AbstractNNFailoverProxyProvider.java:70)
>         at 
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvi
> der.<init>(ConfiguredFailoverProxyProvider.java:44)
>
> This is related to Guava. The version of Guava that is used by Hadoop 3.1.3 
> is 27.0-jre while Accumulo 1.9 still depends (and includes) Guava 14.0. So I 
> set about to build 1.9 with Guava 27.0-jre. I had to set the compiler version 
> to 1.8. As Christopher had mentioned to in a the 1.10 thread, I also ran into 
> problems with modernizer. Without disabling modernizer, the refactor involved 
> looks non-trivial. I also had issues with outdated interfaces in 
> DataoutputHasher.java, CloseWriteAheadLogReferences.java, 
> RemoveCompleteReplicationRecords.java but those were relatively easy fixes. 
> FWIW, I pushed my changes here: 
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Faccumulo%2Fcompare%2Fmaster...arvindshmicrosoft%3Atemp-1.9-guava27&amp;data=02%7C01%7Carvindsh%40microsoft.com%7C87c861e6940f4e6a80fe08d76eac7beb%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C637099559408252616&amp;sdata=cacxPXl9WL0u4NhRAPljtJuo2Gfcf1uWmNACv%2FbJkuY%3D&amp;reserved=0.
>
> So my question is: are these known issues with the current 1.9 branch and 
> Hadoop? Do we want to support Hadoop 3.1 / 3.2 with Accumulo 1.10?
>
> Thank you.
>
> - Arvind.