Yea it would be duplicate jars in both places. It is a shame both share the name "hadoop" since the two features - filesystem and authentication. They end up being two entirely different things both in Hadoop itself and inside of Solr.
Kevin Risden On Thu, Jan 20, 2022 at 4:58 PM David Smiley <[email protected]> wrote: > Separate modules will mean our distro will end up duplicating > hadoop-common and other related JARs for both modules. I was trying to be > practical. But it's not important to me; ok. > > implementation ('org.apache.hadoop:hadoop-common') { transitive = false } // > too many to ignore > implementation ('org.apache.hadoop:hadoop-annotations') > runtimeOnly 'org.apache.htrace:htrace-core4' // note: removed in Hadoop 3.3.2 > runtimeOnly "org.apache.commons:commons-configuration2" > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Thu, Jan 20, 2022 at 4:55 PM Kevin Risden <[email protected]> wrote: > >> My preference would be as a separate HadoopAuthentication or something >> module. HDFS the filesystem / blockcache / etc support is unique and >> separate from the authentication part. It shouldn't all be in one module. >> >> Kevin Risden >> >> >> On Thu, Jan 20, 2022 at 4:48 PM David Smiley <[email protected]> wrote: >> >>> The issue https://issues.apache.org/jira/browse/SOLR-14660 is about >>> moving the HDFS plugins out of core into a module. While a great thing, it >>> still leaves quite a few Hadoop related dependencies in solr-core because >>> Hadoop is not there only for HDFS; it's there for some exotic >>> authentication & authorization plugins. In that JIRA issue I proposed that >>> this module be "hadoop" and have any hadoop related plugins. >>> >>> As a quick experiment, I commented out the hadoop-auth dependency and >>> tried to compile to see what the compiler caught. It exposed the following >>> two Solr plugins: >>> * HadoopAuthPlugin >>> * KerberosPlugin >>> >>> Are we okay with expanding the scope of SOLR-14660 to include these? >>> >>> Note that SOLR-14660 *might* result in 9.0 not including this module in >>> the release distribution if we don't feel the module will be sufficiently >>> ready to release. >>> >>> ~ David Smiley >>> Apache Lucene/Solr Search Developer >>> http://www.linkedin.com/in/davidwsmiley >>> >>
