Just to complement this conversation. I did use the settings.xml (basic changes, specifying profiles with the appropriate repositories)
As initially suspected, the libraries are then added to NARs beyond the nifi-hadoop-libraries-bundle. Using -Pmapr -Dhadoop.version=2.7.0-mapr-1506 results in: $ find -name "*-mapr-*" | grep -v nifi-hadoop-libraries-bundle ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar While using -Pcdh -Dhadoop.version=2.6.0-cdh5.7.1 results in: $ find -name "*-cdh*" | grep -v nifi-hadoop-libraries-bundle ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar This doesn't seem a particularly bad thing. On Thu, Jun 16, 2016 at 1:56 AM, Andre <[email protected]> wrote: > All, > > Moving from GH as suggested by Matt. > > > As part of NIFI-2026 I have been experimenting with ways of using > non-vanilla Hadoop libraries when building NiFi from source. > > The idea is to streamline the overall process one must follow to get > access to HDFS compatible platforms (e.g. MapR-FS) without the need to keep > separate repositories as currently done here: > https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle > > The current commit - finally functional (thanks Matt!) - creates a profile > > custom_hadoop_libraries - more on this below. > > and two convenience properties: > > nifi.hadoop.distro.params - Used to add a java arg within > bootstrap.properties (so far only known to be required by one distro) > > nifi.hadoop.version - Used to pass vendor based version of hadoop > artifacts (e.g. 2.0.0-mr1-cdh4.2.0) > > About the profile: > - The profile enables the individual building nifi-hadoop-libraries-nar to > point mvn to CDH, HDP and MapR repos and use artifacts with the version set > by nifi.hadoop.version > > The profile is disabled by default, therefore, the profile acts as a > fencing mechanisms to ensure convenience binaries should not be tainted > with code of unknown licensing status. > > The used of profile and a custom property also ensure the use of > customised code is restricted to the points where is needed (e.g. while > HDFS libraries may be CDH, Hive libraries may still be vanilla). > > > This is not the only alternative. Joe W has suggested that we possibly > point users to alter their settings.xml (option b), while Apache Spark > simply adds the same repositories to the project main pom.xml ( > https://github.com/apache/spark/blob/branch-1.6/pom.xml#L285) (option c) > > > May I ask you what is your preferred approach / comments? >
