Your hadoop.version could also be set by this profile, right? And conceivably hive.version and hbase.version (et al) if all has been set up correctly (and we can change if not, as we should be setting all dependency versions from properties).
Do you need the java.arg.15 for this? I think it's worth discussing that (blank by default but ready for action) as a general improvement for NiFi, not just for Hadoop flavors but for enabling various things via system properties during build (etc.) Regards, Matt > On Jun 15, 2016, at 8:08 PM, Andre <[email protected]> wrote: > > Just to complement this conversation. > > I did use the settings.xml (basic changes, specifying profiles with the > appropriate repositories) > > As initially suspected, the libraries are then added to NARs beyond the > nifi-hadoop-libraries-bundle. > > Using -Pmapr -Dhadoop.version=2.7.0-mapr-1506 results in: > > $ find -name "*-mapr-*" | grep -v nifi-hadoop-libraries-bundle > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar > > > While using -Pcdh -Dhadoop.version=2.6.0-cdh5.7.1 results in: > > $ find -name "*-cdh*" | grep -v nifi-hadoop-libraries-bundle > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar > > > This doesn't seem a particularly bad thing. > > >> On Thu, Jun 16, 2016 at 1:56 AM, Andre <[email protected]> wrote: >> >> All, >> >> Moving from GH as suggested by Matt. >> >> >> As part of NIFI-2026 I have been experimenting with ways of using >> non-vanilla Hadoop libraries when building NiFi from source. >> >> The idea is to streamline the overall process one must follow to get >> access to HDFS compatible platforms (e.g. MapR-FS) without the need to keep >> separate repositories as currently done here: >> https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle >> >> The current commit - finally functional (thanks Matt!) - creates a profile >> >> custom_hadoop_libraries - more on this below. >> >> and two convenience properties: >> >> nifi.hadoop.distro.params - Used to add a java arg within >> bootstrap.properties (so far only known to be required by one distro) >> >> nifi.hadoop.version - Used to pass vendor based version of hadoop >> artifacts (e.g. 2.0.0-mr1-cdh4.2.0) >> >> About the profile: >> - The profile enables the individual building nifi-hadoop-libraries-nar to >> point mvn to CDH, HDP and MapR repos and use artifacts with the version set >> by nifi.hadoop.version >> >> The profile is disabled by default, therefore, the profile acts as a >> fencing mechanisms to ensure convenience binaries should not be tainted >> with code of unknown licensing status. >> >> The used of profile and a custom property also ensure the use of >> customised code is restricted to the points where is needed (e.g. while >> HDFS libraries may be CDH, Hive libraries may still be vanilla). >> >> >> This is not the only alternative. Joe W has suggested that we possibly >> point users to alter their settings.xml (option b), while Apache Spark >> simply adds the same repositories to the project main pom.xml ( >> https://github.com/apache/spark/blob/branch-1.6/pom.xml#L285) (option c) >> >> >> May I ask you what is your preferred approach / comments? >>
