Andre So are you also supportive of the settings based approach for now?
Thanks Joe On Jun 15, 2016 8:08 PM, "Andre" <[email protected]> wrote: > Just to complement this conversation. > > I did use the settings.xml (basic changes, specifying profiles with the > appropriate repositories) > > As initially suspected, the libraries are then added to NARs beyond the > nifi-hadoop-libraries-bundle. > > Using -Pmapr -Dhadoop.version=2.7.0-mapr-1506 results in: > > $ find -name "*-mapr-*" | grep -v nifi-hadoop-libraries-bundle > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar > > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar > > > While using -Pcdh -Dhadoop.version=2.6.0-cdh5.7.1 results in: > > $ find -name "*-cdh*" | grep -v nifi-hadoop-libraries-bundle > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar > > ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar > > > This doesn't seem a particularly bad thing. > > > On Thu, Jun 16, 2016 at 1:56 AM, Andre <[email protected]> wrote: > > > All, > > > > Moving from GH as suggested by Matt. > > > > > > As part of NIFI-2026 I have been experimenting with ways of using > > non-vanilla Hadoop libraries when building NiFi from source. > > > > The idea is to streamline the overall process one must follow to get > > access to HDFS compatible platforms (e.g. MapR-FS) without the need to > keep > > separate repositories as currently done here: > > https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle > > > > The current commit - finally functional (thanks Matt!) - creates a > profile > > > > custom_hadoop_libraries - more on this below. > > > > and two convenience properties: > > > > nifi.hadoop.distro.params - Used to add a java arg within > > bootstrap.properties (so far only known to be required by one distro) > > > > nifi.hadoop.version - Used to pass vendor based version of hadoop > > artifacts (e.g. 2.0.0-mr1-cdh4.2.0) > > > > About the profile: > > - The profile enables the individual building nifi-hadoop-libraries-nar > to > > point mvn to CDH, HDP and MapR repos and use artifacts with the version > set > > by nifi.hadoop.version > > > > The profile is disabled by default, therefore, the profile acts as a > > fencing mechanisms to ensure convenience binaries should not be tainted > > with code of unknown licensing status. > > > > The used of profile and a custom property also ensure the use of > > customised code is restricted to the points where is needed (e.g. while > > HDFS libraries may be CDH, Hive libraries may still be vanilla). > > > > > > This is not the only alternative. Joe W has suggested that we possibly > > point users to alter their settings.xml (option b), while Apache Spark > > simply adds the same repositories to the project main pom.xml ( > > https://github.com/apache/spark/blob/branch-1.6/pom.xml#L285) (option c) > > > > > > May I ask you what is your preferred approach / comments? > > >
