Joe, I wouldn't say I am supportive of it... But you guys have far more experience in this thing than me, so I am considering all options and trying to keep an open mind. :-)
Had it been my choice - and no side effects are caused by the changes - I rather use Spark's approach and have the repos @ the top level pom so that we can simply point users to: mvn -T2 -DskipTest -Dhadoop.version=whatever But I am happy to try to find a solution that suits the group as a whole and to provide the group with updates as it goes. Cheers On Thu, Jun 16, 2016 at 10:12 AM, Joe Witt <[email protected]> wrote: > Andre > > So are you also supportive of the settings based approach for now? > > Thanks > Joe > On Jun 15, 2016 8:08 PM, "Andre" <[email protected]> wrote: > >> Just to complement this conversation. >> >> I did use the settings.xml (basic changes, specifying profiles with the >> appropriate repositories) >> >> As initially suspected, the libraries are then added to NARs beyond the >> nifi-hadoop-libraries-bundle. >> >> Using -Pmapr -Dhadoop.version=2.7.0-mapr-1506 results in: >> >> $ find -name "*-mapr-*" | grep -v nifi-hadoop-libraries-bundle >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar >> >> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar >> >> >> While using -Pcdh -Dhadoop.version=2.6.0-cdh5.7.1 results in: >> >> $ find -name "*-cdh*" | grep -v nifi-hadoop-libraries-bundle >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar >> >> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar >> >> >> This doesn't seem a particularly bad thing. >> >> >> On Thu, Jun 16, 2016 at 1:56 AM, Andre <[email protected]> wrote: >> >> > All, >> > >> > Moving from GH as suggested by Matt. >> > >> > >> > As part of NIFI-2026 I have been experimenting with ways of using >> > non-vanilla Hadoop libraries when building NiFi from source. >> > >> > The idea is to streamline the overall process one must follow to get >> > access to HDFS compatible platforms (e.g. MapR-FS) without the need to >> keep >> > separate repositories as currently done here: >> > https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle >> > >> > The current commit - finally functional (thanks Matt!) - creates a >> profile >> > >> > custom_hadoop_libraries - more on this below. >> > >> > and two convenience properties: >> > >> > nifi.hadoop.distro.params - Used to add a java arg within >> > bootstrap.properties (so far only known to be required by one distro) >> > >> > nifi.hadoop.version - Used to pass vendor based version of hadoop >> > artifacts (e.g. 2.0.0-mr1-cdh4.2.0) >> > >> > About the profile: >> > - The profile enables the individual building nifi-hadoop-libraries-nar >> to >> > point mvn to CDH, HDP and MapR repos and use artifacts with the version >> set >> > by nifi.hadoop.version >> > >> > The profile is disabled by default, therefore, the profile acts as a >> > fencing mechanisms to ensure convenience binaries should not be tainted >> > with code of unknown licensing status. >> > >> > The used of profile and a custom property also ensure the use of >> > customised code is restricted to the points where is needed (e.g. while >> > HDFS libraries may be CDH, Hive libraries may still be vanilla). >> > >> > >> > This is not the only alternative. Joe W has suggested that we possibly >> > point users to alter their settings.xml (option b), while Apache Spark >> > simply adds the same repositories to the project main pom.xml ( >> > https://github.com/apache/spark/blob/branch-1.6/pom.xml#L285) (option >> c) >> > >> > >> > May I ask you what is your preferred approach / comments? >> > >> >
