OK well let's keep poking at it until we have the best option possible. Will look into sparks approach.
Thanks Joe On Jun 15, 2016 8:27 PM, "Andre" <[email protected]> wrote: > Joe, > > I wouldn't say I am supportive of it... But you guys have far more > experience in this thing than me, so I am considering all options and > trying to keep an open mind. :-) > > Had it been my choice - and no side effects are caused by the changes - I > rather use Spark's approach and have the repos @ the top level pom so that > we can simply point users to: > > mvn -T2 -DskipTest -Dhadoop.version=whatever > > But I am happy to try to find a solution that suits the group as a whole > and to provide the group with updates as it goes. > > Cheers > > > On Thu, Jun 16, 2016 at 10:12 AM, Joe Witt <[email protected]> wrote: > >> Andre >> >> So are you also supportive of the settings based approach for now? >> >> Thanks >> Joe >> On Jun 15, 2016 8:08 PM, "Andre" <[email protected]> wrote: >> >>> Just to complement this conversation. >>> >>> I did use the settings.xml (basic changes, specifying profiles with the >>> appropriate repositories) >>> >>> As initially suspected, the libraries are then added to NARs beyond the >>> nifi-hadoop-libraries-bundle. >>> >>> Using -Pmapr -Dhadoop.version=2.7.0-mapr-1506 results in: >>> >>> $ find -name "*-mapr-*" | grep -v nifi-hadoop-libraries-bundle >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.7.0-mapr-1506.jar >>> >>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.7.0-mapr-1506.jar >>> >>> >>> While using -Pcdh -Dhadoop.version=2.6.0-cdh5.7.1 results in: >>> >>> $ find -name "*-cdh*" | grep -v nifi-hadoop-libraries-bundle >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-common-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-hadoop-bundle/nifi-hadoop-nar/target/classes/META-INF/bundled-dependencies/hadoop-auth-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-mapreduce-client-core-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-annotations-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-api-2.6.0-cdh5.7.1.jar >>> >>> ./nifi-nar-bundles/nifi-hive-bundle/nifi-hive-nar/target/classes/META-INF/bundled-dependencies/hadoop-yarn-common-2.6.0-cdh5.7.1.jar >>> >>> >>> This doesn't seem a particularly bad thing. >>> >>> >>> On Thu, Jun 16, 2016 at 1:56 AM, Andre <[email protected]> wrote: >>> >>> > All, >>> > >>> > Moving from GH as suggested by Matt. >>> > >>> > >>> > As part of NIFI-2026 I have been experimenting with ways of using >>> > non-vanilla Hadoop libraries when building NiFi from source. >>> > >>> > The idea is to streamline the overall process one must follow to get >>> > access to HDFS compatible platforms (e.g. MapR-FS) without the need to >>> keep >>> > separate repositories as currently done here: >>> > https://github.com/xmlking/mapr-nifi-hadoop-libraries-bundle >>> > >>> > The current commit - finally functional (thanks Matt!) - creates a >>> profile >>> > >>> > custom_hadoop_libraries - more on this below. >>> > >>> > and two convenience properties: >>> > >>> > nifi.hadoop.distro.params - Used to add a java arg within >>> > bootstrap.properties (so far only known to be required by one distro) >>> > >>> > nifi.hadoop.version - Used to pass vendor based version of hadoop >>> > artifacts (e.g. 2.0.0-mr1-cdh4.2.0) >>> > >>> > About the profile: >>> > - The profile enables the individual building >>> nifi-hadoop-libraries-nar to >>> > point mvn to CDH, HDP and MapR repos and use artifacts with the >>> version set >>> > by nifi.hadoop.version >>> > >>> > The profile is disabled by default, therefore, the profile acts as a >>> > fencing mechanisms to ensure convenience binaries should not be tainted >>> > with code of unknown licensing status. >>> > >>> > The used of profile and a custom property also ensure the use of >>> > customised code is restricted to the points where is needed (e.g. while >>> > HDFS libraries may be CDH, Hive libraries may still be vanilla). >>> > >>> > >>> > This is not the only alternative. Joe W has suggested that we possibly >>> > point users to alter their settings.xml (option b), while Apache Spark >>> > simply adds the same repositories to the project main pom.xml ( >>> > https://github.com/apache/spark/blob/branch-1.6/pom.xml#L285) (option >>> c) >>> > >>> > >>> > May I ask you what is your preferred approach / comments? >>> > >>> >> >
