Just wanted to mention there is an in-progress PR for native support for Azure blobs, using the Azure storage Java SDK: https://github.com/apache/nifi/pull/1636
It might be more straightforward in your case, no additional libraries configuration required. Andrew On Tue, Apr 4, 2017 at 11:40 AM Bryan Bende <[email protected]> wrote: > Giovanni, > > In the pom.xml at the root of the NiFi source tree: > > <hadoop.version>2.7.3</hadoop.version> > > You can change that to 2.8.0 (if 2.8.0 is released) and then run a > full build, assuming 2.8.0 doesn't break any code that NiFi is using. > > I don't really view this as the same issue as NIFI-1922... even if the > solution to NIFI-1922 was to directly bundle the Azure/Wasb JARs in > NiFi, we would still have to bundle the JARs that are compatible with > the Hadoop client we are using, which is currently 2.7.3. > > In the future when we have an extension registry, we could presumably > publish variations of the nifi-hadoop-nar + nifi-hadoop-libraries-nar > built against different versions of the Hadoop client (2.7.x, 2.8.x, > HDP, CDH, MapR, etc), and with the component versioning work currently > going on in master, it would be easy for people to run as many of > these in parallel as they want. > > For now I think the easiest thing to do is maintain your own build of > nifi-hadoop-libraries-nar by changing the the version mentioned above. > > At some point the NiFi community will likely move to a newer Hadoop > client as they come out (we fairly recently moved from 2.6.x to > 2.7.x), but this a bigger decision that depends on how stable the > client is and what (if any) ramifications it has for compatibility. > > Thanks, > > Bryan > > > > On Tue, Apr 4, 2017 at 11:20 AM, Giovanni Lanzani > <[email protected]> wrote: > > Hi Brian, > > > > Thanks for the reply. > > > > Is there a way to compile NiFi using the Hadoop 2.8.0 libraries? > > > > It's of course unfortunate, but the libraries you mentioned before works > in their very specific version. Once you use a newer version (like > azure-storage-2.2.0) then things seem to break. > > > > Maybe this jira [^1] could be reopened then? 😊 > > > > Cheers, > > > > Giovanni > > > > [^1]: https://issues.apache.org/jira/browse/NIFI-1922 > > > >> -----Original Message----- > >> From: Bryan Bende [mailto:[email protected]] > >> Sent: Tuesday, April 4, 2017 3:59 PM > >> To: [email protected] > >> Subject: Re: GetHDFS from Azure Blob > >> > >> Giovanni, > >> > >> I'm not that familiar with using a key provider, but NiFi currently > bundles the > >> Hadoop 2.7.3 client, and looking at ProviderUtils from 2.7.3, there > doesn't > >> appear to be a method > >> "excludeIncompatibleCredentialProviders": > >> > >> https://github.com/apache/hadoop/blob/release-2.7.3-RC2/hadoop-common- > >> project/hadoop- > >> common/src/main/java/org/apache/hadoop/security/ProviderUtils.java > >> > >> It looks like it is introduced in 2.8.0: > >> > >> https://github.com/apache/hadoop/blob/release-2.8.0-RC3/hadoop-common- > >> project/hadoop- > >> common/src/main/java/org/apache/hadoop/security/ProviderUtils.java > >> > >> Most likely some code that is present in one of the JARs specified > through > >> Additional Resources is dependent on Hadoop 2.8.0, and since NiFi is > bundling > >> 2.7.3, there are some things not lining up. > >> > >> -Bryan > >> > >> > >> On Tue, Apr 4, 2017 at 9:50 AM, Giovanni Lanzani > >> <[email protected]> wrote: > >> > Bryan, > >> > > >> > Allow me to chime in (to ask for help). > >> > > >> > What about when I'm using an encrypted key? > >> > > >> > In my case I have (in core-site.xml) > >> > > >> > <property> > >> > > >> <name> > fs.azure.account.keyprovider.nsanalyticsstorage.blob.core.windows.ne > >> t</name> > >> > > <value>org.apache.hadoop.fs.azure.ShellDecryptionKeyProvider</value> > >> > </property> > >> > > >> > Everything works from the command line (hdfs dfs). > >> > > >> > But NiFi complains with: > >> > > >> > java.lang.NoSuchMethodError: > >> > org.apache.hadoop.security.ProviderUtils.excludeIncompatibleCredential > >> > Providers > >> > > >> > Any ideas? I've already linked hadoop-commons.jar as well (besides > what you > >> suggested below). > >> > > >> > Cheers, > >> > > >> > Giovanni > >> > > >> > > >> >> -----Original Message----- > >> >> From: Bryan Bende [mailto:[email protected]] > >> >> Sent: Tuesday, March 28, 2017 7:41 PM > >> >> To: [email protected] > >> >> Subject: Re: GetHDFS from Azure Blob > >> >> > >> >> Austin, > >> >> > >> >> Can you provide the full error message and stacktrace for the > >> >> IllegalArgumentException from nifi-app.log? > >> >> > >> >> When you start the processor it creates a FileSystem instance based > >> >> on the config files provided to the processor, which in turn causes > >> >> all of the corresponding classes to load. > >> >> > >> >> I'm not that familiar with Azure, but if "Azure blob store" is WASB, > >> >> then I have successfully done the following... > >> >> > >> >> In core-site.xml: > >> >> > >> >> <configuration> > >> >> > >> >> <property> > >> >> <name>fs.defaultFS</name> > >> >> <value>wasb://YOUR_USER@YOUR_HOST/</value> > >> >> </property> > >> >> > >> >> <property> > >> >> <name>fs.azure.account.key.nifi.blob.core.windows.net</name> > >> >> <value>YOUR_KEY</value> > >> >> </property> > >> >> > >> >> <property> > >> >> <name>fs.AbstractFileSystem.wasb.impl</name> > >> >> <value>org.apache.hadoop.fs.azure.Wasb</value> > >> >> </property> > >> >> > >> >> <property> > >> >> <name>fs.wasb.impl</name> > >> >> <value>org.apache.hadoop.fs.azure.NativeAzureFileSystem</value> > >> >> </property> > >> >> > >> >> <property> > >> >> <name>fs.azure.skip.metrics</name> > >> >> <value>true</value> > >> >> </property> > >> >> > >> >> </configuration> > >> >> > >> >> In Additional Resources property of an HDFS processor, point to a > >> >> directory > >> >> with: > >> >> > >> >> azure-storage-2.0.0.jar > >> >> commons-codec-1.6.jar > >> >> commons-lang3-3.3.2.jar > >> >> commons-logging-1.1.1.jar > >> >> guava-11.0.2.jar > >> >> hadoop-azure-2.7.3.jar > >> >> httpclient-4.2.5.jar > >> >> httpcore-4.2.4.jar > >> >> jackson-core-2.2.3.jar > >> >> jsr305-1.3.9.jar > >> >> slf4j-api-1.7.5.jar > >> >> > >> >> > >> >> Thanks, > >> >> > >> >> Bryan > >> >> > >> >> > >> >> On Tue, Mar 28, 2017 at 1:15 PM, Austin Heyne <[email protected]> > wrote: > >> >> > Hi all, > >> >> > > >> >> > Thanks for all the help you've given me so far. Today I'm trying to > >> >> > pull files from an Azure blob store. I've done some reading on this > >> >> > and from previous tickets [1] and guides [2] it seems the > >> >> > recommended approach is to place the required jars, to use the HDFS > >> >> > Azure protocol, in 'Additional Classpath Resoures' and the hadoop > >> >> > core-site and hdfs-site configs into the 'Hadoop Configuration > >> >> > Resources'. I have my local HDFS properly configured to access wasb > >> >> > urls. I'm able to ls, > >> >> copy to and from, etc with out problem. > >> >> > Using the same HDFS config files and trying both all the jars in my > >> >> > hadoop-client/lib directory (hdp) and using the jars recommend in > >> >> > [1] I'm still seeing the "java.lang.IllegalArgumentException: > Wrong FS: " > >> >> > error in my NiFi logs and am unable to pull files from Azure blob > storage. > >> >> > > >> >> > Interestingly, it seems the processor is spinning up way to fast, > >> >> > the errors appear in the log as soon as I start the processor. I'm > >> >> > not sure how it could be loading all of those jars that quickly. > >> >> > > >> >> > Does anyone have any experience with this or recommendations to > try? > >> >> > > >> >> > Thanks, > >> >> > Austin > >> >> > > >> >> > [1] https://issues.apache.org/jira/browse/NIFI-1922 > >> >> > [2] > >> >> > > https://community.hortonworks.com/articles/71916/connecting-to-azur > >> >> > e-d > >> >> > ata-lake-from-a-nifi-dataflow.html > >> >> > > >> >> > >
