Shawn, You are right, the issue is that the JVM only allows a native library to be loaded once, and furthermore if you have native methods that need to be linked to the native library, the native code must be in the same class loader where the native library is loaded (at least from my testing).
What this means is that we can't even load the snappy native library in the system class loader, because then the native methods in Hadoop client will get an unsatisfied link error since they are down in a child class loader. Technically this issue exists even without instance class loading, all it would take is two different NARs that both loaded Hadoop native libraries, its just that instance class loading made it more noticeable because now each HDFS processor has its own class loader. Before all the HDFS processors came from the NAR class loader, so as long as another NAR didn't also load the native libraries, then everything worked. The challenge is that without the instance class loading there have been other issues due to the use of static state in the Hadoop client code. Currently the HDFS processors rely on the static login method of UserGroupInformation when kerberos is enabled, so in order to have multiple HDFS processors with different principals, there has to be instance class loading, otherwise they would overwrite each other while the application is running. -Bryan On Sun, Mar 25, 2018 at 11:34 AM, Shawn Weeks <swe...@weeksconsulting.us> wrote: > Figured I’d throw this out there as I’ve been tracking down this issue for a > couple of weeks now. With the introduction of instance based class loaders > for Hadoop Processors you can no longer using native compression codecs like > Snappy in HDFS Processors like GetHDFS and PutHDFS. This is from what I can > tell caused by the JVM only allowing native libraries to be loaded once > preventing them from being loaded by the instanced class loader. As a work > around the CompressContent Processor works just fine. Here is an example > template demonstrating this behavior for anyone interested. > > > > https://raw.githubusercontent.com/weeksconsulting/nifi-templates/master/Test_Native.xml > > > > Thanks > > Shawn Weeks
