FW: ExecuteScript to list HDFS

[Rui Pedro Ferreira (DSI)]

Hello,

I'm trying to  build a groovy excuteScript to make  some actions on hadoop 
hdfs.  I  follow this answer, from Matt Burgess to   a similar question 
(https://community.hortonworks.com/questions/47493/nifi-executescript-using-external-libarries-with-g.html),
 but with no success.

I test with the script below, using in Module Directory, the classes from nifi:
/apps/nifi-1.9.2/work/nar/extensions/nifi-hadoop-libraries-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/

Always the error is:
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem 
for scheme "hdfs"
        at 
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3266)
        at 
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3286)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
        at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:225)
        at org.apache.hadoop.fs.FileSystem$get.call(Unknown Source)
        at 
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
        at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at 
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
        at Script15.run(Script15.groovy:45)
        at 
org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:321)

in the instruction: fs = FileSystem.get(conf), after receive a successful 
kerberos login.


The script works, adding
@Grab('org.apache.hadoop#hadoop-client;3.0.0')
@Grab('org.apache.hadoop#hadoop-hdfs;3.0.0')
without the nifi stuff,  directly in the file system, but in Nifi if I use The 
Grab Instructions the error is the same.

The nifi script:

import groovy.json.JsonSlurper
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
import org.apache.hadoop.security.AccessControlException
import org.apache.hadoop.security.UserGroupInformation

dir = "/data/table1"
host = "xpto.xpto"
hdfs = "hdfs://" + host + ":8020/"
try{
        conf = new Configuration()
        conf.setClassLoader(Thread.currentThread().getContextClassLoader());
        conf.addResource(new 
Path("/apps/nifi-resources/hive-conf/core-site.xml"));
        conf.addResource(new 
Path("/apps/nifi-resources/hive-conf/hdfs-site.xml"));
        conf.addResource(new 
Path("/apps/nifi-resources/hive-conf/hive-site.xml"));

        //conf.set("fs.default.name", hdfs)
        conf.set("hadoop.security.authentication", "kerberos");

        UserGroupInformation.setConfiguration(conf);
        
UserGroupInformation.loginUserFromKeytab("x...@xpto.xpto<mailto:x...@xpto.xpto>",
 "/apps/nifi-resources/xpto.keytab");

        fs = FileSystem.get(conf)
        status = fs.listStatus( new Path( dir ) )
        status.each { f -> log.info(f.getPath().toString())}
        session.transfer(flowFile, REL_SUCCESS)

}catch (AccessControlException e){
        println("permission denied")
        session.transfer(flowFile, REL_FAILURE)

}

Any help?

Thanks

Rui Ferreira