Hello,
I'm trying to build a groovy excuteScript to make some actions on hadoop
hdfs. I follow this answer, from Matt Burgess to a similar question
(https://community.hortonworks.com/questions/47493/nifi-executescript-using-external-libarries-with-g.html),
but with no success.
I test with the script below, using in Module Directory, the classes from nifi:
/apps/nifi-1.9.2/work/nar/extensions/nifi-hadoop-libraries-nar-1.9.2.nar-unpacked/NAR-INF/bundled-dependencies/
Always the error is:
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem
for scheme "hdfs"
at
org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3266)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3286)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3337)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3305)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:476)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:225)
at org.apache.hadoop.fs.FileSystem$get.call(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:128)
at Script15.run(Script15.groovy:45)
at
org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:321)
in the instruction: fs = FileSystem.get(conf), after receive a successful
kerberos login.
The script works, adding
@Grab('org.apache.hadoop#hadoop-client;3.0.0')
@Grab('org.apache.hadoop#hadoop-hdfs;3.0.0')
without the nifi stuff, directly in the file system, but in Nifi if I use The
Grab Instructions the error is the same.
The nifi script:
import groovy.json.JsonSlurper
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.FileSystem
import org.apache.hadoop.fs.Path
import org.apache.hadoop.security.AccessControlException
import org.apache.hadoop.security.UserGroupInformation
dir = "/data/table1"
host = "xpto.xpto"
hdfs = "hdfs://" + host + ":8020/"
try{
conf = new Configuration()
conf.setClassLoader(Thread.currentThread().getContextClassLoader());
conf.addResource(new
Path("/apps/nifi-resources/hive-conf/core-site.xml"));
conf.addResource(new
Path("/apps/nifi-resources/hive-conf/hdfs-site.xml"));
conf.addResource(new
Path("/apps/nifi-resources/hive-conf/hive-site.xml"));
//conf.set("fs.default.name", hdfs)
conf.set("hadoop.security.authentication", "kerberos");
UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("[email protected]<mailto:[email protected]>",
"/apps/nifi-resources/xpto.keytab");
fs = FileSystem.get(conf)
status = fs.listStatus( new Path( dir ) )
status.each { f -> log.info(f.getPath().toString())}
session.transfer(flowFile, REL_SUCCESS)
}catch (AccessControlException e){
println("permission denied")
session.transfer(flowFile, REL_FAILURE)
}
Any help?
Thanks
Rui Ferreira
