When implementing Nutch 1.0 awhile back, we had to point our scripts to
/opt/freeware/bin to allow Nutch's Hadoop code to utilize the more
Linux-like versions of various system commands (like df) in order for
Hadoop to function properly on AIX. After doing that, we've had no issues
with Nutch's hadoop implementation from 1.0 through 1.4.
Now I'm attempting up migrate my existing, functional configurations from
my Nutch 1.4 installation to Nutch 1.5.1, which is now using Hadoop 1.0.3.
Now when I attempt to run a crawl, I'm getting these errors that seem to
be coming from Hadoop, and seem to want to use a LinuxLoginModule class
(see below)
Is there some configuration setting, plugin or jar file, etc that is
missing from the new version to make this all work on AIX again?
2012-08-02 20:03:21,271 ERROR crawl.Injector - Injector:
java.lang.RuntimeException: java.io.IOException: failure to login
at
org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:546)
at
org.apache.hadoop.mapred.FileInputFormat.addInputPath(FileInputFormat.java:336)
at org.apache.nutch.crawl.Injector.inject(Injector.java:209)
at org.apache.nutch.crawl.Injector.run(Injector.java:248)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Injector.main(Injector.java:238)
Caused by: java.io.IOException: failure to login
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:490)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:452)
at
org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1494)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1395)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
at
org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:542)
... 5 more
Caused by: javax.security.auth.login.LoginException: unable to find
LoginModule class: com.ibm.security.auth.module.LinuxLoginModule
at
javax.security.auth.login.LoginContext.invoke(LoginContext.java:834)
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:209)
at
javax.security.auth.login.LoginContext$5.run(LoginContext.java:732)
at
java.security.AccessController.doPrivileged(AccessController.java:284)
at
javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:729)
at
javax.security.auth.login.LoginContext.login(LoginContext.java:599)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:471)
... 11 more
James
