Hi everyone,
I've spent many hours debugging this failure. I've written a small ORC
compactor using the Java libs. Everything works locally, but deployment to
cloud running in Docker is giving me Kerberos auth failures.
public FileSystem getHadoopFs() {
return new LocalFileSystem() {
@Override
public Configuration getConf() {
Configuration conf = new Configuration();
conf.set("hadoop.security.authorization", "false");
conf.set("hadoop.security.authentication", "simple");
return conf;
}
};
}
I've tried disabling Kerberos like so to no avail. Are there any
recommendations for how to go about this? This is just a standalone jar
running in Docker. Stack trace below.
Exception in thread "main"
org.apache.hadoop.security.KerberosAuthException: failure to login:
javax.security.auth.login.LoginException: java.lang.NullPointerException:
invalid null input: name
at
jdk.security.auth/com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:67)
at
jdk.security.auth/com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:129)
at
java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726)
at
java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665)
at
java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at
java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663)
at
java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574)
at
org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837)
at
org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at com.orc.compaction.utils.FileDump.getReader(FileDump.java:72)
at com.compaction.utils.FileDump.inferSchemaFromBinary(FileDump.java:61)
at com.orc.compaction.utils.FileDump.inferSchemaFromBinary(FileDump.java:47)
at com.orc.compaction.utils.FileDump.inferSchemaFromFiles(FileDump.java:39)
at
com.orc.compaction.CompactMonkey.retrieveWritersPerSchema(CompactMonkey.java:46)
at com,orc.compaction.CompactMonkey.processCompaction(CompactMonkey.java:69)
at com.orc.compaction.App.main(App.java:43)
at
org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847)
at
org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710)
at
org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660)
at
org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
My ORC writer is pretty standard.
public static Writer createOrcWriter(
Properties orcWriterProperties,
Configuration configuration,
Path orcOutputFile,
TypeDescription orcSchema,
FileSystem fs)
throws IOException {
OrcFile.WriterOptions opts =
OrcFile.writerOptions(orcWriterProperties, configuration)
.setSchema(orcSchema)
.useUTCTimestamp(true);
if (fs != null) {
fs.setConf(configuration);
opts.fileSystem(fs);
}
return OrcFile.createWriter(orcOutputFile, opts);
}
