Just following up. I've added the core-default and core-site defaults from hadoop onto the classpath with explicit overrides for kerberos and I seemed to get around this issue.
The downside is, running this on Docker on Kubernetes, the localfilesystem can't seem to find the path I'm trying to write to. Still debugging. On Fri, Apr 16, 2021 at 3:55 AM Ryan Schachte <[email protected]> wrote: > Hi everyone, > > I've spent many hours debugging this failure. I've written a small ORC > compactor using the Java libs. Everything works locally, but deployment to > cloud running in Docker is giving me Kerberos auth failures. > > public FileSystem getHadoopFs() { > return new LocalFileSystem() { > @Override > public Configuration getConf() { > Configuration conf = new Configuration(); > conf.set("hadoop.security.authorization", "false"); > conf.set("hadoop.security.authentication", "simple"); > return conf; > } > }; > } > > > I've tried disabling Kerberos like so to no avail. Are there any > recommendations for how to go about this? This is just a standalone jar > running in Docker. Stack trace below. > > Exception in thread "main" > org.apache.hadoop.security.KerberosAuthException: failure to login: > javax.security.auth.login.LoginException: java.lang.NullPointerException: > invalid null input: name > at > jdk.security.auth/com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:67) > at > jdk.security.auth/com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:129) > at > java.base/javax.security.auth.login.LoginContext.invoke(LoginContext.java:726) > at > java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:665) > at > java.base/javax.security.auth.login.LoginContext$4.run(LoginContext.java:663) > at java.base/java.security.AccessController.doPrivileged(Native Method) > at > java.base/javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:663) > at > java.base/javax.security.auth.login.LoginContext.login(LoginContext.java:574) > at > org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:1926) > at > org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1837) > at > org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571) > at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487) > at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) > at com.orc.compaction.utils.FileDump.getReader(FileDump.java:72) > at com.compaction.utils.FileDump.inferSchemaFromBinary(FileDump.java:61) > at > com.orc.compaction.utils.FileDump.inferSchemaFromBinary(FileDump.java:47) > at com.orc.compaction.utils.FileDump.inferSchemaFromFiles(FileDump.java:39) > at > com.orc.compaction.CompactMonkey.retrieveWritersPerSchema(CompactMonkey.java:46) > at > com,orc.compaction.CompactMonkey.processCompaction(CompactMonkey.java:69) > at com.orc.compaction.App.main(App.java:43) > > at > org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:1847) > at > org.apache.hadoop.security.UserGroupInformation.createLoginUser(UserGroupInformation.java:710) > at > org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:660) > at > org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:571) > at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3487) > at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:3477) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3319) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:227) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:463) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) > > My ORC writer is pretty standard. > > public static Writer createOrcWriter( > Properties orcWriterProperties, > Configuration configuration, > Path orcOutputFile, > TypeDescription orcSchema, > FileSystem fs) > throws IOException { > OrcFile.WriterOptions opts = > OrcFile.writerOptions(orcWriterProperties, configuration) > .setSchema(orcSchema) > .useUTCTimestamp(true); > > if (fs != null) { > fs.setConf(configuration); > opts.fileSystem(fs); > } > return OrcFile.createWriter(orcOutputFile, opts); > } > > > > > > > > > > > > > > >
