My Hadoop 2.8.0's /mr-history/done
directory is owned by the mapred user, who is in the hadoop group, and the directory has the pemissions /mr-history":mapred:hadoop:drwxrwx--- If I run the Hadoop instance without any Kerberos config, and fire up the JobHistory server as the mapred user, everything works. If I flip over to a Kerberised environment, the NameNode and DataNodes, running as the 'hdfs' user, and the Resource and and Node Managers, running as the 'yarn' user, all start up OK and their respective web exposure can be used. When I try to start up the JobHistory server however /bin/su mapred -c '/local/Hadoop/hadoop-2.8.0/sbin/mr-jobhistory-daemon.sh --config /local/Hadoop/hadoop-2.8.0/etc/hadoop/ start historyserver I get a message in the logs telling me that, rather than the mapred user doing things, a user 'jhs' is trying to do stuff, vis 2017-07-20 18:15:09,667 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT] 2017-07-20 18:15:10,062 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user jhs/co246a-9.ecs.vuw.ac...@ecs.vuw.ac.nz using keytab file /local/Hadoop/krb/jhs.service.keytab 2017-07-20 18:15:10,107 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2017-07-20 18:15:10,142 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2017-07-20 18:15:10,142 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobHistoryServer metrics system started 2017-07-20 18:15:10,145 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init 2017-07-20 18:15:10,411 INFO org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://co246a-a.ecs.vuw.ac.nz:9000] 2017-07-20 18:15:10,518 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done] org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error creating done directory: [hdfs://co246a-a.ecs.vuw.ac.nz:9000/mr-history/done] at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:639) at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:585) at org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:550) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:95) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:151) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:231) at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:241) Caused by: org.apache.hadoop.security.AccessControlException: Permission denied: user=jhs, access=EXECUTE, inode="/mr-history":mapred:hadoop:drwxrwx--- But where has the jhs user come from ? Doesn't appear to be set anywhere in any of the config files. According to the hadoop-2.8.0 docs SecureMode page, https://hadoop.apache.org/docs/r2.8.0/hadoop-project-dist/hadoop-common/SecureMode.html ============================================= MapReduce JobHistory Server The MapReduce JobHistory Server keytab file, on that host, should look like the following: $ klist -e -k -t /etc/security/keytab/jhs.service.keytab Keytab name: FILE:/etc/security/keytab/jhs.service.keytab KVNO Timestamp Principal 4 07/18/11 21:08:09 jhs/full.qualified.domain.n...@realm.tld (AES-256 CTS mode with 96-bit SHA-1 HMAC) 4 07/18/11 21:08:09 jhs/full.qualified.domain.n...@realm.tld (AES-128 CTS mode with 96-bit SHA-1 HMAC) 4 07/18/11 21:08:09 jhs/full.qualified.domain.n...@realm.tld (ArcFour with HMAC/md5) 4 07/18/11 21:08:09 host/full.qualified.domain.n...@realm.tld (AES-256 CTS mode with 96-bit SHA-1 HMAC) 4 07/18/11 21:08:09 host/full.qualified.domain.n...@realm.tld (AES-128 CTS mode with 96-bit SHA-1 HMAC) 4 07/18/11 21:08:09 host/full.qualified.domain.n...@realm.tld (ArcFour with HMAC/md5) ============================================= and mine does. The hadoop-2.8.0 docs SecureMode page also suggests that one would need to play around with the hadoop.security.auth_to_local config value, but I haven't had to do that for the nn, dn, rm or nm keytabs. So is there something special about the jhs user ? Or perhaps something special about the other keytab values ? Any clues/insight welcome, Kevin --- Kevin M. Buckley eScience Consultant School of Engineering and Computer Science Victoria University of Wellington New Zealand --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org