I am setting up a 2 nodes hadoop cluster with security, according to the "Hadoop Security Guide" of Cloudera (https://ccp.cloudera.com/display/CDHDOC/CDH3+Security+Guide).
Version info. --------------- hadoop-1.0.0 kerberos-1.8.5 JDK 1.6.0_29 Redhat rhel 4u3 Cluster info. --------------- host "slavez.demodomain" as NameNode host "slavey.demodomain" as DataNode host "master.dedomain" as Kerberos KDC and NTP server (1) Firstly, the NameNode slavez starts, the DataNode slavez can join in, and the HDFS works well. ( I am able to put files from local fs into the HDFS, and to modify file attributes.) (2) Then I starts the JobTracker at slavez, ant it begins waiting for RPC from TaskTracker. (3) But when I starts the TaskTracker at the same node, both the JobTracker and the TaskTracker give out a "security.UserGroupInformation" error for user mapred/slavez.demodomain. The following are detail logs and any suggestion is appreciated! ---------------------------------------------------------------------------------------------- JobTracker starts ---------------------------------------------------------------------------------------------- [mapred@slavez hadoop-1.0.0]$ ./bin/hadoop jobtracker 12/02/10 18:56:53 INFO mapred.JobTracker: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting JobTracker STARTUP_MSG: host = slavez.demodomain/192.168.0.14 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.0 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1214675; compiled by 'hortonfo' on Thu Dec 15 16:36:35 UTC 2011 ************************************************************/ 12/02/10 18:56:53 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 12/02/10 18:56:53 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 12/02/10 18:56:53 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 12/02/10 18:56:53 INFO impl.MetricsSystemImpl: JobTracker metrics system started 12/02/10 18:56:53 INFO impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered. 12/02/10 18:56:54 INFO impl.MetricsSourceAdapter: MBean for source ugi registered. 12/02/10 18:56:54 INFO security.UserGroupInformation: Login successful for user mapred/slavez.demodom...@hadoop.demo.com using keytab file /etc/hadoop/conf/mapred.keytab 12/02/10 18:56:54 INFO delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens 12/02/10 18:56:54 INFO mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, - 1) 12/02/10 18:56:54 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list 12/02/10 18:56:54 INFO delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) 12/02/10 18:56:54 INFO delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens 12/02/10 18:56:54 INFO mapred.JobTracker: Starting jobtracker with owner as mapred 12/02/10 18:56:54 INFO impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9001 registered. 12/02/10 18:56:54 INFO impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9001 registered. 12/02/10 18:56:54 INFO ipc.Server: Starting SocketReader 12/02/10 18:56:54 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter (org.mortbay.log) via org.mortbay.log.Slf4jLog 12/02/10 18:56:55 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 12/02/10 18:56:55 INFO http.HttpServer: Port returned by webServer.getConnectors() [0].getLocalPort() before open() is -1. Opening the listener on 50030 12/02/10 18:56:55 INFO http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030 12/02/10 18:56:55 INFO http.HttpServer: Jetty bound to port 50030 12/02/10 18:56:55 INFO mortbay.log: jetty-6.1.26 12/02/10 18:56:55 WARN mortbay.log: Can't reuse /tmp/Jetty_0_0_0_0_50030_job____yn7qmk, using /tmp/Jetty_0_0_0_0_50030_job____yn7qmk_2577057088123560744 12/02/10 18:56:55 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:50030 12/02/10 18:56:55 INFO impl.MetricsSourceAdapter: MBean for source jvm registered. 12/02/10 18:56:55 INFO impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered. 12/02/10 18:56:55 INFO mapred.JobTracker: JobTracker up at: 9001 12/02/10 18:56:55 INFO mapred.JobTracker: JobTracker webserver: 50030 12/02/10 18:56:55 INFO mapred.JobTracker: Cleaning up the system directory 12/02/10 18:56:55 WARN fs.FileSystem: "slavez.demodomain" is a deprecated filesystem name. Use "hdfs://slavez.demodomain/" instead. 12/02/10 18:56:55 WARN fs.FileSystem: "slavez.demodomain" is a deprecated filesystem name. Use "hdfs://slavez.demodomain/" instead. 12/02/10 18:56:55 INFO mapred.JobTracker: History server being initialized in embedded mode 12/02/10 18:56:55 WARN fs.FileSystem: "slavez.demodomain" is a deprecated filesystem name. Use "hdfs://slavez.demodomain/" instead. 12/02/10 18:56:55 WARN fs.FileSystem: "slavez.demodomain" is a deprecated filesystem name. Use "hdfs://slavez.demodomain/" instead. 12/02/10 18:56:55 INFO mapred.JobHistoryServer: Started job history server at: localhost:50030 12/02/10 18:56:55 INFO mapred.JobTracker: Job History Server web address: localhost:50030 12/02/10 18:56:55 INFO mapred.CompletedJobStatusStore: Completed job store is inactive 12/02/10 18:56:55 INFO mapred.JobTracker: Refreshing hosts information 12/02/10 18:56:55 INFO util.HostsFileReader: Setting the includes file to 12/02/10 18:56:55 INFO util.HostsFileReader: Setting the excludes file to 12/02/10 18:56:55 INFO util.HostsFileReader: Refreshing hosts (include/exclude) list 12/02/10 18:56:55 INFO mapred.JobTracker: Decommissioning 0 nodes 12/02/10 18:56:55 INFO ipc.Server: IPC Server Responder: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server listener on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 0 on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 1 on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 2 on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 3 on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 4 on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 5 on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 6 on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 7 on 9001: starting 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 8 on 9001: starting 12/02/10 18:56:55 INFO mapred.JobTracker: Starting RUNNING 12/02/10 18:56:55 INFO ipc.Server: IPC Server handler 9 on 9001: starting (begin waiting) ---------------------------------------------------------------------------------------------- TaskTracker starts ---------------------------------------------------------------------------------------------- [mapred@slavez hadoop-1.0.0]$ ./bin/hadoop tasktracker 12/02/10 18:57:09 INFO mapred.TaskTracker: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting TaskTracker STARTUP_MSG: host = slavez.demodomain/192.168.0.14 STARTUP_MSG: args = [] STARTUP_MSG: version = 1.0.0 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1214675; compiled by 'hortonfo' on Thu Dec 15 16:36:35 UTC 2011 ************************************************************/ 12/02/10 18:57:09 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 12/02/10 18:57:09 INFO impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered. 12/02/10 18:57:09 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 12/02/10 18:57:09 INFO impl.MetricsSystemImpl: TaskTracker metrics system started 12/02/10 18:57:10 INFO impl.MetricsSourceAdapter: MBean for source ugi registered. 12/02/10 18:57:10 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter (org.mortbay.log) via org.mortbay.log.Slf4jLog 12/02/10 18:57:10 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 12/02/10 18:57:11 INFO mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 12/02/10 18:57:11 WARN security.UserGroupInformation: Terminating renewal thread 12/02/10 18:57:11 INFO security.UserGroupInformation: Asked the TGT renewer thread to terminate 12/02/10 18:57:11 INFO security.UserGroupInformation: Login successful for user mapred/sla...@hadoop.demo.com using keytab file /etc/hadoop/conf/mapred.keytab 12/02/10 18:57:11 INFO mapred.TaskTracker: Starting tasktracker with owner as mapred 12/02/10 18:57:11 INFO mapred.TaskTracker: Good mapred local directories are: /home/hadoop0209/hadoop-cluster/mapred/local 12/02/10 18:57:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 12/02/10 18:57:11 INFO impl.MetricsSourceAdapter: MBean for source jvm registered. 12/02/10 18:57:11 INFO impl.MetricsSourceAdapter: MBean for source TaskTrackerMetrics registered. 12/02/10 18:57:11 INFO ipc.Server: Starting SocketReader 12/02/10 18:57:11 INFO impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort32958 registered. 12/02/10 18:57:11 INFO impl.MetricsSourceAdapter: MBean for source RpcActivityForPort32958 registered. 12/02/10 18:57:11 INFO ipc.Server: IPC Server Responder: starting 12/02/10 18:57:11 INFO ipc.Server: IPC Server listener on 32958: starting 12/02/10 18:57:11 INFO ipc.Server: IPC Server handler 0 on 32958: starting 12/02/10 18:57:11 INFO ipc.Server: IPC Server handler 1 on 32958: starting 12/02/10 18:57:11 INFO ipc.Server: IPC Server handler 2 on 32958: starting 12/02/10 18:57:11 INFO mapred.TaskTracker: TaskTracker up at: localhost.localdomain/127.0.0.1:32958 12/02/10 18:57:11 INFO mapred.TaskTracker: Starting tracker tracker_slavez:localhost.localdomain/127.0.0.1:32958 12/02/10 18:57:11 INFO ipc.Server: IPC Server handler 3 on 32958: starting 12/02/10 18:57:11 ERROR security.UserGroupInformation: PriviledgedActionException as:mapred/sla...@hadoop.demo.com cause:org.apache.hadoop.ipc.RemoteException: Failure to initialize security context 12/02/10 18:57:11 INFO security.UserGroupInformation: Initiating logout for mapred/sla...@hadoop.demo.com 12/02/10 18:57:11 INFO security.UserGroupInformation: Initiating re-login for mapred/sla...@hadoop.demo.com 12/02/10 18:57:13 INFO mapred.TaskTracker: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down TaskTracker at slavez.demodomain/192.168.0.14 ************************************************************/ ---------------------------------------------------------------------------------------------- JobTracker response ---------------------------------------------------------------------------------------------- 12/02/10 18:57:11 ERROR security.UserGroupInformation: PriviledgedActionException as:mapred/slavez.demodom...@hadoop.demo.com cause:javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Key)] 12/02/10 18:57:11 INFO ipc.Server: IPC Server listener on 9001: readAndProcess threw exception javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Key)]. Count of bytes read: 0 javax.security.sasl.SaslException: Failure to initialize security context [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Key)] at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:95) at com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:67) at javax.security.sasl.Sasl.createSaslServer(Sasl.java:491) at org.apache.hadoop.ipc.Server$Connection$1.run(Server.java:989) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs (UserGroupInformation.java:1083) at org.apache.hadoop.ipc.Server$Connection.saslReadAndProcess(Server.java:986) at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1180) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:537) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:344) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask (ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Key) at sun.security.jgss.krb5.Krb5AcceptCredential.getInstance (Krb5AcceptCredential.java:95) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement (Krb5MechFactory.java:111) at sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:178) at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:384) at sun.security.jgss.GSSCredentialImpl.<init>(GSSCredentialImpl.java:42) at sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:139) at com.sun.security.sasl.gsskerb.GssKrb5Server.<init>(GssKrb5Server.java:78) ... 13 more 12/02/10 18:57:22 INFO mapred.JobTracker: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down JobTracker at slavez.demodomain/192.168.0.14 ************************************************************/ -- Ian Jiang
