Looking further @ the resource manager log, the resource manager starts and goes in standby Then it starts IPC Server and IPC Client ( presumably to talk to Zookeeper to elect a active v/s standby Resource Manager). The IPC Server seem to start and then when IPC client tries to connect to server, it seem to detect that there is no kerberos principal defined (for server?). Not sure where is such kerberos principal is defined for IPC client / server that seem to be used for ZooKeeper ?
2014-03-26 01:13:31,851 INFO org.apache.hadoop.security.UserGroupInformation: Login successful for user yarn/[email protected] using keytab file /etc/hadoop/conf/yarn.keytab 2014-03-26 01:13:31,851 INFO org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService: Transitioning to standby 2014-03-26 01:13:31,851 INFO org.apache.hadoop.yarn.server.resourcemanager.RMHAProtocolService: Transitioned to standby 2014-03-26 01:13:31,928 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 23142 2014-03-26 01:13:31,996 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2014-03-26 01:13:31,997 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 23142: starting 2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-cdh5.0.0-beta-1--1, built on 10/28/2013 00:00 GMT 2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=xxxxxx 2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.7.0_51 2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2014-03-26 01:13:32,019 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_51/jre Zookeeper client start command ..... 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/hadoop/lib/native 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.name=Linux 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.arch=amd64 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.6.2.el6.x86_64 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.name=yarn 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.home=/var/lib/hadoop-yarn 2014-03-26 01:13:32,020 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/usr/lib/hadoop-yarn 2014-03-26 01:13:32,021 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=xxxxx:2181,yyyy:2181,zzzz:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@26bc6487 2014-03-26 01:13:32,038 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server xxxx FQDN/xxxx. Will not attempt to authenticate using SASL (unknown error) 2014-03-26 01:13:32,050 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to xxxx/xxxx:2181, initiating session 2014-03-26 01:13:32,060 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server xxxx/xxxx:2181, sessionid = 0x144fbf1d0350000, negotiated timeout = 5000 2014-03-26 01:13:32,076 WARN org.apache.hadoop.ha.ActiveStandbyElector: Ignoring stale result from old client with sessionId 0x144fbf1d0350000 2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Session: 0x144fbf1d0350000 closed 2014-03-26 01:13:32,080 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=xxxx:2181,yyyy:2181,zzzz:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@74c35f27 2014-03-26 01:13:32,081 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down 2014-03-26 01:13:32,082 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server xxxx/xxxx:2181. Will not attempt to authenticate using SASL (unknown error) 2014-03-26 01:13:32,083 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to xxxx/xxxx:2181, initiating session 2014-03-26 01:13:32,088 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server xxxx/xxxx:2181, sessionid = 0x344fbf1fe860001, negotiated timeout = 5000 2014-03-26 01:13:32,095 INFO org.apache.hadoop.ha.ActiveStandbyElector: Session connected. 2014-03-26 01:13:32,096 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8035 2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2014-03-26 01:13:32,161 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8035: starting 2014-03-26 01:13:32,241 WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name 2014-03-26 01:13:32,242 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:yarn/[email protected] (auth:KERBEROS) cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name On Tue, Mar 25, 2014 at 4:55 PM, Manoj Samel <[email protected]>wrote: > Hi, > > Hadoop version is CDH5 Beta1 > > Name node and Resource managers have been configured in HA mode. > > After kerberos is enabled, the resource manager log shows following error > > 2014-03-25 22:21:06,854 WARN org.apache.hadoop.ipc.Client: Exception > encountered while connecting to the server : > java.lang.IllegalArgumentException: Failed to specify server's Kerberos > principal name > 2014-03-25 22:21:06,855 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:yarn/[email protected] (auth:KERBEROS) > cause:java.io.IOException: java.lang.IllegalArgumentException: Failed to > specify server's Kerberos principal name > > The keytab files for hdfs, mapred and yarn are (seemingly) correct and > show correct list of principals using klist. > > Any thoughts, > > Thanks, >
