Hi,
I had setup Hadoop HA Cluster(QJM) with kerberos. Cluster has three journal
nodes. Namenode is able to connectto journal node running on the Namenode. For
other Journal nodes its not able to connect and throws below exception.Can you
please provide inputs for resolving the issue.
2016-05-25 23:09:47,526 INFO org.apache.hadoop.hdfs.server.namenode.NameNode:
STARTUP_MSG:/************************************************************STARTUP_MSG:
Starting NameNodeSTARTUP_MSG: host =
server1.cehdev.com/10.67.169.45STARTUP_MSG: args = []STARTUP_MSG: version =
2.5.2STARTUP_MSG: classpath = STARTUP_MSG: build =
https://git-wip-us.apache.org/repos/asf/hadoop.git -r
cc72e9b000545b86b75a61f4835eb86d57bfafc0; compiled by 'jenkins' on
2014-11-14T23:45ZSTARTUP_MSG: java =
1.7.0_79************************************************************/2016-05-25
23:09:47,550 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered
UNIX signal handlers for [TERM, HUP, INT]2016-05-25 23:09:47,557 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []2016-05-25
23:09:48,147 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded
properties from hadoop-metrics2.properties2016-05-25 23:09:48,307 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).2016-05-25 23:09:48,307 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
started2016-05-25 23:09:48,309 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is
hdfs://hdcluster2016-05-25 23:09:48,310 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use hdcluster
to access this namenode/service.2016-05-25 23:09:49,331 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for user
hdfs/[email protected] using keytab file
/etc/security/keytab/hdfs.service.keytab2016-05-25 23:09:49,392 INFO
org.apache.hadoop.hdfs.DFSUtil: Starting web server as:
HTTP/[email protected] 23:09:49,392 INFO
org.apache.hadoop.hdfs.DFSUtil: Starting Web-server for hdfs at:
http://server1.cehdev.com:500702016-05-25 23:09:49,480 INFO org.mortbay.log:
Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog2016-05-25 23:09:49,487 INFO
org.apache.hadoop.http.HttpRequestLog: Http request log for
http.requests.namenode is not defined2016-05-25 23:09:49,504 INFO
org.apache.hadoop.http.HttpServer2: Added global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)2016-05-25
23:09:49,510 INFO org.apache.hadoop.http.HttpServer2: Added filter
static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context hdfs2016-05-25 23:09:49,510 INFO org.apache.hadoop.http.HttpServer2:
Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs2016-05-25 23:09:49,511 INFO org.apache.hadoop.http.HttpServer2:
Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static2016-05-25 23:09:49,563 INFO org.apache.hadoop.http.HttpServer2:
Added filter 'org.apache.hadoop.hdfs.web.AuthFilter'
(class=org.apache.hadoop.hdfs.web.AuthFilter)2016-05-25 23:09:49,566 INFO
org.apache.hadoop.http.HttpServer2: addJerseyResourcePackage:
packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*2016-05-25 23:09:49,573 INFO
org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to
getDelegationToken2016-05-25 23:09:49,575 INFO
org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to
renewDelegationToken2016-05-25 23:09:49,577 INFO
org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to
cancelDelegationToken2016-05-25 23:09:49,579 INFO
org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to
fsck2016-05-25 23:09:49,581 INFO org.apache.hadoop.http.HttpServer2: Adding
Kerberos (SPNEGO) filter to imagetransfer2016-05-25 23:09:49,593 INFO
org.apache.hadoop.http.HttpServer2: Jetty bound to port 500702016-05-25
23:09:49,593 INFO org.mortbay.log: jetty-6.1.262016-05-25 23:09:49,954 INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Login using keytab /etc/security/keytab/hdfs.service.keytab, for principal
HTTP/[email protected] 23:09:49,965 WARN
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
'signature.secret' configuration not set, using a random value as
secret2016-05-25 23:09:49,967 INFO
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
Login using keytab /etc/security/keytab/hdfs.service.keytab, for principal
HTTP/[email protected] 23:09:49,973 WARN
org.apache.hadoop.security.authentication.server.AuthenticationFilter:
'signature.secret' configuration not set, using a random value as
secret2016-05-25 23:09:50,037 INFO org.mortbay.log: Started
HttpServer2$selectchannelconnectorwithsafestar...@server1.cehdev.com:500702016-05-25
23:09:50,084 WARN org.apache.hadoop.hdfs.server.common.Util: Path
/app/hadoop/dfs/name should be specified as a URI in configuration files.
Please update hdfs configuration.2016-05-25 23:09:50,085 WARN
org.apache.hadoop.hdfs.server.common.Util: Path /app/hadoop/dfs/name should be
specified as a URI in configuration files. Please update hdfs
configuration.2016-05-25 23:09:50,087 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
directory (dfs.namenode.name.dir) configured. Beware of data loss due to lack
of redundant storage directories!2016-05-25 23:09:50,095 WARN
org.apache.hadoop.hdfs.server.common.Util: Path /app/hadoop/dfs/name should be
specified as a URI in configuration files. Please update hdfs
configuration.2016-05-25 23:09:50,096 WARN
org.apache.hadoop.hdfs.server.common.Util: Path /app/hadoop/dfs/name should be
specified as a URI in configuration files. Please update hdfs
configuration.2016-05-25 23:09:50,153 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsLock is
fair:true2016-05-25 23:09:50,219 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=10002016-05-25 23:09:50,220 INFO
org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.namenode.datanode.registration.ip-hostname-check=true2016-05-25
23:09:50,223 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.namenode.startup.delay.block.deletion.sec is set to
000:00:00:00.0002016-05-25 23:09:50,223 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: The block deletion
will start around 2016 May 25 23:09:502016-05-25 23:09:50,226 INFO
org.apache.hadoop.util.GSet: Computing capacity for map BlocksMap2016-05-25
23:09:50,226 INFO org.apache.hadoop.util.GSet: VM type = 64-bit2016-05-25
23:09:50,228 INFO org.apache.hadoop.util.GSet: 2.0% max memory 889 MB = 17.8
MB2016-05-25 23:09:50,228 INFO org.apache.hadoop.util.GSet: capacity =
2^21 = 2097152 entries2016-05-25 23:09:50,235 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.token.enable=true2016-05-25 23:09:50,235 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
dfs.block.access.key.update.interval=600 min(s),
dfs.block.access.token.lifetime=600 min(s),
dfs.encrypt.data.transfer.algorithm=rc42016-05-25 23:09:50,245 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication
= 22016-05-25 23:09:50,246 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
= 5122016-05-25 23:09:50,246 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
= 12016-05-25 23:09:50,246 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
maxReplicationStreams = 22016-05-25 23:09:50,246 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
shouldCheckForEnoughRacks = false2016-05-25 23:09:50,246 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
replicationRecheckInterval = 30002016-05-25 23:09:50,246 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer
= true2016-05-25 23:09:50,246 INFO
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxNumBlocksToLog
= 10002016-05-25 23:09:50,247 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner =
hdfs/[email protected] (auth:KERBEROS)2016-05-25 23:09:50,247 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup =
hadoop2016-05-25 23:09:50,247 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
true2016-05-25 23:09:50,248 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Determined nameservice ID:
hdcluster2016-05-25 23:09:50,248 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: true2016-05-25
23:09:50,250 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append
Enabled: true2016-05-25 23:09:50,297 INFO org.apache.hadoop.util.GSet:
Computing capacity for map INodeMap2016-05-25 23:09:50,297 INFO
org.apache.hadoop.util.GSet: VM type = 64-bit2016-05-25 23:09:50,298 INFO
org.apache.hadoop.util.GSet: 1.0% max memory 889 MB = 8.9 MB2016-05-25
23:09:50,298 INFO org.apache.hadoop.util.GSet: capacity = 2^20 = 1048576
entries2016-05-25 23:09:50,299 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
more than 10 times2016-05-25 23:09:50,309 INFO org.apache.hadoop.util.GSet:
Computing capacity for map cachedBlocks2016-05-25 23:09:50,309 INFO
org.apache.hadoop.util.GSet: VM type = 64-bit2016-05-25 23:09:50,309 INFO
org.apache.hadoop.util.GSet: 0.25% max memory 889 MB = 2.2 MB2016-05-25
23:09:50,309 INFO org.apache.hadoop.util.GSet: capacity = 2^18 = 262144
entries2016-05-25 23:09:50,312 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.99900001287460332016-05-25 23:09:50,312
INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 02016-05-25 23:09:50,312 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension = 300002016-05-25 23:09:50,313 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on namenode is
enabled2016-05-25 23:09:50,314 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use 0.03
of total heap and retry cache entry expiry time is 600000 millis2016-05-25
23:09:50,317 INFO org.apache.hadoop.util.GSet: Computing capacity for map
NameNodeRetryCache2016-05-25 23:09:50,317 INFO org.apache.hadoop.util.GSet: VM
type = 64-bit2016-05-25 23:09:50,317 INFO org.apache.hadoop.util.GSet:
0.029999999329447746% max memory 889 MB = 273.1 KB2016-05-25 23:09:50,317 INFO
org.apache.hadoop.util.GSet: capacity = 2^15 = 32768 entries2016-05-25
23:09:50,323 INFO org.apache.hadoop.hdfs.server.namenode.NNConf: ACLs enabled?
false2016-05-25 23:09:50,323 INFO
org.apache.hadoop.hdfs.server.namenode.NNConf: XAttrs enabled? true2016-05-25
23:09:50,323 INFO org.apache.hadoop.hdfs.server.namenode.NNConf: Maximum size
of an xattr: 163842016-05-25 23:09:50,334 INFO
org.apache.hadoop.hdfs.server.common.Storage: Lock on
/app/hadoop/dfs/name/in_use.lock acquired by nodename
[email protected] 23:09:51,198 WARN
org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be
loaded2016-05-25 23:09:51,785 WARN org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
java.lang.IllegalArgumentException: Server has invalid Kerberos principal:
hdfs/[email protected] 23:09:51,786 WARN
org.apache.hadoop.ipc.Client: Exception encountered while connecting to the
server : java.lang.IllegalArgumentException: Server has invalid Kerberos
principal: hdfs/[email protected] 23:09:51,924 WARN
org.apache.hadoop.hdfs.server.namenode.FSEditLog: Unable to determine input
streams from QJM to [10.67.169.45:8485, 10.67.169.49:8485, 10.67.169.46:8485].
Skipping.org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many
exceptions to achieve quorum size 2/3. 1 successful
responses:10.67.169.45:8485: []2 exceptions thrown:10.67.169.46:8485: Failed on
local exception: java.io.IOException: java.lang.IllegalArgumentException:
Server has invalid Kerberos principal: hdfs/[email protected]; Host
Details : local host is: "server1.cehdev.com/10.67.169.45"; destination host
is: "server2.cehdev.com":8485;10.67.169.49:8485: Failed on local exception:
java.io.IOException: java.lang.IllegalArgumentException: Server has invalid
Kerberos principal: hdfs/[email protected]; Host Details : local
host is: "server1.cehdev.com/10.67.169.45"; destination host is:
"server3.cehdev.com":8485; at
org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81)
at
org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:223)
at
org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:142)
at
org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:471)
at
org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:260)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1430)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1450)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:636)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:279)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:751)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:735)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
Regards,Arul.
