[
https://issues.apache.org/jira/browse/HDFS-7731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14303206#comment-14303206
]
donhoff_h commented on HDFS-7731:
---------------------------------
Hi, Zheng Kai
I'm sure my Kerberos principals and keytab files are configured correctly,
since I can start the secure non-HA cluster without any problem. As to the
hdfs-site.xml, I did not find any problem either. So I list the main part as
below, could you help me to figure out the problem? thanks!
The main part of my hdfs-site.xml
<property>
<name>dfs.nameservices</name>
<value>bgdt-dev-hrb</value>
</property>
<property>
<name>dfs.ha.namenodes.bgdt-dev-hrb</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.bgdt-dev-hrb.nn1</name>
<value>bgdt01.dev.hrb:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.bgdt-dev-hrb.nn2</name>
<value>bgdt02.dev.hrb:9000</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://bgdt01.dev.hrb:8485;bgdt03.dev.hrb:8485;bgdt04.dev.hrb:8485/bgdt-dev-hrb</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.bgdt-dev-hrb</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence
shell(/bin/true)
</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/bgdt/hadoop/hdfs/jn</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///bgdt/hadoop/hdfs/nn</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.name.dir</name>
<value>file:///bgdt/hadoop/hdfs/dn</value>
</property>
<property>
<name>dfs.namenode.http-address.bgdt-dev-hrb.nn1</name>
<value>bgdt01.dev.hrb:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.bgdt-dev-hrb.nn2</name>
<value>bgdt02.dev.hrb:50070</value>
</property>
<property>
<name>dfs.permissions.superusergroup</name>
<value>bgdtgrp</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
<property>
<name>dfs.http.policy</name>
<value>HTTP_ONLY</value>
</property>
<property>
<name>dfs.namenode.https-address.bgdt-dev-hrb.nn1</name>
<value>bgdt01.dev.hrb:50470</value>
</property>
<property>
<name>dfs.namenode.https-address.bgdt-dev-hrb.nn2</name>
<value>bgdt02.dev.hrb:50470</value>
</property>
<property>
<name>dfs.namenode.keytab.file</name>
<value>/etc/hadoop/keytab/hadoop.service.keytab</value>
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/[email protected]</value>
</property>
<property>
<name>dfs.namenode.kerberos.https.principal</name>
<value>host/[email protected]</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.principal</name>
<value>http/[email protected]</value>
</property>
<property>
<name>dfs.web.authentication.kerberos.keytab</name>
<value>/etc/hadoop/keytab/hadoop.service.keytab</value>
</property>
<property>
<name>dfs.journalnode.kerberos.principal</name>
<value>hdfs/[email protected]</value>
</property>
<property>
<name>dfs.journalnode.kerberos.https.principal</name>
<value>host/[email protected]</value>
</property>
<property>
<name>dfs.journalnode.kerberos.internal.spnego.principal</name>
<value>http/[email protected]</value>
</property>
<property>
<name>dfs.journalnode.keytab.file</name>
<value>/etc/hadoop/keytab/hadoop.service.keytab</value>
</property>
> Can not start HA namenode with security enabled
> -----------------------------------------------
>
> Key: HDFS-7731
> URL: https://issues.apache.org/jira/browse/HDFS-7731
> Project: Hadoop HDFS
> Issue Type: Task
> Components: ha, journal-node, namenode, security
> Affects Versions: 2.5.2
> Environment: Redhat6.2 Hadoop2.5.2
> Reporter: donhoff_h
> Labels: hadoop, security
>
> I am converting a secure non-HA cluster into a secure HA cluster. After the
> configuration and started all the journalnodes, I executed the following
> commands on the original NameNode:
> 1. hdfs name -initializeSharedEdits #this step succeeded
> 2. hadoop-daemon.sh start namenode # this step failed.
> So the namenode can not be started. I verified that my principals are right.
> And if I change back to the secure non-HA mode, the namenode can be started.
> The namenode log just reported the following errors and I could not find the
> reason according to this log:
> 2015-02-03 17:42:06,020 INFO org.apache.hadoop.hdfs.server.namenode.FSImage:
> Start loading edits file
> http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3,
>
> http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3
> 2015-02-03 17:42:06,024 INFO
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding
> stream
> 'http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3,
>
> http://bgdt01.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3'
> to transaction ID 68994
> 2015-02-03 17:42:06,024 INFO
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: Fast-forwarding
> stream
> 'http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3'
> to transaction ID 68994
> 2015-02-03 17:42:06,154 ERROR
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream: caught exception
> initializing
> http://bgdt04.dev.hrb:8480/getJournal?jid=bgdt-dev-hrb&segmentTxId=68994&storageInfo=-57%3A876630880%3A0%3ACID-ea4c77aa-882d-4adf-a347-42f1344421f3
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Server not
> found in Kerberos database (7) - UNKNOWN_SERVER)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:464)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog$1.run(EditLogFileInputStream.java:456)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at
> org.apache.hadoop.security.SecurityUtil.doAsUser(SecurityUtil.java:444)
> at
> org.apache.hadoop.security.SecurityUtil.doAsCurrentUser(SecurityUtil.java:438)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream$URLLog.getInputStream(EditLogFileInputStream.java:455)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.init(EditLogFileInputStream.java:141)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOpImpl(EditLogFileInputStream.java:192)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.nextOp(EditLogFileInputStream.java:250)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.skipUntil(EditLogInputStream.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream.nextOp(RedundantEditLogInputStream.java:178)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogInputStream.readOp(EditLogInputStream.java:85)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:184)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:137)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:816)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:676)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:279)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:751)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:735)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
