Thank you Rakesh ,  Problem have been resolved.  
I should not config : default_ccache_name = KEYRING:persistent:%{uid} in 
krb5.conf. It's not work for hadoop.

2016-09-22 

lk_hadoop 



发件人:Rakesh Radhakrishnan <rake...@apache.org>
发送时间:2016-09-21 23:23
主题:Re: hdfs2.7.3 kerberos can not startup
收件人:"kevin"<kiss.kevin...@gmail.com>
抄送:"Wei-Chiu Chuang"<weic...@cloudera.com>,"Brahma Reddy 
Battula"<brahmareddy.batt...@huawei.com>,"user.hadoop"<user@hadoop.apache.org>

I could see "Ticket cache: KEYRING:persistent:1004:1004" in your env.



May be KEYRING persistent cache setting is causing trouble, Kerberos libraries 
to store the krb cache in a location and the Hadoop libraries can't seem to 
access it.


Please refer these links,
https://community.hortonworks.com/questions/818/ipa-kerberos-not-liking-my-kinit-ticket.html

https://community.hortonworks.com/articles/11291/kerberos-cache-in-ipa-redhat-idm-keyring-solved-1.html





Rakesh
Intel


On Wed, Sep 21, 2016 at 2:21 PM, kevin <kiss.kevin...@gmail.com> wrote:

[hadoop@dmp1 ~]$ hdfs dfs -ls /
16/09/20 15:00:44 WARN ipc.Client: Exception encountered while connecting to 
the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: "dmp1.youedata.com/192.168.249.129"; 
destination host is: "dmp1.youedata.com":9000; 
[hadoop@dmp1 ~]$ klist
Ticket cache: KEYRING:persistent:1004:1004
Default principal: had...@example.com


Valid starting       Expires              Service principal
09/20/2016 14:57:34  09/21/2016 14:57:31  krbtgt/example....@example.com
renew until 09/27/2016 14:57:31
[hadoop@dmp1 ~]$ 


I have run kinit had...@example.com before .


2016-09-21 10:14 GMT+08:00 Wei-Chiu Chuang <weic...@cloudera.com>:

You need to run kinit command to authenticate before running hdfs dfs -ls 
command.


Wei-Chiu Chuang


On Sep 20, 2016, at 6:59 PM, kevin <kiss.kevin...@gmail.com> wrote:


Thank you Brahma Reddy Battula.
It's because of my problerm of the hdfs-site config file and https ca 
configuration.
now I can startup namenode and I can see the datanodes from the web.
but When I try hdfs dfs -ls /:


[hadoop@dmp1 hadoop-2.7.3]$ hdfs dfs -ls /
16/09/20 07:56:48 WARN ipc.Client: Exception encountered while connecting to 
the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by 
GSSException: No valid credentials provided (Mechanism level: Failed to find 
any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos 
tgt)]; Host Details : local host is: "dmp1.example.com/192.168.249.129"; 
destination host is: "dmp1.example.com":9000; 


current user is hadoop which startup hdfs , and I have add addprinc hadoop with 
commond :
kadmin.local -q "addprinc hadoop" 




2016-09-20 17:33 GMT+08:00 Brahma Reddy Battula 
<brahmareddy.batt...@huawei.com>:

Seems to be property problem.. it should be principal ( “l” is missed).

<property>
  <name>dfs.secondary.namenode.kerberos.principa</name>
  <value>hadoop/_h...@example.com</value>
</property>


For namenode httpserver start fail, please check rakesh comments..

This is probably due to some missing configuration. 
Could you please re-check the ssl-server.xml, keystore and truststore 
properties:

ssl.server.keystore.location
ssl.server.keystore.keypassword
ssl.client.truststore.location
ssl.client.truststore.password


--Brahma Reddy Battula

From: kevin [mailto:kiss.kevin...@gmail.com] 
Sent: 20 September 2016 16:53
To: Rakesh Radhakrishnan
Cc: user.hadoop
Subject: Re: hdfs2.7.3 kerberos can not startup

thanks, but my issue is name node could  Login successful,but second namenode 
couldn't. and name node got a HttpServer.start() threw a non Bind IOException:

hdfs-site.xml:

<property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
</property>

<property>
  <name>dfs.block.access.token.enable</name>
  <value>true</value>
</property>

<!-- NameNode security config -->
<property>
  <name>dfs.namenode.kerberos.principal</name>
  <value>hadoop/_h...@example.com</value>
</property>
<property>
  <name>dfs.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>
<property>
  <name>dfs.https.port</name>
  <value>50470</value>
</property>
<property>
  <name>dfs.namenode.https-address</name>
  <value>dmp1.example.com:50470</value>
</property>
<property>
  <name>dfs.namenode.kerberos.internal.spnego.principa</name>
  <value>HTTP/_h...@example.com</value>
</property>
<property>
  <name>dfs.web.authentication.kerberos.keytab</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>
<property>
  <name>dfs.http.policy</name>
  <value>HTTPS_ONLY</value>
</property>
<property>
  <name>dfs.https.enable</name>
  <value>true</value>
</property>


<!-- secondary NameNode security config -->
<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>dmp1.example.com:50090</value>
</property>
<property>
  <name>dfs.secondary.namenode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>
<property>
  <name>dfs.secondary.namenode.kerberos.principa</name>
  <value>hadoop/_h...@example.com</value>
</property>      
<property>
  <name>dfs.secondary.namenode.kerberos.internal.spnego.principal</name>
  <value>HTTP/_h...@example.com</value>
</property>
<property>
  <name>dfs.namenode.secondary.https-port</name>
  <value>50470</value>
</property>


<!-- JournalNode security config -->

<property>
  <name>dfs.journalnode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>
<property>
  <name>dfs.journalnode.kerberos.principa</name>
  <value>hadoop/_h...@example.com</value>
</property>      
<property>
  <name>dfs.journalnode.kerberos.internal.spnego.principa</name>
  <value>HTTP/_h...@example.com</value>
</property>
<property>
  <name>dfs.web.authentication.kerberos.keytab</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>


<!-- DataNode security config -->
<property>
  <name>dfs.datanode.kerberos.principal</name>
  <value>hadoop/_h...@example.com</value>
</property>
<property>
  <name>dfs.datanode.keytab.file</name>
  <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>
<property>
  <name>dfs.datanode.data.dir.perm</name>
  <value>700</value>
</property>

<!-- datanode SASL-->
<property>
  <name>dfs.datanode.address</name>
  <value>0.0.0.0:61004</value>
</property>
<property>
  <name>dfs.datanode.http.address</name>
  <value>0.0.0.0:61006</value>
</property>
<property>
  <name>dfs.datanode.https.address</name>
  <value>0.0.0.0:50470</value>
</property>

<property>
  <name>dfs.data.transfer.protection</name>
  <value>integrity</value>
</property>

<property>
     <name>dfs.web.authentication.kerberos.principal</name>
     <value>HTTP/_h...@example.com</value>
</property>
<property>
     <name>dfs.web.authentication.kerberos.keytab</name>
     <value>/etc/hadoop/conf/hdfs.keytab</value>
</property>

and [hadoop@dmp1 hadoop-2.7.3]$ klist -ket /etc/hadoop/conf/hdfs.keytab


Keytab name: FILE:/etc/hadoop/conf/hdfs.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 09/19/2016 16:00:41 hdfs/dmp1.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 hdfs/dmp1.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 hdfs/dmp1.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 16:00:41 hdfs/dmp1.example....@example.com (arcfour-hmac) 
   2 09/19/2016 16:00:41 hdfs/dmp2.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 hdfs/dmp2.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 hdfs/dmp2.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 16:00:41 hdfs/dmp2.example....@example.com (arcfour-hmac) 
   2 09/19/2016 16:00:41 hdfs/dmp3.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 hdfs/dmp3.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 hdfs/dmp3.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 16:00:41 hdfs/dmp3.example....@example.com (arcfour-hmac) 
   2 09/19/2016 16:00:41 HTTP/dmp1.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 HTTP/dmp1.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 HTTP/dmp1.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 16:00:41 HTTP/dmp1.example....@example.com (arcfour-hmac) 
   2 09/19/2016 16:00:41 HTTP/dmp2.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 HTTP/dmp2.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 HTTP/dmp2.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 16:00:41 HTTP/dmp2.example....@example.com (arcfour-hmac) 
   2 09/19/2016 16:00:41 HTTP/dmp3.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 HTTP/dmp3.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 16:00:41 HTTP/dmp3.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 16:00:41 HTTP/dmp3.example....@example.com (arcfour-hmac) 
   2 09/19/2016 20:21:03 hadoop/dmp1.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 20:21:03 hadoop/dmp1.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 20:21:03 hadoop/dmp1.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 20:21:03 hadoop/dmp1.example....@example.com (arcfour-hmac) 
   2 09/19/2016 20:21:03 hadoop/dmp2.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 20:21:03 hadoop/dmp2.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 20:21:03 hadoop/dmp2.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 20:21:03 hadoop/dmp2.example....@example.com (arcfour-hmac) 
   2 09/19/2016 20:21:03 hadoop/dmp3.example....@example.com 
(aes256-cts-hmac-sha1-96) 
   2 09/19/2016 20:21:03 hadoop/dmp3.example....@example.com 
(aes128-cts-hmac-sha1-96) 
   2 09/19/2016 20:21:03 hadoop/dmp3.example....@example.com (des3-cbc-sha1) 
   2 09/19/2016 20:21:03 hadoop/dmp3.example....@example.com (arcfour-hmac) 

2016-09-20 15:52 GMT+08:00 Rakesh Radhakrishnan <rake...@apache.org>:
>>>>>>Caused by: javax.security.auth.login.LoginException: Unable to obtain 
>>>>>>password from user

Could you please check kerberos principal name is specified correctly in
"hdfs-site.xml", which is used to authenticate against Kerberos.

If keytab file defined in "hdfs-site.xml" and doesn't exists or wrong path, you 
will see
this error. So, please verify the path and the keytab filename correctly
configured.

I hope hadoop discussion thread, https://goo.gl/M6l3vv may help you.


>>>>>>>2016-09-20 00:54:06,665 INFO org.apache.hadoop.http.HttpServer2: 
>>>>>>>HttpServer.start() threw a non Bind IOException
java.io.IOException: !JsseListener: java.lang.NullPointerException

This is probably due to some missing configuration. 
Could you please re-check the ssl-server.xml, keystore and truststore 
properties:

ssl.server.keystore.location
ssl.server.keystore.keypassword
ssl.client.truststore.location
ssl.client.truststore.password

Rakesh

On Tue, Sep 20, 2016 at 10:53 AM, kevin <kiss.kevin...@gmail.com> wrote:
hi,all:
My environment : Centos7.2 hadoop2.7.3 jdk1.8
after I config hdfs with kerberos ,I can't start up with sbin/start-dfs.sh

::namenode log as below  

STARTUP_MSG:   build = Unknown -r Unknown; compiled by 'root' on 
2016-09-18T09:05Z
STARTUP_MSG:   java = 1.8.0_102
************************************************************/
2016-09-20 00:54:05,822 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
registered UNIX signal handlers for [TERM, HUP, INT]
2016-09-20 00:54:05,825 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
createNameNode []
2016-09-20 00:54:06,078 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: 
loaded properties from hadoop-metrics2.properties
2016-09-20 00:54:06,149 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: 
Scheduled snapshot period at 10 second(s).
2016-09-20 00:54:06,149 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: 
NameNode metrics system started
2016-09-20 00:54:06,151 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
fs.defaultFS is hdfs://dmp1.example.com:9000
2016-09-20 00:54:06,152 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
Clients are to use dmp1.example.com:9000 to access this namenode/service.
2016-09-20 00:54:06,446 INFO org.apache.hadoop.security.UserGroupInformation: 
Login successful for user hadoop/dmp1.example....@example.com using keytab file 
/etc/hadoop/conf/hdfs.keytab
2016-09-20 00:54:06,472 INFO org.apache.hadoop.hdfs.DFSUtil: Starting web 
server as: HTTP/dmp1.example....@example.com
2016-09-20 00:54:06,475 INFO org.apache.hadoop.hdfs.DFSUtil: Starting 
Web-server for hdfs at: https://dmp1.example.com:50470
2016-09-20 00:54:06,517 INFO org.mortbay.log: Logging to 
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-09-20 00:54:06,533 INFO 
org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable 
to initialize FileSignerSecretProvider, falling back to use random secrets.
2016-09-20 00:54:06,542 INFO org.apache.hadoop.http.HttpRequestLog: Http 
request log for http.requests.namenode is not defined
2016-09-20 00:54:06,546 INFO org.apache.hadoop.http.HttpServer2: Added global 
filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2016-09-20 00:54:06,548 INFO org.apache.hadoop.http.HttpServer2: Added filter 
static_user_filter 
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to 
context hdfs
2016-09-20 00:54:06,548 INFO org.apache.hadoop.http.HttpServer2: Added filter 
static_user_filter 
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to 
context static
2016-09-20 00:54:06,548 INFO org.apache.hadoop.http.HttpServer2: Added filter 
static_user_filter 
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to 
context logs
2016-09-20 00:54:06,653 INFO org.apache.hadoop.http.HttpServer2: Added filter 
'org.apache.hadoop.hdfs.web.AuthFilter' 
(class=org.apache.hadoop.hdfs.web.AuthFilter)
2016-09-20 00:54:06,654 INFO org.apache.hadoop.http.HttpServer2: 
addJerseyResourcePackage: 
packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
 pathSpec=/webhdfs/v1/*
2016-09-20 00:54:06,657 INFO org.apache.hadoop.http.HttpServer2: Adding 
Kerberos (SPNEGO) filter to getDelegationToken
2016-09-20 00:54:06,658 INFO org.apache.hadoop.http.HttpServer2: Adding 
Kerberos (SPNEGO) filter to renewDelegationToken
2016-09-20 00:54:06,658 INFO org.apache.hadoop.http.HttpServer2: Adding 
Kerberos (SPNEGO) filter to cancelDelegationToken
2016-09-20 00:54:06,659 INFO org.apache.hadoop.http.HttpServer2: Adding 
Kerberos (SPNEGO) filter to fsck
2016-09-20 00:54:06,659 INFO org.apache.hadoop.http.HttpServer2: Adding 
Kerberos (SPNEGO) filter to imagetransfer
2016-09-20 00:54:06,665 WARN org.mortbay.log: java.lang.NullPointerException
2016-09-20 00:54:06,665 INFO org.apache.hadoop.http.HttpServer2: 
HttpServer.start() threw a non Bind IOException
java.io.IOException: !JsseListener: java.lang.NullPointerException
at 
org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:516)
at 
org.apache.hadoop.security.ssl.SslSocketConnectorSecure.newServerSocket(SslSocketConnectorSecure.java:47)
at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:914)
at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856)
at 
org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:753)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:639)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:812)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:796)
at 
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1493)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)


::second namenode log as below  

STARTUP_MSG:   build = Unknown -r Unknown; compiled by 'root' on 
2016-09-18T09:05Z
STARTUP_MSG:   java = 1.8.0_102
************************************************************/
2016-09-20 00:54:14,885 INFO 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: registered UNIX 
signal handlers for [TERM, HUP, INT]
2016-09-20 00:54:15,263 FATAL 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Failed to start 
secondary namenode
java.io.IOException: Login failure for hadoop from keytab 
/etc/hadoop/conf/hdfs.keytab: javax.security.auth.login.LoginException: Unable 
to obtain password from user

at 
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:963)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:246)
at 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:217)
at 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:192)
at 
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:671)
Caused by: javax.security.auth.login.LoginException: Unable to obtain password 
from user

at 
com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:897)
at 
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:760)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:617)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at 
org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:954)
... 4 more

Reply via email to