SASL authenticate error when tring to connect zookeeper

[HE Ye]

Hi expert,

Now I  am verifying connection between broker and zookeeper using SASL 
mechanisms and the zookeeper always claims: GSS initiate failed [Caused by 
GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum 
failed)

>From Kerberos log, I see zookeeper sent AS_REQ, broker sent AS_REQ and 
>TGS_REQ. Kerberos server answered them successfully. 

Then through Wireshark, I see broker sent SMPP message to zookeeper, then 
failure happened. Confused by that. It is supposed for broker to send 
KRB_AP_REQ message to zookeeper then, right?

Already confused for a week! Very appreciate if any hint on this issue!

Here is the log:

zookeeper:
[2016-11-15 08:34:21,319] INFO Established session 0x1586868cc200000 with 
negotiated timeout 6000 for client /10.160.32.153:33610 
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-11-15 08:34:21,320] WARN Client failed to SASL authenticate: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
Failure unspecified at GSS-API level (Mechanism level: Checksum failed)] 
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-11-15 08:34:21,320] WARN Closing client connection due to SASL 
authentication failure. (org.apache.zookeeper.server.ZooKeeperServer)
[2016-11-15 08:34:21,320] INFO Closed socket connection for client 
/10.160.32.153:33610 which had sessionid 0x1586868cc200000 
(org.apache.zookeeper.server.NIOServerCnxn)


kafka broker:
[2016-11-15 08:34:21,316] INFO zookeeper state changed (SyncConnected) 
(org.I0Itec.zkclient.ZkClient)
[2016-11-15 08:34:21,321] INFO Unable to read additional data from server 
sessionid 0x1586868cc200000, likely server has closed socket, closing socket 
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-11-15 08:34:21,421] INFO zookeeper state changed (Disconnected) 
(org.I0Itec.zkclient.ZkClient)

Kerberos:
Nov 15 08:34:08 YeKerberosServer-0-0-1 krb5kdc[10896](info): AS_REQ (1 etypes 
{17}) 10.160.32.153: ISSUE: authtime 1479220448, etypes {rep=17 tkt=17 ses=17}, 
zookeeper/kafka.example....@example.com for krbtgt/example....@example.com
Nov 15 08:34:19 YeKerberosServer-0-0-1 krb5kdc[10896](info): AS_REQ (1 etypes 
{17}) 10.160.32.153: NEEDED_PREAUTH: kafka/kafka.example....@example.com for 
krbtgt/example....@example.com, Additional pre-authentication required
Nov 15 08:34:19 YeKerberosServer-0-0-1 krb5kdc[10896](info): AS_REQ (1 etypes 
{17}) 10.160.32.153: ISSUE: authtime 1479220459, etypes {rep=17 tkt=17 ses=17}, 
kafka/kafka.example....@example.com for krbtgt/example....@example.com
Nov 15 08:34:19 YeKerberosServer-0-0-1 krb5kdc[10896](info): TGS_REQ (1 etypes 
{17}) 10.160.32.153: ISSUE: authtime 1479220459, etypes {rep=17 tkt=17 ses=17}, 
kafka/kafka.example....@example.com for zookeeper/10.160.32....@example.com

kafka_server_ jaas.conf
    KafkaServer {
        com.sun.security.auth.module.Krb5LoginModule required
        useKeyTab=true
        storeKey=true
        useTicketCache=false
        serviceName="kafka"
        keyTab="/u/ainet/kafka_2.11-0.10.0.0/config/kafka.keytab"
        principal="kafka/kafka.example....@example.com";
    };

    // Zookeeper client authentication
    Client {
       com.sun.security.auth.module.Krb5LoginModule required
       //useTicketCache=true;
       useKeyTab=true
       storeKey=true
       useTicketCache=false
       serviceName="zookeeper"
       doNotPrompt=true
       refreshKrb5Config=true
       isInitiator=true
       keyTab="/u/ainet/kafka_2.11-0.10.0.0/config/kafka.keytab"
       principal="kafka/kafka.example....@example.com";
    };

zookeeper_server_jaas.conf
    Server {
       com.sun.security.auth.module.Krb5LoginModule required
       useKeyTab=true
       storeKey=true
       useTicketCache=false
       keyTab="/u/ainet/kafka_2.11-0.10.0.0/config/zookeeper.keytab"
       principal="zookeeper/kafka.example....@example.com";
    };

krb5.conf:
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 dns_lookup_realm = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true
 rdns = false
 default_realm = EXAMPLE.COM
 default_ccache_name = KEYRING:persistent:%{uid}
default_tkt_enctypes = aes128-cts
  default_tgs_enctypes = aes128-cts
 permitted_enctypes = aes128-cts
 debug=true

[realms]
 EXAMPLE.COM = {
  kdc = kbserver.example.com
  admin_server = kbserver.example.com
 }

[domain_realm]
 .example.com = EXAMPLE.COM
 example.com = EXAMPLE.COM

[appdefaults]
validate=false


Thanks,
Ye