Thanks, here it is: 1. On the source cluster, will be identical on the target cluster as both use the same Kerberos realm name, though each has its own cluster specific KDC:
$ more /etc/zookeeper/conf/server-jaas.conf > > /** > > * Licensed to the Apache Software Foundation (ASF) under one > > * or more contributor license agreements. See the NOTICE file > > * distributed with this work for additional information > > * regarding copyright ownership. The ASF licenses this file > > * to you under the Apache License, Version 2.0 (the > > * "License"); you may not use this file except in compliance > > * with the License. You may obtain a copy of the License at > > * <p/> > > * http://www.apache.org/licenses/LICENSE-2.0 > > * <p/> > > * Unless required by applicable law or agreed to in writing, software > > * distributed under the License is distributed on an "AS IS" BASIS, > > * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > implied. > > * See the License for the specific language governing permissions and > > * limitations under the License. > > */ > > Server { > > com.sun.security.auth.module.Krb5LoginModule required > > useKeyTab=true > > keyTab="/etc/zookeeper.keytab" > > storeKey=true > > useTicketCache=false > > principal="zookeeper/<source-master-ip>@PGS.dev"; > > }; > > 2. I ran zkCli.sh after authenticating as kerberos principal zookeeper/<host>@PGS.dev got the following: getAcl /hbase > > 'world,'anyone > > : r > > 'sasl,'hbase > > : cdrwa > > 3. I was logged in as Kerberos principal hbase/<host>@PGS.dev when I ran the add_peer command Thanks for taking the time to help me in any way you can. ---- Saad On Wed, May 23, 2018 at 7:24 AM, Reid Chan <reidddc...@outlook.com> wrote: > Three places to check, > > > 1. Would you mind showing your "/etc/zookeeper/conf/server-jaas.conf", > > 2. and using zkCli.sh to getAcl /hbase. > 3. BTW, what was your login principal when executing "add_peer" in > hbase shell. > ________________________________ > From: Saad Mufti <saad.mu...@gmail.com> > Sent: 23 May 2018 01:48:17 > To: user@hbase.apache.org > Subject: HBase Replication Between Two Secure Clusters With Different > Kerberos KDC's > > Hi, > > Here is my scenario, I have two secure/authenticated EMR based HBase > clusters, both have their own cluster dedicated KDC (using EMR support for > this which means we get Kerberos support by just turning on a config flag). > > Now we want to get replication going between them. For other application > reasons, we want both clusters to have the same Kerberos realm, let's say > APP.COM, so Kerberos principals are like a...@app.com . > > I looked around the web and found the instructions at > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/ > bk_hadoop-high-availability/content/hbase-cluster- > replication-among-kerberos-secured-clusters.html > so I tried to follow these directions. Of course the instructions are for > replication between clusters with different realms, so I adapted this by > adding only one principal "krbtgt/app....@app.com" and gave it some > arbitrary password. Followed the rest of the directions as well to pass a > rule property to Zookeeper and the requisite Hadoop property in > core-site.xml . > > After all this, when I set up replication from cluster1 to cluster using > add_peer, I see error messages in the region servers for cluster1 of the > following form: > > > > > 2018-05-22 17:27:45,763 INFO [main-SendThread(xxx.net:2181)] > > zookeeper.ClientCnxn: Opening socket connection to server i > > > > p-10-194-247-88.aolp-ds-dev.us-east-1.ec2.aolcloud.net/xxx.yyy.zzz:2181. > > Will attempt to SASL-authenticate using Login Context section 'Client' > > > > 2018-05-22 17:27:45,764 INFO [main-SendThread(xxx.net:2181)] > > zookeeper.ClientCnxn: Socket connection established to ip-1 > > > > 0-194-247-88.aolp-ds-dev.us-east-1.ec2.aolcloud.net/xxx.yyy.zzz:2181, > > initiating session > > > > 2018-05-22 17:27:45,777 INFO [main-SendThread(xxx.net:2181)] > > zookeeper.ClientCnxn: Session establishment complete on ser > > > > ver xxx.net/xxx.yyy.zzz:2181, sessionid = 0x16388599b300215, negotiated > > timeout = 40000 > > 2018-05-22 17:27:45,779 ERROR [main-SendThread(xxx.net:2181)] > > client.ZooKeeperSaslClient: An error: (java.security.Privil > > > > egedActionException: javax.security.sasl.SaslException: GSS initiate > > failed [Caused by GSSException: No valid credentials provided (Mechanism > > level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)]) > > occurred when evaluating Zookeeper Quorum Member's received SASL token. > > Zookeeper Client will go to AUTH_FAILED state. > > > > 2018-05-22 17:27:45,779 ERROR [main-SendThread(xxx.net:2181)] > > zookeeper.ClientCnxn: SASL authentication with Zookeeper Quorum member > > failed: javax.security.sasl.SaslException: An error: > > (java.security.PrivilegedActionException: > > javax.security.sasl.SaslException: GSS initiate failed [Caused by > > GSSException: No valid credentials provided (Mechanism level: Server not > > found in Kerberos database (7) - LOOKING_UP_SERVER)]) occurred when > > evaluating Zookeeper Quorum Member's received SASL token. Zookeeper > > Client will go to AUTH_FAILED state. > > > > 2018-05-22 17:28:12,574 WARN [main-EventThread] zookeeper.ZKUtil: > > hconnection-0x4dcc1d33-0x16388599b300215, quorum=xyz.net:2181, > > baseZNode=/hbase Unable to set watcher on znode (/hbase/hbaseid) > > > > org.apache.zookeeper.KeeperException$AuthFailedException: > KeeperErrorCode > > = AuthFailed for /hbase/hbaseid > > > > at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:123) > > > > at > > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > > > > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102) > > > > at > > > > > The Zookeeper start command looks like the following: > > /usr/lib/jvm/java-openjdk/bin/java -D*zoo*keeper.log.dir=/var/ > log/*zoo*keeper > > -D*zoo*keeper.root.logger=INFO,ROLLINGFILE -cp /usr/lib/*zoo* > > keeper/bin/../build/classes:/usr/lib/*zoo* > > keeper/bin/../build/lib/*.jar:/usr/lib/*zoo* > > keeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/usr/lib/*zoo* > > keeper/bin/../lib/slf4j-api-1.6.1.jar:/usr/lib/*zoo* > > keeper/bin/../lib/netty-3.10.5.Final.jar:/usr/lib/*zoo* > > keeper/bin/../lib/log4j-1.2.16.jar:/usr/lib/*zoo* > > keeper/bin/../lib/jline-2.11.jar:/usr/lib/*zoo*keeper/bin/../*zoo* > > keeper-3.4.10.jar:/usr/lib/*zoo*keeper/bin/../src/java/lib/*.jar:/etc/ > > *zoo*keeper/conf::/etc/*zoo*keeper/conf:/usr/lib/*zoo*keeper/*:/usr/lib/ > > *zoo*keeper/lib/* -Djava.security.auth.login. > config=/etc/*zoo*keeper/conf/server-jaas.conf > > -D*zoo*keeper.security.auth_to_local=RULE:[2:\$1@\$0](.*@\QAPP.COM > \E$)s/@\ > > APP.COM\E$//DEFAULT -*zoo*keeper.log.threshold=INFO > > -Dcom.sun.management.jmxremote > > -Dcom.sun.management.jmxremote.local.only=false > org.apache.*zoo*keeper.server.quorum.QuorumPeerMain > > /etc/*zoo*keeper/conf/*zoo*.cfg > > > > > The property in core-site looks like the following: > > <property> > > > > <name>hadoop.security.auth_to_local</name> > > > > <value>RULE:[2:\$1@ > > \$0](.*@\\QPGS.dev\\E$)s/@\\QPGS.dev\\E$//DEFAULT</value> > > > > </property> > > > > > > At this point I am not clear how I can get the added Kerberos principal " > krbtgt/app....@app.com" (in both clusters' Kerberos KDC's) to be > authenticated against and for replication to start flowing. > > Any help and/or pointers would be appreciated. > > Thanks. > > ----- > Saad >
