Glad it helped! In general though, in this cases, if you turn on java kerberos debug logging (via something like export HADOOP_OPTS="-Dsun.security.krb5.debug=true"), in your log you would see error like: "unsupported key type found the default TGT: 18"
(which means encryption type 18 per kerberos specification, i.e. aes256-cts-hmac-sha1-96, isn't supported by Java - due to lack of unlimited strength JCE) -Mikhail On Mon, Mar 16, 2015 at 3:19 PM, Manoj Murumkar <[email protected]> wrote: > Thanks Mikhail! It turned out not having unlimited strength JCE files > everywhere on the cluster was root cause (I reproduced it by leaving > default files on regionservers on the newly built cluster). > > On Sun, Mar 15, 2015 at 9:48 PM, Manoj Murumkar <[email protected]> > wrote: > >> Thanks. We took care of above items (all principals were generated using >> Ambari tool) although I am not sure about 3rd item. One of my coworkers >> decided to reinstall HBase, so I am waiting for the cluster to come up to >> see if we still have the issue. Funny thing is, everything else worked in >> secure environment (MR, Hive), just not HBase. The error message didn't >> mention any details why the GSSException was raised (sample below): >> >> >> 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] >> ipc.RpcServer: Kerberos principal name is hbase/ >> [email protected] >> >> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >> ipc.RpcServer: Created SASL server with mechanism = GSSAPI >> >> 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >> ipc.RpcServer: Have read input token of size 627 for processing by >> saslServer.evaluateResponse() >> *2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] >> ipc.RpcServer: RpcServer.listener,port=60020: Caught exception while >> reading:GSS initiate fail* >> >> >> On Sun, Mar 15, 2015 at 2:43 PM, Mikhail Antonov <[email protected]> >> wrote: >> >>> I don't see region server log file in attachment for some reason.. May >>> be the file is too big and gets rejected by mail server? Without it >>> it'd be hard to say. Also master log looks a bit incomplete, are there >>> no more traces? "GSS initiate fail" is normally followed by something >>> like (just for example) - "[Caused by GSSException: No valid >>> credentials provided (Mechanism level: Failed to find any Kerberos >>> tgt)];" or so. Nothing like that? >>> >>> But in general, GSS initialization most often fails for following reasons: >>> >>> - wrong linux path or file permissions on keytab file >>> - typo in kerberos principal/realm (that includes lower/upper case >>> differences) >>> - JCE unlimited strength file aren't installed >>> >>> On Sun, Mar 15, 2015 at 10:04 AM, Manoj Murumkar >>> <[email protected]> wrote: >>> > Attached region server log again. >>> > >>> > Ted, >>> > >>> > Corresponding log for master is: >>> > >>> > java.io.IOException: Couldn't setup connection for >>> > hbase/[email protected] to >>> > hbase/sfdvgctsn001.gid.gap.com@S >>> > >>> > FDVGCT.COM >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) >>> > >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > >>> > at javax.security.auth.Subject.doAs(Subject.java:415) >>> > >>> > at >>> > >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) >>> > >>> > at >>> org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$BlockingStub.openRegion(AdminProtos.java:21176) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:670) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2004) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.master.AssignmentManager.access$300(AssignmentManager.java:119) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.master.AssignmentManager$2.process(AssignmentManager.java:743) >>> > >>> > at >>> > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128) >>> > >>> > at >>> > >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> > >>> > at >>> > >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> > >>> > at java.lang.Thread.run(Thread.java:745) >>> > >>> > Caused by: >>> > >>> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): >>> > GSS initiate failed >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.readStatus(HBaseSaslRpcClient.java:151) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:187) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940) >>> > >>> > at java.security.AccessController.doPrivileged(Native Method) >>> > >>> > at javax.security.auth.Subject.doAs(Subject.java:415) >>> > >>> > at >>> > >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >>> > >>> > at >>> > >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940) >>> > >>> > >>> > On Sun, Mar 15, 2015 at 9:36 AM, Ted Yu <[email protected]> wrote: >>> >> >>> >> bq. Attached both files >>> >> >>> >> I only found hbase-site.xml in the attachment. >>> >> >>> >> On Sun, Mar 15, 2015 at 8:52 AM, Manoj Murumkar < >>> [email protected]> >>> >> wrote: >>> >> >>> >> > Attached both files. >>> >> > >>> >> > >>> >> > >>> >> > On Sat, Mar 14, 2015 at 2:10 AM, Mikhail Antonov < >>> [email protected]> >>> >> > wrote: >>> >> > >>> >> >> Hi, >>> >> >> >>> >> >> Traces (especially one for region server) look a bit incomplete, did >>> >> >> you copy them fully? >>> >> >> >>> >> >> Also may help if you post relevant pieces of hbase-site.xml (with >>> >> >> security configs). >>> >> >> >>> >> >> Thanks, >>> >> >> Mikhail >>> >> >> >>> >> >> >>> >> >> On Fri, Mar 13, 2015 at 11:28 PM, Manoj Murumkar >>> >> >> <[email protected]> wrote: >>> >> >> > Hi, >>> >> >> > >>> >> >> > We have a secured cluster. All components are working well, except >>> >> >> hbase. >>> >> >> > Specifically, this is what I see on regionserver: >>> >> >> > >>> >> >> > 2015-03-14 02:16:11,657 DEBUG [RpcServer.reader=5,port=60020] >>> >> >> > ipc.RpcServer: Kerberos principal name is hbase/ >>> >> >> > [email protected] >>> >> >> > >>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >>> >> >> > ipc.RpcServer: Created SASL server with mechanism = GSSAPI >>> >> >> > >>> >> >> > 2015-03-14 02:16:11,658 DEBUG [RpcServer.reader=5,port=60020] >>> >> >> > ipc.RpcServer: Have read input token of size 627 for processing by >>> >> >> > saslServer.evaluateResponse() >>> >> >> > 2015-03-14 02:16:11,659 DEBUG [RpcServer.reader=5,port=60020] >>> >> >> > ipc.RpcServer: RpcServer.listener,port=60020: Caught exception >>> while >>> >> >> > reading:GSS initiate fail >>> >> >> > >>> >> >> > >>> >> >> > and on the master: >>> >> >> > >>> >> >> > >>> >> >> > java.io.IOException: Couldn't setup connection for >>> >> >> > hbase/sfdvgctmn004. >>> >> >> > [email protected] to hbase/[email protected] >>> >> >> > >>> >> >> > at >>> >> >> > >>> >> >> >>> >> >> >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection$1.run(RpcClient.java:869) >>> >> >> > >>> >> >> > at java.security.AccessController.doPrivileged(Native >>> Method) >>> >> >> > >>> >> >> > at javax.security.auth.Subject.doAs(Subject.java:415) >>> >> >> > >>> >> >> > at >>> >> >> > >>> >> >> >>> >> >> >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >>> >> >> > >>> >> >> > at >>> >> >> > >>> >> >> >>> >> >> >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.handleSaslConnectionFailure(RpcClient.java:841) >>> >> >> > >>> >> >> > at >>> >> >> > >>> >> >> >>> >> >> >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:951) >>> >> >> > >>> >> >> > at >>> >> >> > >>> >> >> >>> >> >> >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094) >>> >> >> > >>> >> >> > at >>> >> >> > >>> >> >> >>> >> >> >>> org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061) >>> >> >> > >>> >> >> > All the keytabs are set properly. Has anyone seen this before? >>> >> >> Appreciate >>> >> >> > the time. >>> >> >> > >>> >> >> > Manoj >>> >> >> >>> >> >> >>> >> >> >>> >> >> -- >>> >> >> Thanks, >>> >> >> Michael Antonov >>> >> >> >>> >> > >>> >> > >>> > >>> > >>> >>> >>> >>> -- >>> Thanks, >>> Michael Antonov >>> >> >> -- Thanks, Michael Antonov
