Hi All, we seem to have overcome this occasional exception by changing
HTable from being called through ConnectionManager vs. directly
instantiating HTable.
OLD BAD:
connection = HConnectionManager.createConnection(config);
protected HTableInterface getHTable(String tableName) throws IOException
{
//Connection might have been closed since we acquired
it...create a
//new one if this is the case
if (connection == null || connection.isClosed())
{
log.info("HConnection null or
closed...reopening");
connection =
HConnectionManager.createConnection(config);
}
return connection.getTable(tableName);
}
NEW GOOD:
public HTableInterface getHTable(byte[] tableName) throws IOException
{
return new HTable(conf, tableName);
}
Comment on this link [0] is why they opted down the initial - BAD path:
". Instances of this class can be constructed directly but it is
encouraged that users get instances via HConnection and HConnectionManager
. See HConnectionManager class comment for an example. "
[0]
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html
We run the workload for a few days and it seems fine. We are not sure why
this occurs but somebody listens this thread may have a better idea.
Thanks for all your help!
Best regards,
Wei
---------------------------------
Wei Tan, PhD
Research Staff Member
IBM T. J. Watson Research Center
http://researcher.ibm.com/person/us-wtan
From: Zesheng Wu <[email protected]>
To: [email protected],
Date: 03/13/2014 09:33 PM
Subject: Re: Occasional GSSException that brings down region server
Thank you all the same :)
2014-03-13 22:50 GMT+08:00 Andrew Purtell <[email protected]>:
> Then nothing immediately comes to mind, sorry.
>
> On Wednesday, March 12, 2014, Zesheng Wu <[email protected]> wrote:
>
> > Hi Andrew,
> >
> > We use keytab file for hbase, there's no ticket cache file under /tmp,
> but
> > we still encounter the same error as above, how about this?
> >
> >
> > 2014-03-12 7:37 GMT+08:00 Andrew Purtell <[email protected]
> <javascript:;>
> > >:
> >
> > > If you might have more than one security enabled Java process
running
> > under
> > > the same UNIX user on the servers, then this and other weirdness can
> > happen
> > > because they share the same ticket cache file in /tmp. Someone does
a
> > > (re)login and another process concurrently accessing the cache gets
> > garbage
> > > or unexpected state.
> > >
> > >
> > > On Mon, Mar 10, 2014 at 2:26 PM, Wei Tan <[email protected]<
javascript:;>>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are running a HBase cluster in these settings and with
kerberos
> > > > enabled.
> > > > HBase: 0.96.1.1
> > > > Zookeeper: 3.4.5
> > > > Hadoop: 1.1.1
> > > >
> > > >
> > > > We constantly put data into HBase and every several hours we get
the
> > > error
> > > > below on a random region server; this error arises and the region
> > server
> > > > kills itself.
> > > >
> > > > ERROR:
> > > > 2014-02-28 09:32:39,755 ERROR
> > > [hconnection-0x116987ad-shared--pool1378-t9]
> > > > security.UserGroupInformation: PriviledgedActionException
> > > > as:XXXXXXXX@DOMAIN cause:javax.security.sasl.SaslException: GSS
> > initiate
> > > > failed [Caused by GSSException: No valid credentials provided
> > (Mechanism
> > > > level: The ticket isn't for us (35) - BAD TGS SERVER NAME)]
> > > >
> > > >
> > > >
> > > > We also tried with multiple version of kdc - all the way up to
latest
> > > > 1.12.1 - still see this error. What is weird is that most put gets
> > > > processed successfully until this error occurs and kills the RS.
> > > >
> > > > Thanks,
> > > > Wei
> > > > ---------------------------------
> > > > Wei Tan, PhD
> > > > Research Staff Member
> > > > IBM T. J. Watson Research Center
> > > > http://researcher.ibm.com/person/us-wtan
> > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> > >
> >
> >
> >
> > --
> > Best Wishes!
> >
> > Yours, Zesheng
> >
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
--
Best Wishes!
Yours, Zesheng
