thrift2 server close the zookeeper session after ten minutes

[赵庆祥]

Hi,

   I’m running a hbase-0.98.17-hadoop2 cluster.

   When I try load data by thrift2 api.

   I got follow error.
Am i misconfig the server. It can reproduce on every trying.

The zookeeper session was closed by 
client.HConnectionManager$HConnectionImplementation after the connection 
established ten minutes.
   The I can not reconnect to the server about 2 hours.



   hbase.ttypes.TIOError: TIOError(_message='Failed after attempts=12, 
exceptions:\nFri Feb 26 11:10:07 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:10:17 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:10:26 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:10:36 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:10:46 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:10:56 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:11:06 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:11:16 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:11:26 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:11:36 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:11:56 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\nFri Feb 26 11:12:16 CST 2016, 
org.apache.hadoop.hbase.client.RpcRetryingCaller@7ba0b738, java.io.IOException: 
hconnection-0x1798ca83 closed\n')

  The thrift2 server log:
  2016-02-26 10:52:48,094 INFO  [pool-2-thread-1] zookeeper.ZooKeeper: Client 
environment:os.arch=amd64
2016-02-26 10:52:48,094 INFO  [pool-2-thread-1] zookeeper.ZooKeeper: Client 
environment:os.version=3.16.0-30-generic
2016-02-26 10:52:48,094 INFO  [pool-2-thread-1] zookeeper.ZooKeeper: Client 
environment:user.name=hadoop
2016-02-26 10:52:48,094 INFO  [pool-2-thread-1] zookeeper.ZooKeeper: Client 
environment:user.home=/home/hadoop
2016-02-26 10:52:48,094 INFO  [pool-2-thread-1] zookeeper.ZooKeeper: Client 
environment:user.dir=/home/hadoop/soft/hbase-0.98.17-hadoop2/conf
2016-02-26 10:52:48,095 INFO  [pool-2-thread-1] zookeeper.ZooKeeper: Initiating 
client connection, 
connectString=artemis-02:2181,artemis-01:2181,artemis-04:2181 
sessionTimeout=9 watcher=hconnection-0x1798ca830x0, 
quorum=artemis-02:2181,artemis-01:2181,artemis-04:2181, baseZNode=/hbase
2016-02-26 10:52:48,107 INFO  [pool-2-thread-1-SendThread(artemis-02:2181)] 
zookeeper.ClientCnxn: Opening socket connection to server 
artemis-02/192.168.132.135:2181. Will not attempt to authenticate using SASL 
(unknown error)
2016-02-26 10:52:48,112 INFO  [pool-2-thread-1-SendThread(artemis-02:2181)] 
zookeeper.ClientCnxn: Socket connection established to 
artemis-02/192.168.132.135:2181, initiating session
2016-02-26 10:52:48,161 INFO  [pool-2-thread-1-SendThread(artemis-02:2181)] 
zookeeper.ClientCnxn: Session establishment complete on server 
artemis-02/192.168.132.135:2181, sessionid = 0x2511eeb753e00ae, negotiated 
timeout = 4
2016-02-26 10:52:48,655 DEBUG [pool-2-thread-1] client.ClientSmallScanner: 
Finished with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', 
STARTKEY => '', ENDKEY => ''}
2016-02-26 11:02:55,825 INFO  [ConnectionCleaner] 
client.HConnectionManager$HConnectionImplementation: Closing zookeeper 
sessionid=0x2511eeb753e00ae
2016-02-26 11:02:55,881 INFO  [ConnectionCleaner] zookeeper.ZooKeeper: Session: 
0x2511eeb753e00ae closed
2016-02-26 11:02:55,881 INFO  [pool-2-thread-1-EventThread] 
zookeeper.ClientCnxn: EventThread shut down
2016-02-26 11:04:47,461 INFO  [pool-2-thread-1] client.RpcRetryingCaller: Call 
exception, tries=10, retries=12, retryTime=16ms, msg=row 
'2F36DD04DA72F574C3FDAFE5FCBAEA30' on table 'iri' at null
2016-02-26 11:05:07,507 INFO  [pool-2-thread-1] client.RpcRetryingCaller: Call 
exception, tries=11, retries=12, retryTime=131162ms, msg=row 
'2F36DD04DA72F574C3FDAFE5FCBAEA30' on table 'iri' at null
2016-02-26 11:11:56,545 INFO  [pool-2-thread-2] client.RpcRetryingCaller: Call 
exception, tries=10, retries=12, retryTime=108936ms, msg=row 
'2F36DD04DA72F574C3FDAFE5FCBAEA30' on table 'iri' at null
2016-02-26 11:12:16,648 INFO  [pool-2-thread-2] client.RpcRetryingCaller: Call 
exception, tries=11, retries=12, 

Re: Compaction settings per table

[Gaurav Agarwal]

Thanks Enis! this is really helpful.
I could not understand your second suggestion; could you please explain it
a bit more?

*For compaction and flush, we have a CompoundConfiguration*
*that is a layered configuration of hbase-site.xml values ->*
*HTD.getConfiguration() values -> HCD.getConfiguration() values.'*


On Fri, Feb 26, 2016 at 4:41 AM, Enis Söztutar  wrote:

> For compaction configurations, you can also set it per table OR even per
> column family.
>
> In java, you can use
>
> HTableDescriptor.setConfiguration() or HColumnDescriptor.setConfiguration()
> to set the specific configuration values that overrides the ones set in
> hbase-site.xml. For compaction and flush, we have a CompoundConfiguration
> that is a layered configuration of hbase-site.xml values ->
> HTD.getConfiguration() values -> HCD.getConfiguration() values.
>
> You can also use hbase shell to set the configuration as well.
>
> Enis
>
> On Thu, Feb 25, 2016 at 4:51 AM, Gaurav Agarwal  wrote:
>
> > Go the answer to memstore size per table
> > via TableDescriptor#setMemStoreFlushSize(long)
> >
> > On Thu, Feb 25, 2016 at 5:38 PM, Gaurav Agarwal 
> wrote:
> >
> > > In addition, is there a way to set memstore flush size per table/cf as
> > > well?
> > >
> > > On Thu, Feb 25, 2016 at 5:20 PM, Gaurav Agarwal 
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> Is there a way to set Compaction configurations differently for each
> of
> > >> my table? Specifically, I want to tweak `
> > >> hbase.hstore.compaction.min.size` parameter for one of my table while
> > >> keeping it to its default value for others.
> > >>
> > >> --cheers, gaurav
> > >>
> > >
> > >
> > >
> > > --
> > > --cheers, gaurav
> > >
> >
> >
> >
> > --
> > --cheers, gaurav
> >
>



-- 
--cheers, gaurav

Re: Compaction settings per table

[Enis Söztutar]

For compaction configurations, you can also set it per table OR even per
column family.

In java, you can use

HTableDescriptor.setConfiguration() or HColumnDescriptor.setConfiguration()
to set the specific configuration values that overrides the ones set in
hbase-site.xml. For compaction and flush, we have a CompoundConfiguration
that is a layered configuration of hbase-site.xml values ->
HTD.getConfiguration() values -> HCD.getConfiguration() values.

You can also use hbase shell to set the configuration as well.

Enis

On Thu, Feb 25, 2016 at 4:51 AM, Gaurav Agarwal  wrote:

> Go the answer to memstore size per table
> via TableDescriptor#setMemStoreFlushSize(long)
>
> On Thu, Feb 25, 2016 at 5:38 PM, Gaurav Agarwal  wrote:
>
> > In addition, is there a way to set memstore flush size per table/cf as
> > well?
> >
> > On Thu, Feb 25, 2016 at 5:20 PM, Gaurav Agarwal 
> wrote:
> >
> >> Hi,
> >>
> >> Is there a way to set Compaction configurations differently for each of
> >> my table? Specifically, I want to tweak `
> >> hbase.hstore.compaction.min.size` parameter for one of my table while
> >> keeping it to its default value for others.
> >>
> >> --cheers, gaurav
> >>
> >
> >
> >
> > --
> > --cheers, gaurav
> >
>
>
>
> --
> --cheers, gaurav
>

Re: Compaction settings per table

[Gaurav Agarwal]

Go the answer to memstore size per table
via TableDescriptor#setMemStoreFlushSize(long)

On Thu, Feb 25, 2016 at 5:38 PM, Gaurav Agarwal  wrote:

> In addition, is there a way to set memstore flush size per table/cf as
> well?
>
> On Thu, Feb 25, 2016 at 5:20 PM, Gaurav Agarwal  wrote:
>
>> Hi,
>>
>> Is there a way to set Compaction configurations differently for each of
>> my table? Specifically, I want to tweak `
>> hbase.hstore.compaction.min.size` parameter for one of my table while
>> keeping it to its default value for others.
>>
>> --cheers, gaurav
>>
>
>
>
> --
> --cheers, gaurav
>



-- 
--cheers, gaurav

Re: Compaction settings per table

[Gaurav Agarwal]

In addition, is there a way to set memstore flush size per table/cf as well?

On Thu, Feb 25, 2016 at 5:20 PM, Gaurav Agarwal  wrote:

> Hi,
>
> Is there a way to set Compaction configurations differently for each of my
> table? Specifically, I want to tweak `hbase.hstore.compaction.min.size` 
> parameter
> for one of my table while keeping it to its default value for others.
>
> --cheers, gaurav
>



-- 
--cheers, gaurav

Compaction settings per table

[Gaurav Agarwal]

Hi,

Is there a way to set Compaction configurations differently for each of my
table? Specifically, I want to tweak
`hbase.hstore.compaction.min.size` parameter
for one of my table while keeping it to its default value for others.

--cheers, gaurav