You get the lease expired from any region hosted on this machine, or only regions from a specific table? If the former, I'd say kill the machine and request a new one. On AES, life is too short to diagnose crap VM's. If you do that and the problem moves to another machine, it must the the region ;)
On Friday, March 6, 2015, Giusto Michele <[email protected]> wrote: > Hi all, > > I am working on a Cloudera CDH5.3.1 cluster (so hbase-0.98.6+cdh5.3.1+74) > and the cluster is running in AWS. The cluster is composed of 6 machines, 2 > masters and 4 slaves. I am having problem on a single slave node (the last > one added, temporarily) where the local RegionServer, once it receives the > first region (even without rows inside), continuously logs messages like: > > 2015-03-06 11:48:29,088 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1688 lease > expired on region > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. > > 2015-03-06 11:48:32,458 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1689 lease > expired on region > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. > > 2015-03-06 11:48:36,089 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1690 lease > expired on region > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. > > 2015-03-06 11:48:39,460 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1691 lease > expired on region > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. > > 2015-03-06 11:48:43,090 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1692 lease > expired on region > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. > > 2015-03-06 11:48:46,459 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1693 lease > expired on region > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. > > 2015-03-06 11:48:50,089 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1694 lease > expired on region > test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. > > > > All the other services are working as expected apart from this problem. I > am not able to understand why this is happening, the only difference I can > see from logs/web UI is that this slave node is identified with its FQDN, > while the others only with the hostname (no domain). > > > > I have checked on the other nodes and they are not reporting similar > messages. Setting log level to TRACE didn’t show nothing else useful, at > least for me. > > > > Finally, the original problem that lead me to find these messages is that > when I map an HBase table as an external table on Hive/Impala, with one > region of this table on the problematic RegionServer, the query remain > pending forever. If the table is not using that RegionServer, queries work > normally. It is important to note that the messages starts immediately > after I create the HBase table, so I believe it is not Impala that creates > problem. > > What does the messages imply? Can the reason of my Impala problem be > related to these messages? > > > > > > Thanks in advance, > > Michele > > > Michele Giusto > > Target Reply > Via Castellanza, 11 > 20151 - Milano - ITALY > phone: +39 02 535761 > [email protected] <javascript:_e(%7B%7D,'cvml','[email protected]');> > www.reply.it > > [image: Target Reply] > > > ------------------------------ > > -- > The information transmitted is intended for the person or entity to which > it is addressed and may contain confidential and/or privileged material. > Any review, retransmission, dissemination or other use of, or taking of any > action in reliance upon, this information by persons or entities other than > the intended recipient is prohibited. If you received this in error, please > contact the sender and delete the material from any computer. >
