Hi all, I am working on a Cloudera CDH5.3.1 cluster (so hbase-0.98.6+cdh5.3.1+74) and the cluster is running in AWS. The cluster is composed of 6 machines, 2 masters and 4 slaves. I am having problem on a single slave node (the last one added, temporarily) where the local RegionServer, once it receives the first region (even without rows inside), continuously logs messages like: 2015-03-06 11:48:29,088 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1688 lease expired on region test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. 2015-03-06 11:48:32,458 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1689 lease expired on region test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. 2015-03-06 11:48:36,089 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1690 lease expired on region test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. 2015-03-06 11:48:39,460 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1691 lease expired on region test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. 2015-03-06 11:48:43,090 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1692 lease expired on region test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. 2015-03-06 11:48:46,459 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1693 lease expired on region test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8. 2015-03-06 11:48:50,089 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1694 lease expired on region test_table2,,1425636551770.739b80dfcac4fbc59ea6a6633b9e86a8.
All the other services are working as expected apart from this problem. I am not able to understand why this is happening, the only difference I can see from logs/web UI is that this slave node is identified with its FQDN, while the others only with the hostname (no domain). I have checked on the other nodes and they are not reporting similar messages. Setting log level to TRACE didn't show nothing else useful, at least for me. Finally, the original problem that lead me to find these messages is that when I map an HBase table as an external table on Hive/Impala, with one region of this table on the problematic RegionServer, the query remain pending forever. If the table is not using that RegionServer, queries work normally. It is important to note that the messages starts immediately after I create the HBase table, so I believe it is not Impala that creates problem. What does the messages imply? Can the reason of my Impala problem be related to these messages? Thanks in advance, Michele Michele Giusto Target Reply Via Castellanza, 11 20151 - Milano - ITALY phone: +39 02 535761 [email protected]<mailto:[email protected]> www.reply.it [Target Reply] ________________________________ -- The information transmitted is intended for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
