this was answered in this thread https://community.hortonworks.com/questions/8757/phoenix-local-indexes.html
On Wed, Jan 6, 2016 at 10:16 AM, Pedro Gandola <[email protected]> wrote: > Hi Guys, > > The issue is a deadlock but it's not related with phoenix and it can be > resolved increasing the number of threads responsible for opening the > regions. > > <property> >> <name>hbase.regionserver.executor.openregion.threads</name> >> <value>100</value> >> </property> > > > Got help from here > <https://community.hortonworks.com/questions/8757/phoenix-local-indexes.html> > . > > Thanks > Cheers > Pedro > > On Tue, Jan 5, 2016 at 10:18 PM, Pedro Gandola <[email protected]> > wrote: > >> Hi Guys, >> >> I have been testing out the Phoenix Local Indexes and I'm facing an issue >> after restart the entire HBase cluster. >> >> *Scenario:* I'm using Phoenix 4.4 and HBase 1.1.1. My test cluster >> contains 10 machines and the main table contains 300 pre-split regions >> which implies 300 regions on local index table as well and to configure >> Phoenix I followed thistutorial >> <http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/configuring-hbase-for-phoenix.html> >> . >> >> When I start a fresh cluster everything is just fine, the local index is >> created and I can insert data and query it using the proper indexes. The >> problem comes when I perform a full restart of the cluster to update some >> configurations in that moment I'm not able to restart the cluster anymore. >> I should do a proper rolling restart but it looks that Ambari is not doing >> it in some situations. >> >> Most of the servers are throwing exceptions like: >> >> INFO [htable-pool7-t1] client.AsyncProcess: #5, >>> table=_LOCAL_IDX_BIDDING_EVENTS, attempt=27/350 failed=1ops, last >>> exception: org.apache.hadoop.hbase.NotServingRegionException: >>> org.apache.hadoop.hbase.NotServingRegionException: Region >>> _LOCAL_IDX_BIDDING_EVENTS,57e4b17e4b17e4ac,1451943466164.253bdee3695b566545329fa3ac86d05e. >>> is not online on ip-10-5-4-24.ec2.internal,16020,1451996088952 >>> at >>> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2898) >>> at >>> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:947) >>> at >>> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:1991) >>> at >>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32213) >>> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) >>> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) >>> at >>> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) >>> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) >>> at java.lang.Thread.run(Thread.java:745) >>> on ip-10-5-4-24.ec2.internal,16020,1451942002174, tracking started >>> null, retrying after=20001ms, replay=1ops >>> INFO >>> [ip-10-5-4-26.ec2.internal,16020,1451996087089-recovery-writer--pool5-t1] >>> client.AsyncProcess: #3, waiting for 2 actions to finish >>> INFO >>> [ip-10-5-4-26.ec2.internal,16020,1451996087089-recovery-writer--pool5-t2] >>> client.AsyncProcess: #4, waiting for 2 actions to finish >> >> >> It looks that they are getting into a state where some region servers are >> waiting for other regions that are not available yet in other servers. >> >> On HBase UI I can see servers stuck on this messages: >> >> *Description:* Replaying edits from >>> hdfs://.../recovered.edits/0000000000000464197 >>> *Status:* Running pre-WAL-restore hook in coprocessors (since 48mins, >>> 45sec ago) >> >> >> Another interesting thing that I noticed is the *empty coprocessor list* for >> the servers that are stuck with 0 regions assigned. >> >> HBase master goes down after logging some of these messages: >> >> GeneralBulkAssigner: Failed bulking assigning N regions >> >> >> I was able to perform full restarts before start using local indexes and >> everything worked fine. This can probably be a misconfiguration from my >> side but I have checked different properties and approaches to restart the >> cluster and I'm unable to do it. >> >> My understanding about local indexes on phoenix (please correct me if I'm >> wrong) is that they are normal HBase tables and phoenix places the regions >> to ensure the proper data locality. Is the data locality fully maintained >> when we lose N region servers and/or the regions are moved? >> >> Any insights would be very helpful. >> >> Thank you >> Cheers >> Pedro >> > >
