A couple of observations 1. I could see GC pauses in the logs but I do not think that could be causing the jvm exists. I have configured the region server heap to be 2 GB. The jconsole indicates that it hardly reaches 1.5 GB. Kindly find some graphs attached.
2. On another run the phoenix client failed with a socket timeout operation. The error is pasted here - http://pastebin.com/mAWwiE2J Are there client timeouts for each region server? It looks like the default is 1 min. How can I increase them? Thanks, Amit. On Thu, Mar 31, 2016 at 2:39 PM, Samir Ahmic <[email protected]> wrote: > Hi Amit, > > Check regionserver logs, usual suspects, are log running GC and HDFS client > related issues. Check for compaction queue. > > Regrads > Samir > > On Thu, Mar 31, 2016 at 10:48 AM, Amit Shah <[email protected]> wrote: > > > Hi, > > > > We have been experimenting hbase (version 1.0) and phoenix (version 4.6) > > for our OLAP workload. In order to precalculate aggregates we have been > > executing an upsert phoenix query that aggregates raw data (over 10 mil > > records) to generate an OLAP cube. > > > > While executing the query, one of the region servers in a cluster of 3 RS > > crashes. I am trying to figure out what could be causing the region > server > > to crash. > > The server shows high disk operations before the jvm crashed. Kindly find > > the disk and other stats attached. > > > > Any suggestions on where could I look into would be helpful. > > > > The upsert query that was executed is > > > > upsert into AGENT_TER_PRO > > > (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT) > > select /*+ INDEX(TRANSACTIONS AG_TER_PRO2) */ > > AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES) > > SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD) > > SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT) AVG_PRICE_PER_UNIT from > > TRANSACTIONS group by AGENT_ID,TERRITORY_ID,PRODUCT_ID; > > > > Thanks, > > Amit. > > > > >
