The attachments you mentioned did go through. For #2, please adjust:
hbase.client.scanner.timeout.period hbase.rpc.timeout Both have default value of 60 seconds. If possible, please pastebin server log snippet before it crashed. Thanks On Thu, Mar 31, 2016 at 3:04 AM, Amit Shah <[email protected]> wrote: > A couple of observations > > 1. I could see GC pauses in the logs but I do not think that could be > causing the jvm exists. I have configured the region server heap to be 2 > GB. The jconsole indicates that it hardly reaches 1.5 GB. Kindly find some > graphs attached. > > 2. On another run the phoenix client failed with a socket timeout > operation. The error is pasted here - http://pastebin.com/mAWwiE2J > Are there client timeouts for each region server? It looks like the > default is 1 min. How can I increase them? > > Thanks, > Amit. > > On Thu, Mar 31, 2016 at 2:39 PM, Samir Ahmic <[email protected]> > wrote: > >> Hi Amit, >> >> Check regionserver logs, usual suspects, are log running GC and HDFS >> client >> related issues. Check for compaction queue. >> >> Regrads >> Samir >> >> On Thu, Mar 31, 2016 at 10:48 AM, Amit Shah <[email protected]> wrote: >> >> > Hi, >> > >> > We have been experimenting hbase (version 1.0) and phoenix (version 4.6) >> > for our OLAP workload. In order to precalculate aggregates we have been >> > executing an upsert phoenix query that aggregates raw data (over 10 mil >> > records) to generate an OLAP cube. >> > >> > While executing the query, one of the region servers in a cluster of 3 >> RS >> > crashes. I am trying to figure out what could be causing the region >> server >> > to crash. >> > The server shows high disk operations before the jvm crashed. Kindly >> find >> > the disk and other stats attached. >> > >> > Any suggestions on where could I look into would be helpful. >> > >> > The upsert query that was executed is >> > >> > upsert into AGENT_TER_PRO >> > >> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT) >> > select /*+ INDEX(TRANSACTIONS AG_TER_PRO2) */ >> > AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES) >> > SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) >> SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD) >> > SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT) AVG_PRICE_PER_UNIT from >> > TRANSACTIONS group by AGENT_ID,TERRITORY_ID,PRODUCT_ID; >> > >> > Thanks, >> > Amit. >> > >> > >> > >
