For Phoenix phoenix.query.maxGlobalMemoryPercentage is 15% of heap https://phoenix.apache.org/tuning.html. Block cache and memstore memory setting are via usual HBase settings and their usage is exposed via jmx at http://<rs_ip>:60030/jmx. Was there any useful info in GC logs? Also 2GB heap is on the low side, can you rerun you test with setting heap to 5 and 10GB?
On Thu, Mar 31, 2016 at 7:01 AM, Amit Shah <amits...@gmail.com> wrote: > Another such instance of the crash is described below. > > > When the regions are evenly distributed across the 3 region servers, one > of the region server crashes without any errors in the logs. It has long GC > pauses. The heap usage on the server had not crossed above 900 MB and the > allocated heap is upto 2 GB. Attached are logs and jconsole screenshot. > > > > Wonder what is causing the GC pauses? Any idea on how is the region server > heap distribution across the block cache, phoenix usage, memstore etc? > > > Thanks, > > Amit. > > > > On Thu, Mar 31, 2016 at 7:14 PM, Amit Shah <amits...@gmail.com> wrote: > >> There have been multiple reasons of the region server jvm crash. For one >> of such errors, the logs are attached. Let me know your inputs. >> >> Thanks, >> Amit. >> >> >> On Thu, Mar 31, 2016 at 6:15 PM, Mujtaba Chohan <mujt...@apache.org> >> wrote: >> >>> Can you attached last couple of hundred lines from RS log before it >>> crashed? Also what's the RS heap size? >>> >>> >>> On Thu, Mar 31, 2016 at 1:48 AM, Amit Shah <amits...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> We have been experimenting hbase (version 1.0) and phoenix (version >>>> 4.6) for our OLAP workload. In order to precalculate aggregates we have >>>> been executing an upsert phoenix query that aggregates raw data (over 10 >>>> mil records) to generate an OLAP cube. >>>> >>>> While executing the query, one of the region servers in a cluster of 3 >>>> RS crashes. I am trying to figure out what could be causing the region >>>> server to crash. >>>> The server shows high disk operations before the jvm crashed. Kindly >>>> find the disk and other stats attached. >>>> >>>> Any suggestions on where could I look into would be helpful. >>>> >>>> The upsert query that was executed is >>>> >>>> upsert into AGENT_TER_PRO >>>> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT) >>>> select /*+ INDEX(TRANSACTIONS AG_TER_PRO2) */ >>>> AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES) >>>> SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD) >>>> SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT) AVG_PRICE_PER_UNIT from >>>> TRANSACTIONS group by AGENT_ID,TERRITORY_ID,PRODUCT_ID; >>>> >>>> Thanks, >>>> Amit. >>>> >>>> >>> >> >
