Hi Amit, I guess processing with HBase + Phoenix is not working for your use-case, it needs a lot of memory and of course swap. I imagine there's no direct solution - but post here if you find one (I imagine some good to try options: splitting the query into smaller ones, salt the table in more buckets or any other smarter idea including rewriting the query to minimize the memory usage - including writing your own aggregation function and doing things optimally given the fact you know your algorithm or workload/ goal).
P.S. I think we know each other, right? Regards, Constantin Pe 1 apr. 2016 4:16 p.m., "Amit Shah" <amits...@gmail.com> a scris: > I tried raising the region server heap memory to 3.5 GB (I have only 8 gb > on my dev server of which I have given 2 GB to the hbase bucket cache). On > jconsole I see the max heap size as 1.78 GB. > What could be the reason of the difference in the heap size? > > On Thu, Mar 31, 2016 at 11:26 PM, Mujtaba Chohan <mujt...@apache.org> > wrote: > >> For Phoenix phoenix.query.maxGlobalMemoryPercentage is 15% of heap >> https://phoenix.apache.org/tuning.html. Block cache and memstore memory >> setting are via usual HBase settings and their usage is exposed via jmx at >> http://<rs_ip>:60030/jmx. Was there any useful info in GC logs? Also 2GB >> heap is on the low side, can you rerun you test with setting heap to 5 and >> 10GB? >> >> On Thu, Mar 31, 2016 at 7:01 AM, Amit Shah <amits...@gmail.com> wrote: >> >>> Another such instance of the crash is described below. >>> >>> >>> When the regions are evenly distributed across the 3 region servers, one >>> of the region server crashes without any errors in the logs. It has long GC >>> pauses. The heap usage on the server had not crossed above 900 MB and the >>> allocated heap is upto 2 GB. Attached are logs and jconsole screenshot. >>> >>> >>> >>> Wonder what is causing the GC pauses? Any idea on how is the region >>> server heap distribution across the block cache, phoenix usage, memstore >>> etc? >>> >>> >>> Thanks, >>> >>> Amit. >>> >>> >>> >>> On Thu, Mar 31, 2016 at 7:14 PM, Amit Shah <amits...@gmail.com> wrote: >>> >>>> There have been multiple reasons of the region server jvm crash. For >>>> one of such errors, the logs are attached. Let me know your inputs. >>>> >>>> Thanks, >>>> Amit. >>>> >>>> >>>> On Thu, Mar 31, 2016 at 6:15 PM, Mujtaba Chohan <mujt...@apache.org> >>>> wrote: >>>> >>>>> Can you attached last couple of hundred lines from RS log before it >>>>> crashed? Also what's the RS heap size? >>>>> >>>>> >>>>> On Thu, Mar 31, 2016 at 1:48 AM, Amit Shah <amits...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> We have been experimenting hbase (version 1.0) and phoenix (version >>>>>> 4.6) for our OLAP workload. In order to precalculate aggregates we have >>>>>> been executing an upsert phoenix query that aggregates raw data (over 10 >>>>>> mil records) to generate an OLAP cube. >>>>>> >>>>>> While executing the query, one of the region servers in a cluster of >>>>>> 3 RS crashes. I am trying to figure out what could be causing the region >>>>>> server to crash. >>>>>> The server shows high disk operations before the jvm crashed. Kindly >>>>>> find the disk and other stats attached. >>>>>> >>>>>> Any suggestions on where could I look into would be helpful. >>>>>> >>>>>> The upsert query that was executed is >>>>>> >>>>>> upsert into AGENT_TER_PRO >>>>>> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT) >>>>>> select /*+ INDEX(TRANSACTIONS AG_TER_PRO2) */ >>>>>> AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES) >>>>>> SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD) >>>>>> SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT) AVG_PRICE_PER_UNIT from >>>>>> TRANSACTIONS group by AGENT_ID,TERRITORY_ID,PRODUCT_ID; >>>>>> >>>>>> Thanks, >>>>>> Amit. >>>>>> >>>>>> >>>>> >>>> >>> >> >
