Hey Constantin, Yeah, I realized that there won't be a quick solution to the issue. There have been patterns that I got to know when trying out various things. As you said I will have to tune the queries and/or the cluster and keep trying.
Regards, Amit P.S - will connect with you separately. On Fri, Apr 1, 2016 at 10:19 PM, Ciureanu Constantin < ciureanu.constan...@gmail.com> wrote: > Hi Amit, > > I guess processing with HBase + Phoenix is not working for your use-case, > it needs a lot of memory and of course swap. I imagine there's no direct > solution - but post here if you find one (I imagine some good to try > options: splitting the query into smaller ones, salt the table in more > buckets or any other smarter idea including rewriting the query to minimize > the memory usage - including writing your own aggregation function and > doing things optimally given the fact you know your algorithm or workload/ > goal). > > P.S. I think we know each other, right? > > Regards, > Constantin > Pe 1 apr. 2016 4:16 p.m., "Amit Shah" <amits...@gmail.com> a scris: > >> I tried raising the region server heap memory to 3.5 GB (I have only 8 gb >> on my dev server of which I have given 2 GB to the hbase bucket cache). On >> jconsole I see the max heap size as 1.78 GB. >> What could be the reason of the difference in the heap size? >> >> On Thu, Mar 31, 2016 at 11:26 PM, Mujtaba Chohan <mujt...@apache.org> >> wrote: >> >>> For Phoenix phoenix.query.maxGlobalMemoryPercentage is 15% of heap >>> https://phoenix.apache.org/tuning.html. Block cache and memstore memory >>> setting are via usual HBase settings and their usage is exposed via jmx at >>> http://<rs_ip>:60030/jmx. Was there any useful info in GC logs? Also >>> 2GB heap is on the low side, can you rerun you test with setting heap to 5 >>> and 10GB? >>> >>> On Thu, Mar 31, 2016 at 7:01 AM, Amit Shah <amits...@gmail.com> wrote: >>> >>>> Another such instance of the crash is described below. >>>> >>>> >>>> When the regions are evenly distributed across the 3 region servers, >>>> one of the region server crashes without any errors in the logs. It has >>>> long GC pauses. The heap usage on the server had not crossed above 900 MB >>>> and the allocated heap is upto 2 GB. Attached are logs and jconsole >>>> screenshot. >>>> >>>> >>>> >>>> Wonder what is causing the GC pauses? Any idea on how is the region >>>> server heap distribution across the block cache, phoenix usage, memstore >>>> etc? >>>> >>>> >>>> Thanks, >>>> >>>> Amit. >>>> >>>> >>>> >>>> On Thu, Mar 31, 2016 at 7:14 PM, Amit Shah <amits...@gmail.com> wrote: >>>> >>>>> There have been multiple reasons of the region server jvm crash. For >>>>> one of such errors, the logs are attached. Let me know your inputs. >>>>> >>>>> Thanks, >>>>> Amit. >>>>> >>>>> >>>>> On Thu, Mar 31, 2016 at 6:15 PM, Mujtaba Chohan <mujt...@apache.org> >>>>> wrote: >>>>> >>>>>> Can you attached last couple of hundred lines from RS log before it >>>>>> crashed? Also what's the RS heap size? >>>>>> >>>>>> >>>>>> On Thu, Mar 31, 2016 at 1:48 AM, Amit Shah <amits...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> We have been experimenting hbase (version 1.0) and phoenix (version >>>>>>> 4.6) for our OLAP workload. In order to precalculate aggregates we have >>>>>>> been executing an upsert phoenix query that aggregates raw data (over 10 >>>>>>> mil records) to generate an OLAP cube. >>>>>>> >>>>>>> While executing the query, one of the region servers in a cluster of >>>>>>> 3 RS crashes. I am trying to figure out what could be causing the region >>>>>>> server to crash. >>>>>>> The server shows high disk operations before the jvm crashed. Kindly >>>>>>> find the disk and other stats attached. >>>>>>> >>>>>>> Any suggestions on where could I look into would be helpful. >>>>>>> >>>>>>> The upsert query that was executed is >>>>>>> >>>>>>> upsert into AGENT_TER_PRO >>>>>>> (AGENT_ID,TERRITORY_ID,PRODUCT_ID,SUM_TOTAL_SALES,SUM_TOTAL_EXPENSES,SUM_UNIT_CNT_SOLD,AVG_PRICE_PER_UNIT) >>>>>>> select /*+ INDEX(TRANSACTIONS AG_TER_PRO2) */ >>>>>>> AGENT_ID,TERRITORY_ID,PRODUCT_ID, sum(TOTAL_SALES) >>>>>>> SUM_TOTAL_SALES,sum(TOTAL_EXPENSES) >>>>>>> SUM_TOTAL_EXPENSES,sum(UNIT_CNT_SOLD) >>>>>>> SUM_UNIT_CNT_SOLD,AVG(PRICE_PER_UNIT) AVG_PRICE_PER_UNIT from >>>>>>> TRANSACTIONS group by AGENT_ID,TERRITORY_ID,PRODUCT_ID; >>>>>>> >>>>>>> Thanks, >>>>>>> Amit. >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
