Hi Anjana, Yes. Agreed, sorry I misread that. In that case OOM should be fine after the fix.
Thank you, Ayoma. On Wed, Dec 16, 2015 at 6:11 PM, Anjana Fernando <anj...@wso2.com> wrote: > Hi Ayoma, > > Thanks for checking up on it, actually "getAllIndexedTables" doesn't > return the Set here, it returns an array that was previously populated in > the refresh operation, so no need to synchronize that method. > > Cheers, > Anjana. > > On Wed, Dec 16, 2015 at 5:44 PM, Ayoma Wijethunga <ay...@wso2.com> wrote: > >> And, missed mentioning that when this this race condition / state >> corruption happens all "get" operations performed on Set/Map get blocked >> resulting in OOM situation. [1 >> <http://mailinator.blogspot.gr/2009/06/beautiful-race-condition.html>] >> has all that explained nicely. I have checked a heap dump in a similar >> situation and if you take one, you will clearly see many threads waiting to >> access this Set instance. >> >> [1] http://mailinator.blogspot.gr/2009/06/beautiful-race-condition.html >> >> On Wed, Dec 16, 2015 at 5:37 PM, Ayoma Wijethunga <ay...@wso2.com> wrote: >> >>> Hi Anjana, >>> >>> Sorry, I didn't notice that you have already replied this thread. >>> >>> However, please consider my point on "getAllIndexedTables" as well. >>> >>> Thank you, >>> Ayoma. >>> >>> On Wed, Dec 16, 2015 at 5:12 PM, Anjana Fernando <anj...@wso2.com> >>> wrote: >>> >>>> Hi Sumedha, >>>> >>>> Thank you for reporting the issue. I've fixed the concurrent >>>> modification exception issue, where, actually both the methods >>>> "addIndexedTable" and "removeIndexedTable" needed to be synchronized, since >>>> they both work on the shared Set object there. >>>> >>>> As for the OOM issue, can you please share a heap dump when the OOM >>>> happened. So we can see what is causing this. And also, I see there are >>>> multiple scripts running at the same time, so this actually can be a >>>> legitimate error also, where the server actually doesn't have enough memory >>>> to continue its operations. @Niranda, please share if there is any info on >>>> tuning Spark's memory requirements. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> On Wed, Dec 16, 2015 at 3:32 PM, Sumedha Rubasinghe <sume...@wso2.com> >>>> wrote: >>>> >>>>> We have DAS Lite included in IoT Server and several summarisation >>>>> scripts deployed. Server is going OOM frequently with following exception. >>>>> >>>>> Shouldn't this[1] method be synchronised? >>>>> >>>>> [1] >>>>> https://github.com/wso2/carbon-analytics/blob/master/components/analytics-core/org.wso2.carbon.analytics.dataservice.core/src/main/java/org/wso2/carbon/analytics/dataservice/core/indexing/AnalyticsIndexedTableStore.java#L45 >>>>> >>>>> >>>>> >>>>>>>>>>> >>>>> [2015-12-16 15:11:00,004] INFO >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>> schedule task for: Light_Sensor_Script for tenant id: -1234 >>>>> [2015-12-16 15:11:00,005] INFO >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>> schedule task for: Magnetic_Sensor_Script for tenant id: -1234 >>>>> [2015-12-16 15:11:00,005] INFO >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>> schedule task for: Pressure_Sensor_Script for tenant id: -1234 >>>>> [2015-12-16 15:11:00,006] INFO >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>> schedule task for: Proximity_Sensor_Script for tenant id: -1234 >>>>> [2015-12-16 15:11:00,006] INFO >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>> schedule task for: Rotation_Sensor_Script for tenant id: -1234 >>>>> [2015-12-16 15:11:00,007] INFO >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>> schedule task for: Temperature_Sensor_Script for tenant id: -1234 >>>>> [2015-12-16 15:11:01,132] ERROR >>>>> {org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter} - Error in >>>>> executing task: null >>>>> java.util.ConcurrentModificationException >>>>> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) >>>>> at java.util.HashMap$KeyIterator.next(HashMap.java:956) >>>>> at java.util.AbstractCollection.toArray(AbstractCollection.java:195) >>>>> at >>>>> org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsIndexedTableStore.refreshIndexedTableArray(AnalyticsIndexedTableStore.java:46) >>>>> at >>>>> org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsIndexedTableStore.addIndexedTable(AnalyticsIndexedTableStore.java:37) >>>>> at >>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.refreshIndexedTableStoreEntry(AnalyticsDataServiceImpl.java:512) >>>>> at >>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.invalidateAnalyticsTableInfo(AnalyticsDataServiceImpl.java:525) >>>>> at >>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.checkAndInvalidateTableInfo(AnalyticsDataServiceImpl.java:504) >>>>> at >>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.setTableSchema(AnalyticsDataServiceImpl.java:495) >>>>> at >>>>> org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelation.insert(AnalyticsRelation.java:162) >>>>> at >>>>> org.apache.spark.sql.sources.InsertIntoDataSource.run(commands.scala:53) >>>>> at >>>>> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57) >>>>> at >>>>> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57) >>>>> at >>>>> org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68) >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88) >>>>> at >>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88) >>>>> at >>>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) >>>>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87) >>>>> at >>>>> org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:950) >>>>> at >>>>> org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:950) >>>>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:144) >>>>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:128) >>>>> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) >>>>> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:755) >>>>> at >>>>> org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:710) >>>>> at >>>>> org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:692) >>>>> at >>>>> org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:199) >>>>> at >>>>> org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:149) >>>>> at >>>>> org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:57) >>>>> at >>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67) >>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>> at >>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> [2015-12-16 15:12:00,001] INFO >>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>> schedule task for: Accelerometer_Sensor_Script for tenant id: -1234 >>>>> >>>>> -- >>>>> /sumedha >>>>> m: +94 773017743 >>>>> b : bit.ly/sumedha >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@wso2.org >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> Ayoma Wijethunga >>> Software Engineer >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> Mobile : +94 (0) 719428123 <+94+(0)+719428123> >>> Blog : http://www.ayomaonline.com >>> LinkedIn: https://www.linkedin.com/in/ayoma >>> >> >> >> >> -- >> Ayoma Wijethunga >> Software Engineer >> WSO2, Inc.; http://wso2.com >> lean.enterprise.middleware >> >> Mobile : +94 (0) 719428123 <+94+(0)+719428123> >> Blog : http://www.ayomaonline.com >> LinkedIn: https://www.linkedin.com/in/ayoma >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- Ayoma Wijethunga Software Engineer WSO2, Inc.; http://wso2.com lean.enterprise.middleware Mobile : +94 (0) 719428123 <+94+(0)+719428123> Blog : http://www.ayomaonline.com LinkedIn: https://www.linkedin.com/in/ayoma
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
