Hi Sumedha, Thank you for reporting the issue. I've fixed the concurrent modification exception issue, where, actually both the methods "addIndexedTable" and "removeIndexedTable" needed to be synchronized, since they both work on the shared Set object there.
As for the OOM issue, can you please share a heap dump when the OOM happened. So we can see what is causing this. And also, I see there are multiple scripts running at the same time, so this actually can be a legitimate error also, where the server actually doesn't have enough memory to continue its operations. @Niranda, please share if there is any info on tuning Spark's memory requirements. Cheers, Anjana. On Wed, Dec 16, 2015 at 3:32 PM, Sumedha Rubasinghe <[email protected]> wrote: > We have DAS Lite included in IoT Server and several summarisation scripts > deployed. Server is going OOM frequently with following exception. > > Shouldn't this[1] method be synchronised? > > [1] > https://github.com/wso2/carbon-analytics/blob/master/components/analytics-core/org.wso2.carbon.analytics.dataservice.core/src/main/java/org/wso2/carbon/analytics/dataservice/core/indexing/AnalyticsIndexedTableStore.java#L45 > > > >>>>>>>>>>> > [2015-12-16 15:11:00,004] INFO > {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the > schedule task for: Light_Sensor_Script for tenant id: -1234 > [2015-12-16 15:11:00,005] INFO > {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the > schedule task for: Magnetic_Sensor_Script for tenant id: -1234 > [2015-12-16 15:11:00,005] INFO > {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the > schedule task for: Pressure_Sensor_Script for tenant id: -1234 > [2015-12-16 15:11:00,006] INFO > {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the > schedule task for: Proximity_Sensor_Script for tenant id: -1234 > [2015-12-16 15:11:00,006] INFO > {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the > schedule task for: Rotation_Sensor_Script for tenant id: -1234 > [2015-12-16 15:11:00,007] INFO > {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the > schedule task for: Temperature_Sensor_Script for tenant id: -1234 > [2015-12-16 15:11:01,132] ERROR > {org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter} - Error in > executing task: null > java.util.ConcurrentModificationException > at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) > at java.util.HashMap$KeyIterator.next(HashMap.java:956) > at java.util.AbstractCollection.toArray(AbstractCollection.java:195) > at > org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsIndexedTableStore.refreshIndexedTableArray(AnalyticsIndexedTableStore.java:46) > at > org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsIndexedTableStore.addIndexedTable(AnalyticsIndexedTableStore.java:37) > at > org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.refreshIndexedTableStoreEntry(AnalyticsDataServiceImpl.java:512) > at > org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.invalidateAnalyticsTableInfo(AnalyticsDataServiceImpl.java:525) > at > org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.checkAndInvalidateTableInfo(AnalyticsDataServiceImpl.java:504) > at > org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.setTableSchema(AnalyticsDataServiceImpl.java:495) > at > org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelation.insert(AnalyticsRelation.java:162) > at org.apache.spark.sql.sources.InsertIntoDataSource.run(commands.scala:53) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57) > at > org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57) > at > org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88) > at > org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88) > at > org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) > at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:950) > at > org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:950) > at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:144) > at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:128) > at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) > at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:755) > at > org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:710) > at > org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:692) > at > org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:199) > at > org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:149) > at > org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:57) > at > org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > [2015-12-16 15:12:00,001] INFO > {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the > schedule task for: Accelerometer_Sensor_Script for tenant id: -1234 > > -- > /sumedha > m: +94 773017743 > b : bit.ly/sumedha > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
