Hi Nirmal, When the purging disabled, if there is already registered purging task then it'll be deleted, and therefore the it's anyhow required to access the task service if it's enabled/disabled.
But we can check the existence of task service, and do the analytics purging related operation if and only if the task service is registered, with this we can resolve the issue irrespective of above configuration. And we can log a warn message if the task service is not registered and purging task is enabled. @Gihan: I think by default the purging needs to be enabled for the continuous operation with RDBMS datasource, without too many data being accumulated in the datasource. Any reason for this to be disabled? Thanks, Sinthuja. On Mon, Jun 29, 2015 at 2:07 PM, Nirmal Fernando <[email protected]> wrote: > That worked Sinthuja! Thanks. However, is it possible to disable the Task > Service initialization if the purging is disabled (which is the default > behaviour)? > > <analytics-data-purging> > > <purging-enable>false</purging-enable> > > <purge-node>true</purge-node> > > <cron-expression>0 0 0 * * ?</cron-expression> > > <purge-include-table-patterns> > > <table>.*</table> > > </purge-include-table-patterns> > > <data-retention-days>365</data-retention-days> > > </analytics-data-purging> > > On Mon, Jun 29, 2015 at 1:57 PM, Sinthuja Ragendran <[email protected]> > wrote: > >> Hi Nirmal, >> >> Thanks for sharing the necessary details. It's due to the data purging >> configuration has been enabled in the analytics-conf.xml which uses the >> task internally. can you please try to comment the analytics purging >> configuration from the repository/conf/analytics/analytics-conf.xml and see? >> >> Thanks, >> Sinthuja. >> >> On Mon, Jun 29, 2015 at 1:44 PM, Nirmal Fernando <[email protected]> wrote: >> >>> Hi Sinthuja, >>> >>> Thanks for the explanation. I think I should have used DAL instead of >>> DAS. Yes, so what I talking here is about the DAL features. Exact error is >>> [1] and reason for this is TaskService being null. Can you please check? >>> >>> [1] >>> >>> 15/06/28 11:54:51 INFO MemoryStore: Block broadcast_0 stored as values >>> in memory (estimated size 3.4 KB, free 265.1 MB) >>> >>> 15/06/28 11:55:02 ERROR Executor: Exception in task 0.0 in stage 0.0 >>> (TID 0) >>> >>> java.lang.NullPointerException >>> >>> at >>> org.wso2.carbon.analytics.dataservice.AnalyticsDataServiceImpl.<init>(AnalyticsDataServiceImpl.java:149) >>> >>> at >>> org.wso2.carbon.analytics.dataservice.AnalyticsServiceHolder.checkAndPopulateCustomAnalyticsDS(AnalyticsServiceHolder.java:79) >>> >>> at >>> org.wso2.carbon.analytics.dataservice.AnalyticsServiceHolder.getAnalyticsDataService(AnalyticsServiceHolder.java:67) >>> >>> at >>> org.wso2.carbon.analytics.spark.core.internal.ServiceHolder.getAnalyticsDataService(ServiceHolder.java:73) >>> >>> at >>> org.wso2.carbon.analytics.spark.core.util.AnalyticsRDD.compute(AnalyticsRDD.java:81) >>> >>> at >>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>> >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>> >>> at >>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>> >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>> >>> at >>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>> >>> at >>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >>> >>> at org.apache.spark.scheduler.Task.run(Task.scala:64) >>> >>> at >>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> >>> at java.lang.Thread.run(Thread.java:745) >>> >>> On Mon, Jun 29, 2015 at 12:16 PM, Sinthuja Ragendran <[email protected]> >>> wrote: >>> >>>> Hi nirmal, >>>> >>>> DAS features such as scripts scheduling, purging,etc are used to submit >>>> the jobs (only spark queries) to external spark cluster, rather those DAS >>>> features jars doesn't need to exists within the external spark cluster >>>> instance. For example, if we consider spark script scheduled execution >>>> scenario which uses Task OSGI services, and the task triggering will be be >>>> occurred wihing DAS node (OSGI env), furthermore when the spark is >>>> configured externally the job will be handed over to the external cluster, >>>> and then results will be given back to DAS node. Therefore I don't think >>>> any of the DAS features jars other than DAL feature jars will be required >>>> to be inside the external spark cluster. >>>> >>>> Can you please explain more on what is your usecase? And how you have >>>> configured the setup with DAS features? >>>> >>>> Thanks, >>>> Sinthuja. >>>> >>>> >>>> On Sunday, June 28, 2015, Nirmal Fernando <[email protected]> wrote: >>>> >>>>> Hi DAS team, >>>>> >>>>> It appears that we have to think and implement DAS features so that >>>>> they will run even in an non-OSGi environment like an external Spark >>>>> scenario. We have some DAS features which are dependent on Task Service >>>>> etc. and they are failing when we use the from within a Spark job which >>>>> runs on an external Spark cluster. >>>>> >>>>> How can we solve this? >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> *Sinthuja Rajendran* >> Associate Technical Lead >> WSO2, Inc.:http://wso2.com >> >> Blog: http://sinthu-rajan.blogspot.com/ >> Mobile: +94774273955 >> >> >> > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *Sinthuja Rajendran* Associate Technical Lead WSO2, Inc.:http://wso2.com Blog: http://sinthu-rajan.blogspot.com/ Mobile: +94774273955
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
