Hi Nirmal, Thanks for sharing the necessary details. It's due to the data purging configuration has been enabled in the analytics-conf.xml which uses the task internally. can you please try to comment the analytics purging configuration from the repository/conf/analytics/analytics-conf.xml and see?
Thanks, Sinthuja. On Mon, Jun 29, 2015 at 1:44 PM, Nirmal Fernando <[email protected]> wrote: > Hi Sinthuja, > > Thanks for the explanation. I think I should have used DAL instead of DAS. > Yes, so what I talking here is about the DAL features. Exact error is [1] > and reason for this is TaskService being null. Can you please check? > > [1] > > 15/06/28 11:54:51 INFO MemoryStore: Block broadcast_0 stored as values in > memory (estimated size 3.4 KB, free 265.1 MB) > > 15/06/28 11:55:02 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID > 0) > > java.lang.NullPointerException > > at > org.wso2.carbon.analytics.dataservice.AnalyticsDataServiceImpl.<init>(AnalyticsDataServiceImpl.java:149) > > at > org.wso2.carbon.analytics.dataservice.AnalyticsServiceHolder.checkAndPopulateCustomAnalyticsDS(AnalyticsServiceHolder.java:79) > > at > org.wso2.carbon.analytics.dataservice.AnalyticsServiceHolder.getAnalyticsDataService(AnalyticsServiceHolder.java:67) > > at > org.wso2.carbon.analytics.spark.core.internal.ServiceHolder.getAnalyticsDataService(ServiceHolder.java:73) > > at > org.wso2.carbon.analytics.spark.core.util.AnalyticsRDD.compute(AnalyticsRDD.java:81) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > > at > org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) > > at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) > > at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) > > at > org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > > at org.apache.spark.scheduler.Task.run(Task.scala:64) > > at > org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > > at java.lang.Thread.run(Thread.java:745) > > On Mon, Jun 29, 2015 at 12:16 PM, Sinthuja Ragendran <[email protected]> > wrote: > >> Hi nirmal, >> >> DAS features such as scripts scheduling, purging,etc are used to submit >> the jobs (only spark queries) to external spark cluster, rather those DAS >> features jars doesn't need to exists within the external spark cluster >> instance. For example, if we consider spark script scheduled execution >> scenario which uses Task OSGI services, and the task triggering will be be >> occurred wihing DAS node (OSGI env), furthermore when the spark is >> configured externally the job will be handed over to the external cluster, >> and then results will be given back to DAS node. Therefore I don't think >> any of the DAS features jars other than DAL feature jars will be required >> to be inside the external spark cluster. >> >> Can you please explain more on what is your usecase? And how you have >> configured the setup with DAS features? >> >> Thanks, >> Sinthuja. >> >> >> On Sunday, June 28, 2015, Nirmal Fernando <[email protected]> wrote: >> >>> Hi DAS team, >>> >>> It appears that we have to think and implement DAS features so that they >>> will run even in an non-OSGi environment like an external Spark scenario. >>> We have some DAS features which are dependent on Task Service etc. and they >>> are failing when we use the from within a Spark job which runs on an >>> external Spark cluster. >>> >>> How can we solve this? >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *Sinthuja Rajendran* Associate Technical Lead WSO2, Inc.:http://wso2.com Blog: http://sinthu-rajan.blogspot.com/ Mobile: +94774273955
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
