Hi Sinthuja,
Thanks for the explanation. I think I should have used DAL instead of DAS.
Yes, so what I talking here is about the DAL features. Exact error is [1]
and reason for this is TaskService being null. Can you please check?
[1]
15/06/28 11:54:51 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 3.4 KB, free 265.1 MB)
15/06/28 11:55:02 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NullPointerException
at
org.wso2.carbon.analytics.dataservice.AnalyticsDataServiceImpl.<init>(AnalyticsDataServiceImpl.java:149)
at
org.wso2.carbon.analytics.dataservice.AnalyticsServiceHolder.checkAndPopulateCustomAnalyticsDS(AnalyticsServiceHolder.java:79)
at
org.wso2.carbon.analytics.dataservice.AnalyticsServiceHolder.getAnalyticsDataService(AnalyticsServiceHolder.java:67)
at
org.wso2.carbon.analytics.spark.core.internal.ServiceHolder.getAnalyticsDataService(ServiceHolder.java:73)
at
org.wso2.carbon.analytics.spark.core.util.AnalyticsRDD.compute(AnalyticsRDD.java:81)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
On Mon, Jun 29, 2015 at 12:16 PM, Sinthuja Ragendran <[email protected]>
wrote:
> Hi nirmal,
>
> DAS features such as scripts scheduling, purging,etc are used to submit
> the jobs (only spark queries) to external spark cluster, rather those DAS
> features jars doesn't need to exists within the external spark cluster
> instance. For example, if we consider spark script scheduled execution
> scenario which uses Task OSGI services, and the task triggering will be be
> occurred wihing DAS node (OSGI env), furthermore when the spark is
> configured externally the job will be handed over to the external cluster,
> and then results will be given back to DAS node. Therefore I don't think
> any of the DAS features jars other than DAL feature jars will be required
> to be inside the external spark cluster.
>
> Can you please explain more on what is your usecase? And how you have
> configured the setup with DAS features?
>
> Thanks,
> Sinthuja.
>
>
> On Sunday, June 28, 2015, Nirmal Fernando <[email protected]> wrote:
>
>> Hi DAS team,
>>
>> It appears that we have to think and implement DAS features so that they
>> will run even in an non-OSGi environment like an external Spark scenario.
>> We have some DAS features which are dependent on Task Service etc. and they
>> are failing when we use the from within a Spark job which runs on an
>> external Spark cluster.
>>
>> How can we solve this?
>>
>> --
>>
>> Thanks & regards,
>> Nirmal
>>
>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>> Mobile: +94715779733
>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
--
Thanks & regards,
Nirmal
Associate Technical Lead - Data Technologies Team, WSO2 Inc.
Mobile: +94715779733
Blog: http://nirmalfdo.blogspot.com/
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
