That worked Sinthuja! Thanks. However, is it possible to disable the Task
Service initialization if the purging is disabled (which is the default
behaviour)?
<analytics-data-purging>
<purging-enable>false</purging-enable>
<purge-node>true</purge-node>
<cron-expression>0 0 0 * * ?</cron-expression>
<purge-include-table-patterns>
<table>.*</table>
</purge-include-table-patterns>
<data-retention-days>365</data-retention-days>
</analytics-data-purging>
On Mon, Jun 29, 2015 at 1:57 PM, Sinthuja Ragendran <[email protected]>
wrote:
> Hi Nirmal,
>
> Thanks for sharing the necessary details. It's due to the data purging
> configuration has been enabled in the analytics-conf.xml which uses the
> task internally. can you please try to comment the analytics purging
> configuration from the repository/conf/analytics/analytics-conf.xml and see?
>
> Thanks,
> Sinthuja.
>
> On Mon, Jun 29, 2015 at 1:44 PM, Nirmal Fernando <[email protected]> wrote:
>
>> Hi Sinthuja,
>>
>> Thanks for the explanation. I think I should have used DAL instead of
>> DAS. Yes, so what I talking here is about the DAL features. Exact error is
>> [1] and reason for this is TaskService being null. Can you please check?
>>
>> [1]
>>
>> 15/06/28 11:54:51 INFO MemoryStore: Block broadcast_0 stored as values in
>> memory (estimated size 3.4 KB, free 265.1 MB)
>>
>> 15/06/28 11:55:02 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID
>> 0)
>>
>> java.lang.NullPointerException
>>
>> at
>> org.wso2.carbon.analytics.dataservice.AnalyticsDataServiceImpl.<init>(AnalyticsDataServiceImpl.java:149)
>>
>> at
>> org.wso2.carbon.analytics.dataservice.AnalyticsServiceHolder.checkAndPopulateCustomAnalyticsDS(AnalyticsServiceHolder.java:79)
>>
>> at
>> org.wso2.carbon.analytics.dataservice.AnalyticsServiceHolder.getAnalyticsDataService(AnalyticsServiceHolder.java:67)
>>
>> at
>> org.wso2.carbon.analytics.spark.core.internal.ServiceHolder.getAnalyticsDataService(ServiceHolder.java:73)
>>
>> at
>> org.wso2.carbon.analytics.spark.core.util.AnalyticsRDD.compute(AnalyticsRDD.java:81)
>>
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>
>> at
>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
>>
>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
>>
>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
>>
>> at
>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>>
>> at org.apache.spark.scheduler.Task.run(Task.scala:64)
>>
>> at
>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> On Mon, Jun 29, 2015 at 12:16 PM, Sinthuja Ragendran <[email protected]>
>> wrote:
>>
>>> Hi nirmal,
>>>
>>> DAS features such as scripts scheduling, purging,etc are used to submit
>>> the jobs (only spark queries) to external spark cluster, rather those DAS
>>> features jars doesn't need to exists within the external spark cluster
>>> instance. For example, if we consider spark script scheduled execution
>>> scenario which uses Task OSGI services, and the task triggering will be be
>>> occurred wihing DAS node (OSGI env), furthermore when the spark is
>>> configured externally the job will be handed over to the external cluster,
>>> and then results will be given back to DAS node. Therefore I don't think
>>> any of the DAS features jars other than DAL feature jars will be required
>>> to be inside the external spark cluster.
>>>
>>> Can you please explain more on what is your usecase? And how you have
>>> configured the setup with DAS features?
>>>
>>> Thanks,
>>> Sinthuja.
>>>
>>>
>>> On Sunday, June 28, 2015, Nirmal Fernando <[email protected]> wrote:
>>>
>>>> Hi DAS team,
>>>>
>>>> It appears that we have to think and implement DAS features so that
>>>> they will run even in an non-OSGi environment like an external Spark
>>>> scenario. We have some DAS features which are dependent on Task Service
>>>> etc. and they are failing when we use the from within a Spark job which
>>>> runs on an external Spark cluster.
>>>>
>>>> How can we solve this?
>>>>
>>>> --
>>>>
>>>> Thanks & regards,
>>>> Nirmal
>>>>
>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>>> Mobile: +94715779733
>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>
>>>>
>>>>
>>
>>
>> --
>>
>> Thanks & regards,
>> Nirmal
>>
>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>> Mobile: +94715779733
>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
>
>
> --
> *Sinthuja Rajendran*
> Associate Technical Lead
> WSO2, Inc.:http://wso2.com
>
> Blog: http://sinthu-rajan.blogspot.com/
> Mobile: +94774273955
>
>
>
--
Thanks & regards,
Nirmal
Associate Technical Lead - Data Technologies Team, WSO2 Inc.
Mobile: +94715779733
Blog: http://nirmalfdo.blogspot.com/
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
