External spark cluster doesn't work with the BAM story due the DAL layer. So, we are dependent on you guys to implement ML-BAM integration :-)
On Tue, Mar 10, 2015 at 10:09 AM, Niranda Perera <[email protected]> wrote: > @supun I suggest you use an external spark cluster for the moment. as of > the current BAM, we still have not finalized external job submission > strategy in a BAM cluster. I will discuss this with Anjana and get back to > you! > > On Tue, Mar 10, 2015 at 10:02 AM, Supun Sethunga <[email protected]> wrote: > >> [looping Nirmal] >> >> On Tue, Mar 10, 2015 at 10:01 AM, Supun Sethunga <[email protected]> wrote: >> >>> [looping Nirmal] >>> >>> Hi Niranda, >>> >>> Thanks for the clarification. >>> >>> @Nirmal : Seems we have to wait till the BAM M2 release for ML-BAM >>> Integration :) . But we still can enable to use an external spark cluster >>> for ML. >>> >>> Regards, >>> Supun >>> >>> On Tue, Mar 10, 2015 at 9:49 AM, Niranda Perera <[email protected]> >>> wrote: >>> >>>> cc+ Anjana >>>> >>>> Hi Supun, >>>> >>>> As per the M1 release, we have only the standalone 'local' mode spark >>>> instance. as you correctly said, we have set the master to 'local' in the >>>> BAM . please refer init() method here [1]. In the init() method, we are >>>> starting a spark context. >>>> >>>> we are still working on spark clustering. the plan is to release it in >>>> the M2. >>>> >>>> [1] >>>> https://github.com/wso2/carbon-analytics/blob/master/components/analytics/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/AnalyticsExecutionService.java >>>> >>>> On Mon, Mar 9, 2015 at 5:48 PM, Supun Sethunga <[email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I tried starting up a BAM server, and submitting a spark-job to the >>>>> Spark which is built in BAM, from an external java client. In doing this, >>>>> I >>>>> was not able to create a Spark Context using BAM's spark instance as >>>>> master. (AFAIK, the way to do this is to create a Spark context inside the >>>>> client, and point BAM's Spark instance as the master, and then submit any >>>>> jobs to the created context.) >>>>> >>>>> Possible reason might be, in BAM, Spark instance (i.e. SparkContext) >>>>> is started locally with one worker thread.(setting master as 'local'). >>>>> Hence the spark instance cannot be accessed by any external client. (Refer >>>>> attached image "BAM-spark.png") >>>>> AFAIK, to be accessible, BAM's Spark instance needs be started in the >>>>> clustering mode. Then any external client also can use the BAM's spark, >>>>> through the "spark.master" URL, which is "spark://localhost:7077" by >>>>> default. (Refer attached image " >>>>> external-spark-cluster.png"). >>>>> >>>>> I'm not sure whether this feature (clustering Spark) is currently >>>>> supported by BAM (in M1 release)? >>>>> >>>>> If so, whats the way forward? >>>>> >>>>> Thanks, >>>>> Supun >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Niranda Perera* >>>> Software Engineer, WSO2 Inc. >>>> Mobile: +94-71-554-8430 >>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > *Niranda Perera* > Software Engineer, WSO2 Inc. > Mobile: +94-71-554-8430 > Twitter: @n1r44 <https://twitter.com/N1R44> > -- Thanks & regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
