Okay Anjana, noted! On Tue, Mar 10, 2015 at 12:10 PM, Anjana Fernando <[email protected]> wrote:
> Hi Niranda, > > When we get the clustering to properly to work, which would be before > Friday's release. It should be straightforward for ML to point to our > master server and submit jobs. So lets wait till we finalize our > implementation, and then ML team can work with us to get their use case > working. > > Cheers, > Anjana. > > On Tue, Mar 10, 2015 at 10:09 AM, Niranda Perera <[email protected]> wrote: > >> @supun I suggest you use an external spark cluster for the moment. as of >> the current BAM, we still have not finalized external job submission >> strategy in a BAM cluster. I will discuss this with Anjana and get back to >> you! >> >> On Tue, Mar 10, 2015 at 10:02 AM, Supun Sethunga <[email protected]> wrote: >> >>> [looping Nirmal] >>> >>> On Tue, Mar 10, 2015 at 10:01 AM, Supun Sethunga <[email protected]> >>> wrote: >>> >>>> [looping Nirmal] >>>> >>>> Hi Niranda, >>>> >>>> Thanks for the clarification. >>>> >>>> @Nirmal : Seems we have to wait till the BAM M2 release for ML-BAM >>>> Integration :) . But we still can enable to use an external spark cluster >>>> for ML. >>>> >>>> Regards, >>>> Supun >>>> >>>> On Tue, Mar 10, 2015 at 9:49 AM, Niranda Perera <[email protected]> >>>> wrote: >>>> >>>>> cc+ Anjana >>>>> >>>>> Hi Supun, >>>>> >>>>> As per the M1 release, we have only the standalone 'local' mode spark >>>>> instance. as you correctly said, we have set the master to 'local' in the >>>>> BAM . please refer init() method here [1]. In the init() method, we are >>>>> starting a spark context. >>>>> >>>>> we are still working on spark clustering. the plan is to release it in >>>>> the M2. >>>>> >>>>> [1] >>>>> https://github.com/wso2/carbon-analytics/blob/master/components/analytics/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/AnalyticsExecutionService.java >>>>> >>>>> On Mon, Mar 9, 2015 at 5:48 PM, Supun Sethunga <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I tried starting up a BAM server, and submitting a spark-job to the >>>>>> Spark which is built in BAM, from an external java client. In doing >>>>>> this, I >>>>>> was not able to create a Spark Context using BAM's spark instance as >>>>>> master. (AFAIK, the way to do this is to create a Spark context inside >>>>>> the >>>>>> client, and point BAM's Spark instance as the master, and then submit any >>>>>> jobs to the created context.) >>>>>> >>>>>> Possible reason might be, in BAM, Spark instance (i.e. SparkContext) >>>>>> is started locally with one worker thread.(setting master as 'local'). >>>>>> Hence the spark instance cannot be accessed by any external client. >>>>>> (Refer >>>>>> attached image "BAM-spark.png") >>>>>> AFAIK, to be accessible, BAM's Spark instance needs be started in the >>>>>> clustering mode. Then any external client also can use the BAM's spark, >>>>>> through the "spark.master" URL, which is "spark://localhost:7077" by >>>>>> default. (Refer attached image " >>>>>> external-spark-cluster.png"). >>>>>> >>>>>> I'm not sure whether this feature (clustering Spark) is currently >>>>>> supported by BAM (in M1 release)? >>>>>> >>>>>> If so, whats the way forward? >>>>>> >>>>>> Thanks, >>>>>> Supun >>>>>> >>>>>> -- >>>>>> *Supun Sethunga* >>>>>> Software Engineer >>>>>> WSO2, Inc. >>>>>> lean | enterprise | middleware >>>>>> Mobile : +94 716546324 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Niranda Perera* >>>>> Software Engineer, WSO2 Inc. >>>>> Mobile: +94-71-554-8430 >>>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>>> >>>> >>>> >>>> >>>> -- >>>> *Supun Sethunga* >>>> Software Engineer >>>> WSO2, Inc. >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> >> >> -- >> *Niranda Perera* >> Software Engineer, WSO2 Inc. >> Mobile: +94-71-554-8430 >> Twitter: @n1r44 <https://twitter.com/N1R44> >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- *Niranda Perera* Software Engineer, WSO2 Inc. Mobile: +94-71-554-8430 Twitter: @n1r44 <https://twitter.com/N1R44>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
