Hi Niranda, When we get the clustering to properly to work, which would be before Friday's release. It should be straightforward for ML to point to our master server and submit jobs. So lets wait till we finalize our implementation, and then ML team can work with us to get their use case working.
Cheers, Anjana. On Tue, Mar 10, 2015 at 10:09 AM, Niranda Perera <[email protected]> wrote: > @supun I suggest you use an external spark cluster for the moment. as of > the current BAM, we still have not finalized external job submission > strategy in a BAM cluster. I will discuss this with Anjana and get back to > you! > > On Tue, Mar 10, 2015 at 10:02 AM, Supun Sethunga <[email protected]> wrote: > >> [looping Nirmal] >> >> On Tue, Mar 10, 2015 at 10:01 AM, Supun Sethunga <[email protected]> wrote: >> >>> [looping Nirmal] >>> >>> Hi Niranda, >>> >>> Thanks for the clarification. >>> >>> @Nirmal : Seems we have to wait till the BAM M2 release for ML-BAM >>> Integration :) . But we still can enable to use an external spark cluster >>> for ML. >>> >>> Regards, >>> Supun >>> >>> On Tue, Mar 10, 2015 at 9:49 AM, Niranda Perera <[email protected]> >>> wrote: >>> >>>> cc+ Anjana >>>> >>>> Hi Supun, >>>> >>>> As per the M1 release, we have only the standalone 'local' mode spark >>>> instance. as you correctly said, we have set the master to 'local' in the >>>> BAM . please refer init() method here [1]. In the init() method, we are >>>> starting a spark context. >>>> >>>> we are still working on spark clustering. the plan is to release it in >>>> the M2. >>>> >>>> [1] >>>> https://github.com/wso2/carbon-analytics/blob/master/components/analytics/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/AnalyticsExecutionService.java >>>> >>>> On Mon, Mar 9, 2015 at 5:48 PM, Supun Sethunga <[email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I tried starting up a BAM server, and submitting a spark-job to the >>>>> Spark which is built in BAM, from an external java client. In doing this, >>>>> I >>>>> was not able to create a Spark Context using BAM's spark instance as >>>>> master. (AFAIK, the way to do this is to create a Spark context inside the >>>>> client, and point BAM's Spark instance as the master, and then submit any >>>>> jobs to the created context.) >>>>> >>>>> Possible reason might be, in BAM, Spark instance (i.e. SparkContext) >>>>> is started locally with one worker thread.(setting master as 'local'). >>>>> Hence the spark instance cannot be accessed by any external client. (Refer >>>>> attached image "BAM-spark.png") >>>>> AFAIK, to be accessible, BAM's Spark instance needs be started in the >>>>> clustering mode. Then any external client also can use the BAM's spark, >>>>> through the "spark.master" URL, which is "spark://localhost:7077" by >>>>> default. (Refer attached image " >>>>> external-spark-cluster.png"). >>>>> >>>>> I'm not sure whether this feature (clustering Spark) is currently >>>>> supported by BAM (in M1 release)? >>>>> >>>>> If so, whats the way forward? >>>>> >>>>> Thanks, >>>>> Supun >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Niranda Perera* >>>> Software Engineer, WSO2 Inc. >>>> Mobile: +94-71-554-8430 >>>> Twitter: @n1r44 <https://twitter.com/N1R44> >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > *Niranda Perera* > Software Engineer, WSO2 Inc. > Mobile: +94-71-554-8430 > Twitter: @n1r44 <https://twitter.com/N1R44> > -- *Anjana Fernando* Senior Technical Lead WSO2 Inc. | http://wso2.com lean . enterprise . middleware
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
