[looping Nirmal] On Tue, Mar 10, 2015 at 10:01 AM, Supun Sethunga <[email protected]> wrote:
> [looping Nirmal] > > Hi Niranda, > > Thanks for the clarification. > > @Nirmal : Seems we have to wait till the BAM M2 release for ML-BAM > Integration :) . But we still can enable to use an external spark cluster > for ML. > > Regards, > Supun > > On Tue, Mar 10, 2015 at 9:49 AM, Niranda Perera <[email protected]> wrote: > >> cc+ Anjana >> >> Hi Supun, >> >> As per the M1 release, we have only the standalone 'local' mode spark >> instance. as you correctly said, we have set the master to 'local' in the >> BAM . please refer init() method here [1]. In the init() method, we are >> starting a spark context. >> >> we are still working on spark clustering. the plan is to release it in >> the M2. >> >> [1] >> https://github.com/wso2/carbon-analytics/blob/master/components/analytics/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/AnalyticsExecutionService.java >> >> On Mon, Mar 9, 2015 at 5:48 PM, Supun Sethunga <[email protected]> wrote: >> >>> Hi, >>> >>> I tried starting up a BAM server, and submitting a spark-job to the >>> Spark which is built in BAM, from an external java client. In doing this, I >>> was not able to create a Spark Context using BAM's spark instance as >>> master. (AFAIK, the way to do this is to create a Spark context inside the >>> client, and point BAM's Spark instance as the master, and then submit any >>> jobs to the created context.) >>> >>> Possible reason might be, in BAM, Spark instance (i.e. SparkContext) is >>> started locally with one worker thread.(setting master as 'local'). Hence >>> the spark instance cannot be accessed by any external client. (Refer >>> attached image "BAM-spark.png") >>> AFAIK, to be accessible, BAM's Spark instance needs be started in the >>> clustering mode. Then any external client also can use the BAM's spark, >>> through the "spark.master" URL, which is "spark://localhost:7077" by >>> default. (Refer attached image " >>> external-spark-cluster.png"). >>> >>> I'm not sure whether this feature (clustering Spark) is currently >>> supported by BAM (in M1 release)? >>> >>> If so, whats the way forward? >>> >>> Thanks, >>> Supun >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> >> >> -- >> *Niranda Perera* >> Software Engineer, WSO2 Inc. >> Mobile: +94-71-554-8430 >> Twitter: @n1r44 <https://twitter.com/N1R44> >> > > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > lean | enterprise | middleware > Mobile : +94 716546324 > -- *Supun Sethunga* Software Engineer WSO2, Inc. lean | enterprise | middleware Mobile : +94 716546324
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
