@supun I suggest you use an external spark cluster for the moment. as of the current BAM, we still have not finalized external job submission strategy in a BAM cluster. I will discuss this with Anjana and get back to you!
On Tue, Mar 10, 2015 at 10:02 AM, Supun Sethunga <[email protected]> wrote: > [looping Nirmal] > > On Tue, Mar 10, 2015 at 10:01 AM, Supun Sethunga <[email protected]> wrote: > >> [looping Nirmal] >> >> Hi Niranda, >> >> Thanks for the clarification. >> >> @Nirmal : Seems we have to wait till the BAM M2 release for ML-BAM >> Integration :) . But we still can enable to use an external spark cluster >> for ML. >> >> Regards, >> Supun >> >> On Tue, Mar 10, 2015 at 9:49 AM, Niranda Perera <[email protected]> wrote: >> >>> cc+ Anjana >>> >>> Hi Supun, >>> >>> As per the M1 release, we have only the standalone 'local' mode spark >>> instance. as you correctly said, we have set the master to 'local' in the >>> BAM . please refer init() method here [1]. In the init() method, we are >>> starting a spark context. >>> >>> we are still working on spark clustering. the plan is to release it in >>> the M2. >>> >>> [1] >>> https://github.com/wso2/carbon-analytics/blob/master/components/analytics/analytics-processors/org.wso2.carbon.analytics.spark.core/src/main/java/org/wso2/carbon/analytics/spark/core/AnalyticsExecutionService.java >>> >>> On Mon, Mar 9, 2015 at 5:48 PM, Supun Sethunga <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> I tried starting up a BAM server, and submitting a spark-job to the >>>> Spark which is built in BAM, from an external java client. In doing this, I >>>> was not able to create a Spark Context using BAM's spark instance as >>>> master. (AFAIK, the way to do this is to create a Spark context inside the >>>> client, and point BAM's Spark instance as the master, and then submit any >>>> jobs to the created context.) >>>> >>>> Possible reason might be, in BAM, Spark instance (i.e. SparkContext) is >>>> started locally with one worker thread.(setting master as 'local'). Hence >>>> the spark instance cannot be accessed by any external client. (Refer >>>> attached image "BAM-spark.png") >>>> AFAIK, to be accessible, BAM's Spark instance needs be started in the >>>> clustering mode. Then any external client also can use the BAM's spark, >>>> through the "spark.master" URL, which is "spark://localhost:7077" by >>>> default. (Refer attached image " >>>> external-spark-cluster.png"). >>>> >>>> I'm not sure whether this feature (clustering Spark) is currently >>>> supported by BAM (in M1 release)? >>>> >>>> If so, whats the way forward? >>>> >>>> Thanks, >>>> Supun >>>> >>>> -- >>>> *Supun Sethunga* >>>> Software Engineer >>>> WSO2, Inc. >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> >>> >>> >>> >>> -- >>> *Niranda Perera* >>> Software Engineer, WSO2 Inc. >>> Mobile: +94-71-554-8430 >>> Twitter: @n1r44 <https://twitter.com/N1R44> >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > lean | enterprise | middleware > Mobile : +94 716546324 > -- *Niranda Perera* Software Engineer, WSO2 Inc. Mobile: +94-71-554-8430 Twitter: @n1r44 <https://twitter.com/N1R44>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
