Re: [Dev] Regarding gmail connector
Which Gmail connector are you referring to? ESB[1] or Ballerina[2] ? [1] https://docs.wso2.com/display/ESBCONNECTORS/Gmail+Connector [2] http://ballerinalang.org/docs/api/0.8/org.wso2.ballerina.connectors.gmail.html On Wed, Apr 26, 2017 at 5:29 PM, Varuni Alwis <var...@wso2.com> wrote: > Hi all, > > I want to send a mail in the html format (not as a text) in the mail body > using the gmail connector? Is that possible with the connector? > > Thanks. > > > *Varuni Alwis* > *Software Engineer - Intern* > *WSO2* > > Email : var...@wso2.com > Mobile : +94 719865395 <+94%2071%20986%205395> > > > > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Please merge the PR
Thanks Thusitha and Maheshika! On Wed, Dec 14, 2016 at 1:15 PM, Maheshika Goonetilleke <mahesh...@wso2.com> wrote: > Hi Thusitha > > Deployed successfully. > > > On Wed, Dec 14, 2016 at 12:21 PM, Thusitha Thilina Dayaratne < > thusit...@wso2.com> wrote: > >> PR Merged. >> >> @Maheshika, could you please release the bundle? >> >> Thanks >> Thusitha >> >> On Sat, Dec 10, 2016 at 7:56 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi Thusitha >>> >>> Made the changes. Can you please review and merge? >>> >>> Thanks, >>> SupunS >>> >>> On Sat, Dec 10, 2016 at 7:46 PM, Thusitha Thilina Dayaratne < >>> thusit...@wso2.com> wrote: >>> >>>> Hi Supun, >>>> >>>> I've put some comments on the PR. Could you please check on them? >>>> >>>> Thanks >>>> Thusitha >>>> >>>> On Sat, Dec 10, 2016 at 7:28 PM, Supun Sethunga <sup...@wso2.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Can you please review and merge the PR [1]. >>>>> >>>>> [1] https://github.com/wso2/orbit/pull/253 >>>>> >>>>> Regards, >>>>> SupunS >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Senior Software Engineer >>>>> WSO2, Inc. >>>>> http://wso2.com/ >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 <+94%2071%20654%206324> >>>>> Blog: http://supunsetunga.blogspot.com >>>>> >>>>> ___ >>>>> Dev mailing list >>>>> Dev@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thusitha Dayaratne >>>> Software Engineer >>>> WSO2 Inc. - lean . enterprise . middleware | wso2.com >>>> >>>> Mobile +94712756809 <+94%2071%20275%206809> >>>> Blog alokayasoya.blogspot.com >>>> Abouthttp://about.me/thusithathilina >>>> <http://wso2.com/signature> >>>> >>>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Senior Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 <+94%2071%20654%206324> >>> Blog: http://supunsetunga.blogspot.com >>> >> >> >> >> -- >> Thusitha Dayaratne >> Software Engineer >> WSO2 Inc. - lean . enterprise . middleware | wso2.com >> >> Mobile +94712756809 <+94%2071%20275%206809> >> Blog alokayasoya.blogspot.com >> Abouthttp://about.me/thusithathilina >> <http://wso2.com/signature> >> >> > > > -- > > Thanks & Best Regards, > > Maheshika Goonetilleke > Senior Engineering Process Coordinator > > *WSO2 Inc* > *email : mahesh...@wso2.com <mahesh...@wso2.com>* > *mobile : +94 773 596707 <+94%2077%20359%206707>* > *www: :http://wso2.com <http://wso2.com/>*lean . enterprise . middleware > > > > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Please merge the PR
Hi Thusitha Made the changes. Can you please review and merge? Thanks, SupunS On Sat, Dec 10, 2016 at 7:46 PM, Thusitha Thilina Dayaratne < thusit...@wso2.com> wrote: > Hi Supun, > > I've put some comments on the PR. Could you please check on them? > > Thanks > Thusitha > > On Sat, Dec 10, 2016 at 7:28 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi, >> >> Can you please review and merge the PR [1]. >> >> [1] https://github.com/wso2/orbit/pull/253 >> >> Regards, >> SupunS >> >> -- >> *Supun Sethunga* >> Senior Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 <+94%2071%20654%206324> >> Blog: http://supunsetunga.blogspot.com >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Thusitha Dayaratne > Software Engineer > WSO2 Inc. - lean . enterprise . middleware | wso2.com > > Mobile +94712756809 <+94%2071%20275%206809> > Blog alokayasoya.blogspot.com > Abouthttp://about.me/thusithathilina > <http://wso2.com/signature> > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Please merge the PR
Hi, Can you please review and merge the PR [1]. [1] https://github.com/wso2/orbit/pull/253 Regards, SupunS -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] How to get a connection from the carbon datasources
Similar approach is used in ml [1] too.
[1]
https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.database/src/main/java/org/wso2/carbon/ml/database/internal/MLDataSource.java#L30
Regards,
On Tue, Sep 27, 2016 at 8:03 AM, Niranda Perera <nira...@wso2.com> wrote:
> Hi all,
>
> I want to use a jdbc connection provided by a carbon-datasource.
>
> I found the following blog from Kishanthan [1], which was done in 2013.
>
> it uses the org.wso2.carbon.tomcat.jndi.CarbonJavaURLContextFactory as
> follows
>
> Hashtable environment = new Hashtable();
> environment.put("java.naming.factory.initialoorg.wso2.carbon.tomcat.jndi.
> CarbonJavaURLContextFactory");
> Context initContext = new InitialContext(environment);
> Object result = initContext.lookup("jdbc/MyCarbonDataSource");
> if (result != null) {
> // Do your work here
> } else {
> System.out.println(“Cannot find MyCarbonDataSource”);
> }
>
> My question is, is there a better way of doing this now (a util method may
> be?) or is this method still applicable?
>
> Best
>
> [1] https://kishanthan.wordpress.com/2013/02/11/access-cabon-data-sources-
> within-webapps-in-wso2-application-server/
>
> --
> *Niranda Perera*
> Software Engineer, WSO2 Inc.
> Mobile: +94-71-554-8430
> Twitter: @n1r44 <https://twitter.com/N1R44>
> https://pythagoreanscript.wordpress.com/
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
--
*Supun Sethunga*
Senior Software Engineer
WSO2, Inc.
http://wso2.com/
lean | enterprise | middleware
Mobile : +94 716546324
Blog: http://supunsetunga.blogspot.com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [DAS][310] Errors while running APIM_INCREMENTAL_PROCESSING_SCRIPT
est_count LONG, resourceTemplate STRING, method STRING, >>>> applicationName STRING, tenantDomain STRING,userAgent STRING, >>>> resourcePath STRING, request INT, applicationId STRING, tier STRING, >>>> throttledOut BOOLEAN, clientIp STRING,applicationOwner STRING, >>>> _timestamp LONG -i",primaryKeys "year, month, day, hour, minute, >>>> consumerKey, context, api_version, userId, hostName, apiPublisher, >>>> resourceTemplate, method, userAgent, clientIp",incrementalProcessing >>>> "APIMGT_PERMINUTE_REQUEST_DATA, HOUR",mergeSchema "false") >>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:764) >>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>> Executor.executeQuery(SparkAnalyticsExecutor.java:721) >>>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso >>>> rService.executeQuery(CarbonAnalyticsProcessorService.java:201) >>>> at org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcesso >>>> rService.executeScript(CarbonAnalyticsProcessorService.java:151) >>>> at org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(A >>>> nalyticsTask.java:60) >>>> at org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute >>>> (TaskQuartzJobAdapter.java:67) >>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executor >>>> s.java:471) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>>> Executor.java:1145) >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>>> lExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> Caused by: java.lang.RuntimeException: Unknown options : >>>> incrementalprocessing >>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>> onProvider.checkParameters(AnalyticsRelationProvider.java:123) >>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>> onProvider.setParameters(AnalyticsRelationProvider.java:113) >>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>> onProvider.createRelation(AnalyticsRelationProvider.java:75) >>>> at org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelati >>>> onProvider.createRelation(AnalyticsRelationProvider.java:45) >>>> at org.apache.spark.sql.execution.datasources.ResolvedDataSourc >>>> e$.apply(ResolvedDataSource.scala:158) >>>> at org.apache.spark.sql.execution.datasources.CreateTempTableUs >>>> ing.run(ddl.scala:92) >>>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes >>>> ult$lzycompute(commands.scala:58) >>>> at org.apache.spark.sql.execution.ExecutedCommand.sideEffectRes >>>> ult(commands.scala:56) >>>> at org.apache.spark.sql.execution.ExecutedCommand.doExecute(com >>>> mands.scala:70) >>>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5. >>>> apply(SparkPlan.scala:132) >>>> at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$5. >>>> apply(SparkPlan.scala:130) >>>> at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperati >>>> onScope.scala:150) >>>> at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.s >>>> cala:130) >>>> at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompu >>>> te(QueryExecution.scala:55) >>>> at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExe >>>> cution.scala:55) >>>> at org.apache.spark.sql.DataFrame.(DataFrame.scala:145) >>>> at org.apache.spark.sql.DataFrame.(DataFrame.scala:130) >>>> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:52) >>>> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:817) >>>> at org.wso2.carbon.analytics.spark.core.internal.SparkAnalytics >>>> Executor.executeQueryLocal(SparkAnalyticsExecutor.java:760) >>>> ... 11 more >>>> >>>> >>>> [1] https://docs.wso2.com/display/AM200/Installing+WSO2+APIM >>>> +Analytics+Features >>>> [2] https://docs.wso2.com/display/DAS310/Incremental+Processing >>>> >>>> Regards, >>>> Malith >>>> -- >>>> Malith Munasinghe | Software Engineer >>>> M: +94 (71) 9401122 >>>> E: mali...@wso2.com >>>> W: http://wso2.com >>>> <http://wso2.com/signature> >>>> >>> >>> >>> >>> -- >>> *Niranda Perera* >>> Software Engineer, WSO2 Inc. >>> Mobile: +94-71-554-8430 >>> Twitter: @n1r44 <https://twitter.com/N1R44> >>> https://pythagoreanscript.wordpress.com/ >>> >> >> >> >> -- >> Rukshan Chathuranga. >> Software Engineer. >> WSO2, Inc. >> > > > > -- > Malith Munasinghe | Software Engineer > M: +94 (71) 9401122 > E: mali...@wso2.com > W: http://wso2.com > <http://wso2.com/signature> > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] WSO2 Machine Learner (ML) 1.2.0 Released!
*WSO2 Machine Learner (ML) 1.2.0 Released!* Machine learning has emerged as a key component of big data analytics space. The goal of WSO2 Machine Learner is to make machine learning accessible to WSO2 big data platform. WSO2 Machine Learner ( http://wso2.com/products/machine-learner/) provides a user-friendly wizard-like interface, which guides users through a set of steps to find and configure machine learning algorithms. The outcome of this process is a model that can be deployed in multiple WSO2 products, such as , WSO2 Data Analytics Server (DAS), WSO2 Enterprise Service Bus (ESB), WSO2 Complex Event Processor (CEP) etc. The novice-friendly machine learning analysis allows developers, data-scientists and database administrators to quickly implement machine learning methods. If you are familiar with WSO2 products, you can utilize WSO2 Machine Learner to build machine learning models for various tasks, such as fraud detection, anomaly detection, classification etc. WSO2 Machine Learner is built upon the award-winning, WSO2 Carbon platform, which is based on the OSGi framework enabling better modularity for your service-oriented architecture (SOA). WSO2 Machine Learner product is powered by WSO2 Carbon <http://wso2.com/products/carbon/>, the SOA middleware component platform and is released under Apache Software License (v2.0) <http://www.apache.org/licenses/LICENSE-2.0.html>, one of the most business-friendly licenses available today. What's New In This Release - Upgrading Spark to 1.6.2 - Integrating with WSO2 Data Analytics Server. - Security Fixes associated with the latest Carbon kernel. *Distribution:* - http://wso2.com/products/machine-learner *Official Documentation:* - https://docs.wso2.com/display/ML120/WSO2+Machine+Learner+Documentation Issues Fixed in This Release - WSO2 ML 1.2.0 Fixed Issues <https://wso2.org/jira/issues/?filter=13325> Known Issues - WSO2 ML 1.2.0 Known Issues <https://wso2.org/jira/issues/?filter=13336> Mailing Lists Join our mailing list and correspond with the developers directly. - Developer List : dev@wso2.org | Subscribe | Mail Archive <http://mail.wso2.org/mailarchive/dev/> Reporting Issues We encourage you to report issues, documentation faults and feature requests regarding WSO2 DAS through the public ML JIRA <https://wso2.org/jira/browse/ML>. You can use the Carbon JIRA <http://www.wso2.org/jira/browse/CARBON> to report any issues related to the Carbon base framework or associated Carbon components. Discussion Forums Alternatively, questions could be raised on http://stackoverflow.com <http://stackoverflow.com/questions/tagged/wso2>. Support We are committed to ensuring that your enterprise middleware deployment is completely supported from evaluation to production. Our unique approach ensures that all support leverages our open development methodology and is provided by the very same engineers who build the technology. For more details and to take advantage of this unique opportunity please visit http://wso2.com/support. Regards, ~ WSO2 ML Team ~ -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [VOTE] Release WSO2 Machine Learner 1.2.0 RC2
Hi all, Thanks for testing and verifying the WSO2 Machine Learner 1.2.0-RC2. We are closing this vote now, as this vote has passed with five +1's and zero -1's. Therefore we are proceeding with the WSO2 Machine Learner 1.2.0 release. Thanks & Regards, WSO2 ML Team On Thu, Sep 15, 2016 at 1:09 PM, Supun Sethunga <sup...@wso2.com> wrote: > [adding dev@] > > > -- Forwarded message -- > From: Sameera Ramasinghe <samee...@wso2.com> > Date: Wed, Sep 14, 2016 at 6:55 PM > Subject: Re: [VOTE] Release WSO2 Machine Learner 1.2.0 RC2 > To: Nirmal Fernando <nir...@wso2.com> > Cc: Supun Sethunga <sup...@wso2.com>, Fazlan Nazeem <fazl...@wso2.com>, > Manorama Perera <manor...@wso2.com>, Upul Bandara <u...@wso2.com> > > > Hi, > > I tested the following > > -ML model generation > -ML prediction through the model > -Viewing the results > > vote- > [+] Stable - go ahead and release > > On Tue, Sep 13, 2016 at 2:19 PM, Nirmal Fernando <nir...@wso2.com> wrote: > >> All, please vote :) >> >> On Fri, Sep 9, 2016 at 5:43 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Please test and vote ;) >>> >>> On Fri, Sep 9, 2016 at 5:41 PM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi Devs, >>>> >>>> This is the 2nd Release Candidate of the WSO2 Machine Learner 1.2.0. This >>>> release includes the following improvements and fixes: >>>> >>>>- ML-DAS integration >>>>- Spark upgrade to v1.6.2 >>>>- Security fixes associated with the WSO2 Carbon 4.4.9 platform >>>>release >>>> >>>> >>>> Please download, test the product and vote. Vote will be open for 72 >>>> hours or as needed. >>>> Refer to github readme for guides. >>>> >>>> The distribution is available to download at [1]. >>>> >>>> Fixed Issue in [2]. Tag to be vote upon [3]. >>>> >>>> [1] https://github.com/wso2/product-ml/releases/download/v1. >>>> 2.0-rc2/wso2ml-1.2.0-rc2.zip >>>> [2] https://wso2.org/jira/issues/?filter=13325 >>>> [3] https://github.com/wso2/product-ml/releases/tag/v1.2.0-rc2 >>>> >>>> Please vote as follows. >>>> [+] Stable - go ahead and release >>>> [-] Broken - do not release (please explain why) >>>> >>>> Thanks, >>>> WSO2 ML Team >>>> >>>> -- >>>> *Supun Sethunga* >>>> Senior Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> Blog: http://supunsetunga.blogspot.com >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Senior Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> Blog: http://supunsetunga.blogspot.com >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > *Sameera Ramasinghe* > Software Engineer, WSO2 Inc.; http://wso2.com > mobile: *+94 714489682* > > > > -- > *Supun Sethunga* > Senior Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > Blog: http://supunsetunga.blogspot.com > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Fwd: [VOTE] Release WSO2 Machine Learner 1.2.0 RC2
[adding dev@] -- Forwarded message -- From: Sameera Ramasinghe <samee...@wso2.com> Date: Wed, Sep 14, 2016 at 6:55 PM Subject: Re: [VOTE] Release WSO2 Machine Learner 1.2.0 RC2 To: Nirmal Fernando <nir...@wso2.com> Cc: Supun Sethunga <sup...@wso2.com>, Fazlan Nazeem <fazl...@wso2.com>, Manorama Perera <manor...@wso2.com>, Upul Bandara <u...@wso2.com> Hi, I tested the following -ML model generation -ML prediction through the model -Viewing the results vote- [+] Stable - go ahead and release On Tue, Sep 13, 2016 at 2:19 PM, Nirmal Fernando <nir...@wso2.com> wrote: > All, please vote :) > > On Fri, Sep 9, 2016 at 5:43 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Please test and vote ;) >> >> On Fri, Sep 9, 2016 at 5:41 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi Devs, >>> >>> This is the 2nd Release Candidate of the WSO2 Machine Learner 1.2.0. This >>> release includes the following improvements and fixes: >>> >>>- ML-DAS integration >>>- Spark upgrade to v1.6.2 >>>- Security fixes associated with the WSO2 Carbon 4.4.9 platform >>>release >>> >>> >>> Please download, test the product and vote. Vote will be open for 72 >>> hours or as needed. >>> Refer to github readme for guides. >>> >>> The distribution is available to download at [1]. >>> >>> Fixed Issue in [2]. Tag to be vote upon [3]. >>> >>> [1] https://github.com/wso2/product-ml/releases/download/v1. >>> 2.0-rc2/wso2ml-1.2.0-rc2.zip >>> [2] https://wso2.org/jira/issues/?filter=13325 >>> [3] https://github.com/wso2/product-ml/releases/tag/v1.2.0-rc2 >>> >>> Please vote as follows. >>> [+] Stable - go ahead and release >>> [-] Broken - do not release (please explain why) >>> >>> Thanks, >>> WSO2 ML Team >>> >>> -- >>> *Supun Sethunga* >>> Senior Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> Blog: http://supunsetunga.blogspot.com >>> >> >> >> >> -- >> *Supun Sethunga* >> Senior Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> Blog: http://supunsetunga.blogspot.com >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *Sameera Ramasinghe* Software Engineer, WSO2 Inc.; http://wso2.com mobile: *+94 714489682* -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [Vote] Release WSO2 Data Analytics Server (DAS) 3.1.0-RC3
Tested following ML scenarios: - Creating/Deleting Datasets from CSV files / DAS Tables - Creating/Deleting Projects - Creating/Deleting Analyzes - Training/Deleting/Publishing/Downloading Models. - Predicting using Models [+] Stable - Go ahead and release Regards Supun On Wed, Sep 14, 2016 at 2:52 AM, Anjana Fernando <anj...@wso2.com> wrote: > Hi, > > I've tested the following:- > > * Basic Indexing/Searching Operations > * Spark SQL scheduled query execution in background/foreground > * Activity Monitoring with multiple streams > * Samples > - Wikipedia > - SmartHome > > [+] Stable - Go ahead and release > > Cheers, > Anjana. > > On Fri, Sep 9, 2016 at 4:54 PM, Niranda Perera <nira...@wso2.com> wrote: > >> Hi Devs, >> >> This is the third release candidate (RC) of WSO2 Data Analytics Server >> 3.1.0 release. >> >> New / Improvements In This Release after RC2 >> >>- Bug fixes in ML integration >>- Spark configuration parameters for long running jobs >> >> Issues Fixed in This Release >> >>- WSO2 DAS 3.1.0 Fixed Issues >><https://wso2.org/jira/issues/?filter=13152> >> >> Known Issues >> >>- WSO2 DAS 3.1.0 Known Issues >><https://wso2.org/jira/issues/?filter=13154> >> >> Source and distribution packages: >> >>- https://github.com/wso2/product-das/releases/tag/v3.1.0-RC3 >> >> Please download, test, and vote. The README file under the distribution >> contains guide and instructions on how to try it out locally. >> >> [+] Stable - Go ahead and release >> [-] Broken - Do not release (explain why) >> >> This vote will be open for 72 hours or as needed. >> >> Regards, >> WSO2 DAS Team >> >> -- >> *Niranda Perera* >> Software Engineer, WSO2 Inc. >> Mobile: +94-71-554-8430 >> Twitter: @n1r44 <https://twitter.com/N1R44> >> https://pythagoreanscript.wordpress.com/ >> > > > > -- > *Anjana Fernando* > Associate Director / Architect > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [VOTE] Release WSO2 Machine Learner 1.2.0 RC2
Tested the Samples. [+] Stable - go ahead and release Regards, Supun On Fri, Sep 9, 2016 at 5:41 PM, Supun Sethunga <sup...@wso2.com> wrote: > Hi Devs, > > This is the 2nd Release Candidate of the WSO2 Machine Learner 1.2.0. This > release includes the following improvements and fixes: > >- ML-DAS integration >- Spark upgrade to v1.6.2 >- Security fixes associated with the WSO2 Carbon 4.4.9 platform release > > > Please download, test the product and vote. Vote will be open for 72 hours > or as needed. > Refer to github readme for guides. > > The distribution is available to download at [1]. > > Fixed Issue in [2]. Tag to be vote upon [3]. > > [1] https://github.com/wso2/product-ml/releases/download/ > v1.2.0-rc2/wso2ml-1.2.0-rc2.zip > [2] https://wso2.org/jira/issues/?filter=13325 > [3] https://github.com/wso2/product-ml/releases/tag/v1.2.0-rc2 > > Please vote as follows. > [+] Stable - go ahead and release > [-] Broken - do not release (please explain why) > > Thanks, > WSO2 ML Team > > -- > *Supun Sethunga* > Senior Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > Blog: http://supunsetunga.blogspot.com > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [VOTE] Release WSO2 Machine Learner 1.2.0 RC2
Hi Devs, This is the 2nd Release Candidate of the WSO2 Machine Learner 1.2.0. This release includes the following improvements and fixes: - ML-DAS integration - Spark upgrade to v1.6.2 - Security fixes associated with the WSO2 Carbon 4.4.9 platform release Please download, test the product and vote. Vote will be open for 72 hours or as needed. Refer to github readme for guides. The distribution is available to download at [1]. Fixed Issue in [2]. Tag to be vote upon [3]. [1] https://github.com/wso2/product-ml/releases/download/v1.2.0-rc2/wso2ml-1.2.0-rc2.zip [2] https://wso2.org/jira/issues/?filter=13325 [3] https://github.com/wso2/product-ml/releases/tag/v1.2.0-rc2 Please vote as follows. [+] Stable - go ahead and release [-] Broken - do not release (please explain why) Thanks, WSO2 ML Team -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [Architecture] [VOTE] Release WSO2 Machine Learner 1.2.0 RC1
We have encountered an issue in ML-DAS integration. Hence calling off this vote. We'll be calling a new vote soon. Regards. Supun On Wed, Sep 7, 2016 at 3:40 PM, Upul Bandara <u...@wso2.com> wrote: > +1 for CEP Extension > > On Wed, Sep 7, 2016 at 2:59 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Tested all the samples. >> >> [+] Stable - go ahead and release >> >> >> >> On Wed, Sep 7, 2016 at 1:55 PM, Nirmal Fernando <nir...@wso2.com> wrote: >> >>> [+] Stable - go ahead and release >>> >>> On Tue, Sep 6, 2016 at 6:09 PM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi Devs, >>>> >>>> This is the 1st Release Candidate of the WSO2 Machine Learner 1.2.0. This >>>> release includes the following improvements and fixes: >>>> >>>>- ML-DAS integration >>>>- Spark upgrade to v1.6.2 >>>>- Security fixes associated with the WSO2 Carbon 4.4.9 platform >>>>release >>>> >>>> >>>> Please download, test the product and vote. Vote will be open for 72 >>>> hours or as needed. >>>> Refer to github readme for guides. >>>> >>>> The distribution is available to download at [1]. >>>> >>>> Fixed Issue in [2]. Tag to be vote upon [3]. >>>> >>>> [1] https://github.com/wso2/product-ml/releases/download/v1. >>>> 2.0-rc1/wso2ml-1.2.0-rc1.zip >>>> [2] https://wso2.org/jira/issues/?filter=13325 >>>> [3] https://github.com/wso2/product-ml/releases/tag/v1.2.0-rc1 >>>> >>>> Please vote as follows. >>>> [+] Stable - go ahead and release >>>> [-] Broken - do not release (please explain why) >>>> >>>> Thanks, >>>> WSO2 ML Team >>>> >>>> -- >>>> *Supun Sethunga* >>>> Senior Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> Blog: http://supunsetunga.blogspot.com >>>> >>>> ___ >>>> Architecture mailing list >>>> architect...@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >>> ___ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Supun Sethunga* >> Senior Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> Blog: http://supunsetunga.blogspot.com >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > Upul Bandara, > Associate Technical Lead, WSO2, Inc., > Mob: +94 715 468 345. > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [Architecture] [VOTE] Release WSO2 Machine Learner 1.2.0 RC1
Tested all the samples. [+] Stable - go ahead and release On Wed, Sep 7, 2016 at 1:55 PM, Nirmal Fernando <nir...@wso2.com> wrote: > [+] Stable - go ahead and release > > On Tue, Sep 6, 2016 at 6:09 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Devs, >> >> This is the 1st Release Candidate of the WSO2 Machine Learner 1.2.0. This >> release includes the following improvements and fixes: >> >>- ML-DAS integration >>- Spark upgrade to v1.6.2 >>- Security fixes associated with the WSO2 Carbon 4.4.9 platform >>release >> >> >> Please download, test the product and vote. Vote will be open for 72 >> hours or as needed. >> Refer to github readme for guides. >> >> The distribution is available to download at [1]. >> >> Fixed Issue in [2]. Tag to be vote upon [3]. >> >> [1] https://github.com/wso2/product-ml/releases/download/v1. >> 2.0-rc1/wso2ml-1.2.0-rc1.zip >> [2] https://wso2.org/jira/issues/?filter=13325 >> [3] https://github.com/wso2/product-ml/releases/tag/v1.2.0-rc1 >> >> Please vote as follows. >> [+] Stable - go ahead and release >> [-] Broken - do not release (please explain why) >> >> Thanks, >> WSO2 ML Team >> >> -- >> *Supun Sethunga* >> Senior Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> Blog: http://supunsetunga.blogspot.com >> >> ___ >> Architecture mailing list >> architect...@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [VOTE] Release WSO2 Machine Learner 1.2.0 RC1
Hi Devs, This is the 1st Release Candidate of the WSO2 Machine Learner 1.2.0. This release includes the following improvements and fixes: - ML-DAS integration - Spark upgrade to v1.6.2 - Security fixes associated with the WSO2 Carbon 4.4.9 platform release Please download, test the product and vote. Vote will be open for 72 hours or as needed. Refer to github readme for guides. The distribution is available to download at [1]. Fixed Issue in [2]. Tag to be vote upon [3]. [1] https://github.com/wso2/product-ml/releases/download/v1.2.0-rc1/wso2ml-1.2.0-rc1.zip [2] https://wso2.org/jira/issues/?filter=13325 [3] https://github.com/wso2/product-ml/releases/tag/v1.2.0-rc1 Please vote as follows. [+] Stable - go ahead and release [-] Broken - do not release (please explain why) Thanks, WSO2 ML Team -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML][DAS] Database name not found
> > or the snapshot of both (1.1.1 and 5.1.0). Correction: "or the snapshot of both (1.1.1 and 3.1.0)" On Thu, Sep 1, 2016 at 10:07 AM, Supun Sethunga <sup...@wso2.com> wrote: > Hi Dimuth, > > I can't find datasource property named as "WSO2_ANALYTICS_FS_DB" in both >> analytics-datasources.xml files. > > Sharing WSO2_ANALYTICS_EVENT_STORE_DB and > WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB > would be enough. I think the Doc needs to be updated. > @Rukshani, please note. > > Once you point the above two Data sources to the same DB, you should be > able to see the DAS tables in ML. If not, can you please share your > datasource configuration in both servers? > > Anyway, ML 1.1.1-SNAPSHOT is using carbon-analytics v1.2.4 and DAS 3.0.1 > is using carbon-analytics v1.0.5. Hence you will get errors when the > dataset is created, saying the table is corrupt. So use either the latest > released versions of both ML and DAS (1.1.0 and 3.0.1), or the snapshot of > both (1.1.1 and 5.1.0). > > Regards, > Supun > > > > On Wed, Aug 31, 2016 at 4:29 PM, Dimuth Menikgama <dim...@wso2.com> wrote: > >> I am trying to follow this documentation on Machine learning titled as >> "Integration with WSO2 Data Analytics Server" . >> [1] https://docs.wso2.com/display/ML110/Integration+with+WSO2+Da >> ta+Analytics+Server >> <https://docs.wso2.com/display/ML110/Integration+with+WSO2+Data+Analytics+Server> >> >> I can't find datasource property named as "WSO2_ANALYTICS_FS_DB" in both >> analytics-datasources.xml files. >> >> When I continue the process to create a dataset ignoring above issue, I >> can't see any DAS tables in ML. >> >> I am using ML 1.1.1 and DAS 3.0.1 versions. >> >> Thanks & Regards, >> Dimuth. >> >> >> <https://docs.wso2.com/display/ML110/Integration+with+WSO2+Data+Analytics+Server> >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Supun Sethunga* > Senior Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > Blog: http://supunsetunga.blogspot.com > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML][DAS] Database name not found
Hi Dimuth, I can't find datasource property named as "WSO2_ANALYTICS_FS_DB" in both > analytics-datasources.xml files. Sharing WSO2_ANALYTICS_EVENT_STORE_DB and WSO2_ANALYTICS_PROCESSED_DATA_STORE_DB would be enough. I think the Doc needs to be updated. @Rukshani, please note. Once you point the above two Data sources to the same DB, you should be able to see the DAS tables in ML. If not, can you please share your datasource configuration in both servers? Anyway, ML 1.1.1-SNAPSHOT is using carbon-analytics v1.2.4 and DAS 3.0.1 is using carbon-analytics v1.0.5. Hence you will get errors when the dataset is created, saying the table is corrupt. So use either the latest released versions of both ML and DAS (1.1.0 and 3.0.1), or the snapshot of both (1.1.1 and 5.1.0). Regards, Supun On Wed, Aug 31, 2016 at 4:29 PM, Dimuth Menikgama <dim...@wso2.com> wrote: > I am trying to follow this documentation on Machine learning titled as > "Integration with WSO2 Data Analytics Server" . > [1] https://docs.wso2.com/display/ML110/Integration+with+WSO2+ > Data+Analytics+Server > <https://docs.wso2.com/display/ML110/Integration+with+WSO2+Data+Analytics+Server> > > I can't find datasource property named as "WSO2_ANALYTICS_FS_DB" in both > analytics-datasources.xml files. > > When I continue the process to create a dataset ignoring above issue, I > can't see any DAS tables in ML. > > I am using ML 1.1.1 and DAS 3.0.1 versions. > > Thanks & Regards, > Dimuth. > > > <https://docs.wso2.com/display/ML110/Integration+with+WSO2+Data+Analytics+Server> > > _______ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML/DAS] java.lang.StackOverflowError in Recommendation Algorithm in spark 1.6.2
Well, no, its a property set for the spark context. Basically what it does is, it stores a snapshot of RDD's state in the file system, very similar to the CEP state persistence. And when we set checkpointing in the spark context, it applies to all the RRDs created in that context AFAIK. On Thu, Sep 1, 2016 at 7:48 AM, Nirmal Fernando <nir...@wso2.com> wrote: > 'checkpointing' is an algorithm property right? We can add it as a > hyperparameter configuration ? What do we specifically need to do as ML > server? > > On Wed, Aug 31, 2016 at 10:47 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> We can reduce the default one, but usually a will user increase/change >> that when tuning hyper-parameters to increase the accuracy. So we need a >> solution that would work globally (for any value). A typical 'user'' >> cannot/shouldn't enable checkpointing, as IMO its a server configuration. >> >> anyway, the default one is 20, which is still in the lower side :) >> >> On Wed, Aug 31, 2016 at 7:15 PM, Nirmal Fernando <nir...@wso2.com> wrote: >> >>> Can't we reduce the default number of iterations? and document how to >>> enable 'check pointing'. >>> >>> On Wed, Aug 31, 2016 at 7:03 PM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi all, >>>> >>>> We are encountering $subject in ML, for the default hyper-parameter >>>> values. A similar issue has been reported in [1], but with a different >>>> algorithm. >>>> >>>> This occurs when the number of iterations for model training is large. >>>> The solution suggested at [1] (setting a checkpoint directory) works for >>>> our scenario, and is the only solid solution we have for the moment. But as >>>> mentioned in [2], checkpointing add some overhead for spark operations, and >>>> requires some tuning based on the use case. Therefore, I'm not sure is it a >>>> good idea to enable checkpointing in ML, as it would affect DAS's >>>> performance. (This checkpointing is done for the Spark Context, and it is >>>> shared by both ML and DAS) >>>> >>>> Other option would be to, set checkpointing at the start of the >>>> Recommendation algorithm, and once the model is trained, then unset >>>> checkpointing. Since we are encountering this issue only at this particular >>>> algorithm, it is not needed to be done for any other algorithm. >>>> >>>> Would like to know what would be the best approach? >>>> >>>> [1] https://issues.apache.org/jira/browse/SPARK-13546 >>>> [2] http://spark.apache.org/docs/1.6.2/streaming-programming >>>> -guide.html#checkpointing >>>> >>>> >>>> >>>> *Stack Trace:* >>>> >>>> Caused by: java.lang.StackOverflowError >>>> at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Obje >>>> ctInputStream.java:2606) >>>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1506) >>>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>>> am.java:1774) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>>> m.java:2000) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.j >>>> ava:1924) >>>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>>> am.java:1801) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>>> m.java:2000) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.j >>>> ava:1924) >>>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>>> am.java:1801) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>>> m.java:2000) >>>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.j >>>> ava:1924) >>>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>>> am.java:1801) >>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>>> at sun.reflec
Re: [Dev] [ML/DAS] java.lang.StackOverflowError in Recommendation Algorithm in spark 1.6.2
We can reduce the default one, but usually a will user increase/change that when tuning hyper-parameters to increase the accuracy. So we need a solution that would work globally (for any value). A typical 'user'' cannot/shouldn't enable checkpointing, as IMO its a server configuration. anyway, the default one is 20, which is still in the lower side :) On Wed, Aug 31, 2016 at 7:15 PM, Nirmal Fernando <nir...@wso2.com> wrote: > Can't we reduce the default number of iterations? and document how to > enable 'check pointing'. > > On Wed, Aug 31, 2016 at 7:03 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi all, >> >> We are encountering $subject in ML, for the default hyper-parameter >> values. A similar issue has been reported in [1], but with a different >> algorithm. >> >> This occurs when the number of iterations for model training is large. >> The solution suggested at [1] (setting a checkpoint directory) works for >> our scenario, and is the only solid solution we have for the moment. But as >> mentioned in [2], checkpointing add some overhead for spark operations, and >> requires some tuning based on the use case. Therefore, I'm not sure is it a >> good idea to enable checkpointing in ML, as it would affect DAS's >> performance. (This checkpointing is done for the Spark Context, and it is >> shared by both ML and DAS) >> >> Other option would be to, set checkpointing at the start of the >> Recommendation algorithm, and once the model is trained, then unset >> checkpointing. Since we are encountering this issue only at this particular >> algorithm, it is not needed to be done for any other algorithm. >> >> Would like to know what would be the best approach? >> >> [1] https://issues.apache.org/jira/browse/SPARK-13546 >> [2] http://spark.apache.org/docs/1.6.2/streaming-programming >> -guide.html#checkpointing >> >> >> >> *Stack Trace:* >> >> Caused by: java.lang.StackOverflowError >> at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Obje >> ctInputStream.java:2606) >> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1506) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1774) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >> .java:1058) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.
[Dev] [ML/DAS] java.lang.StackOverflowError in Recommendation Algorithm in spark 1.6.2
$colon.readObject(List.scala:362) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) at scala.collection.immutable.$colon$colon.readObject(List.scala:362) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) at scala.collection.immutable.$colon$colon.readObject(List.scala:362) at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) Regards, Supun -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSoC 2016]Proposal 03: [ML]Visual(Deep) Neural Network Builder -Project Progress
Hi Lakini, Thanks for the update. Can you please schedule a final demo with end to end implementation, early next week? Please note that this demo and the quality of deliverables (specially the code, documentations, etc) will have a big impact on the final evaluations. Regards, Supun On Fri, Aug 12, 2016 at 8:54 AM, Lakini Senanayaka < lakinisenanayak...@gmail.com> wrote: > Hi, > > I have taken only the Accuracy,Precision,F1 Score and Recall of neural > networks. > The values are listed below. > > *Accuracy: 0.9712* > *Precision: 0.9711317415957886* > *Recall: 0.9708490075847962* > *F1 Score: 0.9709903540085949* > > These are the hyper parameters user has set for the MNIST dataset for the > UI. > > *Neural Network Type = Feed Forward Network* > *Seed = 123* > *Learning Rate = 0.006* > *Batch Size = 128* > *nEpoches = 15* > *Iteration = 1* > *Optimization Algorithm = Storchastic Gradient Descent* > *Updater = Nesterovs* > *Momentum = 0.9* > *Pretrain = False* > *Back Propargation = True* > > *Input Layer:* > * Input Nodes = 784* > > *1st Hidden Layer:* > * Number of Hidden Nodes = 1000* > * WeightInit = Xavier* > * Activation = RELU* > > *Output Layer:* > * Output Nodes = 10* > * WeightInit = Xavier* > * Activation = Softmax* > * Loss Function = Negetive Log Likelihood* > > > For the training purposes and for the time being I have used the Dataset > of MNIST which is available in DL4J library. > > *DataSetIterator trainIter = new MnistDataSetIterator(bachSize, true, > (int)seed);* > *DataSetIterator testIter = new MnistDataSetIterator(bachSize, false, > (int) seed);* > > I will try to use the dataset from the WSO2 ML Server.I couldn't try it > yesterday. > > The training time for the above scenario is 08 min 45s . > I will share the code once I complete reading data set from the WSO2 ML > Sever. > > Thank you. > > On Thu, Aug 11, 2016 at 8:43 AM, Upul Bandara <u...@wso2.com> wrote: > >> Sounds good >> >> Can you please share us the performance numbers (and confusion matrix) >> you got with the MINIST? >> Also, how did you do: >> load MNIST data into WSO2 ML Server? >> split training/testing subsets? >> Tune hyper-parameters? >> >> What was the training time? >> >> Thanks, >> Upul >> >> >> On Thu, Aug 11, 2016 at 6:59 AM, Lakini Senanayaka < >> lakinisenanayak...@gmail.com> wrote: >> >>> Hi. >>> >>> Thank you very much for sharing the code. >>> I have solved many of the issues I had with the UI. >>> >>> I have tested the UI with MNIST dataset.It gave the correct results and >>> the UI is working well with large datasets. >>> >>> The UI gets unresponsive for sometimes as the AJAX call takes nearly 5 >>> minutes to give the response.Currently, I am handling that issue and I have >>> to call the dataset from the ML server. >>> >>> I will share my documentation as soon as possible. >>> >>> Thank you. >>> >>> On Tue, Aug 9, 2016 at 9:52 AM, Upul Bandara <u...@wso2.com> wrote: >>> >>>> Following jag file ( please look at function loadDatasets(), function >>>> updateDatasets(), function deleteDataset(datasetId) and etc) has everything >>>> you need to know to get dataset from ML server to your UI. Please kindly >>>> note that it is very difficult for us to give exact code snippet to meet >>>> your requirement. As a student, you should be able to go through available >>>> code which shows how to use APIs and uses those APIs in your applications. >>>> >>>> I think the API doc Supun shared with you and the following jag file >>>> will help you to understand how to extract dataset and versions from the ML >>>> Server and displaying those in your UI. >>>> >>>> BTW, did you manage to test your UI with large datasets such as MNIST. >>>> MNIST is an image dataset but you can convert it to standard CSV format and >>>> can be used to train NN using your UI. >>>> >>>> https://github.com/wso2/carbon-ml/blob/master/apps/ml/site/d >>>> ata/datasets.jag >>>> >>>> On Mon, Aug 8, 2016 at 6:31 AM, Lakini Senanayaka < >>>> lakinisenanayak...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Thank you very much. >>>>> >>>>> But Upul told me about a code snippet which I can use inside the API >>>>> implementation in the
Re: [Dev] Fwd: GSOC2016: [ML][CEP] [SAMOA]Predictive analytic with online data for WSO2 Machine Learner-Samoa Integration
a.version'
> at org.wso2.siddhi.core.util.parser.QueryParser.parse(
> QueryParser.java:132)
> at org.wso2.siddhi.core.util.parser.ExecutionPlanParser.
> parse(ExecutionPlanParser.java:145)
> ... 77 more
> Caused by: org.wso2.siddhi.core.exception.ExecutionPlanCreationException:
> com.typesafe.config.ConfigException$Missing: No configuration setting
> found for key 'akka.version'
> at org.wso2.siddhi.core.query.processor.stream.
> AbstractStreamProcessor.initProcessor(AbstractStreamProcessor.java:79)
> at org.wso2.siddhi.core.util.parser.SingleInputStreamParser.
> generateProcessor(SingleInputStreamParser.java:182)
> at org.wso2.siddhi.core.util.parser.SingleInputStreamParser.
> parseInputStream(SingleInputStreamParser.java:93)
> at org.wso2.siddhi.core.util.parser.InputStreamParser.
> parse(InputStreamParser.java:58)
> at org.wso2.siddhi.core.util.parser.QueryParser.parse(
> QueryParser.java:85)
> ... 78 more
> Caused by: com.typesafe.config.ConfigException$Missing: No configuration
> setting found for key 'akka.version'
> at com.typesafe.config.impl.SimpleConfig.findKey(
> SimpleConfig.java:124)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:145)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:151)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
> at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
> at com.typesafe.config.impl.SimpleConfig.getString(
> SimpleConfig.java:206)
> at akka.actor.ActorSystem$Settings.(ActorSystem.scala:169)
> at akka.actor.ActorSystemImpl.(ActorSystem.scala:505)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:142)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:119)
> at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$
> doCreateActorSystem(AkkaUtils.scala:121)
> at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(
> AkkaUtils.scala:53)
> at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(
> AkkaUtils.scala:52)
> at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.
> apply$mcVI$sp(Utils.scala:1988)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1979)
> at org.apache.spark.util.AkkaUtils$.createActorSystem(
> AkkaUtils.scala:55)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:266)
> at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:193)
> at org.apache.spark.SparkContext.createSparkEnv(SparkContext.
> scala:288)
> at org.apache.spark.SparkContext.(SparkContext.scala:457)
> at org.apache.spark.api.java.JavaSparkContext.(
> JavaSparkContext.scala:59)
> at org.gsoc.carbon.ml.siddhi.extension.streaming.algorithm.
> StreamingLinearRegression.(StreamingLinearRegression.java:60)
> at org.gsoc.carbon.ml.siddhi.extension.streaming.
> StreamingLinearRegressionStreamProcessor.init(
> StreamingLinearRegressionStreamProcessor.java:83)
> at org.wso2.siddhi.core.query.processor.stream.
> AbstractStreamProcessor.initProcessor(AbstractStreamProcessor.java:65)
> ... 82 more
>
>
> thakn you.
> BR,
> Mahesh.
> [1] https://github.com/dananjayamahesh/streaming/
> blob/master/src/main/java/org/gsoc/carbon/ml/siddhi/
> extension/streaming/algorithm/StreamingLinearRegression.java
>
>
> On Mon, Aug 8, 2016 at 4:00 PM, Supun Sethunga <sup...@wso2.com> wrote:
>
>> Hi Mahesh,
>>
>> Couple of issues I noticed:
>>
>>- Your siddhi-extension is using spark 1.6.1 dependency. But in the
>>logs, it says the version of spark is 1.4.1. You can see the following
>> line
>>in the logs.
>>
>> *TID: [-1234] [] [2016-08-08 13:05:11,699] INFO
>> {org.apache.spark.SparkContext} - Running Spark version 1.4.1 *
>>
>>
>>- In each of the extension, you are starting a spark-context, and a
>>new spark application. But spark only allows to create one spark context
>>per JVM.
>>
>> Can you fix those and check whether the issue is still exists? Also
>> please use a fresh CEP pack when testing. For now, you can avoid the second
>> issue by creating only one execution plan and calling only one algorithm at
>> a time.
>>
>> Btw, it would be easy for us to reproduce the issue and check whats
>> happening, if you can include all dependencies inside the jar itself.
>> Otherwise, it's a nightmare to find and add the missing dependencies one by
>> one.
>>
>> Regards,
>> Supun
>>
>>
>> On Mon, Aug 8, 2016 at 3:43 PM, Mahesh Dananjaya <
>> dananjay
Re: [Dev] Fwd: GSOC2016: [ML][CEP] [SAMOA]Predictive analytic with online data for WSO2 Machine Learner-Samoa Integration
>>> Did you add samoa jars in ML?
>>>>>>>>
>>>>>>>> On Fri, Aug 5, 2016 at 12:20 PM, Mahesh Dananjaya <
>>>>>>>> dananjayamah...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Supun,
>>>>>>>>> This is the error i am getting while run the extension in the
>>>>>>>>> cabon-ml side,
>>>>>>>>> plase reffer to link [1] for the class.
>>>>>>>>>
>>>>>>>>> ERROR
>>>>>>>>> {org.wso2.carbon.ml.siddhi.extension.streaming.samoa.StreamingClusteringTaskBuilder}
>>>>>>>>> - Fail to initialize the task
>>>>>>>>>
>>>>>>>>> java.lang.Exception: Class not found: StreamingClusteringTask
>>>>>>>>>
>>>>>>>>>at com.github.javacliparser.ClassOption.cliStringToObject(Class
>>>>>>>>> Option.java:136)
>>>>>>>>>
>>>>>>>>>at org.wso2.carbon.ml.siddhi.extension.streaming.samoa.Streamin
>>>>>>>>> gClusteringTaskBuilder.initClusteringTask(StreamingClusterin
>>>>>>>>> gTaskBuilder.java:129)
>>>>>>>>>
>>>>>>>>>at org.wso2.carbon.ml.siddhi.extension.streaming.samoa.Streamin
>>>>>>>>> gClusteringTaskBuilder.initTask(StreamingClusteringTaskBuild
>>>>>>>>> er.java:100)
>>>>>>>>>
>>>>>>>>>at org.wso2.carbon.ml.siddhi.extension.streaming.samoa.Streamin
>>>>>>>>> gClustering.run(StreamingClustering.java:77)
>>>>>>>>>
>>>>>>>>>at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>
>>>>>>>>> [2016-08-04 16:23:07,437] INFO {org.wso2.carbon.ml.siddhi.ext
>>>>>>>>> ension.streaming.samoa.StreamingClusteringTaskBuilder} - Fail to
>>>>>>>>> initialize the taskjava.lang.Exception: Class not found:
>>>>>>>>> StreamingClusteringTask
>>>>>>>>>
>>>>>>>>> +++Please refeer link [1] for the
>>>>>>>>> StreamingClusteringTask.
>>>>>>>>>
>>>>>>>>> Then again for StreamingClusteringStream class while i bypass the
>>>>>>>>> String query in the initTask(). please refer link [2] for the class.
>>>>>>>>>
>>>>>>>>> Exception in thread "Thread-60" java.lang.IllegalArgumentException:
>>>>>>>>> Problems with option: streamTrain
>>>>>>>>>
>>>>>>>>>at com.github.javacliparser.ClassOption.setValueViaCLIString(Cl
>>>>>>>>> assOption.java:64)
>>>>>>>>>
>>>>>>>>>at com.github.javacliparser.AbstractOption.resetToDefault(Abstr
>>>>>>>>> actOption.java:90)
>>>>>>>>>
>>>>>>>>>at com.github.javacliparser.AbstractClassOption.(Abstract
>>>>>>>>> ClassOption.java:84)
>>>>>>>>>
>>>>>>>>>at com.github.javacliparser.AbstractClassOption.(Abstract
>>>>>>>>> ClassOption.java:63)
>>>>>>>>>
>>>>>>>>>at com.github.javacliparser.ClassOption.(ClassOption.java
>>>>>>>>> :38)
>>>>>>>>>
>>>>>>>>>at org.wso2.carbon.ml.siddhi.extension.streaming.samoa.Streamin
>>>>>>>>> gClusteringTask.(StreamingClusteringTask.java:52)
>>>>>>>>>
>>>>>>>>>at org.wso2.carbon.ml.siddhi.extension.streaming.samoa.Streamin
>>>>>>>>> gClusteringTaskBuilder.initClusteringTask(StreamingClusterin
>>>>>>>>> gTaskBuilder.java:140)
>>>>>>>>>
>>>>>>>>>at org.wso2.carbon.ml.siddhi.extension.streaming.samoa.Streamin
>>>>>>>>> gClusteringTaskBuilder.initTask(StreamingClusteringTaskBuild
>>>>>>>>> er.java:103)
>>>>>>>>>
>>>>>>>>>at org.wso2.carbon.ml.siddhi.extension.streami
Re: [Dev] [GSoC 2016]Proposal 03: [ML]Visual(Deep) Neural Network Builder -Project Progress
Hi Lakini, Please find all the APIs and their samples at [1]. Additionally, you can always refer the existing UI to get an idea. for eg: you can check the dataset page of the existing ML UI to see how the datasets are retrieved. [1] https://docs.wso2.com/display/ML110/REST+API+Guides Regards, Supun. On Sat, Aug 6, 2016 at 11:06 AM, Lakini Senanayaka < lakinisenanayak...@gmail.com> wrote: > Hi Upul, > > Could you please explain me the way to get the dataset from the carbon ML > to the API which I am implementing. > > Thank you. > > On Thu, Aug 4, 2016 at 4:37 PM, Lakini Senanayaka < > lakinisenanayak...@gmail.com> wrote: > >> Hi, >> >> This is the meeting minute for today's GSoC project review meeting. >> >> Upul reviewed the current progress of the project, "(Deep) Neural Network >> Builder for WSO2 Machine Learner". >> Below mentioned are the decisions made and the new tasks allocated at the >> meeting. >> >> Assigned action items: >> -Making the UI more user-friendly. >> Ex: Physical lines of connections between layers should follow >> the movements of the layers. >> -Train the feed forward network for MNIST dataset. >> -Testing the accuracy of the neural network model for different >> optimization and updater algorithms. >> -Handling Exceptions. >> -Trying to show graphs related to the model, along with the accuracy >> of the model in the output. >> -Start writing documentation. >> -Showing a demo for the ML team. >> >> Decisions made: >> >> -Higher priority should be given in implementing feed-forward >> network than RNN. >> -Upul will provide me necessary instructions on the way to access >> the dataset which is inserted through the WSO2 ML console as NN builder UI >> needs to access that dataset and use as the training dataset. >> >> Thank you. >> >> On Wed, Aug 3, 2016 at 5:59 AM, Lakini Senanayaka < >> lakinisenanayak...@gmail.com> wrote: >> >>> Hi, >>> >>> Thank you very much for your kindness. >>> >>> I can schedule a meeting on this Thursday(4th-August-2016). >>> >>> Thank you. >>> >>> On Tue, Aug 2, 2016 at 10:39 AM, Upul Bandara <u...@wso2.com> wrote: >>> >>>> Hi, >>>> >>>> As you know we have only two weeks to complete GSOC projects. >>>> Unfortunately, this project is a little bit lagging behind and it looks >>>> like you have to put some additional effort during the last two week. In >>>> order to support you, we though it is better to have a small F2F meeting >>>> with you and it will help you to sort out any issues currently you are >>>> facing with. >>>> >>>> So can you please schedule a meeting with the ML team? >>>> >>>> Thanks, >>>> Upul >>>> >>>> On Sat, Jul 30, 2016 at 5:45 PM, Nirmal Fernando <nir...@wso2.com> >>>> wrote: >>>> >>>>> >>>>> On Sat, Jul 30, 2016 at 1:43 PM, Lakini Senanayaka < >>>>> lakinisenanayak...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have solved the above problem.I have referred the mail thread *[Dev] >>>>>> "Error 403 - Forbidden" when session expires in admin console*[1] in >>>>>> dev mailing list.I couldn't upload jaggery files to the console and >>>>>> when >>>>>> I was trying to do it, it gave the* Error 403-Forbidden.* >>>>>> >>>>>> I have copied the jaggery files to >>>>>> */repository/deployment/server/jaggerapps* directory as >>>>>> Supun instructed and it worked. >>>>>> >>>>>> Although I have implemented the back end for feed forward neural >>>>>> network still it has some problems.It doesn't work well.Do I need to add >>>>>> the neural network algorithms to the >>>>>> *org.wso2.carbon.ml.core.spark.algorithms* in Carbon ML ?Do I need >>>>>> to follow the same pattern which is used by the Carbon ML when >>>>>> implementing >>>>>> the back end coding and APIs? >>>>>> >>>>> >>>>> Yes please. >>>>> >>>>>> >>>>>> I will submit the demo ASAP. >>>>>> >>>>>> [1]https://mail.google.com/mail/u/0/#search/Re%3A+%5BDev%5D+ >>>>>> %22Error+403++Forbidden%22+when+session+expires+in+admin%09c >>>>>> onsole/155bfcbf7e98992f?projector=1 >>>>>> >>>>>> Thank you. >>>>>> -- >>>>>> KIND Regards, >>>>>> *Lakini Senanayaka.* >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Upul Bandara, >>>> Associate Technical Lead, WSO2, Inc., >>>> Mob: +94 715 468 345. >>>> >>> >>> >>> >>> -- >>> KIND Regards, >>> *Lakini Senanayaka.* >>> >>> >> >> >> -- >> KIND Regards, >> *Lakini Senanayaka.* >> >> > > > -- > KIND Regards, > *Lakini Senanayaka.* > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSoC 2016] Final Evaluation
[adding ml gsoc students] On Wed, Aug 3, 2016 at 10:46 PM, Imesh Gunaratne <im...@wso2.com> wrote: > Dear Students, > > This is to give you a heads up on $subject. > > We have seen in the past students being inactive time to time and trying > to complete the deliverables at the last minute when the submission > deadline is nearby. > > Please note that at the final evaluation we will be evaluating your > overall effort throughout the project period in addition to the > deliverables that you submit. On high level this would include following: > >- Planning >- Researching >- Problem solving >- Initiatives >- Level of communication >- Community interaction >- Continuous effort on design & implementation >- Deliverables >- Documentation >- Quality of code & all deliverables > > GSoC student manual [1] will be really helpful for you to understand this > better. > > [1] > http://write.flossmanuals.net/gsocstudentguide/what-is-google-summer-of-code/ > > Thanks > > -- > *Imesh Gunaratne* > Software Architect > WSO2 Inc: http://wso2.com > T: +94 11 214 5345 M: +94 77 374 2057 > W: https://medium.com/@imesh TW: @imesh > lean. enterprise. middleware > > > _______ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Fwd: GSOC2016: [ML][CEP] [SAMOA]Predictive analytic with online data for WSO2 Machine Learner-Samoa Integration
Hi Mahesh, samoa dependency version in siddhi-extension should be *0.4.0-incubating-SNAPSHOT*. That should solve the issue Regards, Supun On Thu, Aug 4, 2016 at 11:50 AM, Mahesh Dananjaya <dananjayamah...@gmail.com > wrote: > Hi all, > samoa is in my local repository and dependencies works fine with all other > extensions that add samoa 0.4.0-incubator dependencies. But only when i > build carbon-ml, it gives priority for the remote repository for looking > samoa. SO any help with this to give priority for local m2 repo before > carbon-ml building is looking for the remote one. I am getting the error > because of this. maven option -U also not seems to be working here. any > help please. > > [ERROR] Failed to execute goal on project > org.wso2.carbon.ml.siddhi.extension: Could not resolve dependencies for > project > org.wso2.carbon.ml:org.wso2.carbon.ml.siddhi.extension:bundle:1.1.2-SNAPSHOT: > The following artifacts could not be resolved: > org.apache.samoa:samoa-api:jar:0.4.0-incubating, > org.apache.samoa:samoa-local:jar:0.4.0-incubating: Could not find artifact > org.apache.samoa:samoa-api:jar:0.4.0-incubating in wso2-nexus ( > http://maven.wso2.org/nexus/content/groups/wso2-public/) > > Since samoa will be used for future work, is it possible to add that in > the relevant wso2 repo. Because there are still no samoa 0.4.0-incubator > maven repo. only 0.3.0. we cannot continue our work with the 0.3.0 since it > is outdated. > > thank you. > regards, > Mahesh. > > On Wed, Aug 3, 2016 at 4:29 PM, Miyuru Dayarathna <miyu...@wso2.com> > wrote: > >> Adding Jayan to this email thread. >> >> -- >> Thanks, >> Miyuru Dayarathna >> Senior Technical Lead >> Mobile: +94713527783 >> Blog: http://miyurublog.blogspot.com >> >> On Wed, Aug 3, 2016 at 3:11 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi Mahesh, >>> >>> you should build carbon-ml *without *-U option. -U means you force mvn >>> to look for updates in remote repo. Rather run it with -o option. Also, can >>> you double check whether the dependencies are defined correctly (group >>> Id's, versions etc). >>> >>> Alternatively, it seems there is a samoa released version in mvn repo. >>> Maybe you could try that one as well. But that's v0.3.0.. >>> >>> [1] https://mvnrepository.com/artifact/org.apache.samoa >>> >>> On Wed, Aug 3, 2016 at 2:54 PM, Mahesh Dananjaya < >>> dananjayamah...@gmail.com> wrote: >>> >>>> Hi Supun, >>>> I just neeed a little help. I am in the process of integrating my samoa >>>> core functions and extension into carbon-ml's siddhi extension. For samoa i >>>> am using locally built samoa project to provide samoa 0.4.0 >>>> dependencies,since we dont have it in the maven repo or else where. But >>>> when i build carbon-ml by adding samoa dependencies, it seems to be maven >>>> search for remote location, not the local maven repo first. I am running >>>> maven with -U option. But still the problems occurs. Is there any specific >>>> thing in carbon-ml like settings to search remote before local one? I just >>>> need to give local maven repo for the dependency. My extension seperately >>>> working fine, so there is no problem wihat the local dependencies outside >>>> carbon-ml. So can you please help me with this.thank you. >>>> regards, >>>> Mahesh. >>>> >>>> On Fri, Jul 22, 2016 at 3:17 PM, Srinath Perera <srin...@wso2.com> >>>> wrote: >>>> >>>>> Hi Mahesh, >>>>> >>>>> On Thu, Jul 21, 2016 at 2:10 PM, Mahesh Dananjaya < >>>>> dananjayamah...@gmail.com> wrote: >>>>> >>>>>> Hi All, >>>>>> I am onto connecting cep streams with samoa streams to data analysis >>>>>> using samoa framework. To connect samoa with cep siddhi event streams >>>>>> what >>>>>> i we can do is that try to convert cep streams into samoa streams or else >>>>>> writing wrpper for samoa for cep streasm to be used. In both cases i >>>>>> have >>>>>> to covert siddhi cep streasm into samoa streams. Samoa is using MOA to >>>>>> analyse data. Moo contains ML framework to analyse stream data. Samoa is >>>>>> wrapping MOA withsome of its classes. >>>>>> >>>>>> Samoa streams is based on MOA, Instance and InstanceStreams. Samoa
Re: [Dev] Fwd: GSOC2016: [ML][CEP] [SAMOA]Predictive analytic with online data for WSO2 Machine Learner-Samoa Integration
.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>> ... 7 more >>>>>> >>>>>> >>>>>> can i actually call the Task like this. >>>>>> >>>>>> BR, >>>>>> Mahesh. >>>>>> >>>>>> [1] >>>>>> https://samoa.incubator.apache.org/documentation/Prequential-Evaluation-Task.html >>>>>> [2] >>>>>> https://github.com/apache/incubator-samoa/blob/releases/0.4.0-incubating-RC0/samoa-api/src/main/java/org/apache/samoa/tasks/ClusteringEvaluation.java >>>>>> [3] >>>>>> https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc/samoa/streaming/src/main/java/org/gsoc/samoa/streaming >>>>>> [4] >>>>>> https://github.com/apache/incubator-samoa/tree/releases/0.4.0-incubating-RC0/samoa-local/src/main/java/org/apache/samoa >>>>>> >>>>>> >>>>>> On Thu, Jul 14, 2016 at 3:47 PM, Mahesh Dananjaya < >>>>>> dananjayamah...@gmail.com> wrote: >>>>>> >>>>>>> Hi srinath, >>>>>>> sure.i am working on it.thank you. >>>>>>> regards, >>>>>>> Mahesh. >>>>>>> >>>>>>> On Thu, Jul 14, 2016 at 11:12 AM, Srinath Perera <srin...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Mahesh, >>>>>>>> >>>>>>>> Let's focus on getting SAOMA work with CEP. It is OK to be limited >>>>>>>> to few algorithms. >>>>>>>> >>>>>>>> --Srinath >>>>>>>> >>>>>>>> On Thu, Jul 14, 2016 at 10:49 AM, Mahesh Dananjaya < >>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Maheshakya, >>>>>>>>> I think we can build new tasks [1] like the one in execution plan >>>>>>>>> in cep with samoa. I will try to build a one. >>>>>>>>> regards, >>>>>>>>> Mahesh. >>>>>>>>> [1] >>>>>>>>> https://samoa.incubator.apache.org/documentation/Developing-New-Tasks-in-SAMOA.html >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jul 14, 2016 at 10:35 AM, Mahesh Dananjaya < >>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Maheshakya, >>>>>>>>>> I am building and running samoa to see its functionality. In >>>>>>>>>> samoa still we have limited supports in algorithms. Samoa supports >>>>>>>>>> only >>>>>>>>>> classification and clustering with streams. It also use kind of >>>>>>>>>> StreamProcessor, like the one we use in StreamProcessor extension. >>>>>>>>>> I was >>>>>>>>>> getting started with Samoa referring to this page [1]. Then i ran >>>>>>>>>> couple of >>>>>>>>>> examples to identified the flow. Samoa use hadoop framework instead >>>>>>>>>> spark >>>>>>>>>> for distribution. But i am using it in a local mode. When i see the >>>>>>>>>> Samoa >>>>>>>>>> core there is only limited algorithms. IMO if we are going to use >>>>>>>>>> Samoa we >>>>>>>>>> have to limit the functionality and algorithms [2]. When i go to >>>>>>>>>> developer >>>>>>>>>> corner in [3], it seems to be something like CEP extension that we >>>>>>>>>> are >>>>>>>>>> using currenlty. SO in Samoa though the algorihtms are limited, they >>>>>>>>>> have >>>>>>>>>> implemented streaming support for them. Therefore if we integrate it >>>>>>>>>> into >>>>>>>>>> CEP we have to look for how to handle streams and algorithms in >>>>>>>>>> Samoa side. >>>>>>>>>> Is it good for your side to have both hadoop and spark running >>>>>>>>>> background.thank you. >>>>>>>>>> regards, >>>>>>>>>> Mahesh. >>>>>>>>>> >>>>>>>>>> [1] https://samoa.incubator.apache.org/documentation/Home.html >>>>>>>>>> [2] >>>>>>>>>> https://samoa.incubator.apache.org/documentation/api/current/index.html >>>>>>>>>> [3] >>>>>>>>>> https://samoa.incubator.apache.org/documentation/SAMOA-Topology.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Srinath Perera, Ph.D. >>>>>>>>http://people.apache.org/~hemapani/ >>>>>>>>http://srinathsview.blogspot.com/ >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> ___ >>>>>> Dev mailing list >>>>>> Dev@wso2.org >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pruthuvi Maheshakya Wijewardena >>>>> >>>>> mahesha...@wso2.com >>>>> +94711228855 >>>>> >>>>> >>>>> >>>> >>> >> >> >> -- >> >> Srinath Perera, Ph.D. >>http://people.apache.org/~hemapani/ >>http://srinathsview.blogspot.com/ >> > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [VOTE] Release WSO2 Machine Learner 1.1.1 RC1
[-] Calling off the vote as we have encountered some issues with the ml ui jaggery app. url mappings are broken. We will start a new vote soon. Regards, Supun On Sat, Jul 30, 2016 at 8:30 AM, Supun Sethunga <sup...@wso2.com> wrote: > Hi Devs, > > This is the 1st Release Candidate of the WSO2 Machine Learner 1.1.1. This > is a patch release and following main fixes were included: > >- ML-DAS integration >- Security fixes associated with the WSO2 Carbon 4.4.7 platform release > > > Please download, test the product and vote. Vote will be open for 72 hours > or as needed. > Refer to github readme for guides. > > The distribution is available to download at [1]. > > Fixed Issue in [2]. Tag to be vote upon [3]. > > [1] > https://github.com/wso2/product-ml/releases/download/v1.1.1-rc1/wso2ml-1.1.1-rc1.zip > > [2] https://wso2.org/jira/issues/?filter=13175 > [3] *https://github.com/wso2/product-ml/releases/tag/v1.1.1-rc1 > <https://github.com/wso2/product-ml/releases/tag/v1.1.1-rc1>* > > Please vote as follows. > [+] Stable - go ahead and release > [-] Broken - do not release (please explain why) > > Thanks, > WSO2 ML Team > > -- > *Supun Sethunga* > Senior Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > Blog: http://supunsetunga.blogspot.com > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [VOTE] Release WSO2 Machine Learner 1.1.1 RC1
Hi Devs, This is the 1st Release Candidate of the WSO2 Machine Learner 1.1.1. This is a patch release and following main fixes were included: - ML-DAS integration - Security fixes associated with the WSO2 Carbon 4.4.7 platform release Please download, test the product and vote. Vote will be open for 72 hours or as needed. Refer to github readme for guides. The distribution is available to download at [1]. Fixed Issue in [2]. Tag to be vote upon [3]. [1] https://github.com/wso2/product-ml/releases/download/v1.1.1-rc1/wso2ml-1.1.1-rc1.zip [2] https://wso2.org/jira/issues/?filter=13175 [3] *https://github.com/wso2/product-ml/releases/tag/v1.1.1-rc1 <https://github.com/wso2/product-ml/releases/tag/v1.1.1-rc1>* Please vote as follows. [+] Stable - go ahead and release [-] Broken - do not release (please explain why) Thanks, WSO2 ML Team -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [Orbit] Please merge PR for spark-mllib_2.10 version 1.4.1.wso2v3
e is: 401, ReasonPhrase: Unauthorized. >>> >>> at >>> org.apache.maven.plugin.deploy.DeployMojo.deployProject(DeployMojo.java:284) >>> >>> at org.apache.maven.plugin.deploy.DeployMojo.execute(DeployMojo.java:169) >>> >>> at >>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) >>> >>> at >>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) >>> >>> ... 19 more >>> >>> Caused by: >>> org.apache.maven.artifact.deployer.ArtifactDeploymentException: Failed to >>> deploy artifacts: Could not transfer artifact >>> org.wso2.orbit.org.apache.spark:spark-mllib_2.10:jar:1.4.1.wso2v3 from/to >>> wso2.releases ( >>> http://maven.wso2.org/nexus/content/repositories/releases/): Failed to >>> transfer file: >>> http://maven.wso2.org/nexus/content/repositories/releases/org/wso2/orbit/org/apache/spark/spark-mllib_2.10/1.4.1.wso2v3/spark-mllib_2.10-1.4.1.wso2v3.jar. >>> Return code is: 401, ReasonPhrase: Unauthorized. >>> >>> at >>> org.apache.maven.artifact.deployer.DefaultArtifactDeployer.deploy(DefaultArtifactDeployer.java:141) >>> >>> at >>> org.apache.maven.plugin.deploy.AbstractDeployMojo.deploy(AbstractDeployMojo.java:171) >>> >>> at >>> org.apache.maven.plugin.deploy.DeployMojo.deployProject(DeployMojo.java:250) >>> >>> ... 22 more >>> >>> Caused by: org.sonatype.aether.deployment.DeploymentException: Failed to >>> deploy artifacts: Could not transfer artifact >>> org.wso2.orbit.org.apache.spark:spark-mllib_2.10:jar:1.4.1.wso2v3 from/to >>> wso2.releases ( >>> http://maven.wso2.org/nexus/content/repositories/releases/): Failed to >>> transfer file: >>> http://maven.wso2.org/nexus/content/repositories/releases/org/wso2/orbit/org/apache/spark/spark-mllib_2.10/1.4.1.wso2v3/spark-mllib_2.10-1.4.1.wso2v3.jar. >>> Return code is: 401, ReasonPhrase: Unauthorized. >>> >>> at >>> org.sonatype.aether.impl.internal.DefaultDeployer.deploy(DefaultDeployer.java:280) >>> >>> at >>> org.sonatype.aether.impl.internal.DefaultDeployer.deploy(DefaultDeployer.java:211) >>> >>> at >>> org.sonatype.aether.impl.internal.DefaultRepositorySystem.deploy(DefaultRepositorySystem.java:443) >>> >>> at >>> org.apache.maven.artifact.deployer.DefaultArtifactDeployer.deploy(DefaultArtifactDeployer.java:137) >>> >>> ... 24 more >>> >>> Caused by: org.sonatype.aether.transfer.ArtifactTransferException: Could >>> not transfer artifact >>> org.wso2.orbit.org.apache.spark:spark-mllib_2.10:jar:1.4.1.wso2v3 from/to >>> wso2.releases ( >>> http://maven.wso2.org/nexus/content/repositories/releases/): Failed to >>> transfer file: >>> http://maven.wso2.org/nexus/content/repositories/releases/org/wso2/orbit/org/apache/spark/spark-mllib_2.10/1.4.1.wso2v3/spark-mllib_2.10-1.4.1.wso2v3.jar. >>> Return code is: 401, ReasonPhrase: Unauthorized. >>> >>> at >>> org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:951) >>> >>> at >>> org.sonatype.aether.connector.wagon.WagonRepositoryConnector$4.wrap(WagonRepositoryConnector.java:941) >>> >>> at >>> org.sonatype.aether.connector.wagon.WagonRepositoryConnector$PutTask.run(WagonRepositoryConnector.java:837) >>> >>> at >>> org.sonatype.aether.connector.wagon.WagonRepositoryConnector.put(WagonRepositoryConnector.java:467) >>> >>> at >>> org.sonatype.aether.impl.internal.DefaultDeployer.deploy(DefaultDeployer.java:274) >>> >>> ... 27 more >>> >>> Caused by: org.apache.maven.wagon.TransferFailedException: Failed to >>> transfer file: >>> http://maven.wso2.org/nexus/content/repositories/releases/org/wso2/orbit/org/apache/spark/spark-mllib_2.10/1.4.1.wso2v3/spark-mllib_2.10-1.4.1.wso2v3.jar. >>> Return code is: 401, ReasonPhrase: Unauthorized. >>> >>> at >>> org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.put(AbstractHttpClientWagon.java:626) >>> >>> at >>> org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.put(AbstractHttpClientWagon.java:524) >>> >>> at >>> org.apache.maven.wagon.shared.http4.AbstractHttpClientWagon.put(AbstractHttpClientWagon.java:505) >>> >>> at >>> org.apache.maven.wagon.sh
Re: [Dev] [Orbit] Please merge PR for spark-mllib_2.10 version 1.4.1.wso2v3
Hi Manoj, Did the changes as per the comment. Can you please review it? Thanks, Supun On Mon, Jul 25, 2016 at 6:25 PM, Manoj Kumara <ma...@wso2.com> wrote: > Hi Supun, > > Added a comment. Please resolve that. > > Regards, > manoj > > *Manoj Kumara* > WSO2 Inc. *| **lean. enterprise. middleware.* > *Mobile:* +94 713 448188 > > On Mon, Jul 25, 2016 at 5:23 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Carbon Team, >> >> Can you please review and merge PR[1]? This is a fix for [2]. >> >> [1] https://github.com/wso2/orbit/pull/222 >> [2] https://wso2.org/jira/browse/CEP-1471 >> >> Regards, >> Supun >> >> -- >> *Supun Sethunga* >> Senior Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> Blog: http://supunsetunga.blogspot.com >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [Orbit] Please merge PR for spark-mllib_2.10 version 1.4.1.wso2v3
Hi Carbon Team, Can you please review and merge PR[1]? This is a fix for [2]. [1] https://github.com/wso2/orbit/pull/222 [2] https://wso2.org/jira/browse/CEP-1471 Regards, Supun -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] DAS event receiving gets blocked under high load in minimum HA
Hi, Thanks Suho for the quick fix. I tested with the latest siddhi (with 50+ million events, spanning over 1.5 hours ), in both single node and minimum HA setups, and the issue no longer exists. Thanks, Supun On Mon, Jul 4, 2016 at 12:31 PM, Supun Sethunga <sup...@wso2.com> wrote: > Execution plan: > > [1] > https://github.com/wso2/analytics-esb/blob/master/features/org.wso2.analytics.esb.feature/src/main/capp/realtime/ESBAnalyticsExecutionPlan/ESBAnalytics.siddhiql > > Thanks, > Supun > > On Mon, Jul 4, 2016 at 12:30 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Attached the thread dump. >> >> [1] is the execution plan >> >> >> On Fri, Jul 1, 2016 at 7:37 PM, Sriskandarajah Suhothayan <s...@wso2.com> >> wrote: >> >>> Can you reproduce this and get a thread dump & share that with us when >>> it got stuck. >>> >>> Regards >>> Suho >>> >>> On Fri, Jul 1, 2016 at 7:25 PM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi, >>>> >>>> While running a load test with the latest analytics-esb, noticed that >>>> the event receiving gets blocked after some time (within 15 mins), under >>>> high load. This was a minimum HA cluster, and we persist only per-second >>>> stats. Until the receiving got blocked, events received at a throughput of >>>> around 9000. Also, I couldn't reproduce this in a single node setup. >>>> >>>> Since there weren't any changes to execution plans, I tried reverting >>>> the siddhi (core, query-api, query-compiler) to an older build (Jun 23, >>>> 2016), and re-ran the test, and did not encounter the issue... >>>> >>>> Thanks, >>>> Supun >>>> >>>> -- >>>> *Supun Sethunga* >>>> Senior Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> Blog: http://supunsetunga.blogspot.com >>>> >>> >>> >>> >>> -- >>> >>> *S. Suhothayan* >>> Technical Lead & Team Lead of WSO2 Complex Event Processor >>> *WSO2 Inc. *http://wso2.com >>> * <http://wso2.com/>* >>> lean . enterprise . middleware >>> >>> >>> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >>> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >>> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >>> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>* >>> >> >> >> >> -- >> *Supun Sethunga* >> Senior Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> Blog: http://supunsetunga.blogspot.com >> > > > > -- > *Supun Sethunga* > Senior Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > Blog: http://supunsetunga.blogspot.com > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] DAS event receiving gets blocked under high load in minimum HA
Execution plan: [1] https://github.com/wso2/analytics-esb/blob/master/features/org.wso2.analytics.esb.feature/src/main/capp/realtime/ESBAnalyticsExecutionPlan/ESBAnalytics.siddhiql Thanks, Supun On Mon, Jul 4, 2016 at 12:30 PM, Supun Sethunga <sup...@wso2.com> wrote: > Attached the thread dump. > > [1] is the execution plan > > > On Fri, Jul 1, 2016 at 7:37 PM, Sriskandarajah Suhothayan <s...@wso2.com> > wrote: > >> Can you reproduce this and get a thread dump & share that with us when it >> got stuck. >> >> Regards >> Suho >> >> On Fri, Jul 1, 2016 at 7:25 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi, >>> >>> While running a load test with the latest analytics-esb, noticed that >>> the event receiving gets blocked after some time (within 15 mins), under >>> high load. This was a minimum HA cluster, and we persist only per-second >>> stats. Until the receiving got blocked, events received at a throughput of >>> around 9000. Also, I couldn't reproduce this in a single node setup. >>> >>> Since there weren't any changes to execution plans, I tried reverting >>> the siddhi (core, query-api, query-compiler) to an older build (Jun 23, >>> 2016), and re-ran the test, and did not encounter the issue... >>> >>> Thanks, >>> Supun >>> >>> -- >>> *Supun Sethunga* >>> Senior Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> Blog: http://supunsetunga.blogspot.com >>> >> >> >> >> -- >> >> *S. Suhothayan* >> Technical Lead & Team Lead of WSO2 Complex Event Processor >> *WSO2 Inc. *http://wso2.com >> * <http://wso2.com/>* >> lean . enterprise . middleware >> >> >> *cell: (+94) 779 756 757 <%28%2B94%29%20779%20756%20757> | blog: >> http://suhothayan.blogspot.com/ <http://suhothayan.blogspot.com/>twitter: >> http://twitter.com/suhothayan <http://twitter.com/suhothayan> | linked-in: >> http://lk.linkedin.com/in/suhothayan <http://lk.linkedin.com/in/suhothayan>* >> > > > > -- > *Supun Sethunga* > Senior Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > Blog: http://supunsetunga.blogspot.com > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] DAS event receiving gets blocked under high load in minimum HA
Hi, While running a load test with the latest analytics-esb, noticed that the event receiving gets blocked after some time (within 15 mins), under high load. This was a minimum HA cluster, and we persist only per-second stats. Until the receiving got blocked, events received at a throughput of around 9000. Also, I couldn't reproduce this in a single node setup. Since there weren't any changes to execution plans, I tried reverting the siddhi (core, query-api, query-compiler) to an older build (Jun 23, 2016), and re-ran the test, and did not encounter the issue... Thanks, Supun -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi, I will check and let you know Regards, Supun On Tue, Jun 28, 2016 at 5:33 AM, Misgana Negassi < negas...@tf.uni-freiburg.de> wrote: > Hi Supun, > > These are the links for the code I changed. > > > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.database/src/main/java/org/wso2/carbon/ml/database/DatabaseService.java#L422 > > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.database/src/main/java/org/wso2/carbon/ml/database/internal/MLDatabaseService.java#L2173 > > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.database/src/main/java/org/wso2/carbon/ml/database/internal/MLDatabaseService.java#L2506 > > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.commons/src/main/java/org/wso2/carbon/ml/commons/domain/Workflow.java#L267 > > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.commons/src/main/java/org/wso2/carbon/ml/commons/domain/Workflow.java#L280 > > Best, > Misgana > > > On 27.06.2016 08:53, Supun Sethunga wrote: > > Hi Misgana, > > Can you commit all the changes to your fork of carbon-ml, and share us the > link. So that I can check whats happening. > > Also, do I need to manually edit the machinelearner.xml file(s) to include >> Stacking? > > Yes. No need to add hyper-parameters, since we can't predefine them. > > Thanks, > Supun > > On Mon, Jun 27, 2016 at 4:33 AM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> Hello Supun, >> sorry to bother you again with the same problem but I am not making any >> progress right now :( >> I can't get the sample script running. >> >> I tried two approaches: >> A) Using the server provided and applying patches. >> Problem: I can not apply my patch. I suspect the problem is that my patch >> is of a newer version. The server's carbon ml is 1.1.1. and mine is 1.1.2. >> Can I somewhere change the version number when I build my project? >> >> B) Building carbon and then product-ml and then unpacking >> modules/distribution/target/wso2ml-2.0.0-SNAPSHOT.zip >> I can start the server with the right components except for one exception: >> >> log4j:ERROR Could not instantiate appender named "CARBON_MEMORY". >> log4j:ERROR Could not instantiate class >> [org.wso2.carbon.logging.service.appender.CarbonMemoryAppender]. >> java.lang.ClassNotFoundException: >> org.wso2.carbon.logging.service.appender.CarbonMemoryAppender cannot be >> found by org.wso2.carbon.logging_4.4.3 >> at >> org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501) >> >> >> However, when I try to run my example script there seems to be nothing >> happening on the server side. No exceptions but also the debugger does not >> stop when I expect it would. It seems something is crashing but the >> exceptions are swallowed. Is there a way to increase the debug output? >> >> Also, do I need to manually edit the machinelearner.xml file(s) to >> include Stacking? >> >> >> On 21.06.2016 05:41, Supun Sethunga wrote: >> >> Hi Misgana, >> >> I have two issues: >> 1. Issue: >> >> Exception in thread "pool-17-thread-7" java.lang.IllegalArgumentException: >>> No enum constant org.wso2.carbon.ml.commons.con >>> stants.MLConstants.SUPERVISED_ALGORITHM.STACKING >>> at java.lang.Enum.valueOf(Enum.java:238) >>> at org.wso2.carbon.ml.commons.constants.MLConstants$SUPERVISED_ >>> ALGORITHM.valueOf(MLConstants.java:225) >>> at org.wso2.carbon.ml.core.spark.algorithms.SupervisedSparkMode >>> lBuilder.build(SupervisedSparkModelBuilder.java:158) >> >> Which approach did you try? putting the jar as a patch (patch)? or >> build the carbon-ml and product ml respectively? If it was patching method, >> did you build org.wso2.carbon.ml.commons as well? >> >> Caused by: org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >>> An error occurred while inserting hyper parameter to the database: Value >>> too long for column "VALUE VARCHAR(50)": "'rO0ABXNyABNqYXZhLnV0aWwuQXJy >>> YXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHACdwQCc3IAEWphdmEu >>> dXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2F... (372)"; SQL statement: >>> INSERT INTO ML_HYPER_PARAMETER(ANALYSIS_ID, ALGORITHM_NAME, NAME, >>> VALUE, LAST_MODIFIED_TIME) VALUES(?,?,?,?, CURRENT_TIMESTAMP()) [90005-140] >>> The Logic is: The hyperparameters of base-algorithms are seri
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Misgana, Can you commit all the changes to your fork of carbon-ml, and share us the link. So that I can check whats happening. Also, do I need to manually edit the machinelearner.xml file(s) to include > Stacking? Yes. No need to add hyper-parameters, since we can't predefine them. Thanks, Supun On Mon, Jun 27, 2016 at 4:33 AM, Misgana Negassi < negas...@tf.uni-freiburg.de> wrote: > Hello Supun, > sorry to bother you again with the same problem but I am not making any > progress right now :( > I can't get the sample script running. > > I tried two approaches: > A) Using the server provided and applying patches. > Problem: I can not apply my patch. I suspect the problem is that my patch > is of a newer version. The server's carbon ml is 1.1.1. and mine is 1.1.2. > Can I somewhere change the version number when I build my project? > > B) Building carbon and then product-ml and then unpacking > modules/distribution/target/wso2ml-2.0.0-SNAPSHOT.zip > I can start the server with the right components except for one exception: > > log4j:ERROR Could not instantiate appender named "CARBON_MEMORY". > log4j:ERROR Could not instantiate class > [org.wso2.carbon.logging.service.appender.CarbonMemoryAppender]. > java.lang.ClassNotFoundException: > org.wso2.carbon.logging.service.appender.CarbonMemoryAppender cannot be > found by org.wso2.carbon.logging_4.4.3 > at > org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:501) > > > However, when I try to run my example script there seems to be nothing > happening on the server side. No exceptions but also the debugger does not > stop when I expect it would. It seems something is crashing but the > exceptions are swallowed. Is there a way to increase the debug output? > > Also, do I need to manually edit the machinelearner.xml file(s) to include > Stacking? > > > On 21.06.2016 05:41, Supun Sethunga wrote: > > Hi Misgana, > > I have two issues: > 1. Issue: > > Exception in thread "pool-17-thread-7" java.lang.IllegalArgumentException: >> No enum constant org.wso2.carbon.ml.commons.con >> stants.MLConstants.SUPERVISED_ALGORITHM.STACKING >> at java.lang.Enum.valueOf(Enum.java:238) >> at org.wso2.carbon.ml.commons.constants.MLConstants$SUPERVISED_ >> ALGORITHM.valueOf(MLConstants.java:225) >> at org.wso2.carbon.ml.core.spark.algorithms.SupervisedSparkMode >> lBuilder.build(SupervisedSparkModelBuilder.java:158) > > Which approach did you try? putting the jar as a patch (patch)? or > build the carbon-ml and product ml respectively? If it was patching method, > did you build org.wso2.carbon.ml.commons as well? > > Caused by: org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: >> An error occurred while inserting hyper parameter to the database: Value >> too long for column "VALUE VARCHAR(50)": "'rO0ABXNyABNqYXZhLnV0aWwuQXJy >> YXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHACdwQCc3IAEWphdmEu >> dXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2F... (372)"; SQL statement: >> INSERT INTO ML_HYPER_PARAMETER(ANALYSIS_ID, ALGORITHM_NAME, NAME, VALUE, >> LAST_MODIFIED_TIME) VALUES(?,?,?,?, CURRENT_TIMESTAMP()) [90005-140] >> The Logic is: The hyperparameters of base-algorithms are serialized and >> passed in the product-ml hyperparamters script file.This is a serialized >> string of List of Maps, where each map contains hyperparameters of each >> base-algorithm and will be deserialized in buildStackingModel() in >> SupervisedSparkModelBuilder and fed to Stacking train(). >> Current idea to solve it is: >> 1. Increase column capacity more than VALUE VARCHAR(50) >> 2. Change serialization logic >> What do you think? > > > I believe we don't need to pass a serialized a map and put the string in > DB. Can we store the hyperparamerets as follows, in the ML_HYPER_PARAMETER > table? > > *ANALYSIS_ID* *ALGORITHM_NAME* *NAME* *VALUE* *LAST_MODIFIED_TIME* > 123 Meta_Algorithm_1 Learning Rate 0.001 > 123 Meta_Algorithm_1 Num_Trees 10 > 123 Meta_Algorithm_1 Max_Depth 0.001 > 123 Meta_Algorithm_1 Seed 4567 > 123 Meta_Algorithm_2 Learning Rate 0.001 > 123 Meta_Algorithm_2 Iterations 100 > 123 Meta_Algorithm_2 Seed 789 > … … … … … > 123 Base_Algorithm Learning Rate 0.001 > 123 Base_Algorithm Iterations 100 > 123 Base_Algorithm Seed 6325 > > Do you see any complications/issues in doing so? > > > > Thanks, > Supun > > On Mon, Jun 20, 2016 at 6:58 PM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> Hi Supun, >> >> I have two issues: >> 1. Issue: >> >> Exception in thread "poo
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Misgana, I have two issues: 1. Issue: Exception in thread "pool-17-thread-7" java.lang.IllegalArgumentException: > No enum constant org.wso2.carbon.ml.commons.con > stants.MLConstants.SUPERVISED_ALGORITHM.STACKING > at java.lang.Enum.valueOf(Enum.java:238) > at org.wso2.carbon.ml.commons.constants.MLConstants$SUPERVISED_ > ALGORITHM.valueOf(MLConstants.java:225) > at org.wso2.carbon.ml.core.spark.algorithms.SupervisedSparkMode > lBuilder.build(SupervisedSparkModelBuilder.java:158) Which approach did you try? putting the jar as a patch (patch)? or build the carbon-ml and product ml respectively? If it was patching method, did you build org.wso2.carbon.ml.commons as well? Caused by: org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: > An error occurred while inserting hyper parameter to the database: Value > too long for column "VALUE VARCHAR(50)": "'rO0ABXNyABNqYXZhLnV0aWwuQXJy > YXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHACdwQCc3IAEWphdmEu > dXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2F... (372)"; SQL statement: > INSERT INTO ML_HYPER_PARAMETER(ANALYSIS_ID, ALGORITHM_NAME, NAME, VALUE, > LAST_MODIFIED_TIME) VALUES(?,?,?,?, CURRENT_TIMESTAMP()) [90005-140] > The Logic is: The hyperparameters of base-algorithms are serialized and > passed in the product-ml hyperparamters script file.This is a serialized > string of List of Maps, where each map contains hyperparameters of each > base-algorithm and will be deserialized in buildStackingModel() in > SupervisedSparkModelBuilder and fed to Stacking train(). > Current idea to solve it is: > 1. Increase column capacity more than VALUE VARCHAR(50) > 2. Change serialization logic > What do you think? I believe we don't need to pass a serialized a map and put the string in DB. Can we store the hyperparamerets as follows, in the ML_HYPER_PARAMETER table? *ANALYSIS_ID* *ALGORITHM_NAME* *NAME* *VALUE* *LAST_MODIFIED_TIME* 123 Meta_Algorithm_1 Learning Rate 0.001 123 Meta_Algorithm_1 Num_Trees 10 123 Meta_Algorithm_1 Max_Depth 0.001 123 Meta_Algorithm_1 Seed 4567 123 Meta_Algorithm_2 Learning Rate 0.001 123 Meta_Algorithm_2 Iterations 100 123 Meta_Algorithm_2 Seed 789 … … … … … 123 Base_Algorithm Learning Rate 0.001 123 Base_Algorithm Iterations 100 123 Base_Algorithm Seed 6325 Do you see any complications/issues in doing so? Thanks, Supun On Mon, Jun 20, 2016 at 6:58 PM, Misgana Negassi < negas...@tf.uni-freiburg.de> wrote: > Hi Supun, > > I have two issues: > 1. Issue: > > Exception in thread "pool-17-thread-7" java.lang.IllegalArgumentException: > No enum constant > org.wso2.carbon.ml.commons.constants.MLConstants.SUPERVISED_ALGORITHM.STACKING > at java.lang.Enum.valueOf(Enum.java:238) > at > org.wso2.carbon.ml.commons.constants.MLConstants$SUPERVISED_ALGORITHM.valueOf(MLConstants.java:225) > at > org.wso2.carbon.ml.core.spark.algorithms.SupervisedSparkModelBuilder.build(SupervisedSparkModelBuilder.java:158) > > > It doesn't recognize STACKING although I have built carbonml as you have > described. > > 2. Issue > > > Caused by: > org.wso2.carbon.ml.database.exceptions.DatabaseHandlerException: An error > occurred while inserting hyper parameter to the database: Value too long > for column "VALUE VARCHAR(50)": > "'rO0ABXNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHACdwQCc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2F... > (372)"; SQL statement: > INSERT INTO ML_HYPER_PARAMETER(ANALYSIS_ID, ALGORITHM_NAME, NAME, VALUE, > LAST_MODIFIED_TIME) VALUES(?,?,?,?, CURRENT_TIMESTAMP()) [90005-140] > > The Logic is: The hyperparameters of base-algorithms are serialized and > passed in the product-ml hyperparamters script file.This is a serialized > string of List of Maps, where each map contains hyperparameters of each > base-algorithm and will be deserialized in buildStackingModel() in > SupervisedSparkModelBuilder and fed to Stacking train(). > > Current idea to solve it is: > 1. Increase column capacity more than VALUE VARCHAR(50) > 2. Change serialization logic > > What do you think? > > Regards, > Misgana > > > > > Can you specify why what was the error? Is there any error/stack trace? >> > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Error while registering UDF method in Spark
Thanks Krishantha, that solved the issue. Regards, Supun On Wed, Jun 15, 2016 at 10:46 AM, Krishantha Samaraweera < krishan...@wso2.com> wrote: > Hi Supun, > > "$jacocoInit" method injection happens at class instrumentation phase > hence excluding the class though filter file may not work. Filter works > only when report generation happens and it exclude all classes included in > filter files from report generation. > > So you need to edit instrumenation.txt file and include classes you want > to instrument specifically. > > Thanks, > Krishantha. > > On Wed, Jun 15, 2016 at 9:10 AM, Supun Sethunga <sup...@wso2.com> wrote: > >> Any update on this? >> >> Believe this gonna be an issue for most of the analytics products (if not >> all). >> >> Regards, >> Supun >> >> On Thu, Jun 9, 2016 at 3:06 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi Dharshana/Sachith, >>> >>> We are observing the same error in analytics-esb server as well. Even >>> though the exception is harmless directly, it causes any Test cases which >>> extend *ServerStartupBaseTest*[1] to be failed (coz of the "ERROR" >>> string in the logs). >>> >>> I tried excluding the UDF class in the filter.txt, but had no luck. >>> Same testcases passes if we disable the coverage. :( >>> >>> Any idea of a solution? >>> >>> [1] >>> https://github.com/wso2/carbon-platform-integration-utils/blob/master/common-framework-tests/src/main/java/org/wso2/carbon/integration/common/tests/ServerStartupBaseTest.java >>> >>> >>> Thanks, >>> Supun >>> >>> On Sun, Jun 5, 2016 at 4:17 PM, Dharshana Warusavitharana < >>> dharsha...@wso2.com> wrote: >>> >>>> Hi Nirmal, >>>> >>>> Yes this is the nature of jacoco in coverage. We are using offline >>>> instrumentation which adds few data in to classes to determine whether they >>>> are loaded in to JVM >>>> >>>> in /src/test/resource directory you have two files called filter.txt >>>> and instrumentation.txt >>>> >>>> You can remove any pattern in filter.txt by >>>> >>>> -org.wso2.carbon.throttle.core.impl.ipbase.* >>>> -org.wso2.uri.template.parser.* >>>> Thank you, >>>> Dharshana. >>>> >>>> On Sun, Jun 5, 2016 at 6:28 AM, Nirmal Fernando <nir...@wso2.com> >>>> wrote: >>>> >>>>> Can we skip certain classes getting this $jacocoInit method? >>>>> >>>>> On Sat, Jun 4, 2016 at 11:38 PM, Sachith Withana <sach...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Fazlan, >>>>>> >>>>>> To add to what Gimantha said, since we register methods, only the >>>>>> jacoco injected method would fail to register. >>>>>> All the other methods would be registered as UDFs. >>>>>> >>>>>> Regards, >>>>>> Sachith >>>>>> >>>>>> On Sat, Jun 4, 2016 at 12:35 PM, Gimantha Bandara <giman...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Fazlan, >>>>>>> >>>>>>> Spark UDFs are the Java methods that we define. Jacoco actually >>>>>>> injects a static method called "$jacocoInit" to every test class, so >>>>>>> every >>>>>>> UDF class will contain an additional method called "$jacocoInit". Spark >>>>>>> cannot determine the datatypes passed in the parameters and registering >>>>>>> "$jacocoInit" as a spark UDF fails. Thats why that error comes, and >>>>>>> since >>>>>>> it is not a method that we define as a spark UDF, it is harmless. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> On Sat, Jun 4, 2016 at 10:12 PM, Nirmal Fernando <nir...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Fazlan, >>>>>>>> >>>>>>>> AFAIK this is an unharmful error and Sachith should know some >>>>>>>> context. >>>>>>>> >>>>>>>> On Sat, Jun 4, 2016 at 10:09 PM, Fazlan Nazeem <fazl...@wso2.com> >>>>>
Re: [Dev] Error while registering UDF method in Spark
Any update on this?
Believe this gonna be an issue for most of the analytics products (if not
all).
Regards,
Supun
On Thu, Jun 9, 2016 at 3:06 PM, Supun Sethunga <sup...@wso2.com> wrote:
> Hi Dharshana/Sachith,
>
> We are observing the same error in analytics-esb server as well. Even
> though the exception is harmless directly, it causes any Test cases which
> extend *ServerStartupBaseTest*[1] to be failed (coz of the "ERROR"
> string in the logs).
>
> I tried excluding the UDF class in the filter.txt, but had no luck. Same
> testcases passes if we disable the coverage. :(
>
> Any idea of a solution?
>
> [1]
> https://github.com/wso2/carbon-platform-integration-utils/blob/master/common-framework-tests/src/main/java/org/wso2/carbon/integration/common/tests/ServerStartupBaseTest.java
>
>
> Thanks,
> Supun
>
> On Sun, Jun 5, 2016 at 4:17 PM, Dharshana Warusavitharana <
> dharsha...@wso2.com> wrote:
>
>> Hi Nirmal,
>>
>> Yes this is the nature of jacoco in coverage. We are using offline
>> instrumentation which adds few data in to classes to determine whether they
>> are loaded in to JVM
>>
>> in /src/test/resource directory you have two files called filter.txt and
>> instrumentation.txt
>>
>> You can remove any pattern in filter.txt by
>>
>> -org.wso2.carbon.throttle.core.impl.ipbase.* -org.wso2.uri.template.parser.*
>> Thank you,
>> Dharshana.
>>
>> On Sun, Jun 5, 2016 at 6:28 AM, Nirmal Fernando <nir...@wso2.com> wrote:
>>
>>> Can we skip certain classes getting this $jacocoInit method?
>>>
>>> On Sat, Jun 4, 2016 at 11:38 PM, Sachith Withana <sach...@wso2.com>
>>> wrote:
>>>
>>>> Hi Fazlan,
>>>>
>>>> To add to what Gimantha said, since we register methods, only the
>>>> jacoco injected method would fail to register.
>>>> All the other methods would be registered as UDFs.
>>>>
>>>> Regards,
>>>> Sachith
>>>>
>>>> On Sat, Jun 4, 2016 at 12:35 PM, Gimantha Bandara <giman...@wso2.com>
>>>> wrote:
>>>>
>>>>> Hi Fazlan,
>>>>>
>>>>> Spark UDFs are the Java methods that we define. Jacoco actually
>>>>> injects a static method called "$jacocoInit" to every test class, so every
>>>>> UDF class will contain an additional method called "$jacocoInit". Spark
>>>>> cannot determine the datatypes passed in the parameters and registering
>>>>> "$jacocoInit" as a spark UDF fails. Thats why that error comes, and since
>>>>> it is not a method that we define as a spark UDF, it is harmless.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> On Sat, Jun 4, 2016 at 10:12 PM, Nirmal Fernando <nir...@wso2.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Fazlan,
>>>>>>
>>>>>> AFAIK this is an unharmful error and Sachith should know some context.
>>>>>>
>>>>>> On Sat, Jun 4, 2016 at 10:09 PM, Fazlan Nazeem <fazl...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We are observing some errors related to UDF registration in API-M
>>>>>>> analytics Test integration phase. The error keeps on printing till the
>>>>>>> Spark Client is started. This error is only present during the test
>>>>>>> phase.
>>>>>>> It does not occur when the server is started separately. DAS team, Any
>>>>>>> idea
>>>>>>> on how we could get rid of this? Jenkins log in [1].
>>>>>>>
>>>>>>> [2016-06-04 02:23:32,998] ERROR
>>>>>>> {org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor}
>>>>>>> - Error while registering the UDF method: $jacocoInit, Cannot
>>>>>>> determine the return DataType: class [Z
>>>>>>>
>>>>>>> INFO
>>>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] -
>>>>>>> org.wso2.carbon.analytics.spark.core.exception.AnalyticsUDFException:
>>>>>>> Cannot determine the return DataType: class [Z
>>>>>>> INFO
>>>>>>> [org.wso2.carbon.automation.extensions.servers
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
on whole dataset and store > list base models. > >For test() method, we create a level1_test_dataset by combining > predictions of basemodels (trained on the whole dataset) on the > level0_test_dataset. Finally, we get final predictions invoking Predictor > predict() method ona meta-algorithm using level1_test_dataset. > The output will be of form JavaPairRDD<Double, Double> PredictionAndLabels. > > > 3. Refactored Methods/Classes: > > MLCONSTANTS:[3] Added hyperparameters for Stacking > Util:[4] Class with helper methods > BaseModelBuilder: for building models (*will commit once done*) > > [1] > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/SupervisedSparkModelBuilder.java#L829 > [2] > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/Stacking.java > [3] > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.commons/src/main/java/org/wso2/carbon/ml/commons/constants/MLConstants.java > [4] > https://github.com/zemoel/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/utils/Util.java > > On 13.06.2016 05:46, Supun Sethunga wrote: > > Hi Misgana, > > How is the work going so far? Can you give a brief update on the current > status, and what is left to be done? > > Thanks, > Supun > > On Wed, Jun 8, 2016 at 5:54 PM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> Hi Supun, >> >> Thank you for your suggestions. I would like to abstract from the >> concrete the level0 models and concrete level1 model and needed a suitable >> interface for that. MLModel seemed a good choice also in perspective of >> later integration of stacking. >> >> From my point of view the native spark models are not suitable because >> they don't share a suitable interface like MLModel. My approach is to use >> those native spark models and convert them to carbonml type models using >> eg. MLRandomForest.setModel() for RandomForestModel. >> >> Also now I found a way to use the Predictor without a configuration >> context by passing an empty encoding. >> >> I will put extra effort to meet the deadline and create code which is >> scalable to later change/integration. I apologize for any miscommunication >> that may arise from my side. >> >> Best regards, >> Misgana >> >> On 08.06.2016 05:35, Supun Sethunga wrote: >> >> Hi, >> >> Also just a gentle reminder.., we have just under two weeks for the >> mid-term evaluation. We need to have some end to end working scenario of >> stacking by then. So lets put some extra effort and try to complete one >> scenario. >> >> Thanks, >> Supun >> >> On Tue, Jun 7, 2016 at 10:48 PM, Supun Sethunga < <sup...@wso2.com> >> sup...@wso2.com> wrote: >> >>> Hi, >>> >>> I have restructured my code, see [1][2]. It is a naive implementation if >>>> my logic works. >>> >>> Why there are two files/implementations? which is the correct one? Say, >>> if I want to try out your implementation, which one should I run? >>> >>> When creating an instance of Stacking class, I need to pass an argument >>>> of type MLModelConfigurationContext. and getcontext() method doesn't work >>>> as I am in a static main method. >>>> Question: 1. What is the best way to get context argument? >>>> 2. Is the design of Stacking class fine? >>> >>> >>> Why would you need to use MLModelConfigurationContext. and getcontext() >>> methods? Those are utility methods used at the ML server runtime to >>> temporary store configurations. Let's not worry about those, as they are >>> part of the integration phase. First, try to implement the stacking with >>> native spark-mllib libraries, and re-use methods/components in ML server >>> *ONLY >>> IF * they are necessary or they are re-usable (i.e: if some method is >>> already available, which you need). >>> >>> Regards, >>> Supun >>> >>> On Tue, Jun 7, 2016 at 8:31 PM, Misgana Negassi < >>> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >>> >>>> >>>> Hi Supun, >>>> >>>> My dependencies problems are solved, thanks! >>>> >>>> I have restructured my code
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Misgana, How is the work going so far? Can you give a brief update on the current status, and what is left to be done? Thanks, Supun On Wed, Jun 8, 2016 at 5:54 PM, Misgana Negassi <negas...@tf.uni-freiburg.de > wrote: > Hi Supun, > > Thank you for your suggestions. I would like to abstract from the concrete > the level0 models and concrete level1 model and needed a suitable interface > for that. MLModel seemed a good choice also in perspective of later > integration of stacking. > > From my point of view the native spark models are not suitable because > they don't share a suitable interface like MLModel. My approach is to use > those native spark models and convert them to carbonml type models using > eg. MLRandomForest.setModel() for RandomForestModel. > > Also now I found a way to use the Predictor without a configuration > context by passing an empty encoding. > > I will put extra effort to meet the deadline and create code which is > scalable to later change/integration. I apologize for any miscommunication > that may arise from my side. > > Best regards, > Misgana > > On 08.06.2016 05:35, Supun Sethunga wrote: > > Hi, > > Also just a gentle reminder.., we have just under two weeks for the > mid-term evaluation. We need to have some end to end working scenario of > stacking by then. So lets put some extra effort and try to complete one > scenario. > > Thanks, > Supun > > On Tue, Jun 7, 2016 at 10:48 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi, >> >> I have restructured my code, see [1][2]. It is a naive implementation if >>> my logic works. >> >> Why there are two files/implementations? which is the correct one? Say, >> if I want to try out your implementation, which one should I run? >> >> When creating an instance of Stacking class, I need to pass an argument >>> of type MLModelConfigurationContext. and getcontext() method doesn't work >>> as I am in a static main method. >>> Question: 1. What is the best way to get context argument? >>> 2. Is the design of Stacking class fine? >> >> >> Why would you need to use MLModelConfigurationContext. and getcontext() >> methods? Those are utility methods used at the ML server runtime to >> temporary store configurations. Let's not worry about those, as they are >> part of the integration phase. First, try to implement the stacking with >> native spark-mllib libraries, and re-use methods/components in ML server >> *ONLY >> IF * they are necessary or they are re-usable (i.e: if some method is >> already available, which you need). >> >> Regards, >> Supun >> >> On Tue, Jun 7, 2016 at 8:31 PM, Misgana Negassi < >> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >> >>> >>> Hi Supun, >>> >>> My dependencies problems are solved, thanks! >>> >>> I have restructured my code, see [1][2]. It is a naive implementation if >>> my logic works. >>> >>> >>> When creating an instance of Stacking class, I need to pass an argument >>> of type MLModelConfigurationContext. and getcontext() method doesn't work >>> as I am in a static main method. >>> Question: 1. What is the best way to get context argument? >>> 2. Is the design of Stacking class fine? >>> >>> I appreciate your feedback! >>> Misgana >>> >>> >>> >>> >>> >>> [1] >>> https://github.com/zemoel/ensemble-methods/blob/master/src/main/java/Stacking.java >>> [2] >>> https://github.com/zemoel/ensemble-methods/blob/master/src/main/java/ReadCSV.java#L188 >>> >>> >>> On 04.06.2016 06:18, Supun Sethunga wrote: >>> >>> Hi, >>> >>> Can you check whether you have defined the relevant repositories in the >>> pom.xml? If haven't, please do so as in [1]. >>> >>> If that didn't work out, can you try checking out the source code of >>> [2], and build it locally, and then build your code? >>> >>> [1] https://github.com/wso2/carbon-ml/blob/master/pom.xml#L59 >>> [2] >>> https://github.com/wso2/carbon-metrics/tree/v1.1.0/components/org.wso2.carbon.metrics.manager >>> >>> Regards, >>> >>> On Fri, Jun 3, 2016 at 7:06 PM, Misgana Negassi < >>> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >>> >>>> Hi Supun, >>>> >>>> This is [1] the predict() method invoked. >>&
Re: [Dev] Error while registering UDF method in Spark
>> org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:819) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:771) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:214) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:433) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:451) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerService(BundleContextImpl.java:950) >>>>>> INFO >>>>>> [org.wso2.carbon.automation.extensions.servers.utils.ServerLogReader] - >>>>>> at >>>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceComponent.activate(Analytics >>>>>> >>>>>> >>>>>> [1] https://wso2.org/jenkins/job/analytics-apim/136/consoleText >>>>>> -- >>>>>> Thanks & Regards, >>>>>> >>>>>> Fazlan Nazeem >>>>>> >>>>>> *Software Engineer* >>>>>> >>>>>> *WSO2 Inc* >>>>>> Mobile : +94772338839 >>>>>> <%2B94%20%280%29%20773%20451194> >>>>>> fazl...@wso2.com >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Thanks & regards, >>>>> Nirmal >>>>> >>>>> Team Lead - WSO2 Machine Learner >>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>> Mobile: +94715779733 >>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>> >>>>> >>>>> >>>>> ___ >>>>> Dev mailing list >>>>> Dev@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Gimantha Bandara >>>> Software Engineer >>>> WSO2. Inc : http://wso2.com >>>> Mobile : +94714961919 >>>> >>> >>> >>> >>> -- >>> Sachith Withana >>> Software Engineer; WSO2 Inc.; http://wso2.com >>> E-mail: sachith AT wso2.com >>> M: +94715518127 >>> Linked-In: <http://goog_416592669> >>> https://lk.linkedin.com/in/sachithwithana >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > > Dharshana Warusavitharana > Senior Software Engineer , Test Automation > WSO2 Inc. http://wso2.com > email : dharsha...@wso2.com <dharsha...@wso2.com> > Tel : +94 11 214 5345 > Fax :+94 11 2145300 > cell : +94770342233 > blog : http://dharshanaw.blogspot.com > > lean . enterprise . middleware > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSoC 2016]Proposal 03: [ML]Visual(Deep) Neural Network Builder -Project Progress
rading this? >>>>>>> >>>>>>> Thank you. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jun 1, 2016 at 2:19 PM, Upul Bandara <u...@wso2.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Good progress. >>>>>>>> >>>>>>>> The key parts of the project are designing the front-end of NN >>>>>>>> builder, communicating between the front-end and the Deeplearning4J >>>>>>>> back-end. >>>>>>>> So at this stage of the project, it is better to put your effort on >>>>>>>> above the components. Later, we can integrate what you have built with >>>>>>>> WSO2 >>>>>>>> Machine Learning server. >>>>>>>> >>>>>>>> Once you have completed a basic NN builder (ability to build a >>>>>>>> simple feedforward is enough) we would like to have a quick demo. >>>>>>>> >>>>>>>> Let me know if you have further questions or clarifications. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Upul >>>>>>>> >>>>>>>> On Wed, Jun 1, 2016 at 10:42 AM, Lakini Senanayaka < >>>>>>>> lakinisenanayak...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I'm sorry for the late response.Thank you very much for the last >>>>>>>>> email.I have gone through JQueryUI and I have implemented some samples >>>>>>>>> using JQueryUI.I have gone through the WSO2 machine learner and I >>>>>>>>> have >>>>>>>>> identified the place to insert this jaggery page(replace the >>>>>>>>> hyper-parameter page in the ML UI). >>>>>>>>> Currently I'm designing the sketch of the Deep Neural Network >>>>>>>>> Builder-the front end. >>>>>>>>> I'll send my sketch before tonight.Based on your comments I can >>>>>>>>> start implementations. >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jun 1, 2016 at 9:29 AM, Supun Sethunga <sup...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Lakini, >>>>>>>>>> >>>>>>>>>> Any update on the progress? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Supun >>>>>>>>>> >>>>>>>>>> On Fri, May 27, 2016 at 5:34 PM, Supun Sethunga <sup...@wso2.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Lakini, >>>>>>>>>>> >>>>>>>>>>> Sorry for the delayed response. As the first part of the >>>>>>>>>>> project, you can start the work on the UI/drag and drop feature, as >>>>>>>>>>> the >>>>>>>>>>> core of your project is based around that. Therefore, shall we try >>>>>>>>>>> to get a >>>>>>>>>>> basic version of drag and drop UI, by the mid-term evaluation? (No >>>>>>>>>>> need to >>>>>>>>>>> connect the UI with the dl4j, for the first phase) So this would be >>>>>>>>>>> include: >>>>>>>>>>> >>>>>>>>>>>- A link/re-direct to go to Visual builder, upon selecting >>>>>>>>>>>the Neural Netowrks (as the algorithm) >>>>>>>>>>>- The drawing/dropping area and a panel to pick the objects >>>>>>>>>>>to be dragged (objects as in, nodes, layers, links, etc) >>>>>>>>>>> >>>>>>>>>>> This drag and drop page should replace the hyper-parameter page >
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi, Also just a gentle reminder.., we have just under two weeks for the mid-term evaluation. We need to have some end to end working scenario of stacking by then. So lets put some extra effort and try to complete one scenario. Thanks, Supun On Tue, Jun 7, 2016 at 10:48 PM, Supun Sethunga <sup...@wso2.com> wrote: > Hi, > > I have restructured my code, see [1][2]. It is a naive implementation if >> my logic works. > > Why there are two files/implementations? which is the correct one? Say, if > I want to try out your implementation, which one should I run? > > When creating an instance of Stacking class, I need to pass an argument of >> type MLModelConfigurationContext. and getcontext() method doesn't work as I >> am in a static main method. >> Question: 1. What is the best way to get context argument? >> 2. Is the design of Stacking class fine? > > > Why would you need to use MLModelConfigurationContext. and getcontext() > methods? Those are utility methods used at the ML server runtime to > temporary store configurations. Let's not worry about those, as they are > part of the integration phase. First, try to implement the stacking with > native spark-mllib libraries, and re-use methods/components in ML server *ONLY > IF * they are necessary or they are re-usable (i.e: if some method is > already available, which you need). > > Regards, > Supun > > On Tue, Jun 7, 2016 at 8:31 PM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> >> Hi Supun, >> >> My dependencies problems are solved, thanks! >> >> I have restructured my code, see [1][2]. It is a naive implementation if >> my logic works. >> >> >> When creating an instance of Stacking class, I need to pass an argument >> of type MLModelConfigurationContext. and getcontext() method doesn't work >> as I am in a static main method. >> Question: 1. What is the best way to get context argument? >> 2. Is the design of Stacking class fine? >> >> I appreciate your feedback! >> Misgana >> >> >> >> >> >> [1] >> https://github.com/zemoel/ensemble-methods/blob/master/src/main/java/Stacking.java >> [2] >> https://github.com/zemoel/ensemble-methods/blob/master/src/main/java/ReadCSV.java#L188 >> >> >> On 04.06.2016 06:18, Supun Sethunga wrote: >> >> Hi, >> >> Can you check whether you have defined the relevant repositories in the >> pom.xml? If haven't, please do so as in [1]. >> >> If that didn't work out, can you try checking out the source code of [2], >> and build it locally, and then build your code? >> >> [1] https://github.com/wso2/carbon-ml/blob/master/pom.xml#L59 >> [2] >> https://github.com/wso2/carbon-metrics/tree/v1.1.0/components/org.wso2.carbon.metrics.manager >> >> Regards, >> >> On Fri, Jun 3, 2016 at 7:06 PM, Misgana Negassi < >> negas...@tf.uni-freiburg.de> wrote: >> >>> Hi Supun, >>> >>> This is [1] the predict() method invoked. >>> >>> I added the dependencies and run mvn clean install and am trying to >>> debug this error now: >>> >>> >>> >>> *Failed to execute goal on project ensemble-methods: Could not resolve >>> dependencies for project >>> org.wso2.carbon.ml:ensemble-methods:jar:1.0-SNAPSHOT: The following >>> artifacts could not be resolved: >>> org.wso2.carbon.metrics:org.wso2.carbon.metrics.manager:jar:1.1.0 *Best >>> >>> >>> >>> >>> [1] >>> https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/impl/Predictor.java#L80 >>> >>> On 03.06.2016 14:11, Supun Sethunga wrote: >>> >>> Hi, >>> >>> Can you specify which predict() method, to be precise? >>> >>> Anyway, looking at the error trace, this seems to be a version mismatch >>> for the dependency "org.wso2.carbon.metrics ". Can you try adding v1.1.0 >>> ? >>> i.e: org.wso2.carbon.metrics >>> org.wso2.carbon.metrics.manager >>> 1.1.0 >>> >>> Regards, >>> >>> On Fri, Jun 3, 2016 at 5:21 PM, Misgana Negassi < >>> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >>> >>>> Hi Supun, >>>> I get this error when calling predict() method. >>>> >>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>> org/wso2/carbon/metric
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi, I have restructured my code, see [1][2]. It is a naive implementation if my > logic works. Why there are two files/implementations? which is the correct one? Say, if I want to try out your implementation, which one should I run? When creating an instance of Stacking class, I need to pass an argument of > type MLModelConfigurationContext. and getcontext() method doesn't work as I > am in a static main method. > Question: 1. What is the best way to get context argument? > 2. Is the design of Stacking class fine? Why would you need to use MLModelConfigurationContext. and getcontext() methods? Those are utility methods used at the ML server runtime to temporary store configurations. Let's not worry about those, as they are part of the integration phase. First, try to implement the stacking with native spark-mllib libraries, and re-use methods/components in ML server *ONLY IF * they are necessary or they are re-usable (i.e: if some method is already available, which you need). Regards, Supun On Tue, Jun 7, 2016 at 8:31 PM, Misgana Negassi <negas...@tf.uni-freiburg.de > wrote: > > Hi Supun, > > My dependencies problems are solved, thanks! > > I have restructured my code, see [1][2]. It is a naive implementation if > my logic works. > > > When creating an instance of Stacking class, I need to pass an argument of > type MLModelConfigurationContext. and getcontext() method doesn't work as I > am in a static main method. > Question: 1. What is the best way to get context argument? > 2. Is the design of Stacking class fine? > > I appreciate your feedback! > Misgana > > > > > > [1] > https://github.com/zemoel/ensemble-methods/blob/master/src/main/java/Stacking.java > [2] > https://github.com/zemoel/ensemble-methods/blob/master/src/main/java/ReadCSV.java#L188 > > > On 04.06.2016 06:18, Supun Sethunga wrote: > > Hi, > > Can you check whether you have defined the relevant repositories in the > pom.xml? If haven't, please do so as in [1]. > > If that didn't work out, can you try checking out the source code of [2], > and build it locally, and then build your code? > > [1] https://github.com/wso2/carbon-ml/blob/master/pom.xml#L59 > [2] > https://github.com/wso2/carbon-metrics/tree/v1.1.0/components/org.wso2.carbon.metrics.manager > > Regards, > > On Fri, Jun 3, 2016 at 7:06 PM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> Hi Supun, >> >> This is [1] the predict() method invoked. >> >> I added the dependencies and run mvn clean install and am trying to debug >> this error now: >> >> >> >> *Failed to execute goal on project ensemble-methods: Could not resolve >> dependencies for project >> org.wso2.carbon.ml:ensemble-methods:jar:1.0-SNAPSHOT: The following >> artifacts could not be resolved: >> org.wso2.carbon.metrics:org.wso2.carbon.metrics.manager:jar:1.1.0 *Best >> >> >> >> >> [1] >> https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/impl/Predictor.java#L80 >> >> On 03.06.2016 14:11, Supun Sethunga wrote: >> >> Hi, >> >> Can you specify which predict() method, to be precise? >> >> Anyway, looking at the error trace, this seems to be a version mismatch >> for the dependency "org.wso2.carbon.metrics ". Can you try adding v1.1.0? >> i.e: org.wso2.carbon.metrics >> org.wso2.carbon.metrics.manager >> 1.1.0 >> >> Regards, >> >> On Fri, Jun 3, 2016 at 5:21 PM, Misgana Negassi < >> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >> >>> Hi Supun, >>> I get this error when calling predict() method. >>> >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/wso2/carbon/metrics/manager/Level >>> at >>> org.wso2.carbon.ml.core.impl.Predictor.getTimer(Predictor.java:301) >>> at org.wso2.carbon.ml.core.impl.Predictor.predict(Predictor.java:83) >>> at testing.main(testing.java:27) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at >>> com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) >>> Caused by: java.lang.ClassNotFoundException: >>> org.wso2.carbon.metrics.manager.Le
Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner
Thanks Mahesh! The graphs look promising! :) So by looking at graph, LR with SGD can train a model within 60 secs (6*10^10 nano sec), using about 900,000 data points . Means, this online training can handle events/data points coming at rate of 15,000 per second (or more) , if the batch size is set to 900,000 (or less) or window size is set to 60 secs (or less). This is great IMO! On Sat, Jun 4, 2016 at 10:51 AM, Mahesh Dananjaya <dananjayamah...@gmail.com > wrote: > Hi Maheshakya, > As you requested i can change other parameters as well such as feature > size(p). Initially i did it with p=3;sure thing. Anyway you can see and run > the code if you want. source is at [1]. the test timing is called with > random data as you requested if you set args[0] to 1. And you can find the > extension and streaming algorithms in gsoc/ directiry[2]. thank you. > BR, > Mahesh. > [1] > https://github.com/dananjayamahesh/GSOC2016/blob/master/spark-examples/first-example/src/main/java/org/sparkexample/StreamingLinearRegression.java > [2] https://github.com/dananjayamahesh/GSOC2016/tree/master/gsoc > > On Sat, Jun 4, 2016 at 10:39 AM, Mahesh Dananjaya < > dananjayamah...@gmail.com> wrote: > >> Hi supun, >> Though i pushed it yesterday, there was some problems with the network. >> now you can see them in the repo location [1].I added some Matlab plot you >> can see the patter there.you can use ml also. Ok sure thing. I can prepare >> a report or else blog if you want. files are as follows. The y axis is in >> ns and x axis is in batch size. And also i added two pplots as jpegs[2], so >> you can easily compare. >> lr_timing_1000.txt -> batch size incremented by 1000 >> lr_timing_1.txt -> batch size incremented by 1 >> lr_timing_power10.txt -> batch size incremented by power of 10 >> >> In here independent variable is only tha batch size.If you want i can >> send you making other parameters such as step size, number of iteration, >> feature vector size as independent variables. please let me know if you >> want further info. thank you. >> regards, >> Mahesh. >> >> >> [1 >> ]https://github.com/dananjayamahesh/GSOC2016/tree/master/spark-examples/first-example/output >> [2] >> https://github.com/dananjayamahesh/GSOC2016/blob/master/spark-examples/first-example/output/lr_timing_1.jpg >> >> On Sat, Jun 4, 2016 at 9:58 AM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi Mahesh, >>> >>> I have added those timing reports to my repo [1]. >>> >>> Whats the file name? :) >>> >>> Btw, can you compile simple doc (gdoc) with the above results, and bring >>> everything to one place? That way it is easy to compare, and keep track. >>> >>> Thanks, >>> Supun >>> >>> On Fri, Jun 3, 2016 at 7:23 PM, Mahesh Dananjaya < >>> dananjayamah...@gmail.com> wrote: >>> >>>> Hi Maheshkya, >>>> I have added those timing reports to my repo [1].please have a look at. >>>> three files are there. one is using incremet as 1000 for batch sizes >>>> (lr_timing_1000). Otherone is using incremet by 1 (lr_timing_1) >>>> upto 1 million in both scenarios.you can see the reports and figures in the >>>> location [2] in the repo. i also added the streaminglinearregression >>>> classes in the repo gsoc folder.thank you. >>>> regards, >>>> Mahesh. >>>> [1]https://github.com/dananjayamahesh/GSOC2016 >>>> [2] >>>> https://github.com/dananjayamahesh/GSOC2016/tree/master/spark-examples/first-example/output >>>> >>>> On Mon, May 30, 2016 at 9:24 AM, Maheshakya Wijewardena < >>>> mahesha...@wso2.com> wrote: >>>> >>>>> Hi Mahesh, >>>>> >>>>> Thank you for the update. I will look into your implementation. >>>>> >>>>> And i will be able to send you the timing/performances analysis report >>>>>> tomorrow for the SGD functions >>>>>> >>>>> >>>>> Great. Sent those asap so that we can proceed. >>>>> >>>>> Best regards. >>>>> >>>>> On Sun, May 29, 2016 at 6:56 PM, Mahesh Dananjaya < >>>>> dananjayamah...@gmail.com> wrote: >>>>> >>>>>> >>>>>> Hi maheshakay, >>>>>> I have implemented the linear regression with cep siddhi event stream >>>>>> with taking batch sizes as parameters from the cep. Now we can tr
Re: [Dev] Fwd: GSOC2016: [ML][CEP] Predictive analytic with online data for WSO2 Machine Learner
gt;>>>>>>>>>>>>>>>>>>>>> proposal in plain Java. It's better if you can create a >>>>>>>>>>>>>>>>>>>>>> git repo and share >>>>>>>>>>>>>>>>>>>>>> your code once you have made some progress. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Were you able debug and understand the flow of the ML >>>>>>>>>>>>>>>>>>>>>> siddhi extension? I hope you haven't encountered more >>>>>>>>>>>>>>>>>>>>>> errors after >>>>>>>>>>>>>>>>>>>>>> switching the released version of CEP. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Is this Friday okay for you? Afternoon at 2:00 pm? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Best regards. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>> http://spark.apache.org/docs/latest/programming-guide.html >>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>> http://spark.apache.org/docs/latest/mllib-guide.html >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Wed, May 4, 2016 at 1:07 PM, Mahesh Dananjaya < >>>>>>>>>>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>&
Re: [Dev] [GSoC 2016]Proposal 03: [ML]Visual(Deep) Neural Network Builder -Project Progress
Hi Lakini, Any update on the progress? Regards, Supun On Fri, May 27, 2016 at 5:34 PM, Supun Sethunga <sup...@wso2.com> wrote: > Hi Lakini, > > Sorry for the delayed response. As the first part of the project, you can > start the work on the UI/drag and drop feature, as the core of your project > is based around that. Therefore, shall we try to get a basic version of > drag and drop UI, by the mid-term evaluation? (No need to connect the UI > with the dl4j, for the first phase) So this would be include: > >- A link/re-direct to go to Visual builder, upon selecting the Neural >Netowrks (as the algorithm) >- The drawing/dropping area and a panel to pick the objects to be >dragged (objects as in, nodes, layers, links, etc) > > This drag and drop page should replace the hyper-parameter page in the ML > UI. More precisely, In the Machine Learner wizard, when a user picks the > algorithm name as Neural Network, and proceed, this drag and drop page > should be prompted instead of the hyper-parameters page. You can create the > overall page as a jaggery page (.jag). It would be easier for you to get a > copy of an existing page, and modifying the content. That will preserve the > existing styles, session handling etc. > > Please feel free to raise any question you come across during implementing. > > Regards, > Supun > > On Tue, May 24, 2016 at 8:09 PM, Lakini Senanayaka < > lakinisenanayak...@gmail.com> wrote: > >> Hi, >> >> Thank you very much for the suggestions.I will be using JQueryUI. >> >> As my coding period started yesterday according to the GSoC Schedule I >> would like to know the workload that should be completed >> before the midterm evaluation. Please advise me accordingly. >> >> Thank you. >> >> >> On Tue, May 24, 2016 at 10:34 AM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi Lakini, >>> >>> Yes you can use any of them, with open source licence. But I would >>> prefer JQueryUI, as its a well known library, and is already been used by >>> some of the wso2 products. >>> >>> Regards, >>> Supun >>> >>> On Tue, May 24, 2016 at 9:29 AM, Lakini Senanayaka < >>> lakinisenanayak...@gmail.com> wrote: >>> >>>> Hi, >>>> >>>> I have found some libraries to build drag and drop UI for our front end >>>> .They are JQury UI[1],Dragula[2],Draggablily[3]. >>>> I hope I can do the developments using these libraries. >>>> >>>> [1]jqueryui <http://jqueryui.com/> >>>> [2]dragula <https://bevacqua.github.io/dragula/> >>>> [3]draggabilly <http://draggabilly.desandro.com/> >>>> >>>> Could you please tell me if you have any other suggestions? >>>> >>>> On Sat, May 21, 2016 at 11:05 PM, Lakini Senanayaka < >>>> lakinisenanayak...@gmail.com> wrote: >>>> >>>>> Hi , >>>>> >>>>> This is my weekly progress update of my project. >>>>> >>>>> I have solved all the problems I have mentioned in the last email.I >>>>> could find a class BaseDatasetIterator which is in DL4J where we can find >>>>> inbuilt iterators for CurvesDataSetIterator, IrisDataSetIterator, >>>>> MnistDataSetIterator, MovingWindowBaseDataSetIterator, >>>>> RawMnistDataSetIterator .There is no iterator for CIFAR dataset yet. >>>>> >>>>> I have gone through the whole documentation except Deeplearning4j on >>>>> Spark.The latest documentation is very understandable than the earlier >>>>> one.I have studied neural networks- Restricted Boltzmann Machines, >>>>> Convolutional Nets (ConvNets), >>>>> Long Short-Term Memory Units (LSTMs), >>>>> Denoising Autoencoders, >>>>> Recurrent Nets and LSTMs, >>>>> Multilayer Neural Nets, >>>>> Deep-Belief Network, >>>>> Deep AutoEncoder, >>>>> Stacked Denoising Autoencoders >>>>> >>>>> I have run the example codes[1] and I have understood the >>>>> implementations.I have rerun the codes with modifying different parameter >>>>> values like number and size of the hidden layers, learning rate, >>>>> momentum, >>>>> weight distribution and various types of regularization and checked the &g
Re: [Dev] [GSoC 2016]Proposal 03: [ML]Visual(Deep) Neural Network Builder -Project Progress
Hi Lakini, Sorry for the delayed response. As the first part of the project, you can start the work on the UI/drag and drop feature, as the core of your project is based around that. Therefore, shall we try to get a basic version of drag and drop UI, by the mid-term evaluation? (No need to connect the UI with the dl4j, for the first phase) So this would be include: - A link/re-direct to go to Visual builder, upon selecting the Neural Netowrks (as the algorithm) - The drawing/dropping area and a panel to pick the objects to be dragged (objects as in, nodes, layers, links, etc) This drag and drop page should replace the hyper-parameter page in the ML UI. More precisely, In the Machine Learner wizard, when a user picks the algorithm name as Neural Network, and proceed, this drag and drop page should be prompted instead of the hyper-parameters page. You can create the overall page as a jaggery page (.jag). It would be easier for you to get a copy of an existing page, and modifying the content. That will preserve the existing styles, session handling etc. Please feel free to raise any question you come across during implementing. Regards, Supun On Tue, May 24, 2016 at 8:09 PM, Lakini Senanayaka < lakinisenanayak...@gmail.com> wrote: > Hi, > > Thank you very much for the suggestions.I will be using JQueryUI. > > As my coding period started yesterday according to the GSoC Schedule I > would like to know the workload that should be completed > before the midterm evaluation. Please advise me accordingly. > > Thank you. > > > On Tue, May 24, 2016 at 10:34 AM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Lakini, >> >> Yes you can use any of them, with open source licence. But I would prefer >> JQueryUI, as its a well known library, and is already been used by some of >> the wso2 products. >> >> Regards, >> Supun >> >> On Tue, May 24, 2016 at 9:29 AM, Lakini Senanayaka < >> lakinisenanayak...@gmail.com> wrote: >> >>> Hi, >>> >>> I have found some libraries to build drag and drop UI for our front end >>> .They are JQury UI[1],Dragula[2],Draggablily[3]. >>> I hope I can do the developments using these libraries. >>> >>> [1]jqueryui <http://jqueryui.com/> >>> [2]dragula <https://bevacqua.github.io/dragula/> >>> [3]draggabilly <http://draggabilly.desandro.com/> >>> >>> Could you please tell me if you have any other suggestions? >>> >>> On Sat, May 21, 2016 at 11:05 PM, Lakini Senanayaka < >>> lakinisenanayak...@gmail.com> wrote: >>> >>>> Hi , >>>> >>>> This is my weekly progress update of my project. >>>> >>>> I have solved all the problems I have mentioned in the last email.I >>>> could find a class BaseDatasetIterator which is in DL4J where we can find >>>> inbuilt iterators for CurvesDataSetIterator, IrisDataSetIterator, >>>> MnistDataSetIterator, MovingWindowBaseDataSetIterator, >>>> RawMnistDataSetIterator .There is no iterator for CIFAR dataset yet. >>>> >>>> I have gone through the whole documentation except Deeplearning4j on >>>> Spark.The latest documentation is very understandable than the earlier >>>> one.I have studied neural networks- Restricted Boltzmann Machines, >>>> Convolutional Nets (ConvNets), >>>> Long Short-Term Memory Units (LSTMs), >>>> Denoising Autoencoders, >>>> Recurrent Nets and LSTMs, >>>> Multilayer Neural Nets, >>>> Deep-Belief Network, >>>> Deep AutoEncoder, >>>> Stacked Denoising Autoencoders >>>> >>>> I have run the example codes[1] and I have understood the >>>> implementations.I have rerun the codes with modifying different parameter >>>> values like number and size of the hidden layers, learning rate, momentum, >>>> weight distribution and various types of regularization and checked the >>>> performance. >>>> >>>> I have implemented a convolution net[3] and trained it using LFW >>>> dataset[2] and a Recurrent net[4].Still I have a problem in vectorizing >>>> CIFRA-10 data set. >>>> >>>> Currently I am researching about libraries to build drag & drop UI >>>> for front end. >>>> >>>> [1]dl4j-0.4-examples >>>> <https://github.com/deeplearning4j/dl4j-0.4-examples> >>>> [2]LFW Face Dataset &l
Re: [Dev] [ml][cep][gsoc-6] Initial Project Meeting-Predictive Analysis with online data
Hi Mahesh, Before you start on the siddhi extension, can you also do a small evaluation on the rate that each of those SGD algorithms train models, and share the results? (i.e: how long does it take to train a model, for different sample sizes.). AFAIU, the size of the 'K' or the window length (or the worst case, whether to train as streaming or as batch), is strictly determined by the above results. Regards, Supun On Wed, May 25, 2016 at 7:05 PM, Mahesh Dananjaya <dananjayamah...@gmail.com > wrote: > Hi Maheshakya, > this is the one i need to post.please check. > > Today we had a initial project meeting with wso2 ml team. what we > discussed architecture, best approaches and scope of the entire project. As > i underestood there will be three main component in the design. > 1. CEP Siddhi Extension for the CEP streaming data interface > 2. Core with Sample data points, recently saved model > 3. Apache Spark MLLib algorithm wrapper > > As long as we need to facilitate the streaming data support for the cep > siddhi, what i need initially is to developed a siddhi extension to get > events streams into my program. As we discussed the best approach is the > CEP Siddhi extension with Stream Processor extension to get cep event > streams into the program to incrementally learn the model to predictions > and analysis [1]. Extension will be the interface for cep to send data to > program. There can be different interface for different applications to > use the program. This stream data from cep is taken as a batches rather > than single events.And also there will be outpur from my program to cep > which are the recent model information and parameters such as MSE, Cluster > center etc. After that there can be two approaches that we have not > finalized. > > 1. Collect K-Size batch from incoming data and learn model with that mini > batch and store the model.In this case memory requirement depends on K and > the number of features of the event. But this way we can achieve high level > of streaming perspectives such as data obsolesces and data horizon while > keeping relevant data while removing irrelevant data from the model > training. > 2. Collect data into large memory and use moving window of K size n shift. > Where the n is the number if points that the window is moved after one > calculation. In that case we need a large memory. > > Another approach that raised in that store the events/data points in a > database and use them later. As we discussed there can be two approaches to > send the updated/learned model into customer side. Time based and size > based approach. In that case there can be a time window (one day, one > week, etc.) or a batch size (or both) in the K-size batch approach. > > Then the other component, the wrapper around the spakl mllib SGD based > algorithms to for incremental learning. As i realized there will be memory > constraints and other constraints when we incrementally learn models with > stream data coming out of the cep, basically from the machine that cep is > deployed. Therefore we need to look into timing and performance while we > are using those algorithms on large data sets over time frequently. > Initially what we supposed to do is that develop that extension for > cep/siddhi to get stream data/events/sample points. after that we can move > with for mllib libs. Now we have three algorithms, linear regression, > k-mean clustering and logistic regression though we intially look only into > fisrt two.so this wees will be spent to develop that extension. thank you. > > regards, > Mahesh. > > [1] > https://docs.wso2.com/display/CEP400/Writing+a+Custom+Stream+Processor+Extension > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSoC 2016]Proposal 03: [ML]Visual(Deep) Neural Network Builder -Project Progress
Hi Lakini, Yes you can use any of them, with open source licence. But I would prefer JQueryUI, as its a well known library, and is already been used by some of the wso2 products. Regards, Supun On Tue, May 24, 2016 at 9:29 AM, Lakini Senanayaka < lakinisenanayak...@gmail.com> wrote: > Hi, > > I have found some libraries to build drag and drop UI for our front end > .They are JQury UI[1],Dragula[2],Draggablily[3]. > I hope I can do the developments using these libraries. > > [1]jqueryui <http://jqueryui.com/> > [2]dragula <https://bevacqua.github.io/dragula/> > [3]draggabilly <http://draggabilly.desandro.com/> > > Could you please tell me if you have any other suggestions? > > On Sat, May 21, 2016 at 11:05 PM, Lakini Senanayaka < > lakinisenanayak...@gmail.com> wrote: > >> Hi , >> >> This is my weekly progress update of my project. >> >> I have solved all the problems I have mentioned in the last email.I >> could find a class BaseDatasetIterator which is in DL4J where we can find >> inbuilt iterators for CurvesDataSetIterator, IrisDataSetIterator, >> MnistDataSetIterator, MovingWindowBaseDataSetIterator, >> RawMnistDataSetIterator .There is no iterator for CIFAR dataset yet. >> >> I have gone through the whole documentation except Deeplearning4j on >> Spark.The latest documentation is very understandable than the earlier >> one.I have studied neural networks- Restricted Boltzmann Machines, >> Convolutional Nets (ConvNets), >> Long Short-Term Memory Units (LSTMs), >> Denoising Autoencoders, >> Recurrent Nets and LSTMs, >> Multilayer Neural Nets, >> Deep-Belief Network, >> Deep AutoEncoder, >> Stacked Denoising Autoencoders >> >> I have run the example codes[1] and I have understood the >> implementations.I have rerun the codes with modifying different parameter >> values like number and size of the hidden layers, learning rate, momentum, >> weight distribution and various types of regularization and checked the >> performance. >> >> I have implemented a convolution net[3] and trained it using LFW >> dataset[2] and a Recurrent net[4].Still I have a problem in vectorizing >> CIFRA-10 data set. >> >> Currently I am researching about libraries to build drag & drop UI for >> front end. >> >> [1]dl4j-0.4-examples >> <https://github.com/deeplearning4j/dl4j-0.4-examples> >> [2]LFW Face Dataset <http://vis-www.cs.umass.edu/lfw/> >> [3]ConvolutionNetLFW >> <https://github.com/Lakini/Deeplearning4Java/blob/master/src/main/java/org/deeplearning4j/examples/convolution/ConvolutionNetLFW.java> >> [4]RecurrentNetworkExample >> <https://github.com/Lakini/Deeplearning4Java/blob/master/src/main/java/org/deeplearning4j/examples/recurrent/basic/RecurrentNetworkExample.java> >> >> >> Could you please guide me to do the next step of my project . >> >> Thank you. >> >> Sincerely, >> *Lakini Senanayaka.* >> >> > > > -- > Thank you. > > Sincerely, > *Lakini Senanayaka.* > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Release tag's not available in ML repo
Hi Manoj, What is the naming convention for this released tag? Btw, seems it is not mentioned in the release process docs [1].. If this is a required step, shall we update the docs too? [1] https://docs.wso2.com/display/Carbon445/Releasing+a+Git+Repository On Wed, May 11, 2016 at 12:31 PM, Manoj Kumara <ma...@wso2.com> wrote: > Great Thanks Supun. > > *Manoj Kumara* > WSO2 Inc. *| **lean. enterprise. middleware.* > *Mobile:* +94 713 448188 > > On Wed, May 11, 2016 at 12:30 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Manoj, >> >> Yes, latest RCs contain the released versions of dependencies. Please go >> ahead with 1.0.0-RC2 and 1.1.0-RC5. >> >> We'll create a release tag as well. >> >> Regards, >> Supun >> >> On Wed, May 11, 2016 at 12:16 PM, Manoj Kumara <ma...@wso2.com> wrote: >> >>> Hi ML Team, >>> >>> I'll looking into released tags of each product to issue an security >>> patch and noticed that tag correspond to 1.0.0 and 1.1.0 version is not >>> available in the repo. >>> >>> Can I assume the latest RC tag contain the released version of each >>> dependency repo's. IMO this tag need to be created as well for supporting >>> purposes. >>> >>> Regards, >>> Manoj >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Can you try doing a "mvn clean"? Do you get any compilation failures? Regards, Supun On Thu, May 12, 2016 at 7:48 PM, Misgana Negassi < negas...@tf.uni-freiburg.de> wrote: > Hi Supun, > > Thank you for the supportive hangout session. > > I had one question I forgot to ask. > > When I was importing the carbon-ml as maven project to Eclipse I had this > error message. > *Multiple annotations found at this line:* > *- Plugin execution not covered by lifecycle configuration: > org.apache.felix:maven-scr-plugin:1.7.2:scr (execution: > generate-scr-scrdescriptor, phase: process-* > * classes)* > > > *- maven-remote-resources-plugin (goal "process") is ignored by m2e. *How > did you solve this problem if you have enountered it? > > Best, > Misgana > > > On 09.05.2016 06:12, Supun Sethunga wrote: > > Hi Misgana, > > I committed the code for reading a csv file. My next task will be sampling >> and starting to implement an ensemble method(Stacking). > > I went through the code. Would like to suggest a small thing. Most of the > Spark algorithms need JavaRDDs as the input for datasets. Hence reading > your file as a JavaRDD is the better approach than reading it > as a list of labelled points. Please refer [1] and [2] for an example. > > - How to decide which models to use for an ensemble and which parameters? > > Type of Model/Algorithm has to be a user input. The parameters will > depend on the algorithm user picks. > > - Should the ensemble methods be implemented as a wrapper around the >> base-models? > > Yes. You can use the existing algorithms in WSO2 Machine Learner, as the > base-models. (I have shared that in my previous mail) > > >> - Which library to use for matrix operations? Is Apache >> commons.math.Linearalgebra ok? > > Yes Apache commons.math.* would be fine. Infact you can use any library > with open-source licence. > > > What do you think about a hangout session to clarify stuff and get to know >> each other.:) > > Of course! Please arrange some time slot (Hope it will be IST time zone: > GMT+5.30 friendly :) ) and send me a calendar invite. > > > [1] > https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/utils/MLUtils.java#L58 > [2] > https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/SupervisedSparkModelBuilder.java#L87 > > Regards, > Supun > > On Sat, May 7, 2016 at 8:46 PM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> >> Hi Supun, >> I committed the code for reading a csv file. My next task will be >> sampling and starting to implement an ensemble method(Stacking). >> I have some questions about: >> - How to decide which models to use for an ensemble and which >> parameters? >> - Should the ensemble methods be implemented as a wrapper around the >> base-models? >> - Which library to use for matrix operations? Is Apache >> commons.math.Linearalgebra ok? >> >> What do you think about a hangout session to clarify stuff and get to >> know each other.:) >> >> Have a nice weekend! >> Misgana >> >> >> On 05.05.2016 19:59, Misgana Negassi wrote: >> >> Hi Supun, >> Thank you for the detailed explanation. >> >> I switched to intelliJ IDEA as an IDE with Ubuntu 14.04. The import >> errors are resolved after the project was imported as maven project. This >> took a while because of persistent pom.xml errors -- I am still in the >> process of reading about maven, Spark, REST and the carbon-Architecture. >> I have created an independent maven-project[1] for the implementation of >> the ensemble-methods. Currently I am writing code for reading a CSV file >> using the Apache library and converting it into Java RDD. I will commit >> once I am done with it. >> >> >> >> Regards, >> Misgana >> >> >> [1]https://github.com/zemoel/ensemble-methods >> >> >> On 04.05.2016 06:33, Supun Sethunga wrote: >> >> Hi Misgana, >> >> Seems you have misunderstood the "carbon" architecture. Let me explain >> it. *carbon-ml* repo contains the source code of the osgi bundles (i.e: >> jar libraries) which contain actual implementation/logic (such as, the >> implementation of importing datasets, creating projects, building models, >> and etc). Hence, these are similar to any other third-party library, which >> you have to invoke using their APIs. (They don't have ma
Re: [Dev] Release tag's not available in ML repo
Hi Manoj, Yes, latest RCs contain the released versions of dependencies. Please go ahead with 1.0.0-RC2 and 1.1.0-RC5. We'll create a release tag as well. Regards, Supun On Wed, May 11, 2016 at 12:16 PM, Manoj Kumara <ma...@wso2.com> wrote: > Hi ML Team, > > I'll looking into released tags of each product to issue an security patch > and noticed that tag correspond to 1.0.0 and 1.1.0 version is not available > in the repo. > > Can I assume the latest RC tag contain the released version of each > dependency repo's. IMO this tag need to be created as well for supporting > purposes. > > Regards, > Manoj > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Misgana, I committed the code for reading a csv file. My next task will be sampling > and starting to implement an ensemble method(Stacking). I went through the code. Would like to suggest a small thing. Most of the Spark algorithms need JavaRDDs as the input for datasets. Hence reading your file as a JavaRDD is the better approach than reading it as a list of labelled points. Please refer [1] and [2] for an example. - How to decide which models to use for an ensemble and which parameters? Type of Model/Algorithm has to be a user input. The parameters will depend on the algorithm user picks. - Should the ensemble methods be implemented as a wrapper around the > base-models? Yes. You can use the existing algorithms in WSO2 Machine Learner, as the base-models. (I have shared that in my previous mail) > - Which library to use for matrix operations? Is Apache > commons.math.Linearalgebra ok? Yes Apache commons.math.* would be fine. Infact you can use any library with open-source licence. What do you think about a hangout session to clarify stuff and get to know > each other.:) Of course! Please arrange some time slot (Hope it will be IST time zone: GMT+5.30 friendly :) ) and send me a calendar invite. [1] https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/utils/MLUtils.java#L58 [2] https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/SupervisedSparkModelBuilder.java#L87 Regards, Supun On Sat, May 7, 2016 at 8:46 PM, Misgana Negassi <negas...@tf.uni-freiburg.de > wrote: > > Hi Supun, > I committed the code for reading a csv file. My next task will be sampling > and starting to implement an ensemble method(Stacking). > I have some questions about: > - How to decide which models to use for an ensemble and which parameters? > - Should the ensemble methods be implemented as a wrapper around the > base-models? > - Which library to use for matrix operations? Is Apache > commons.math.Linearalgebra ok? > > What do you think about a hangout session to clarify stuff and get to know > each other.:) > > Have a nice weekend! > Misgana > > > On 05.05.2016 19:59, Misgana Negassi wrote: > > Hi Supun, > Thank you for the detailed explanation. > > I switched to intelliJ IDEA as an IDE with Ubuntu 14.04. The import errors > are resolved after the project was imported as maven project. This took a > while because of persistent pom.xml errors -- I am still in the process of > reading about maven, Spark, REST and the carbon-Architecture. > I have created an independent maven-project[1] for the implementation of > the ensemble-methods. Currently I am writing code for reading a CSV file > using the Apache library and converting it into Java RDD. I will commit > once I am done with it. > > > > Regards, > Misgana > > > [1]https://github.com/zemoel/ensemble-methods > > > On 04.05.2016 06:33, Supun Sethunga wrote: > > Hi Misgana, > > Seems you have misunderstood the "carbon" architecture. Let me explain it. > *carbon-ml* repo contains the source code of the osgi bundles (i.e: jar > libraries) which contain actual implementation/logic (such as, the > implementation of importing datasets, creating projects, building models, > and etc). Hence, these are similar to any other third-party library, which > you have to invoke using their APIs. (They don't have main classes, they > have APIs). This repo also contains a REST API, which exposes the > above-mentioned APIs as a RESTful service. It also includes the source code > for the UI, which invokes those REST APIs, behalf of the user. Please refer > [1] to get the overall idea. > Whereas, *product-ml* repo contains the source code, which collects the > necessary libraries (such as the libraries of carbon-ml, REST API, UI) and > bundles it all together, to create the final binary distribution. > > *carbon-ml* already contains the implementations of number of algorithms > [2]. Your ultimate goal is to add three more such implementations to that > repo (i.e: the three ensemble methods). In doing so, you don't need to > re-implement the logics of importing datasets, creating projects and etc.. > As those have already been implemented and you can use those methods from > their APIs. (Please refer any of the current algorithms to get an idea..). > > But, since it can be difficult to implement the ensemble logic and > integrate it to carbon-ml repo, *at the same time*, We recommend you to *first > implement your logic in a separate java client.* This has to be an > independent maven project. The whole purpose of this java client is to > implement your logic independently and tes
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Misgana, As you progress, please keep us posted too. It would be nice if you can share your code as well (Github project). You can take a fork of repo [1], and start working on your fork. [1] https://github.com/wso2/carbon-ml On Mon, Apr 25, 2016 at 7:57 PM, Misgana Negassi < negas...@tf.uni-freiburg.de> wrote: > Hi Supun, > > Thank you for accepting me for this project!I am excited to work on it and > start right away with the links you sent. > > Best, > Misgana > > > > On 25.04.2016 12:06, Supun Sethunga wrote: > > Hi Misgana, > > Congratulations for getting accepted for the gsoc 2016! Hope you are ready > to get started with the project. > > To get more familiarized with the code, I'm sharing the implementations of > the current algorithms [1]. For your ensemble method, you need to add three > more cases (for the three types of ensembles) for the method [2]. You may > try out adding a new algorithm to he existing flow, and see how it works. > Please feel free to raise any questions/issues you come across. > > [1] > https://github.com/wso2/carbon-ml/tree/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms > [2] > <https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/SupervisedSparkModelBuilder.java#L101> > https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/SupervisedSparkModelBuilder.java#L101 > > Regards, > Supun > > On Thu, Mar 24, 2016 at 9:31 PM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> Hi Supun, >> >> Thank you for your support and advice in this proposal process! >> >> In the case you are interested, I am attaching my report paper with >> contains my work with ensemble methods particularly Stacking. >> >> Best, >> Misgana >> >> >> On 24.03.2016 04:12, Supun Sethunga wrote: >> >> Looks good! Please go ahead and submit to GSoC. >> >> Thanks, >> Supun >> >> On Thu, Mar 24, 2016 at 4:02 AM, Misgana Negassi < >> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >> >>> Hi Supun, >>> >>> I have added the changes you recommended. Could you kindly give me a >>> feedback? >>> >>> Best, >>> Misgana >>> >>> On 23.03.2016 15:04, Supun Sethunga wrote: >>> >>> Hi Misgana, >>> >>> I went through your proposal. Overall it looks good. Here are a few >>> comments I would like to point out: >>> >>>- Its better to have some sort of an architecture diagram, >>>explaining your solution in a higher level. >>>- In the timeline, better to break down the "Week 13 (May 23 June >>>20, 2016)" into three sub-levels, and allocate timeslots for each of the >>>three methods (Stacking, Boosting and Bagging) separately. That would >>> make >>>it easy for you to work on those methods separately, as well as to track >>>the progress. >>>- In the timeline, can you double check the "week" numbers..? for >>>eg; in [*Week 1-3 (May 23 June 20, 2016*], I guess it should be "*Week >>>1-4*" (there are four weeks in the mentioned duration). Similarly, >>>check the others too. >>> >>> Please share us the draft proposal once you fix those. >>> >>> Thanks, >>> Supun >>> >>> On Wed, Mar 23, 2016 at 7:17 PM, Misgana Negassi < >>> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >>> >>>> Hi Supun, >>>> >>>> I am attaching my proposal draft. I am very grateful for your comments. >>>> >>>> Thanks, >>>> Misgana >>>> >>>> >>>> On 23.03.2016 04:54, Supun Sethunga wrote: >>>> >>>> Hi Misgana, >>>> >>>> As we have mentioned in the project proposal as well, the main >>>> objective is to integrate ensemble support for the existing flow of the >>>> WSO2 Machine Learner. We are focusing on the three methods: Bagging, >>>> Boosting and Stacking. (On technique per each of these methods) >>>> >>>> If you haven't tried out already, you can get to know the Machine >>>> Learner product by downloading it and running it (Please use link [1] to >>>> download). Official documentation [2] an
Re: [Dev] ESB Analytics - Verifying the common production use cases
JIRA: https://wso2.org/jira/browse/ANLYESB-15 On Mon, Apr 18, 2016 at 2:24 PM, Srinath Perera <srin...@wso2.com> wrote: > Could you open a Jira ( blocker)? We need to figure out a solution for the > project as well. > > --Srinath > > > > On Mon, Apr 18, 2016 at 12:17 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Kasun, >> >> We ran several rounds of load tests for ESB analytics (covering few >> scenarios with several proxy services, APIs, Sequences, etc). In the first >> round, we ran the tests with a low TPS (around 10-15) to test and validate >> the use cases, such as the counts, delays, drilling down to >> mediator/sequence/message level, etc. These scenarios worked as expected. >> >> On the second round, we ran the same tests with a higher TPS (gradually >> increasing). In the process, we identified a bottleneck at the DB level >> under a high TPS (TPS of over 50, which is not that high), resulting the >> stat generation to be slowed down with time. This occurs as DAS is both >> reading and writing from/to the same table (row events) at the same time, >> at a rapid rate, causing the IO operations to be slowed down (due to table >> locks and etc.), and hence the summarizing process takes longer time, in >> each iteration. (most of the consumed time for summarizing, around 80% is >> taken for reading the raw data and store them as event-wise data in DB) >> Therefore, we are currently doing an enhancement to improve performance, >> by summarizing the data (summarize up to per-second info) on the fly with >> siddhi and store it, rather than spark doing it as a batch operation. This >> should eliminate roughly around 80% (above mentioned) of time taken to >> read/write row data from DB. >> >> Will give an update after re-running the load test with this enhancement. >> >> Regards, >> Supun >> >> On Mon, Apr 18, 2016 at 11:40 AM, Kasun Indrasiri <ka...@wso2.com> wrote: >> >>> Any updates on this? >>> >>> We must test the ESB analytics/ ESB 5 alpha release with the common >>> production use cases + high load. >>> >>> On Mon, Mar 28, 2016 at 2:38 PM, Buddhima Wijeweera <buddh...@wso2.com> >>> wrote: >>> >>>> Hi All, >>>> >>>> We are currently in the process of testing statistics & tracing for >>>> various scenarios. >>>> Initially we planned to cover on the areas we have identified for >>>> functionality testing. >>>> Then we planned to move on to common scenarios, and performance testing >>>> as well. >>>> >>>> Thank you, >>>> >>>> On Mon, Mar 28, 2016 at 2:34 PM, Srinath Perera <srin...@wso2.com> >>>> wrote: >>>> >>>>> Please run ESB benchmark, get the events comes out of it to DAS, and >>>>> use that for testing. >>>>> >>>>> --Srinath >>>>> >>>>> On Mon, Mar 28, 2016 at 10:02 AM, Kasun Indrasiri <ka...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Buddhima/VirajS, >>>>>> >>>>>> Once the end to end functionalities are implemented on the ESB >>>>>> analytics side, can we start testing the current implementation for >>>>>> production use cases of ESB. Basically we need to cover all the common >>>>>> integration scenarios and stress test the implemented with high load too. >>>>>> >>>>>> Thanks, >>>>>> Kasun >>>>>> >>>>>> -- >>>>>> Kasun Indrasiri >>>>>> Software Architect >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> cell: +94 77 556 5206 >>>>>> Blog : http://kasunpanorama.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>> Site: http://home.apache.org/~hemapani/ >>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>> Phone: 0772360902 >>>>> >>>> >>>> >>>> >>>> -- >>>> Buddhima Wijeweera >>>> Software Engineer; WSO2 Inc.; http://wso2.com , >>>> >>>> Mobile: +94 71 427 9966 >>>> Email: buddh...@wso2.com >>>> Blog: https://buddhimawijeweera.wordpress.com >>>> GitHub Profile: https://github.com/Buddhima >>>> >>> >>> >>> >>> -- >>> Kasun Indrasiri >>> Software Architect >>> WSO2, Inc.; http://wso2.com >>> lean.enterprise.middleware >>> >>> cell: +94 77 556 5206 >>> Blog : http://kasunpanorama.blogspot.com/ >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://home.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] ESB Analytics - Verifying the common production use cases
Hi Kasun, We ran several rounds of load tests for ESB analytics (covering few scenarios with several proxy services, APIs, Sequences, etc). In the first round, we ran the tests with a low TPS (around 10-15) to test and validate the use cases, such as the counts, delays, drilling down to mediator/sequence/message level, etc. These scenarios worked as expected. On the second round, we ran the same tests with a higher TPS (gradually increasing). In the process, we identified a bottleneck at the DB level under a high TPS (TPS of over 50, which is not that high), resulting the stat generation to be slowed down with time. This occurs as DAS is both reading and writing from/to the same table (row events) at the same time, at a rapid rate, causing the IO operations to be slowed down (due to table locks and etc.), and hence the summarizing process takes longer time, in each iteration. (most of the consumed time for summarizing, around 80% is taken for reading the raw data and store them as event-wise data in DB) Therefore, we are currently doing an enhancement to improve performance, by summarizing the data (summarize up to per-second info) on the fly with siddhi and store it, rather than spark doing it as a batch operation. This should eliminate roughly around 80% (above mentioned) of time taken to read/write row data from DB. Will give an update after re-running the load test with this enhancement. Regards, Supun On Mon, Apr 18, 2016 at 11:40 AM, Kasun Indrasiri <ka...@wso2.com> wrote: > Any updates on this? > > We must test the ESB analytics/ ESB 5 alpha release with the common > production use cases + high load. > > On Mon, Mar 28, 2016 at 2:38 PM, Buddhima Wijeweera <buddh...@wso2.com> > wrote: > >> Hi All, >> >> We are currently in the process of testing statistics & tracing for >> various scenarios. >> Initially we planned to cover on the areas we have identified for >> functionality testing. >> Then we planned to move on to common scenarios, and performance testing >> as well. >> >> Thank you, >> >> On Mon, Mar 28, 2016 at 2:34 PM, Srinath Perera <srin...@wso2.com> wrote: >> >>> Please run ESB benchmark, get the events comes out of it to DAS, and use >>> that for testing. >>> >>> --Srinath >>> >>> On Mon, Mar 28, 2016 at 10:02 AM, Kasun Indrasiri <ka...@wso2.com> >>> wrote: >>> >>>> Hi Buddhima/VirajS, >>>> >>>> Once the end to end functionalities are implemented on the ESB >>>> analytics side, can we start testing the current implementation for >>>> production use cases of ESB. Basically we need to cover all the common >>>> integration scenarios and stress test the implemented with high load too. >>>> >>>> Thanks, >>>> Kasun >>>> >>>> -- >>>> Kasun Indrasiri >>>> Software Architect >>>> WSO2, Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> cell: +94 77 556 5206 >>>> Blog : http://kasunpanorama.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> >>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>> Site: http://home.apache.org/~hemapani/ >>> Photos: http://www.flickr.com/photos/hemapani/ >>> Phone: 0772360902 >>> >> >> >> >> -- >> Buddhima Wijeweera >> Software Engineer; WSO2 Inc.; http://wso2.com , >> >> Mobile: +94 71 427 9966 >> Email: buddh...@wso2.com >> Blog: https://buddhimawijeweera.wordpress.com >> GitHub Profile: https://github.com/Buddhima >> > > > > -- > Kasun Indrasiri > Software Architect > WSO2, Inc.; http://wso2.com > lean.enterprise.middleware > > cell: +94 77 556 5206 > Blog : http://kasunpanorama.blogspot.com/ > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [DAS] Adding a relative path in external-spark-classpath.conf doesn't work
Thanks Nirmal for the fix!
I'm closing the Jira [1]
[1] https://wso2.org/jira/browse/DAS-397
On Wed, Apr 13, 2016 at 9:17 AM, Nirmal Fernando <nir...@wso2.com> wrote:
> Thanks, Niranda for merging.
>
> On Wed, Apr 13, 2016 at 8:48 AM, Niranda Perera <nira...@wso2.com> wrote:
>
>> Hi Nirmal,
>>
>> Thanks for pointing this out. I will merge the PR.
>>
>> Rgds
>> On Apr 13, 2016 08:35, "Nirmal Fernando" <nir...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> * Even though we have implemented the support for relative paths at
>>> repository/conf/analytics/spark/external-spark-classpath.conf file, it
>>> isn't working properly.
>>>
>>> if (fileExists(line)) {
>>>
>>> scp = scp + separator + line;
>>>
>>> } *else if (fileExists(carbonHome + File.separator +
>>> line)) {*
>>>
>>> *scp = scp + separator + carbonHome + File.separator
>>> + line;*
>>>
>>> *} *else {
>>>
>>> throw new IOException("File not found : " + line);
>>>
>>> }
>>>
>>>
>>>
>>>
>>>
>>> private static boolean fileExists(String path) {
>>>
>>> File tempFile = new File(path);
>>>
>>> return tempFile.exists() && !tempFile.isDirectory();
>>>
>>>}
>>> * We check for file.exists in order to determine the existence of the
>>> file and it'll be true even if it's a relative path in some cases (where
>>> current execution directory = carbon.home).
>>> * But Spark needs the path to be absolute.
>>> * Hence the fix was to check whether the path is absolute too.
>>>
>>> https://github.com/wso2/carbon-analytics/pull/170/files
>>>
>>> Please review and merge this PR.
>>>
>>> --
>>>
>>> Thanks & regards,
>>> Nirmal
>>>
>>> Team Lead - WSO2 Machine Learner
>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>> Mobile: +94715779733
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>>
>>>
>
>
> --
>
> Thanks & regards,
> Nirmal
>
> Team Lead - WSO2 Machine Learner
> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
> Mobile: +94715779733
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>
--
*Supun Sethunga*
Software Engineer
WSO2, Inc.
http://wso2.com/
lean | enterprise | middleware
Mobile : +94 716546324
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [DAS] Error in Searching Records by Query
Hi Lochana, Can you try with "time: [1422247320 TO 1482554520] AND *_webappName* :" (note the underscore). "webappName" with underscore-prefix means to match the entire word, while just the "webappName"says to match any part of the word. Regards, Supun On Wed, Apr 6, 2016 at 1:58 PM, Lochana Ranaweera <locha...@wso2.com> wrote: > Hi all, > > I'm using the data explorer of the WSO2 DAS to search for records by a > query. The table is as follows: > > > > > In this table the column 'webappName' is of type STRING and it is declared > as a FACET field in order to search for webapps by the name. > > A list of records corresponding to the 'examples' webapp can be inquired > using the given query; *time: [1422247320 TO 1482554520] AND webappName: > "examples"*. However, the result for the query, *time: [1422247320 TO > 1482554520] AND webappName: "/"*, is the entire set of records in the > table, but not the records corresponding to webapp ' / ' in the given time > range. > > The following queries too resulted in the entire set of records in the > table: > >- time: [1422247320 TO 1482554520] AND webappName: "a" >- time: [1422247320 TO 1482554520] AND webappName: "A" >- time: [1422247320 TO 1482554520] AND webappName: "%" >- time: [1422247320 TO 1482554520] AND webappName: "*" >- time: [1422247320 TO 1482554520] AND webappName: "$" > > All the records in the given table satisfy the time range in the queries. > Except in the query, *time: [1422247320 TO 1482554520] AND webappName: > "examples"*,it seems the condition joined by the 'AND' is ignored when > webapp name takes the form " / " ," a ", " A ", " $ " , " * " or " %" . > > Appreciate your feedback on this matter. > > Regards, > -- > Lochana Ranaweera > Intern Software Engineer > WSO2 Inc: http://wso2.com > Blog: https://lochanaranaweera.wordpress.com/ > Mobile: +94716487055 <http://tel%2B716487055> > > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Regarding [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Ishani, Thank you for your interest. However, this project is aimed at GSoC 2016, with a tartget duration of 3 months. And currently the dealine for proposal submission has ended. Hence, I'm afraid it won't be possible to do the above project as you final year research. But you can come up with your own research topic around the similar field (propbably some real world application of ML), which I think would be a better option. And if you need any help, we are happy to provide any assistence. Thanks, Supun On Wed, Apr 6, 2016 at 9:56 AM, Ishani Pathinayake < ishanipathinay...@gmail.com> wrote: > Hi, > > I'm a 4th year undergraduate from SLIIT faculty of > computing(specializing in Software Engineering). I'm interested in doing > the above mentioned project as the final year research. I would like to > know more on this topic and the procedure to be followed when implementing > it. looking forward to your reply. > > Thank you. > > -- > *Cheers !* > Ishani Pathinayake. > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Applying for GSoC 2016 ----> Visual (Deep) Neural Network Builder
ea >>>>>> about the proposal I would like to point, Nirmal's proposal submitted to >>>>>> 2011 GSOC >>>>>> >>>>>> >>>>>> <http://nirmalfdo.blogspot.com/2011/04/apache-tuscany-develop-simple-tool-that.html> >>>>>> http://nirmalfdo.blogspot.com/2011/04/apache-tuscany-develop-simple-tool-that.html >>>>>> >>>>>> If you need further help, please let us know. >>>>>> >>>>>> Thanks, >>>>>> Upul >>>>>> >>>>>> On Wed, Mar 9, 2016 at 9:32 AM, Vishma Senadhi Dias < >>>>>> <vishma...@cse.mrt.ac.lk>vishma...@cse.mrt.ac.lk> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I am Vishma, a third year undergraduate from Department of Computer >>>>>>> Science and Engineering, University of Moratuwa, Sri Lanka. I would >>>>>>> like to >>>>>>> apply for the Visual (Deep) Neural Network Builder project. It would be >>>>>>> very helpful if I could get any instructions on starting things up. >>>>>>> >>>>>>> Thanks in advance, >>>>>>> Vishma. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Upul Bandara, >>>>>> Associate Technical Lead, WSO2, Inc., >>>>>> Mob: +94 715 468 345 <%2B94%20715%20468%20345>. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Upul Bandara, >>>>> Associate Technical Lead, WSO2, Inc., >>>>> Mob: +94 715 468 345 <%2B94%20715%20468%20345>. >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Upul Bandara, >>>> Associate Technical Lead, WSO2, Inc., >>>> Mob: +94 715 468 345 <%2B94%20715%20468%20345>. >>>> >>>> >>>> >>> >>> >>> -- >>> Upul Bandara, >>> Associate Technical Lead, WSO2, Inc., >>> Mob: +94 715 468 345. >>> >>> >>> >> >> >> -- >> Upul Bandara, >> Associate Technical Lead, WSO2, Inc., >> Mob: +94 715 468 345. >> > > > > -- > Upul Bandara, > Associate Technical Lead, WSO2, Inc., > Mob: +94 715 468 345. > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSoC 2016]Proposal 03:[ML] Visual (Deep) Neural Network Builder
Hi Lakini, Please complete any changes and submit your final proposal to GSoC, before the deadline (Friday). Regards, Supun On Thu, Mar 24, 2016 at 10:17 AM, Lakini Senanayaka < lakinisenanayak...@gmail.com> wrote: > Thank you for the comments. > Yes.I'll change my proposal and I'll send it soon. > > On Thu, Mar 24, 2016 at 9:59 AM, Upul Bandara <u...@wso2.com> wrote: > >> We have added few comments. >> Can you please address those comments? >> >> Thanks, >> Upul >> >> On Thu, Mar 24, 2016 at 9:04 AM, Upul Bandara <u...@wso2.com> wrote: >> >>> Please grant us comment rights. >>> >>> On Thu, Mar 24, 2016 at 9:02 AM, Upul Bandara <u...@wso2.com> wrote: >>> >>>> Yes, just started to read it and will provide feedback ASAP >>>> >>>> On Thu, Mar 24, 2016 at 8:55 AM, Lakini Senanayaka < >>>> lakinisenanayak...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Hope you got my mail about the proposal. >>>>> >>>>> On Wed, Mar 23, 2016 at 5:18 PM, Lakini Senanayaka < >>>>> lakinisenanayak...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Could you please check my proposal[1] for [ML] Visual (Deep) Neural >>>>>> Network Builder.I kindly request your feedback and comments on it. >>>>>> >>>>>> [1]Proposal 03:[ML] Visual (Deep) Neural Network Builder >>>>>> <https://docs.google.com/document/d/179c0nQAl_7CCoQZjlPgPiloq2nAco1mUfT8WYOKLf90/edit?usp=sharing> >>>>>> >>>>>> >>>>>> -- >>>>>> Thank you. >>>>>> >>>>>> Sincerely, >>>>>> *Lakini Senanayaka.* >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thank you. >>>>> >>>>> Sincerely, >>>>> *Lakini Senanayaka.* >>>>> >>>>> >>>> >>>> >>>> -- >>>> Upul Bandara, >>>> Mob: +94 715 468 345. >>>> >>> >>> >>> >>> -- >>> Upul Bandara, >>> Associate Technical Lead, WSO2, Inc., >>> Mob: +94 715 468 345. >>> >> >> >> >> -- >> Upul Bandara, >> Associate Technical Lead, WSO2, Inc., >> Mob: +94 715 468 345. >> > > > > -- > Thank you. > > Sincerely, > *Lakini Senanayaka.* > > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Introduction for GSoC
;>> >>>>>> I have developed several software for small companies as a before >>>>>> becoming a student at IIT. >>>>>> >>>>>> I am also enrolled as IEEE Student member. >>>>>> >>>>>> I have done my advanced level in Mathematics field that would help >>>>>> me to go on algorithmic or problem solving type situations. >>>>>> >>>>>> I have studied and done research on Machine Learning and Natural >>>>>> Language Processing for past 2 years from mit online and mostly on >>>>>> coursera. >>>>>> >>>>>> Thank you. >>>>>> >>>>>> >>>>>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=emailclient_term=oa-2115-e> >>>>>> Scanned >>>>>> for viruses. www.avast.com >>>>>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=emailclient_term=oa-2115-e> >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *CD Athuraliya* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> lean . enterprise . middleware >>>>> Mobile: +94 716288847 <94716288847> >>>>> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter >>>>> <https://twitter.com/cdathuraliya> | Blog >>>>> <https://cdathuraliya.wordpress.com/> >>>>> >>>>> >>>>> >>>>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=emailclient_term=oa-2115-e> >>>>> Scanned >>>>> for viruses. www.avast.com >>>>> <https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=emailclient_term=oa-2115-e> >>>>> >>>> >>>> >>>> >>>> -- >>>> *CD Athuraliya* >>>> Software Engineer >>>> WSO2, Inc. >>>> lean . enterprise . middleware >>>> Mobile: +94 716288847 <94716288847> >>>> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter >>>> <https://twitter.com/cdathuraliya> | Blog >>>> <https://cdathuraliya.wordpress.com/> >>>> >>> >>> >> >> >> -- >> *CD Athuraliya* >> Software Engineer >> WSO2, Inc. >> lean . enterprise . middleware >> Mobile: +94 716288847 <94716288847> >> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter >> <https://twitter.com/cdathuraliya> | Blog >> <https://cdathuraliya.wordpress.com/> >> > > > > -- > *CD Athuraliya* > Software Engineer > WSO2, Inc. > lean . enterprise . middleware > Mobile: +94 716288847 <94716288847> > LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter > <https://twitter.com/cdathuraliya> | Blog > <https://cdathuraliya.wordpress.com/> > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Fwd: GSOC2016: Proposal 6: [ML]
;>>>>>>>>>> [4][5]. >>>>>>>>>>> >>>>>>>>>>> Best regards. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/regression/LinearRegressionWithSGD.html >>>>>>>>>>> [2] >>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/clustering/KMeans.html >>>>>>>>>>> [3] >>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/classification/LogisticRegressionWithSGD.html >>>>>>>>>>> [4] >>>>>>>>>>> https://docs.wso2.com/display/CEP310/Working+with+Event+Streams >>>>>>>>>>> [5] >>>>>>>>>>> https://docs.wso2.com/display/CEP310/Working+with+Execution+Plans >>>>>>>>>>> >>>>>>>>>>> On Fri, Mar 4, 2016 at 11:26 AM, Mahesh Dananjaya < >>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi maheshakya, >>>>>>>>>>>> give me sometime to go through your ML package. Do current >>>>>>>>>>>> product have any stream data support?. i did some university >>>>>>>>>>>> projects >>>>>>>>>>>> related to machine learning with regressions,modelling, factor >>>>>>>>>>>> analysis, >>>>>>>>>>>> cluster analysis and classification problems (Discriminant >>>>>>>>>>>> Analysis) with >>>>>>>>>>>> SVM (Support Vector machines), Neural networks, LS classification >>>>>>>>>>>> and >>>>>>>>>>>> ML(Maximum likelihood). give me sometime to see how wso2 >>>>>>>>>>>> architecture >>>>>>>>>>>> works.then i can come up with good architecture.thank you. >>>>>>>>>>>> BR, >>>>>>>>>>>> Mahesh. >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Mar 2, 2016 at 2:41 PM, Mahesh Dananjaya < >>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Maheshakya, >>>>>>>>>>>>> Thank you for the resources. I will go through this and >>>>>>>>>>>>> looking forward to this proposed project.Thank you. >>>>>>>>>>>>> BR, >>>>>>>>>>>>> Mahesh. >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Mar 2, 2016 at 1:52 PM, Maheshakya Wijewardena < >>>>>>>>>>>>> mahesha...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Mahesh, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for the interest for this project. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We would like to know what type of similar projects you have >>>>>>>>>>>>>> worked on. You may have seen that WSO2 Machine Learner supports >>>>>>>>>>>>>> several >>>>>>>>>>>>>> learning algorithms at the moment[1]. This project intends to >>>>>>>>>>>>>> leverage the >>>>>>>>>>>>>> existing algorithms in WSO2 Machine Learner to support streaming >>>>>>>>>>>>>> data. As >>>>>>>>>>>>>> an initiative, first you can get an idea about what WSO2 Machine >>>>>>>>>>>>>> Learner >>>>>>>>>>>>>> does and how it operates. You can download WSO2 Machine Learner >>>>>>>>>>>>>> from >>>>>>>>>>>>>> product page[2] and the the source code [3]. ML is using Apache >>>>>>>>>>>>>> Spark >>>>>>>>>>>>>> MLLib[4] for its' algorithms so it's better to read and >>>>>>>>>>>>>> understand what it >>>>>>>>>>>>>> does as well. >>>>>>>>>>>>>> >>>>>>>>>>>>>> In order to get an idea about the deliverables and the scope >>>>>>>>>>>>>> of this project, try to understand how Spark streaming[5] (see >>>>>>>>>>>>>> examples) >>>>>>>>>>>>>> handles streaming data. Also, have a look in the streaming >>>>>>>>>>>>>> algorithms[6][7] >>>>>>>>>>>>>> supported by MLLib. There are two approaches discussed to employ >>>>>>>>>>>>>> incremental learning in ML in the project proposals page. These >>>>>>>>>>>>>> streaming >>>>>>>>>>>>>> algorithms can be directly used in the first approach. For the >>>>>>>>>>>>>> other >>>>>>>>>>>>>> approach, the your implementation should contain a procedure to >>>>>>>>>>>>>> create mini >>>>>>>>>>>>>> batches from streaming data with relevant sizes (i.e. a moving >>>>>>>>>>>>>> window) and >>>>>>>>>>>>>> do periodic retraining of the same algorithm. >>>>>>>>>>>>>> >>>>>>>>>>>>>> To start with the project, you will need to come up with a >>>>>>>>>>>>>> suitable plan and an architecture first. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please watch the video referenced in the proposal (reference: >>>>>>>>>>>>>> 5). It will help you getting a better idea about machine learning >>>>>>>>>>>>>> algorithms with streaming data. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Let us know if you need any help with these. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best regards >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/Machine+Learner+Algorithms >>>>>>>>>>>>>> [2] http://wso2.com/products/machine-learner/ >>>>>>>>>>>>>> [3] >>>>>>>>>>>>>> https://docs.wso2.com/display/ML110/Building+from+Source#BuildingfromSource-Downloadingthesourcecheckout >>>>>>>>>>>>>> [4] https://spark.apache.org/docs/1.4.1/mllib-guide.html >>>>>>>>>>>>>> [5] >>>>>>>>>>>>>> https://spark.apache.org/docs/1.4.1/streaming-programming-guide.html >>>>>>>>>>>>>> [6] >>>>>>>>>>>>>> https://spark.apache.org/docs/1.4.1/mllib-linear-methods.html#streaming-linear-regression >>>>>>>>>>>>>> [7] >>>>>>>>>>>>>> https://spark.apache.org/docs/1.4.1/mllib-clustering.html#streaming-k-means >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Mar 2, 2016 at 1:19 PM, Mahesh Dananjaya < >>>>>>>>>>>>>> dananjayamah...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> I am interesting on contribute to proposal 6: "Predictive >>>>>>>>>>>>>>> analytic with online data for WSO2 Machine Learner" for GSOC2 >>>>>>>>>>>>>>> this time. >>>>>>>>>>>>>>> Since i have been engaging with some similar projects i think >>>>>>>>>>>>>>> it will be a >>>>>>>>>>>>>>> great experience for me. Please let me know what you think and >>>>>>>>>>>>>>> what you >>>>>>>>>>>>>>> suggest. I have been going through your documents.thank you. >>>>>>>>>>>>>>> regards, >>>>>>>>>>>>>>> Mahesh Dananjaya. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ___ >>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>> Dev@wso2.org >>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>>>>> +94711228855 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>>>> mahesha...@wso2.com >>>>>>>>>>> +94711228855 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>>>> mahesha...@wso2.com >>>>>>>>> +94711228855 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Pruthuvi Maheshakya Wijewardena >>>>>>> mahesha...@wso2.com >>>>>>> +94711228855 >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Pruthuvi Maheshakya Wijewardena >>>>> mahesha...@wso2.com >>>>> +94711228855 >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Pruthuvi Maheshakya Wijewardena >>> mahesha...@wso2.com >>> +94711228855 >>> >>> >>> >> > > > -- > Pruthuvi Maheshakya Wijewardena > mahesha...@wso2.com > +94711228855 > > > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSoC 2016] [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Hasitha, Please submit your final proposal to GSoC, before the deadline. Regards, Supun On Wed, Mar 23, 2016 at 2:03 PM, Hasitha Jayasundara < hasithajayasunda...@gmail.com> wrote: > Thank you very much > > On Wed, Mar 23, 2016 at 2:00 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> This Looks good! Please go ahead and submit the final draft to GSoC, on >> or before Friday :) >> >> On Wed, Mar 23, 2016 at 1:50 PM, Hasitha Jayasundara < >> hasithajayasunda...@gmail.com> wrote: >> >>> I made the changes.Really sorry that I had to change the timeline >>> several times.Since you have requested a rough timeline I only added some >>> main topics in the previous drafts.Could you please check the new one and >>> add some comments. >>> >>> https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing >>> >>> On Tue, Mar 22, 2016 at 11:19 PM, Supun Sethunga <sup...@wso2.com> >>> wrote: >>> >>>> Coding period (implementations) should be within 23rd May to 23rd >>>> August :) >>>> >>>> Btw, rather than having one slot as "Implement ensemble methods", can >>>> you break it down to the three methods, and have three separate slots? That >>>> way it would be easy for you to work on the three methods separately, as >>>> well as track down the progress.. >>>> >>>> Regards, >>>> Supun >>>> >>>> On Mon, Mar 21, 2016 at 4:57 PM, Hasitha Jayasundara < >>>> hasithajayasunda...@gmail.com> wrote: >>>> >>>>> Am I violating google GSoC calendar in my timeline? >>>>> >>>>> On Mon, Mar 21, 2016 at 4:47 PM, Hasitha Jayasundara < >>>>> hasithajayasunda...@gmail.com> wrote: >>>>> >>>>>> Made some adjustments.Could you please check >>>>>> >>>>>> https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing >>>>>> >>>>>> On Mon, Mar 21, 2016 at 4:24 PM, Hasitha Jayasundara < >>>>>> hasithajayasunda...@gmail.com> wrote: >>>>>> >>>>>>> I 'll adjust them >>>>>>> >>>>>>> >>>>>>> On Mon, Mar 21, 2016 at 4:07 PM, Supun Sethunga <sup...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Hasitha, >>>>>>>> >>>>>>>> Project timeline should tally with the gsoc timelines [1]. I've >>>>>>>> added a few comments in the doc itself too. Can you please adjust >>>>>>>> those? >>>>>>>> >>>>>>>> [1] https://developers.google.com/open-source/gsoc/timeline >>>>>>>> >>>>>>>> On Mon, Mar 21, 2016 at 3:39 PM, Hasitha Jayasundara < >>>>>>>> hasithajayasunda...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Added timeline >>>>>>>>> >>>>>>>>> https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing >>>>>>>>> >>>>>>>>> On Mon, Mar 21, 2016 at 10:35 AM, Supun Sethunga <sup...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Hasitha, >>>>>>>>>> >>>>>>>>>> Overall looks good to me! Added a few comments, please fix them >>>>>>>>>> too. >>>>>>>>>> >>>>>>>>>> Also, can you share a rough timeline too? If you feel like the >>>>>>>>>> current project goals does not fit into the deadlines, we can revise >>>>>>>>>> the >>>>>>>>>> goals before starting the project, so that it would fit within the >>>>>>>>>> timeline. >>>>>>>>>> >>>>>>>>>> BTW, please CC to WSO2 Developers mailing list (dev@wso2.org) >>>>>>>>>> for all project-related discussions. You might have to subscribe via >>>>>>>>>> http://wso2.com/mail/ >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>&g
Re: [Dev] GSoC project for building an ensemble of models
Hi Anurag, Please submit your final proposal to GSoC, before the deadline. Regards, Supun On Tue, Mar 22, 2016 at 11:12 PM, Supun Sethunga <sup...@wso2.com> wrote: > Hi Anurag, > > Looks good! May be you can also improve the diagram to reflect the > description of the three methods (which is right above the diagram). > > Anyway, please create the final proposal and submit before Friday, 25th. > > Regards, > Supun > > On Tue, Mar 22, 2016 at 1:52 PM, Anurag Kashyap <anurag1...@gmail.com> > wrote: > >> Hi, >> I have made a few changes to the proposal and elaborated a few points. >> Kindly give your valuable feedback. >> >> https://docs.google.com/document/d/1L132ApiOm4m0DbyVZ-TYCedsRrt7oE_cgOzAuScvFiQ/edit?usp=sharing >> >> Regards, >> Anurag Kashyap >> >> On Mon, Mar 21, 2016 at 4:43 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi Anurag, >>> >>> We have added a few comments. Can you please fix those and share the >>> updated one? >>> >>> Thanks, >>> Supun >>> >>> On Mon, Mar 21, 2016 at 1:29 PM, Nirmal Fernando <nir...@wso2.com> >>> wrote: >>> >>>> Thanks. >>>> >>>> On Mon, Mar 21, 2016 at 1:27 PM, Anurag Kashyap <anurag1...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> I've changed the settings. You should be able to comment or edit the >>>>> proposa >>>>> <https://docs.google.com/document/d/1L132ApiOm4m0DbyVZ-TYCedsRrt7oE_cgOzAuScvFiQ/edit?usp=sharing>l >>>>> now. >>>>> Thanks >>>>> >>>>> On Mon, Mar 21, 2016 at 3:52 PM, Nirmal Fernando <nir...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Anurag, >>>>>> >>>>>> Thanks for your proposal. Can you please give us 'commenting' right? >>>>>> >>>>>> On Mon, Mar 21, 2016 at 1:03 PM, Anurag Kashyap <anurag1...@gmail.com >>>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> Thank you so much for the valuable input. This >>>>>>> <https://docs.google.com/document/d/1L132ApiOm4m0DbyVZ-TYCedsRrt7oE_cgOzAuScvFiQ/edit?usp=sharing> >>>>>>> is a tentative draft of my proposal. >>>>>>> >>>>>>> Regards, >>>>>>> Anurag >>>>>>> >>>>>>> On Mon, Mar 21, 2016 at 1:03 PM, Supun Sethunga <sup...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Anurag, >>>>>>>> >>>>>>>> Have you started on the project proposal? If so, could you please >>>>>>>> share it (the draft) with us? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> On Mon, Mar 21, 2016 at 10:03 AM, Nirmal Fernando <nir...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 21, 2016 at 10:01 AM, Supun Sethunga <sup...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Anurag, >>>>>>>>>> >>>>>>>>>> Can you help me figure out the significance of the >>>>>>>>>>> serialVersionUID field? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> All the algorithm classes including the one you mentioned >>>>>>>>>> implement the Serializable interface (Because these generated models >>>>>>>>>> can be >>>>>>>>>> exported and imported as java objects). "serialVersionUID" is >>>>>>>>>> used when serializing and deserializing the objects. If we do not >>>>>>>>>> define a "serialVersionUID", >>>>>>>>>> then java runtime will generate them during serialization and >>>>>>>>>> deserialization. And if these two serialVersionUID's does not >>>>>>>>>> matches, then the deserialization fails. As a result, the java >>>>>>>>>> obje
Re: [Dev] Getting "Program too complex: too big jump offset" error in a jaggery file
Thanks all for the suggestions!
it indeed is a limitation with js/rhino. Refactoring solves the issue.
Thanks,
Supun
On Thu, Mar 24, 2016 at 3:13 PM, Nipuna Chandradasa <nipu...@wso2.com>
wrote:
> Hi,
>
> I have come across this issue before. this is because of the jaggery file
> size is exceeding the max size that rhino engine can compile that is 70kb
> (Not sure whether this is the correct size) as i know. So the solution is
> either modularize your code in to several files and combined them together
> using include() or as Dakshika mentioned use already existing java methods.
>
> Regards,
>
> On Thu, Mar 24, 2016 at 2:34 PM, Dakshika Jayathilaka <daksh...@wso2.com>
> wrote:
>
>> Hi,
>>
>> IMO you need to write your main logic on java and use it on jaggery
>> file[1].
>>
>> [1]
>> http://madhukaudantha.blogspot.com/2014/02/using-java-method-from-jaggery-file.html
>>
>> Regards,
>>
>> *Dakshika Jayathilaka*
>> PMC Member & Committer of Apache Stratos
>> Senior Software Engineer
>> WSO2, Inc.
>> lean.enterprise.middleware
>> 0771100911
>>
>> On Wed, Mar 23, 2016 at 4:58 PM, Akalanka Pagoda Arachchi <
>> darsha...@wso2.com> wrote:
>>
>>> This does look like a bug [1], however, is also due to reaching the
>>> limit of instruction size. Few suggestions are there in [2], may be you can
>>> try them out.
>>>
>>> [1] - https://bugzilla.mozilla.org/show_bug.cgi?id=331837
>>> [2] -
>>> http://community.sahipro.com/forums/discussion/1316/funtion-too-long-getting-compilation-error
>>>
>>> Thanks,
>>> Akalanka.
>>>
>>> On Wed, Mar 23, 2016 at 4:44 PM, Supun Sethunga <sup...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a jag file containing ~1900 LOC [1]. To that, I added few more
>>>> stringify() calls. At one point, the jag app throws the below exception. If
>>>> I remove any of the stringify() methods, it works fine. (This error is
>>>> thrown for other functions such as well, like array looping or accessing
>>>> array elements and etc).
>>>>
>>>> Wonder whether there is a limit of instructions/logics per single jag
>>>> file?
>>>>
>>>> Appreciate any help/workaround on this.
>>>>
>>>>
>>>> *Stack trace:*
>>>>
>>>> [2016-03-23 16:32:13,498] ERROR
>>>> {org.jaggeryjs.jaggery.core.manager.WebAppManager} -
>>>> org.mozilla.javascript.WrappedException: Wrapped
>>>> org.jaggeryjs.scriptengine.exceptions.ScriptException:
>>>> org.mozilla.javascript.WrappedException: Wrapped
>>>> org.jaggeryjs.scriptengine.exceptions.ScriptException:
>>>> org.jaggeryjs.scriptengine.exceptions.ScriptException:
>>>> org.mozilla.javascript.EvaluatorException: Encountered code generation
>>>> error while compiling function "null": Program too complex: too big jump
>>>> offset (/portal/controllers/apis/esbanalytics.jag#2)
>>>> (/portal/routers/apis.jag#22) (/portal/routers/tenant.jag#67)
>>>> org.jaggeryjs.scriptengine.exceptions.ScriptException:
>>>> org.mozilla.javascript.WrappedException: Wrapped
>>>> org.jaggeryjs.scriptengine.exceptions.ScriptException:
>>>> org.mozilla.javascript.WrappedException: Wrapped
>>>> org.jaggeryjs.scriptengine.exceptions.ScriptException:
>>>> org.jaggeryjs.scriptengine.exceptions.ScriptException:
>>>> org.mozilla.javascript.EvaluatorException: Encountered code generation
>>>> error while compiling function "null": Program too complex: too big jump
>>>> offset (/portal/controllers/apis/esbanalytics.jag#2)
>>>> (/portal/routers/apis.jag#22) (/portal/routers/tenant.jag#67)
>>>> at
>>>> org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:571)
>>>> at
>>>> org.jaggeryjs.scriptengine.engine.RhinoEngine.exec(RhinoEngine.java:273)
>>>> at
>>>> org.jaggeryjs.jaggery.core.manager.WebAppManager.exec(WebAppManager.java:587)
>>>> at
>>>> org.jaggeryjs.jaggery.core.manager.WebAppManager.execute(WebAppManager.java:507)
>>>> at
>>>> org.jaggeryjs.jaggery.core.JaggeryServlet.doGet(JaggeryServlet.java:24)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>>
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Deng, Looks good!. Please go ahead and submit to GSoC. Regards, Supun On Thu, Mar 24, 2016 at 10:47 AM, tbc.dengwe...@outlook.com < tbc.dengwe...@outlook.com> wrote: > Hi Supun, > > Thansk for advise. I have added some figures. Have a look. > > https://drive.google.com/file/d/0BwujRZWY9rKPemVkbVYwTmd6bGM/view?usp=sharing > thanks > Deng > > -- > tbc.dengwe...@outlook.com > > > *From:* Supun Sethunga <sup...@wso2.com> > *Date:* 2016-03-23 22:11 > *To:* tbc.dengwe...@outlook.com > *CC:* WSO2 Developers' List <dev@wso2.org> > *Subject:* Re: Re: [GSOC2016]Proposal 4: [ML] Ensemble Methods Support > for WSO2 Machine Learner > Hi Deng, > > Couple of comments: > >- Can you add sort of an architecture diagram, explaining your >solution? (which visually explains implementation in a higher level) >- >- In the Road map, It makes more sense to mention the starting-date >and end-date for each of the phases (instead of the duration). Also you can >add time-slots for other steps (like doing the background research, code >cleanup, documenting and etc..) too, to the same timeline. > > Rest of the proposal looks good to me. :) > > Thanks, > Supun > > > On Wed, Mar 23, 2016 at 6:57 PM, tbc.dengwe...@outlook.com < > tbc.dengwe...@outlook.com> wrote: > >> Hello supuns， >> I have modified my modified my proposal. Please check it. The link is >> below: >> https://drive.google.com/open?id=0BwujRZWY9rKPemVkbVYwTmd6bGM >> thanks >> Deng. >> >> -- >> tbc.dengwe...@outlook.com >> >> >> *From:* Supun Sethunga <sup...@wso2.com> >> *Date:* 2016-03-23 02:35 >> *To:* tbcTobeContinued <tbc.dengwe...@outlook.com> >> *CC:* Nirmal Fernando <nir...@wso2.com>; WSO2 Developers' List >> <dev@wso2.org> >> *Subject:* Re: [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for >> WSO2 Machine Learner >> Hi, >> >> Thanks for your interest and the proposal. In the "Proposal 4: [ML] >> Ensemble Methods Support for WSO2 Machine Learner", we are focusing on the >> three methods: Bagging, Boosting and Stacking. You may not need to >> implement several techniques for Boosting, but rather one technique for >> each of the above three methods. >> >> Can you update the proposal accordingly? >> >> Thanks, >> Supun >> >> On Tue, Mar 22, 2016 at 8:30 PM, Nirmal Fernando <nir...@wso2.com> wrote: >> >>> Thanks >>> >>> On Tue, Mar 22, 2016 at 8:22 PM, tbcTobeContinued < >>> tbc.dengwe...@outlook.com> wrote: >>> >>>> Sorry,here is the link: >>>> >>>> https://drive.google.com/file/d/0BwujRZWY9rKPemVkbVYwTmd6bGM/view?usp=sharing >>>> >>>> 在 2016年3月22日，下午10:40，Nirmal Fernando <nir...@wso2.com> 写道： >>>> >>>> Thanks for the proposal Deng, we'll check and provide feedback. It'll >>>> be easier, if you could share a google document with us, with comment >>>> rights. >>>> >>>> On Tue, Mar 22, 2016 at 7:42 PM, tbc.dengwe...@outlook.com < >>>> tbc.dengwe...@outlook.com> wrote: >>>> >>>>> Hi,all >>>>> I want to implemet AbaBoost for WSO2 Machine Leaner. here is my >>>>> proposal, please give some feedback to improve it. >>>>> Thanks >>>>> wicky >>>>> >>>>> ------ >>>>> tbc.dengwe...@outlook.com >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> >> > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Looks good! Please go ahead and submit to GSoC. Thanks, Supun On Thu, Mar 24, 2016 at 4:02 AM, Misgana Negassi < negas...@tf.uni-freiburg.de> wrote: > Hi Supun, > > I have added the changes you recommended. Could you kindly give me a > feedback? > > Best, > Misgana > > On 23.03.2016 15:04, Supun Sethunga wrote: > > Hi Misgana, > > I went through your proposal. Overall it looks good. Here are a few > comments I would like to point out: > >- Its better to have some sort of an architecture diagram, explaining >your solution in a higher level. >- In the timeline, better to break down the "Week 13 (May 23 June >20, 2016)" into three sub-levels, and allocate timeslots for each of the >three methods (Stacking, Boosting and Bagging) separately. That would make >it easy for you to work on those methods separately, as well as to track >the progress. >- In the timeline, can you double check the "week" numbers..? for eg; >in [*Week 1-3 (May 23 June 20, 2016*], I guess it should be "*Week >1-4*" (there are four weeks in the mentioned duration). Similarly, >check the others too. > > Please share us the draft proposal once you fix those. > > Thanks, > Supun > > On Wed, Mar 23, 2016 at 7:17 PM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> Hi Supun, >> >> I am attaching my proposal draft. I am very grateful for your comments. >> >> Thanks, >> Misgana >> >> >> On 23.03.2016 04:54, Supun Sethunga wrote: >> >> Hi Misgana, >> >> As we have mentioned in the project proposal as well, the main objective >> is to integrate ensemble support for the existing flow of the WSO2 Machine >> Learner. We are focusing on the three methods: Bagging, Boosting and >> Stacking. (On technique per each of these methods) >> >> If you haven't tried out already, you can get to know the Machine >> Learner product by downloading it and running it (Please use link [1] to >> download). Official documentation [2] and blog [3] will help you on how to >> use the product. You can also go through the source code of WSO2 ML ([4] >> and [5]), and get familiarized with the current implementations. >> >> Meantime, as Nirmal mentioned, can you please send us the draft of the >> proposal so that we can review it and give you a feedback? >> >> [1] <http://wso2.com/products/machine-learner/> >> http://wso2.com/products/machine-learner/ >> [2] <https://docs.wso2.com/display/ML100/Introducing+Machine+Learner> >> https://docs.wso2.com/display/ML100/Introducing+Machine+Learner >> [3] >> <http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html> >> http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html >> [4] <https://github.com/wso2/carbon-ml>https://github.com/wso2/carbon-ml >> [5] <https://github.com/wso2/product-ml> >> https://github.com/wso2/product-ml >> >> Thanks, >> Supun >> >> On Wed, Mar 23, 2016 at 7:20 AM, Nirmal Fernando < <nir...@wso2.com> >> nir...@wso2.com> wrote: >> >>> Thanks, Misgana for your interest in a WSO2 ML GSoC project. Whilst I >>> let Supun give you some more information on the project, I encourage you to >>> create a draft proposal and send us for review. >>> >>> On Wed, Mar 23, 2016 at 2:58 AM, Misgana Negassi < >>> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >>> >>>> Hallo! >>>> >>>> I am Misgana, hailing from Freiburg, Germany and I am interested in >>>> working with you on the Ensemble methods . I have already implemented >>>> Stacking in python(code available in github/zemoel) and compared it to >>>> other ensemble methods such as Ensemble Selection on AUC performance >>>> measures. The comparison also included using above mentioned methods as >>>> part of an automated machine learning platform(Autosklearn). >>>> >>>> I am currently working on my proposal and would be grateful for your >>>> reply. >>>> >>>> Misgana >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: <http://nirmalfdo.blogspot.com/>http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> >> >> > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > <http://wso2.com/>http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Misgana, I went through your proposal. Overall it looks good. Here are a few comments I would like to point out: - Its better to have some sort of an architecture diagram, explaining your solution in a higher level. - In the timeline, better to break down the "Week 13 (May 23 June 20, 2016)" into three sub-levels, and allocate timeslots for each of the three methods (Stacking, Boosting and Bagging) separately. That would make it easy for you to work on those methods separately, as well as to track the progress. - In the timeline, can you double check the "week" numbers..? for eg; in [*Week 1-3 (May 23 June 20, 2016*], I guess it should be "*Week 1-4*" (there are four weeks in the mentioned duration). Similarly, check the others too. Please share us the draft proposal once you fix those. Thanks, Supun On Wed, Mar 23, 2016 at 7:17 PM, Misgana Negassi < negas...@tf.uni-freiburg.de> wrote: > Hi Supun, > > I am attaching my proposal draft. I am very grateful for your comments. > > Thanks, > Misgana > > > On 23.03.2016 04:54, Supun Sethunga wrote: > > Hi Misgana, > > As we have mentioned in the project proposal as well, the main objective > is to integrate ensemble support for the existing flow of the WSO2 Machine > Learner. We are focusing on the three methods: Bagging, Boosting and > Stacking. (On technique per each of these methods) > > If you haven't tried out already, you can get to know the Machine > Learner product by downloading it and running it (Please use link [1] to > download). Official documentation [2] and blog [3] will help you on how to > use the product. You can also go through the source code of WSO2 ML ([4] > and [5]), and get familiarized with the current implementations. > > Meantime, as Nirmal mentioned, can you please send us the draft of the > proposal so that we can review it and give you a feedback? > > [1] <http://wso2.com/products/machine-learner/> > http://wso2.com/products/machine-learner/ > [2] <https://docs.wso2.com/display/ML100/Introducing+Machine+Learner> > https://docs.wso2.com/display/ML100/Introducing+Machine+Learner > [3] > <http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html> > http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html > [4] <https://github.com/wso2/carbon-ml>https://github.com/wso2/carbon-ml > [5] <https://github.com/wso2/product-ml> > https://github.com/wso2/product-ml > > Thanks, > Supun > > On Wed, Mar 23, 2016 at 7:20 AM, Nirmal Fernando <nir...@wso2.com> wrote: > >> Thanks, Misgana for your interest in a WSO2 ML GSoC project. Whilst I let >> Supun give you some more information on the project, I encourage you to >> create a draft proposal and send us for review. >> >> On Wed, Mar 23, 2016 at 2:58 AM, Misgana Negassi < >> <negas...@tf.uni-freiburg.de>negas...@tf.uni-freiburg.de> wrote: >> >>> Hallo! >>> >>> I am Misgana, hailing from Freiburg, Germany and I am interested in >>> working with you on the Ensemble methods . I have already implemented >>> Stacking in python(code available in github/zemoel) and compared it to >>> other ensemble methods such as Ensemble Selection on AUC performance >>> measures. The comparison also included using above mentioned methods as >>> part of an automated machine learning platform(Autosklearn). >>> >>> I am currently working on my proposal and would be grateful for your >>> reply. >>> >>> Misgana >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > <http://wso2.com/>http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Deng, Couple of comments: - Can you add sort of an architecture diagram, explaining your solution? (which visually explains implementation in a higher level) - - In the Road map, It makes more sense to mention the starting-date and end-date for each of the phases (instead of the duration). Also you can add time-slots for other steps (like doing the background research, code cleanup, documenting and etc..) too, to the same timeline. Rest of the proposal looks good to me. :) Thanks, Supun On Wed, Mar 23, 2016 at 6:57 PM, tbc.dengwe...@outlook.com < tbc.dengwe...@outlook.com> wrote: > Hello supuns， > I have modified my modified my proposal. Please check it. The link is > below: > https://drive.google.com/open?id=0BwujRZWY9rKPemVkbVYwTmd6bGM > thanks > Deng. > > -- > tbc.dengwe...@outlook.com > > > *From:* Supun Sethunga <sup...@wso2.com> > *Date:* 2016-03-23 02:35 > *To:* tbcTobeContinued <tbc.dengwe...@outlook.com> > *CC:* Nirmal Fernando <nir...@wso2.com>; WSO2 Developers' List > <dev@wso2.org> > *Subject:* Re: [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for > WSO2 Machine Learner > Hi, > > Thanks for your interest and the proposal. In the "Proposal 4: [ML] > Ensemble Methods Support for WSO2 Machine Learner", we are focusing on the > three methods: Bagging, Boosting and Stacking. You may not need to > implement several techniques for Boosting, but rather one technique for > each of the above three methods. > > Can you update the proposal accordingly? > > Thanks, > Supun > > On Tue, Mar 22, 2016 at 8:30 PM, Nirmal Fernando <nir...@wso2.com> wrote: > >> Thanks >> >> On Tue, Mar 22, 2016 at 8:22 PM, tbcTobeContinued < >> tbc.dengwe...@outlook.com> wrote: >> >>> Sorry,here is the link: >>> >>> https://drive.google.com/file/d/0BwujRZWY9rKPemVkbVYwTmd6bGM/view?usp=sharing >>> >>> 在 2016年3月22日，下午10:40，Nirmal Fernando <nir...@wso2.com> 写道： >>> >>> Thanks for the proposal Deng, we'll check and provide feedback. It'll be >>> easier, if you could share a google document with us, with comment rights. >>> >>> On Tue, Mar 22, 2016 at 7:42 PM, tbc.dengwe...@outlook.com < >>> tbc.dengwe...@outlook.com> wrote: >>> >>>> Hi,all >>>> I want to implemet AbaBoost for WSO2 Machine Leaner. here is my >>>> proposal, please give some feedback to improve it. >>>> Thanks >>>> wicky >>>> >>>> -- >>>> tbc.dengwe...@outlook.com >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Getting "Program too complex: too big jump offset" error in a jaggery file
er.java:148) at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126) ... 70 more Caused by: org.jaggeryjs.scriptengine.exceptions.ScriptException: org.mozilla.javascript.EvaluatorException: Encountered code generation error while compiling function "null": Program too complex: too big jump offset (/portal/controllers/apis/esbanalytics.jag#2) at org.jaggeryjs.scriptengine.cache.CacheManager.cacheScript(CacheManager.java:91) at org.jaggeryjs.scriptengine.cache.CacheManager.getScriptObject(CacheManager.java:102) at org.jaggeryjs.scriptengine.engine.RhinoEngine.execScript(RhinoEngine.java:562) ... 77 more Caused by: org.mozilla.javascript.EvaluatorException: Encountered code generation error while compiling function "null": Program too complex: too big jump offset (/portal/controllers/apis/esbanalytics.jag#2) at org.mozilla.javascript.DefaultErrorReporter.runtimeError(DefaultErrorReporter.java:77) at org.mozilla.javascript.Context.reportRuntimeError(Context.java:913) at org.mozilla.javascript.optimizer.Codegen.reportClassFileFormatException(Codegen.java:180) at org.mozilla.javascript.optimizer.Codegen.generateCode(Codegen.java:308) at org.mozilla.javascript.optimizer.Codegen.compileToClassFile(Codegen.java:166) at org.mozilla.javascript.optimizer.ClassCompiler.compileToClassFiles(ClassCompiler.java:155) at org.jaggeryjs.scriptengine.cache.CacheManager.cacheScript(CacheManager.java:84) ... 79 more [1] https://github.com/wso2/analytics-esb/blob/master/features/org.wso2.carbon.analytics.esb.feature/src/main/jaggeryapi/esbanalytics.jag Thanks, Supun -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSoC 2016] [ML] Ensemble Methods Support for WSO2 Machine Learner
This Looks good! Please go ahead and submit the final draft to GSoC, on or before Friday :) On Wed, Mar 23, 2016 at 1:50 PM, Hasitha Jayasundara < hasithajayasunda...@gmail.com> wrote: > I made the changes.Really sorry that I had to change the timeline several > times.Since you have requested a rough timeline I only added some main > topics in the previous drafts.Could you please check the new one and add > some comments. > > https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing > > On Tue, Mar 22, 2016 at 11:19 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Coding period (implementations) should be within 23rd May to 23rd August >> :) >> >> Btw, rather than having one slot as "Implement ensemble methods", can you >> break it down to the three methods, and have three separate slots? That way >> it would be easy for you to work on the three methods separately, as well >> as track down the progress.. >> >> Regards, >> Supun >> >> On Mon, Mar 21, 2016 at 4:57 PM, Hasitha Jayasundara < >> hasithajayasunda...@gmail.com> wrote: >> >>> Am I violating google GSoC calendar in my timeline? >>> >>> On Mon, Mar 21, 2016 at 4:47 PM, Hasitha Jayasundara < >>> hasithajayasunda...@gmail.com> wrote: >>> >>>> Made some adjustments.Could you please check >>>> >>>> https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing >>>> >>>> On Mon, Mar 21, 2016 at 4:24 PM, Hasitha Jayasundara < >>>> hasithajayasunda...@gmail.com> wrote: >>>> >>>>> I 'll adjust them >>>>> >>>>> >>>>> On Mon, Mar 21, 2016 at 4:07 PM, Supun Sethunga <sup...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Hasitha, >>>>>> >>>>>> Project timeline should tally with the gsoc timelines [1]. I've added >>>>>> a few comments in the doc itself too. Can you please adjust those? >>>>>> >>>>>> [1] https://developers.google.com/open-source/gsoc/timeline >>>>>> >>>>>> On Mon, Mar 21, 2016 at 3:39 PM, Hasitha Jayasundara < >>>>>> hasithajayasunda...@gmail.com> wrote: >>>>>> >>>>>>> Added timeline >>>>>>> >>>>>>> https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing >>>>>>> >>>>>>> On Mon, Mar 21, 2016 at 10:35 AM, Supun Sethunga <sup...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Hasitha, >>>>>>>> >>>>>>>> Overall looks good to me! Added a few comments, please fix them too. >>>>>>>> >>>>>>>> Also, can you share a rough timeline too? If you feel like the >>>>>>>> current project goals does not fit into the deadlines, we can revise >>>>>>>> the >>>>>>>> goals before starting the project, so that it would fit within the >>>>>>>> timeline. >>>>>>>> >>>>>>>> BTW, please CC to WSO2 Developers mailing list (dev@wso2.org) for >>>>>>>> all project-related discussions. You might have to subscribe via >>>>>>>> http://wso2.com/mail/ >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Supun >>>>>>>> >>>>>>>> On Sun, Mar 20, 2016 at 6:02 PM, Hasitha Jayasundara < >>>>>>>> hasithajayasunda...@gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing >>>>>>>>> Coluld you please check this and give me some comments how can i >>>>>>>>> improve >>>>>>>>> >>>>>>>>> On Wed, Mar 16, 2016 at 9:02 AM, Supun Sethunga <sup...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Hasitha, >>>>>>>>>> >>>>>>>>>> Out of ensembling method available, following are the three main >>>>>>>>>> t
Re: [Dev] Regarding Proposal 6: [ML] Predictive analytics with online data for WSO2 Machine Learner
gt; Hi, >>>> >>>> >>>> https://docs.wso2.com/display/GSoC/Project+Proposals+for+2016#ProjectProposalsfor2016-Proposal6:[ML]PredictiveanalyticswithonlinedataforWSO2MachineLearner >>>> >>>> I'm a 4th year undergraduate from SLIIT faculty of >>>> computing(specializing in Software Engineering) who is interested in doing >>>> the aforementioned project as the final year research. I would like to know >>>> more details on this topic and the procedure to be followed when >>>> implementing it. Please be kind enough to reply as soon as possible. >>>> >>>> Thank you >>>> >>>> -- >>>> Best Regards, >>>> Heshani Herath >>>> >>> >>> >>> >>> -- >>> Pruthuvi Maheshakya Wijewardena >>> mahesha...@wso2.com >>> +94711228855 >>> >>> >>> >> >> >> -- >> Best Regards, >> Heshani Herath >> > > > > -- > Pruthuvi Maheshakya Wijewardena > mahesha...@wso2.com > +94711228855 > > > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Misgana, As we have mentioned in the project proposal as well, the main objective is to integrate ensemble support for the existing flow of the WSO2 Machine Learner. We are focusing on the three methods: Bagging, Boosting and Stacking. (On technique per each of these methods) If you haven't tried out already, you can get to know the Machine Learner product by downloading it and running it (Please use link [1] to download). Official documentation [2] and blog [3] will help you on how to use the product. You can also go through the source code of WSO2 ML ([4] and [5]), and get familiarized with the current implementations. Meantime, as Nirmal mentioned, can you please send us the draft of the proposal so that we can review it and give you a feedback? [1] http://wso2.com/products/machine-learner/ [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner [3] http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html [4] https://github.com/wso2/carbon-ml [5] https://github.com/wso2/product-ml Thanks, Supun On Wed, Mar 23, 2016 at 7:20 AM, Nirmal Fernando <nir...@wso2.com> wrote: > Thanks, Misgana for your interest in a WSO2 ML GSoC project. Whilst I let > Supun give you some more information on the project, I encourage you to > create a draft proposal and send us for review. > > On Wed, Mar 23, 2016 at 2:58 AM, Misgana Negassi < > negas...@tf.uni-freiburg.de> wrote: > >> Hallo! >> >> I am Misgana, hailing from Freiburg, Germany and I am interested in >> working with you on the Ensemble methods . I have already implemented >> Stacking in python(code available in github/zemoel) and compared it to >> other ensemble methods such as Ensemble Selection on AUC performance >> measures. The comparison also included using above mentioned methods as >> part of an automated machine learning platform(Autosklearn). >> >> I am currently working on my proposal and would be grateful for your >> reply. >> >> Misgana >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2016]Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi, Thanks for your interest and the proposal. In the "Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner", we are focusing on the three methods: Bagging, Boosting and Stacking. You may not need to implement several techniques for Boosting, but rather one technique for each of the above three methods. Can you update the proposal accordingly? Thanks, Supun On Tue, Mar 22, 2016 at 8:30 PM, Nirmal Fernando <nir...@wso2.com> wrote: > Thanks > > On Tue, Mar 22, 2016 at 8:22 PM, tbcTobeContinued < > tbc.dengwe...@outlook.com> wrote: > >> Sorry,here is the link: >> >> https://drive.google.com/file/d/0BwujRZWY9rKPemVkbVYwTmd6bGM/view?usp=sharing >> >> 在 2016年3月22日，下午10:40，Nirmal Fernando <nir...@wso2.com> 写道： >> >> Thanks for the proposal Deng, we'll check and provide feedback. It'll be >> easier, if you could share a google document with us, with comment rights. >> >> On Tue, Mar 22, 2016 at 7:42 PM, tbc.dengwe...@outlook.com < >> tbc.dengwe...@outlook.com> wrote: >> >>> Hi,all >>> I want to implemet AbaBoost for WSO2 Machine Leaner. here is my >>> proposal, please give some feedback to improve it. >>> Thanks >>> wicky >>> >>> -- >>> tbc.dengwe...@outlook.com >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] GSoC project for building an ensemble of models
Hi Anurag, Looks good! May be you can also improve the diagram to reflect the description of the three methods (which is right above the diagram). Anyway, please create the final proposal and submit before Friday, 25th. Regards, Supun On Tue, Mar 22, 2016 at 1:52 PM, Anurag Kashyap <anurag1...@gmail.com> wrote: > Hi, > I have made a few changes to the proposal and elaborated a few points. > Kindly give your valuable feedback. > > https://docs.google.com/document/d/1L132ApiOm4m0DbyVZ-TYCedsRrt7oE_cgOzAuScvFiQ/edit?usp=sharing > > Regards, > Anurag Kashyap > > On Mon, Mar 21, 2016 at 4:43 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Anurag, >> >> We have added a few comments. Can you please fix those and share the >> updated one? >> >> Thanks, >> Supun >> >> On Mon, Mar 21, 2016 at 1:29 PM, Nirmal Fernando <nir...@wso2.com> wrote: >> >>> Thanks. >>> >>> On Mon, Mar 21, 2016 at 1:27 PM, Anurag Kashyap <anurag1...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> I've changed the settings. You should be able to comment or edit the >>>> proposa >>>> <https://docs.google.com/document/d/1L132ApiOm4m0DbyVZ-TYCedsRrt7oE_cgOzAuScvFiQ/edit?usp=sharing>l >>>> now. >>>> Thanks >>>> >>>> On Mon, Mar 21, 2016 at 3:52 PM, Nirmal Fernando <nir...@wso2.com> >>>> wrote: >>>> >>>>> Hi Anurag, >>>>> >>>>> Thanks for your proposal. Can you please give us 'commenting' right? >>>>> >>>>> On Mon, Mar 21, 2016 at 1:03 PM, Anurag Kashyap <anurag1...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> Thank you so much for the valuable input. This >>>>>> <https://docs.google.com/document/d/1L132ApiOm4m0DbyVZ-TYCedsRrt7oE_cgOzAuScvFiQ/edit?usp=sharing> >>>>>> is a tentative draft of my proposal. >>>>>> >>>>>> Regards, >>>>>> Anurag >>>>>> >>>>>> On Mon, Mar 21, 2016 at 1:03 PM, Supun Sethunga <sup...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Anurag, >>>>>>> >>>>>>> Have you started on the project proposal? If so, could you please >>>>>>> share it (the draft) with us? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> On Mon, Mar 21, 2016 at 10:03 AM, Nirmal Fernando <nir...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Mar 21, 2016 at 10:01 AM, Supun Sethunga <sup...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Anurag, >>>>>>>>> >>>>>>>>> Can you help me figure out the significance of the >>>>>>>>>> serialVersionUID field? >>>>>>>>> >>>>>>>>> >>>>>>>>> All the algorithm classes including the one you mentioned >>>>>>>>> implement the Serializable interface (Because these generated models >>>>>>>>> can be >>>>>>>>> exported and imported as java objects). "serialVersionUID" is >>>>>>>>> used when serializing and deserializing the objects. If we do not >>>>>>>>> define a "serialVersionUID", >>>>>>>>> then java runtime will generate them during serialization and >>>>>>>>> deserialization. And if these two serialVersionUID's does not >>>>>>>>> matches, then the deserialization fails. As a result, the java object >>>>>>>>> will >>>>>>>>> not be readable. You can check [1] for further details. >>>>>>>>> >>>>>>>>> Also, I wanted to know if I should include personal details too in >>>>>>>>>> the proposal- such as my university and where I'm currently working. >>>>>>>>> >>>>>>>>> You may include. You can check out a sample gsoc proposal by >>>>>>>>> Nirmal (nir...@wso2.com) at [2]. >>>>>>>>> >>>>>>>>>
Re: [Dev] [GSoC 2016] [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Hasitha, Project timeline should tally with the gsoc timelines [1]. I've added a few comments in the doc itself too. Can you please adjust those? [1] https://developers.google.com/open-source/gsoc/timeline On Mon, Mar 21, 2016 at 3:39 PM, Hasitha Jayasundara < hasithajayasunda...@gmail.com> wrote: > Added timeline > > https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing > > On Mon, Mar 21, 2016 at 10:35 AM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Hasitha, >> >> Overall looks good to me! Added a few comments, please fix them too. >> >> Also, can you share a rough timeline too? If you feel like the current >> project goals does not fit into the deadlines, we can revise the goals >> before starting the project, so that it would fit within the timeline. >> >> BTW, please CC to WSO2 Developers mailing list (dev@wso2.org) for all >> project-related discussions. You might have to subscribe via >> http://wso2.com/mail/ >> >> Thanks, >> Supun >> >> On Sun, Mar 20, 2016 at 6:02 PM, Hasitha Jayasundara < >> hasithajayasunda...@gmail.com> wrote: >> >>> >>> https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing >>> Coluld you please check this and give me some comments how can i improve >>> >>> On Wed, Mar 16, 2016 at 9:02 AM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi Hasitha, >>>> >>>> Out of ensembling method available, following are the three main types >>>> that we are interested in: >>>> >>>>- Stacking - Training multiple algorithms (called base-learners) on >>>>the same dataset, and combining them using another algorithm >>>>(meta-learner). >>>>- Bagging - Training a single algorithm over subsets of data. >>>>- Boosting - Training multiple algorithms on the same data, and >>>>combining them over a weighted average (giving higher priority to >>>>misclassified data points). >>>> >>>> You can do some background reading on those three topics to get a good >>>> understanding on ensembling methods. There are good online resources >>>> available. >>>> >>>> or if you can could you please provide me a time to a google hangout? >>>> >>>> Yes sure. Can you please set up a meeting? You can check my google >>>> calendar for free time slots. (I might not be available on 18-20 March) >>>> >>>> P.S: Don't call us sir, just call us by name :) >>>> Also, please CC "dev@wso2.org" mailing list for all project related >>>> emails. >>>> >>>> Regards, >>>> Supun >>>> >>>> On Tue, Mar 15, 2016 at 1:36 AM, Hasitha Jayasundara < >>>> hasithajayasunda...@gmail.com> wrote: >>>> >>>>> Dear Sir, >>>>> >>>>> I have gone through the WSo2 ML algorithms(Linear Regression,Lasso >>>>> regression...)and now i have the idea about how the platform is >>>>> working.Since I am new to Ensembling and there's less resources for >>>>> learning Ensembling,can you provide me some resources or links to learn >>>>> the >>>>> concept Ensembling,or if you can could you please provide me a time to a >>>>> google hangout?Thank you. >>>>> >>>>> On Tue, Mar 8, 2016 at 10:41 AM, Hasitha Jayasundara < >>>>> hasithajayasunda...@gmail.com> wrote: >>>>> >>>>>> Thank you very much sir.I 'll let you know if there's any issue. >>>>>> >>>>>> On Tue, Mar 8, 2016 at 10:03 AM, Supun Sethunga <sup...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> [looping dev] >>>>>>> >>>>>>> On Tue, Mar 8, 2016 at 10:01 AM, Supun Sethunga <sup...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Hasitha, >>>>>>>> >>>>>>>> Thank you for your interest in the above project. As we have >>>>>>>> mentioned in the project proposal as well, the main objective is to >>>>>>>> integrate ensemble support for the existing flow of the WSO2 Machine >>>>>>>> Learner. We are focusing on the three methods: Bagging, Boosting and >
Re: [Dev] GSoC 2016 - Ensemble Methods Support for WSO2 Machine Learner
Hi, Can you please share your proposal with us to have a review before the submission? Thanks, Supun On Mon, Mar 7, 2016 at 5:20 PM, Dilan Tharaka <tharakamd...@cse.mrt.ac.lk> wrote: > thank you for the reply. I'll go through the steps you mentioned and get > back to you if there's any trouble > > On 7 March 2016 at 17:16, Supun Sethunga <sup...@wso2.com> wrote: > >> [Adding WSO2 Developers mailing list] >> >> Hi Dilan, >> >> Thank you for your interest in the above project. As we have mentioned >> in the project proposal as well, the main objective is to integrate >> ensemble support for the existing flow of the WSO2 Machine Learner. We are >> focusing on the three methods: Bagging, Boosting and Stacking. >> >> To start with, you can get to know the Machine Learner product by >> downloading it and running it (Please use link [1] to download). Official >> documentation [2] and blog [3] will help you on how to use the product. As >> the next step, you can go through the source code of WSO2 ML ([4] and [5]). >> >> Please feel free to raise if you have any questions or any unclear points. >> >> [1] http://wso2.com/products/machine-learner/ >> [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner >> [3] >> http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html >> [4] https://github.com/wso2/carbon-ml >> [5] https://github.com/wso2/product-ml >> >> Regards, >> Supun >> >> On Mon, Mar 7, 2016 at 2:19 PM, Dilan Tharaka <tharakamd...@cse.mrt.ac.lk >> > wrote: >> >>> Hi, >>> >>> I'm Dilan Tharaka an undergraduate of University of Moratuwa. I went >>> through WSO2 GSoC ideas list and found the idea "Ensemble Methods Support >>> for WSO2 Machine Learner" is interesting. I have Java, Web development and >>> Unix knowledge. And I've used Spark and Weka libraries in machine leaning. >>> I would be thank full if you could help me in getting started with this >>> project. >>> >>> Thank you, >>> Regards, >>> Dilan Tharaka, >>> Undergraduate, >>> Department of Computer Science and Engineering >>> University of Moratuwa >>> >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > Thank you, > Regards, > Dilan Tharaka, > Undergraduate - batch 12 (Batch Representative), > Department of Computer Science and Engineering > University of Moratuwa > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi, Can you please share your proposal with us to have a review before the submission? Thanks, Supun On Wed, Mar 16, 2016 at 8:38 AM, Supun Sethunga <sup...@wso2.com> wrote: > Hi Dulaj, > > Thank you for your interest in the above project. As we have mentioned in > the project proposal as well, the main objective is to integrate ensemble > support for the existing flow of the WSO2 Machine Learner. We are focusing > on the three methods: Bagging, Boosting and Stacking. > > To start with, you can get to know the Machine Learner product by > downloading it and running it (Please use link [1] to download). Official > documentation [2] and blog [3] will help you on how to use the product. As > the next step, you can go through the source code of WSO2 ML ([4] and [5]), > and get familiarized with the current implementations. > > Please feel free to raise if you have any questions or any unclear points. > > [1] http://wso2.com/products/machine-learner/ > [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner > [3] > http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html > [4] https://github.com/wso2/carbon-ml > [5] https://github.com/wso2/product-ml > > Regards, > Supun > > On Tue, Mar 15, 2016 at 11:28 AM, Dulaj Rajitha < > dulaj.rajitha...@cse.mrt.ac.lk> wrote: > >> *Hi, >> * >> I'm Dulaj Rajitha, 3rd year (Semester 6) undergraduate, university of >> Moratuwa and I found this project is interesting. >> I have working experience in machine learning throughout my internship and >> experience in working on Apache Spark ML and MlLib with java 8. >> So please provide me some guide lines on how to proceed and what are the >> technologies that are required to get familiar with. >> >> --- >> Regards, >> Dulaj Rajitha,*Undergraduate >> **Department of Computer Science and Engineering >> **University of Moratuwa.* >> >> > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] GSoC project for building an ensemble of models
Hi Anurag, We have added a few comments. Can you please fix those and share the updated one? Thanks, Supun On Mon, Mar 21, 2016 at 1:29 PM, Nirmal Fernando <nir...@wso2.com> wrote: > Thanks. > > On Mon, Mar 21, 2016 at 1:27 PM, Anurag Kashyap <anurag1...@gmail.com> > wrote: > >> Hi, >> I've changed the settings. You should be able to comment or edit the >> proposa >> <https://docs.google.com/document/d/1L132ApiOm4m0DbyVZ-TYCedsRrt7oE_cgOzAuScvFiQ/edit?usp=sharing>l >> now. >> Thanks >> >> On Mon, Mar 21, 2016 at 3:52 PM, Nirmal Fernando <nir...@wso2.com> wrote: >> >>> Hi Anurag, >>> >>> Thanks for your proposal. Can you please give us 'commenting' right? >>> >>> On Mon, Mar 21, 2016 at 1:03 PM, Anurag Kashyap <anurag1...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> Thank you so much for the valuable input. This >>>> <https://docs.google.com/document/d/1L132ApiOm4m0DbyVZ-TYCedsRrt7oE_cgOzAuScvFiQ/edit?usp=sharing> >>>> is a tentative draft of my proposal. >>>> >>>> Regards, >>>> Anurag >>>> >>>> On Mon, Mar 21, 2016 at 1:03 PM, Supun Sethunga <sup...@wso2.com> >>>> wrote: >>>> >>>>> Hi Anurag, >>>>> >>>>> Have you started on the project proposal? If so, could you please >>>>> share it (the draft) with us? >>>>> >>>>> Thanks, >>>>> >>>>> On Mon, Mar 21, 2016 at 10:03 AM, Nirmal Fernando <nir...@wso2.com> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Mar 21, 2016 at 10:01 AM, Supun Sethunga <sup...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Anurag, >>>>>>> >>>>>>> Can you help me figure out the significance of the serialVersionUID >>>>>>>> field? >>>>>>> >>>>>>> >>>>>>> All the algorithm classes including the one you mentioned >>>>>>> implement the Serializable interface (Because these generated models >>>>>>> can be >>>>>>> exported and imported as java objects). "serialVersionUID" is used >>>>>>> when serializing and deserializing the objects. If we do not define a >>>>>>> "serialVersionUID", >>>>>>> then java runtime will generate them during serialization and >>>>>>> deserialization. And if these two serialVersionUID's does not >>>>>>> matches, then the deserialization fails. As a result, the java object >>>>>>> will >>>>>>> not be readable. You can check [1] for further details. >>>>>>> >>>>>>> Also, I wanted to know if I should include personal details too in >>>>>>>> the proposal- such as my university and where I'm currently working. >>>>>>> >>>>>>> You may include. You can check out a sample gsoc proposal by Nirmal ( >>>>>>> nir...@wso2.com) at [2]. >>>>>>> >>>>>>> BTW, please CC to WSO2 Developers mailing list (dev@wso2.org) for >>>>>>> all project-related discussions. >>>>>>> >>>>>> >>>>>> You may have to subscribe to dev@, please refer to >>>>>> http://wso2.com/mail/ >>>>>> >>>>>>> >>>>>>> [1] >>>>>>> http://javarevisited.blogspot.com/2014/05/why-use-serialversionuid-inside-serializable-class-in-java.html >>>>>>> [2] http://nirmalfdo.blogspot.com/search/label/GSoC-2011 >>>>>>> >>>>>>> Regards >>>>>>> Supun >>>>>>> >>>>>>> >>>>>>> On Thu, Mar 17, 2016 at 6:26 PM, Anurag Kashyap < >>>>>>> anurag1...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Supun, >>>>>>>> I was going through this >>>>>>>> <https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/RidgeRegression.java> >>>>>>>> link. Can you help me figure out the significance of the >>>>>>>>
Re: [Dev] [GSoC 2016] [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Hasitha, Overall looks good to me! Added a few comments, please fix them too. Also, can you share a rough timeline too? If you feel like the current project goals does not fit into the deadlines, we can revise the goals before starting the project, so that it would fit within the timeline. BTW, please CC to WSO2 Developers mailing list (dev@wso2.org) for all project-related discussions. You might have to subscribe via http://wso2.com/mail/ Thanks, Supun On Sun, Mar 20, 2016 at 6:02 PM, Hasitha Jayasundara < hasithajayasunda...@gmail.com> wrote: > > https://docs.google.com/document/d/1bAJ2DYuaIIzhIWpxyKJl03v2aNRVl9miUg6PvNHyesA/edit?usp=sharing > Coluld you please check this and give me some comments how can i improve > > On Wed, Mar 16, 2016 at 9:02 AM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Hasitha, >> >> Out of ensembling method available, following are the three main types >> that we are interested in: >> >>- Stacking - Training multiple algorithms (called base-learners) on >>the same dataset, and combining them using another algorithm >>(meta-learner). >>- Bagging - Training a single algorithm over subsets of data. >>- Boosting - Training multiple algorithms on the same data, and >>combining them over a weighted average (giving higher priority to >>misclassified data points). >> >> You can do some background reading on those three topics to get a good >> understanding on ensembling methods. There are good online resources >> available. >> >> or if you can could you please provide me a time to a google hangout? >> >> Yes sure. Can you please set up a meeting? You can check my google >> calendar for free time slots. (I might not be available on 18-20 March) >> >> P.S: Don't call us sir, just call us by name :) >> Also, please CC "dev@wso2.org" mailing list for all project related >> emails. >> >> Regards, >> Supun >> >> On Tue, Mar 15, 2016 at 1:36 AM, Hasitha Jayasundara < >> hasithajayasunda...@gmail.com> wrote: >> >>> Dear Sir, >>> >>> I have gone through the WSo2 ML algorithms(Linear Regression,Lasso >>> regression...)and now i have the idea about how the platform is >>> working.Since I am new to Ensembling and there's less resources for >>> learning Ensembling,can you provide me some resources or links to learn the >>> concept Ensembling,or if you can could you please provide me a time to a >>> google hangout?Thank you. >>> >>> On Tue, Mar 8, 2016 at 10:41 AM, Hasitha Jayasundara < >>> hasithajayasunda...@gmail.com> wrote: >>> >>>> Thank you very much sir.I 'll let you know if there's any issue. >>>> >>>> On Tue, Mar 8, 2016 at 10:03 AM, Supun Sethunga <sup...@wso2.com> >>>> wrote: >>>> >>>>> [looping dev] >>>>> >>>>> On Tue, Mar 8, 2016 at 10:01 AM, Supun Sethunga <sup...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Hasitha, >>>>>> >>>>>> Thank you for your interest in the above project. As we have >>>>>> mentioned in the project proposal as well, the main objective is to >>>>>> integrate ensemble support for the existing flow of the WSO2 Machine >>>>>> Learner. We are focusing on the three methods: Bagging, Boosting and >>>>>> Stacking. >>>>>> >>>>>> To start with, you can get to know the Machine Learner product by >>>>>> downloading it and running it (Please use link [1] to download). Official >>>>>> documentation [2] and blog [3] will help you on how to use the product. >>>>>> As >>>>>> the next step, you can go through the source code of WSO2 ML ([4] and >>>>>> [5]), >>>>>> and get familiarized with the current implementations. >>>>>> >>>>>> Please feel free to raise if you have any questions or any unclear >>>>>> points. >>>>>> >>>>>> [1] http://wso2.com/products/machine-learner/ >>>>>> [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner >>>>>> [3] >>>>>> http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html >>>>>> [4] https://github.com/wso2/carbon-ml >>>>>> [5] https://github.com/wso2/product-ml >>>>>> >>>>>> Regards, >>>&
Re: [Dev] GSoC project for building an ensemble of models
Hi Anurag, Have you started on the project proposal? If so, could you please share it (the draft) with us? Thanks, On Mon, Mar 21, 2016 at 10:03 AM, Nirmal Fernando <nir...@wso2.com> wrote: > > > On Mon, Mar 21, 2016 at 10:01 AM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Anurag, >> >> Can you help me figure out the significance of the serialVersionUID field? >> >> >> All the algorithm classes including the one you mentioned implement the >> Serializable interface (Because these generated models can be exported and >> imported as java objects). "serialVersionUID" is used when serializing >> and deserializing the objects. If we do not define a "serialVersionUID", >> then java runtime will generate them during serialization and >> deserialization. And if these two serialVersionUID's does not matches, >> then the deserialization fails. As a result, the java object will not be >> readable. You can check [1] for further details. >> >> Also, I wanted to know if I should include personal details too in the >>> proposal- such as my university and where I'm currently working. >> >> You may include. You can check out a sample gsoc proposal by Nirmal ( >> nir...@wso2.com) at [2]. >> >> BTW, please CC to WSO2 Developers mailing list (dev@wso2.org) for all >> project-related discussions. >> > > You may have to subscribe to dev@, please refer to http://wso2.com/mail/ > >> >> [1] >> http://javarevisited.blogspot.com/2014/05/why-use-serialversionuid-inside-serializable-class-in-java.html >> [2] http://nirmalfdo.blogspot.com/search/label/GSoC-2011 >> >> Regards >> Supun >> >> >> On Thu, Mar 17, 2016 at 6:26 PM, Anurag Kashyap <anurag1...@gmail.com> >> wrote: >> >>> Hi Supun, >>> I was going through this >>> <https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/RidgeRegression.java> >>> link. Can you help me figure out the significance of the serialVersionUID >>> field? If that field is necessary then anonymous classes will have to be >>> implemented instead of lambda expressions in several places. Also, I wanted >>> to know if I should include personal details too in the proposal- such as >>> my university and where I'm currently working. >>> >>> >>> >>> >>> >>> >>> >>> On Mon, Mar 14, 2016 at 5:57 PM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi Anurag, >>>> >>>> Yes, the master branch is using java 8, hence you can use lambda >>>> expressions. >>>> >>>> Regards, >>>> Supun >>>> >>>> On Mon, Mar 14, 2016 at 3:24 PM, Anurag Kashyap <anurag1...@gmail.com> >>>> wrote: >>>> >>>>> Hi Supun, >>>>> One more question. I couldn't find if the version of Java being used >>>>> is Java 8 or a previous version. If it is Java 8, Lambda calculus can be >>>>> used to make some things easier, especially while working with spark. >>>>> Thanks and Regards, >>>>> Anurag Kashyap >>>>> >>>>> >>>>> On Mon, Mar 7, 2016 at 8:05 PM, Supun Sethunga <sup...@wso2.com> >>>>> wrote: >>>>> >>>>>> [Adding WSO2 Developers mailing list] >>>>>> >>>>>> Hi Anurag, >>>>>> >>>>>> Please find my comments inline. >>>>>> >>>>>> First, which boosting model are we going to target-Ada boosting or >>>>>>> some other model, or various models? >>>>>> >>>>>> You can pick any method as you wish. >>>>>> >>>>>> >>>>>> Second, what method would you suggest I use to combine the results of >>>>>>> the models in stacking- is it going to be a single layer LR model? >>>>>> >>>>>> This has to be a user input (i.e. user should be able to pick any >>>>>> algorithm for the base-learners as well as for the meta-learner). >>>>>> >>>>>> Regards, >>>>>> Supun >>>>>> >>>>>> On Mon, Mar 7, 2016 at 5:15 PM, Anurag Kashyap <anurag1...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Supun, >>>>
Re: [Dev] GSoC project for building an ensemble of models
Hi Anurag, Can you help me figure out the significance of the serialVersionUID field? All the algorithm classes including the one you mentioned implement the Serializable interface (Because these generated models can be exported and imported as java objects). "serialVersionUID" is used when serializing and deserializing the objects. If we do not define a "serialVersionUID", then java runtime will generate them during serialization and deserialization. And if these two serialVersionUID's does not matches, then the deserialization fails. As a result, the java object will not be readable. You can check [1] for further details. Also, I wanted to know if I should include personal details too in the > proposal- such as my university and where I'm currently working. You may include. You can check out a sample gsoc proposal by Nirmal ( nir...@wso2.com) at [2]. BTW, please CC to WSO2 Developers mailing list (dev@wso2.org) for all project-related discussions. [1] http://javarevisited.blogspot.com/2014/05/why-use-serialversionuid-inside-serializable-class-in-java.html [2] http://nirmalfdo.blogspot.com/search/label/GSoC-2011 Regards Supun On Thu, Mar 17, 2016 at 6:26 PM, Anurag Kashyap <anurag1...@gmail.com> wrote: > Hi Supun, > I was going through this > <https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/spark/algorithms/RidgeRegression.java> > link. Can you help me figure out the significance of the serialVersionUID > field? If that field is necessary then anonymous classes will have to be > implemented instead of lambda expressions in several places. Also, I wanted > to know if I should include personal details too in the proposal- such as > my university and where I'm currently working. > > > > > > > > On Mon, Mar 14, 2016 at 5:57 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Anurag, >> >> Yes, the master branch is using java 8, hence you can use lambda >> expressions. >> >> Regards, >> Supun >> >> On Mon, Mar 14, 2016 at 3:24 PM, Anurag Kashyap <anurag1...@gmail.com> >> wrote: >> >>> Hi Supun, >>> One more question. I couldn't find if the version of Java being used is >>> Java 8 or a previous version. If it is Java 8, Lambda calculus can be used >>> to make some things easier, especially while working with spark. >>> Thanks and Regards, >>> Anurag Kashyap >>> >>> >>> On Mon, Mar 7, 2016 at 8:05 PM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> [Adding WSO2 Developers mailing list] >>>> >>>> Hi Anurag, >>>> >>>> Please find my comments inline. >>>> >>>> First, which boosting model are we going to target-Ada boosting or some >>>>> other model, or various models? >>>> >>>> You can pick any method as you wish. >>>> >>>> >>>> Second, what method would you suggest I use to combine the results of >>>>> the models in stacking- is it going to be a single layer LR model? >>>> >>>> This has to be a user input (i.e. user should be able to pick any >>>> algorithm for the base-learners as well as for the meta-learner). >>>> >>>> Regards, >>>> Supun >>>> >>>> On Mon, Mar 7, 2016 at 5:15 PM, Anurag Kashyap <anurag1...@gmail.com> >>>> wrote: >>>> >>>>> Hi Supun, >>>>> I've begun reading the source rom link 3. I had a couple of >>>>> questions. First, which boosting model are we going to target-Ada boosting >>>>> or some other model, or various models? Second, what method would you >>>>> suggest I use to combine the results of the models in stacking- is it >>>>> going >>>>> to be a single layer LR model? >>>>> >>>>> >>>>> >>>>> On Sun, Mar 6, 2016 at 6:25 PM, Supun Sethunga <sup...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Anurag, >>>>>> >>>>>> Thank you for your interest in the above project. You can start by >>>>>> getting to know the Machine Learner product by downloading it and running >>>>>> it (Please use link [1] to download). Official documentation [2] will >>>>>> help >>>>>> you on using the product. You can find the source code base at [3] and >>>>>> [4]. >>>>>> As we have mentioned in the project proposal as wel
Re: [Dev] [GSoC 2016] [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Hasitha, Out of ensembling method available, following are the three main types that we are interested in: - Stacking - Training multiple algorithms (called base-learners) on the same dataset, and combining them using another algorithm (meta-learner). - Bagging - Training a single algorithm over subsets of data. - Boosting - Training multiple algorithms on the same data, and combining them over a weighted average (giving higher priority to misclassified data points). You can do some background reading on those three topics to get a good understanding on ensembling methods. There are good online resources available. or if you can could you please provide me a time to a google hangout? Yes sure. Can you please set up a meeting? You can check my google calendar for free time slots. (I might not be available on 18-20 March) P.S: Don't call us sir, just call us by name :) Also, please CC "dev@wso2.org" mailing list for all project related emails. Regards, Supun On Tue, Mar 15, 2016 at 1:36 AM, Hasitha Jayasundara < hasithajayasunda...@gmail.com> wrote: > Dear Sir, > > I have gone through the WSo2 ML algorithms(Linear Regression,Lasso > regression...)and now i have the idea about how the platform is > working.Since I am new to Ensembling and there's less resources for > learning Ensembling,can you provide me some resources or links to learn the > concept Ensembling,or if you can could you please provide me a time to a > google hangout?Thank you. > > On Tue, Mar 8, 2016 at 10:41 AM, Hasitha Jayasundara < > hasithajayasunda...@gmail.com> wrote: > >> Thank you very much sir.I 'll let you know if there's any issue. >> >> On Tue, Mar 8, 2016 at 10:03 AM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> [looping dev] >>> >>> On Tue, Mar 8, 2016 at 10:01 AM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi Hasitha, >>>> >>>> Thank you for your interest in the above project. As we have mentioned >>>> in the project proposal as well, the main objective is to integrate >>>> ensemble support for the existing flow of the WSO2 Machine Learner. We are >>>> focusing on the three methods: Bagging, Boosting and Stacking. >>>> >>>> To start with, you can get to know the Machine Learner product by >>>> downloading it and running it (Please use link [1] to download). Official >>>> documentation [2] and blog [3] will help you on how to use the product. As >>>> the next step, you can go through the source code of WSO2 ML ([4] and [5]), >>>> and get familiarized with the current implementations. >>>> >>>> Please feel free to raise if you have any questions or any unclear >>>> points. >>>> >>>> [1] http://wso2.com/products/machine-learner/ >>>> [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner >>>> [3] >>>> http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html >>>> [4] https://github.com/wso2/carbon-ml >>>> [5] https://github.com/wso2/product-ml >>>> >>>> Regards, >>>> Supun >>>> >>>> On Tue, Mar 8, 2016 at 9:53 AM, Hasitha Jayasundara < >>>> hasithajayasunda...@gmail.com> wrote: >>>> >>>>> Dear Sir, >>>>> >>>>> I am an undergraduate of University of Moratuwa department of >>>>> Electronic and Telecommunication Engineering. I am very much interested in >>>>> machine learning knowledge and i would like to start the project' Ensemble >>>>> Methods Support for WSO2 Machine Learner'.So please provide me some guide >>>>> lines and materials for study and get a clear understanding about the >>>>> mentioned project. >>>>> >>>>> Thank you >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> *Supun Sethunga* >>>> Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Proposal 4: [ML] Ensemble Methods Support for WSO2 Machine Learner
Hi Dulaj, Thank you for your interest in the above project. As we have mentioned in the project proposal as well, the main objective is to integrate ensemble support for the existing flow of the WSO2 Machine Learner. We are focusing on the three methods: Bagging, Boosting and Stacking. To start with, you can get to know the Machine Learner product by downloading it and running it (Please use link [1] to download). Official documentation [2] and blog [3] will help you on how to use the product. As the next step, you can go through the source code of WSO2 ML ([4] and [5]), and get familiarized with the current implementations. Please feel free to raise if you have any questions or any unclear points. [1] http://wso2.com/products/machine-learner/ [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner [3] http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html [4] https://github.com/wso2/carbon-ml [5] https://github.com/wso2/product-ml Regards, Supun On Tue, Mar 15, 2016 at 11:28 AM, Dulaj Rajitha < dulaj.rajitha...@cse.mrt.ac.lk> wrote: > *Hi, > * > I'm Dulaj Rajitha, 3rd year (Semester 6) undergraduate, university of > Moratuwa and I found this project is interesting. > I have working experience in machine learning throughout my internship and > experience in working on Apache Spark ML and MlLib with java 8. > So please provide me some guide lines on how to proceed and what are the > technologies that are required to get familiar with. > > --- > Regards, > Dulaj Rajitha,*Undergraduate > **Department of Computer Science and Engineering > **University of Moratuwa.* > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] GSoC project for building an ensemble of models
Hi Anurag, Yes, the master branch is using java 8, hence you can use lambda expressions. Regards, Supun On Mon, Mar 14, 2016 at 3:24 PM, Anurag Kashyap <anurag1...@gmail.com> wrote: > Hi Supun, > One more question. I couldn't find if the version of Java being used is > Java 8 or a previous version. If it is Java 8, Lambda calculus can be used > to make some things easier, especially while working with spark. > Thanks and Regards, > Anurag Kashyap > > > On Mon, Mar 7, 2016 at 8:05 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> [Adding WSO2 Developers mailing list] >> >> Hi Anurag, >> >> Please find my comments inline. >> >> First, which boosting model are we going to target-Ada boosting or some >>> other model, or various models? >> >> You can pick any method as you wish. >> >> >> Second, what method would you suggest I use to combine the results of the >>> models in stacking- is it going to be a single layer LR model? >> >> This has to be a user input (i.e. user should be able to pick any >> algorithm for the base-learners as well as for the meta-learner). >> >> Regards, >> Supun >> >> On Mon, Mar 7, 2016 at 5:15 PM, Anurag Kashyap <anurag1...@gmail.com> >> wrote: >> >>> Hi Supun, >>> I've begun reading the source rom link 3. I had a couple of questions. >>> First, which boosting model are we going to target-Ada boosting or some >>> other model, or various models? Second, what method would you suggest I use >>> to combine the results of the models in stacking- is it going to be a >>> single layer LR model? >>> >>> >>> >>> On Sun, Mar 6, 2016 at 6:25 PM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi Anurag, >>>> >>>> Thank you for your interest in the above project. You can start by >>>> getting to know the Machine Learner product by downloading it and running >>>> it (Please use link [1] to download). Official documentation [2] will help >>>> you on using the product. You can find the source code base at [3] and [4]. >>>> As we have mentioned in the project proposal as well, the main >>>> objective is to integrate ensemble support for the existing flow of the >>>> WSO2 Machine Learner. We are focusing on the three methods: Bagging, >>>> Boosting and Stacking. >>>> >>>> Please feel free to raise, if you have any questions or any unclear >>>> points. >>>> >>>> [1] http://wso2.com/products/machine-learner/ >>>> [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner >>>> [3] https://github.com/wso2/carbon-ml >>>> [4] https://github.com/wso2/product-ml >>>> >>>> Regards, >>>> Supun >>>> >>>> On Sun, Mar 6, 2016 at 12:08 PM, Anurag Kashyap <anurag1...@gmail.com> >>>> wrote: >>>> >>>>> Hi Supun, >>>>> I'm currently working as a research assistant at NTU, Singapore for >>>>> my bachelor thesis in my final year, I looked at the list of GSoC >>>>> projects for WSO2 and I'm particularly interested in the project *Ensemble >>>>> Methods Support for WSO2 Machine Learner.* I am currently working on >>>>> Ensemble methods for multi-label classification, so I have some experience >>>>> in this field and I've read several scholarly articles about their >>>>> implementation and benefit. I believe I can contribute to this project as >>>>> a >>>>> GSoC intern. Eagerly awaiting a response! >>>>> >>>>> Yours Sincerely, >>>>> *Anurag Kashyap* >>>>> *Research Assistant* >>>>> *Robotics-|| * >>>>> *Nanyang Technological University, Singapore* >>>>> >>>>> [image: Blog] <https://thelinuxkernel.wordpress.com> >>>>> >>>> >>>> >>>> >>>> -- >>>> *Supun Sethunga* >>>> Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> >>> >>> >>> >>> -- >>> Yours sincerely, >>> >>> *Anurag Kashyap* >>> *MSc.(Tech.)* Information Systems, >>> *BITS Pilani K K Birla Goa Campus* >>> >>> [image: Blog] <https://thelinuxkernel.wordpress.com> >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > Yours sincerely, > > *Anurag Kashyap* > *MSc.(Tech.)* Information Systems, > *BITS Pilani K K Birla Goa Campus* > > [image: Blog] <https://thelinuxkernel.wordpress.com> > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC'16] Proposal 3 : Visual (Deep) Neural Network Builder
Hi Vimuth, For project Proposal 3, as we have mentioned in the project proposal as well, the main objective is to integrate ensemble support for the existing flow of the WSO2 Machine Learner. We are focusing on the three methods: Bagging, Boosting and Stacking. Since you have already run the product ML, you can also go through the source code of WSO2 ML ([1] and [2]), and get familiarized with the current implementations. Please feel free to raise if you have any questions or any unclear points. [1] https://github.com/wso2/carbon-ml [2] https://github.com/wso2/product-ml Regards, Supun On Wed, Mar 9, 2016 at 11:31 AM, Isuru Udana <isu...@wso2.com> wrote: > [Looping project mentors] > > On Wed, Mar 9, 2016 at 11:11 AM, vimuth fernando <vim...@gmail.com> wrote: > >> Hi, >> >> I am a full time MSc student in the Dept. of Computer Science at >> University of Moratuwa. I am thinking about taking part in GSOC this year >> and saw that WSO2 has some interesting project ideas especially in the >> Machine Learner project. >> >> I am interested in working on either Proposal 3 (Visual Neural Network >> Builder) or Proposal 4 (Ensemble Methods Support for WSO2 Machine Learner). >> I am open to working on either project. >> >> In Proposal 3, am i correct in understanding that the goal is to build >> the UI that will use the algorithms implemented in spark? >> >> I have started to familiarize myself with the WSO2 Machine Learner using >> the samples provided with the documentation. >> >> Your help in preparing a good proposal is much appreciated. >> >> Thank you very much. >> >> -- >> Vimuth Dinuka Fernando >> Research Assistant, Department Of Computer Science and Engineering, >> University Of Moratuwa, >> Sri Lanka >> >> phone: 0777559890 >> >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Isuru Udana* > Associate Technical Lead > WSO2 Inc.; http://wso2.com > email: isu...@wso2.com cell: +94 77 3791887 > blog: http://mytecheye.blogspot.com/ > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSoC 2016] [ML] Ensemble Methods Support for WSO2 Machine Learner
[looping dev] On Tue, Mar 8, 2016 at 10:01 AM, Supun Sethunga <sup...@wso2.com> wrote: > Hi Hasitha, > > Thank you for your interest in the above project. As we have mentioned in > the project proposal as well, the main objective is to integrate ensemble > support for the existing flow of the WSO2 Machine Learner. We are focusing > on the three methods: Bagging, Boosting and Stacking. > > To start with, you can get to know the Machine Learner product by > downloading it and running it (Please use link [1] to download). Official > documentation [2] and blog [3] will help you on how to use the product. As > the next step, you can go through the source code of WSO2 ML ([4] and [5]), > and get familiarized with the current implementations. > > Please feel free to raise if you have any questions or any unclear points. > > [1] http://wso2.com/products/machine-learner/ > [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner > [3] > http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html > [4] https://github.com/wso2/carbon-ml > [5] https://github.com/wso2/product-ml > > Regards, > Supun > > On Tue, Mar 8, 2016 at 9:53 AM, Hasitha Jayasundara < > hasithajayasunda...@gmail.com> wrote: > >> Dear Sir, >> >> I am an undergraduate of University of Moratuwa department of Electronic >> and Telecommunication Engineering. I am very much interested in machine >> learning knowledge and i would like to start the project' Ensemble Methods >> Support for WSO2 Machine Learner'.So please provide me some guide lines and >> materials for study and get a clear understanding about the mentioned >> project. >> >> Thank you >> >> > > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] GSoC project for building an ensemble of models
[Adding WSO2 Developers mailing list] Hi Anurag, Please find my comments inline. First, which boosting model are we going to target-Ada boosting or some > other model, or various models? You can pick any method as you wish. Second, what method would you suggest I use to combine the results of the > models in stacking- is it going to be a single layer LR model? This has to be a user input (i.e. user should be able to pick any algorithm for the base-learners as well as for the meta-learner). Regards, Supun On Mon, Mar 7, 2016 at 5:15 PM, Anurag Kashyap <anurag1...@gmail.com> wrote: > Hi Supun, > I've begun reading the source rom link 3. I had a couple of questions. > First, which boosting model are we going to target-Ada boosting or some > other model, or various models? Second, what method would you suggest I use > to combine the results of the models in stacking- is it going to be a > single layer LR model? > > > > On Sun, Mar 6, 2016 at 6:25 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Anurag, >> >> Thank you for your interest in the above project. You can start by >> getting to know the Machine Learner product by downloading it and running >> it (Please use link [1] to download). Official documentation [2] will help >> you on using the product. You can find the source code base at [3] and [4]. >> As we have mentioned in the project proposal as well, the main objective >> is to integrate ensemble support for the existing flow of the WSO2 Machine >> Learner. We are focusing on the three methods: Bagging, Boosting and >> Stacking. >> >> Please feel free to raise, if you have any questions or any unclear >> points. >> >> [1] http://wso2.com/products/machine-learner/ >> [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner >> [3] https://github.com/wso2/carbon-ml >> [4] https://github.com/wso2/product-ml >> >> Regards, >> Supun >> >> On Sun, Mar 6, 2016 at 12:08 PM, Anurag Kashyap <anurag1...@gmail.com> >> wrote: >> >>> Hi Supun, >>> I'm currently working as a research assistant at NTU, Singapore for my >>> bachelor thesis in my final year, I looked at the list of GSoC projects >>> for WSO2 and I'm particularly interested in the project *Ensemble >>> Methods Support for WSO2 Machine Learner.* I am currently working on >>> Ensemble methods for multi-label classification, so I have some experience >>> in this field and I've read several scholarly articles about their >>> implementation and benefit. I believe I can contribute to this project as a >>> GSoC intern. Eagerly awaiting a response! >>> >>> Yours Sincerely, >>> *Anurag Kashyap* >>> *Research Assistant* >>> *Robotics-|| * >>> *Nanyang Technological University, Singapore* >>> >>> [image: Blog] <https://thelinuxkernel.wordpress.com> >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > Yours sincerely, > > *Anurag Kashyap* > *MSc.(Tech.)* Information Systems, > *BITS Pilani K K Birla Goa Campus* > > [image: Blog] <https://thelinuxkernel.wordpress.com> > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] GSoC 2016 - Ensemble Methods Support for WSO2 Machine Learner
[Adding WSO2 Developers mailing list] Hi Dilan, Thank you for your interest in the above project. As we have mentioned in the project proposal as well, the main objective is to integrate ensemble support for the existing flow of the WSO2 Machine Learner. We are focusing on the three methods: Bagging, Boosting and Stacking. To start with, you can get to know the Machine Learner product by downloading it and running it (Please use link [1] to download). Official documentation [2] and blog [3] will help you on how to use the product. As the next step, you can go through the source code of WSO2 ML ([4] and [5]). Please feel free to raise if you have any questions or any unclear points. [1] http://wso2.com/products/machine-learner/ [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner [3] http://supunsetunga.blogspot.com/2015/09/building-your-first-predictive-model.html [4] https://github.com/wso2/carbon-ml [5] https://github.com/wso2/product-ml Regards, Supun On Mon, Mar 7, 2016 at 2:19 PM, Dilan Tharaka <tharakamd...@cse.mrt.ac.lk> wrote: > Hi, > > I'm Dilan Tharaka an undergraduate of University of Moratuwa. I went > through WSO2 GSoC ideas list and found the idea "Ensemble Methods Support > for WSO2 Machine Learner" is interesting. I have Java, Web development and > Unix knowledge. And I've used Spark and Weka libraries in machine leaning. > I would be thank full if you could help me in getting started with this > project. > > Thank you, > Regards, > Dilan Tharaka, > Undergraduate, > Department of Computer Science and Engineering > University of Moratuwa > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] GSoC project for building an ensemble of models
Hi Anurag, Thank you for your interest in the above project. You can start by getting to know the Machine Learner product by downloading it and running it (Please use link [1] to download). Official documentation [2] will help you on using the product. You can find the source code base at [3] and [4]. As we have mentioned in the project proposal as well, the main objective is to integrate ensemble support for the existing flow of the WSO2 Machine Learner. We are focusing on the three methods: Bagging, Boosting and Stacking. Please feel free to raise, if you have any questions or any unclear points. [1] http://wso2.com/products/machine-learner/ [2] https://docs.wso2.com/display/ML100/Introducing+Machine+Learner [3] https://github.com/wso2/carbon-ml [4] https://github.com/wso2/product-ml Regards, Supun On Sun, Mar 6, 2016 at 12:08 PM, Anurag Kashyap <anurag1...@gmail.com> wrote: > Hi Supun, > I'm currently working as a research assistant at NTU, Singapore for my > bachelor thesis in my final year, I looked at the list of GSoC projects > for WSO2 and I'm particularly interested in the project *Ensemble Methods > Support for WSO2 Machine Learner.* I am currently working on Ensemble > methods for multi-label classification, so I have some experience in this > field and I've read several scholarly articles about their implementation > and benefit. I believe I can contribute to this project as a GSoC intern. > Eagerly awaiting a response! > > Yours Sincerely, > *Anurag Kashyap* > *Research Assistant* > *Robotics-|| * > *Nanyang Technological University, Singapore* > > [image: Blog] <https://thelinuxkernel.wordpress.com> > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML][GSOC] Ensemble learning for WSO2 ML
Hi Minudika, Which options should be available for the user to get decisions? For > example, if the user is going to use the bagging method, the number of > samples can be pre-defined by the user. I think that depends on the implementation. As you've mentioned, # of samples, would definitely has to be a user-input. Other than that, sample size, algorithm to be used, its hyper-parameters, aggregation criteria (if there are multiple ways of aggregating), etc might have to be taken from the user. Similarly, for stacking, we might have to get from user: the # of models, algorithm for each model, hyper-parameters for each model, algorithm for aggregation, and etc.. Regards, Supun On Wed, Mar 2, 2016 at 3:16 AM, Minudika Malshan <minudika...@gmail.com> wrote: > Hi, > > Please help me to be clarified on this. > Apart from the implementation of those ensemble methods at the back end, > we're supposed to develop some UI features. > Which options should be available for the user to get decisions? For > example, if the user is going to use the bagging method, the number of > samples can be pre-defined by the user. > > Regards, > Minudika > > Minudika Malshan > Undergraduate > Department of Computer Science and Engineering > University of Moratuwa. > > > > > On Mon, Feb 29, 2016 at 11:04 AM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi Minudika, >> >> Thank you for your interest in the project. >> >> GBT and Random Forest are well known ensemble methods, and are readily >> available as a single algorithm OOB in spark. So we need not be >> implementing them again. You may treat them as any other simpler algorithm, >> for the project. >> >> Let me clarify the few things. For ensemble methods, you can consider the >> following three options: >> >>- Stacking - Training multiple algos on the same data, and combining >>them using another algo. >>- Bagging - Training a single algo over subsets of data. >>- Boosting - Training multiple algos on the same data, and combining >>them over a weighted average. >> >> Personally I would prefer picking Stacking (since Boosting is a special >> case of Stacking, later would cover both) and Bagging for >> implementation, but you may pick appropriately. AFAIK these three methods >> are not available OOB in spark. (except for Boosting in GBT and and bagging >> in Random Forest). Expectation of the project is to implement such a >> logic, where a user can use any algorithm(s), pick the ensemble method, and >> train a model. >> >> For bagging, you can use sampling techniques available in spark (eg: >> rdd.sample(), df.sample() etc ) [1]. >> >> Please do let us know if you need further clarifications. >> >> [1] http://spark.apache.org/docs/latest/api/java/ >> >> Regards, >> Supun >> >> On Mon, Feb 29, 2016 at 12:07 AM, Minudika Malshan <minudika...@gmail.com >> > wrote: >> >>> Hi, >>> >>> I found out that spark.ml Lib supports two ensemble algorithms, GBT and >>> Random Forest. >>> Will it be possible to implement Bagging and boosting methods using ml >>> Lib features? >>> >>> Also I'm grateful if you can give me some resources to getting started >>> with implementation of Bagging method using ml Lib functionalities. If >>> there's any other library which is allowed to use for this implementation, >>> please let me know. >>> >>> Thanks and regards. >>> Minudika >>> >>> Minudika Malshan >>> Undergraduate >>> Department of Computer Science and Engineering >>> University of Moratuwa. >>> >>> >>> >>> ___ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML][GSOC] Ensemble learning for WSO2 ML
Hi Minudika, Thank you for your interest in the project. GBT and Random Forest are well known ensemble methods, and are readily available as a single algorithm OOB in spark. So we need not be implementing them again. You may treat them as any other simpler algorithm, for the project. Let me clarify the few things. For ensemble methods, you can consider the following three options: - Stacking - Training multiple algos on the same data, and combining them using another algo. - Bagging - Training a single algo over subsets of data. - Boosting - Training multiple algos on the same data, and combining them over a weighted average. Personally I would prefer picking Stacking (since Boosting is a special case of Stacking, later would cover both) and Bagging for implementation, but you may pick appropriately. AFAIK these three methods are not available OOB in spark. (except for Boosting in GBT and and bagging in Random Forest). Expectation of the project is to implement such a logic, where a user can use any algorithm(s), pick the ensemble method, and train a model. For bagging, you can use sampling techniques available in spark (eg: rdd.sample(), df.sample() etc ) [1]. Please do let us know if you need further clarifications. [1] http://spark.apache.org/docs/latest/api/java/ Regards, Supun On Mon, Feb 29, 2016 at 12:07 AM, Minudika Malshan <minudika...@gmail.com> wrote: > Hi, > > I found out that spark.ml Lib supports two ensemble algorithms, GBT and > Random Forest. > Will it be possible to implement Bagging and boosting methods using ml > Lib features? > > Also I'm grateful if you can give me some resources to getting started > with implementation of Bagging method using ml Lib functionalities. If > there's any other library which is allowed to use for this implementation, > please let me know. > > Thanks and regards. > Minudika > > Minudika Malshan > Undergraduate > Department of Computer Science and Engineering > University of Moratuwa. > > > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [DAS] Compressed Event Analytics Relation Provider
Added the changes as per code review. Please merge [1]. [1] https://github.com/wso2/carbon-analytics/pull/159 Thanks, Supun On Fri, Feb 26, 2016 at 11:30 AM, Anjana Fernando <anj...@wso2.com> wrote: > Please schedule a code review for this. > > Cheers, > Anjana. > > On Fri, Feb 26, 2016 at 11:03 AM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi, >> >> Please find the PR [1] for adding string compression to the event >> analytics relational provider. >> >> [1] https://github.com/wso2/carbon-analytics/pull/159 >> >> Regards, >> Supun >> >> On Wed, Feb 24, 2016 at 3:35 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Thanks! >>> >>> Regards, >>> Supun >>> >>> On Wed, Feb 24, 2016 at 2:20 PM, Anjana Fernando <anj...@wso2.com> >>> wrote: >>> >>>> Merged the PR. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> On Wed, Feb 24, 2016 at 1:00 PM, Supun Sethunga <sup...@wso2.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have done the changes suggested during the code review. Please >>>>> review and merge [1]. >>>>> >>>>> Will send a separate PR for Compressing the json string. >>>>> >>>>> >>>>> Thanks, >>>>> Supun >>>>> >>>>> On Fri, Feb 19, 2016 at 3:37 PM, Anjana Fernando <anj...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Supun, >>>>>> >>>>>> Can you please schedule a code review for this. >>>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> On Tue, Feb 16, 2016 at 4:30 PM, Supun Sethunga <sup...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Attached [1] is the PR for $subject. Please review and merge. >>>>>>> >>>>>>> [1] https://github.com/wso2/carbon-analytics/pull/156 >>>>>>> >>>>>>> Thanks, >>>>>>> Supun >>>>>>> >>>>>>> -- >>>>>>> *Supun Sethunga* >>>>>>> Software Engineer >>>>>>> WSO2, Inc. >>>>>>> http://wso2.com/ >>>>>>> lean | enterprise | middleware >>>>>>> Mobile : +94 716546324 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Senior Technical Lead >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> http://wso2.com/ >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [DAS] Compressed Event Analytics Relation Provider
Hi, Please find the PR [1] for adding string compression to the event analytics relational provider. [1] https://github.com/wso2/carbon-analytics/pull/159 Regards, Supun On Wed, Feb 24, 2016 at 3:35 PM, Supun Sethunga <sup...@wso2.com> wrote: > Thanks! > > Regards, > Supun > > On Wed, Feb 24, 2016 at 2:20 PM, Anjana Fernando <anj...@wso2.com> wrote: > >> Merged the PR. >> >> Cheers, >> Anjana. >> >> On Wed, Feb 24, 2016 at 1:00 PM, Supun Sethunga <sup...@wso2.com> wrote: >> >>> Hi, >>> >>> I have done the changes suggested during the code review. Please review >>> and merge [1]. >>> >>> Will send a separate PR for Compressing the json string. >>> >>> >>> Thanks, >>> Supun >>> >>> On Fri, Feb 19, 2016 at 3:37 PM, Anjana Fernando <anj...@wso2.com> >>> wrote: >>> >>>> Hi Supun, >>>> >>>> Can you please schedule a code review for this. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> On Tue, Feb 16, 2016 at 4:30 PM, Supun Sethunga <sup...@wso2.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Attached [1] is the PR for $subject. Please review and merge. >>>>> >>>>> [1] https://github.com/wso2/carbon-analytics/pull/156 >>>>> >>>>> Thanks, >>>>> Supun >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> http://wso2.com/ >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> *Supun Sethunga* >>> Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> >> >> >> >> -- >> *Anjana Fernando* >> Senior Technical Lead >> WSO2 Inc. | http://wso2.com >> lean . enterprise . middleware >> > > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [DAS] Compressed Event Analytics Relation Provider
Thanks! Regards, Supun On Wed, Feb 24, 2016 at 2:20 PM, Anjana Fernando <anj...@wso2.com> wrote: > Merged the PR. > > Cheers, > Anjana. > > On Wed, Feb 24, 2016 at 1:00 PM, Supun Sethunga <sup...@wso2.com> wrote: > >> Hi, >> >> I have done the changes suggested during the code review. Please review >> and merge [1]. >> >> Will send a separate PR for Compressing the json string. >> >> >> Thanks, >> Supun >> >> On Fri, Feb 19, 2016 at 3:37 PM, Anjana Fernando <anj...@wso2.com> wrote: >> >>> Hi Supun, >>> >>> Can you please schedule a code review for this. >>> >>> Cheers, >>> Anjana. >>> >>> On Tue, Feb 16, 2016 at 4:30 PM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Hi, >>>> >>>> Attached [1] is the PR for $subject. Please review and merge. >>>> >>>> [1] https://github.com/wso2/carbon-analytics/pull/156 >>>> >>>> Thanks, >>>> Supun >>>> >>>> -- >>>> *Supun Sethunga* >>>> Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Senior Technical Lead >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Compressing JSON payload in ESB Analytics
Hi Viraj/Buddhima, It seems like if we convert the compressed string to UTF-8 string (as in [1]), we can't decompress it. AFAIU, we need to send the byte array after compressing, then only we can use that byte array to decompress. Were you able to convert it back to string? [1] https://github.com/wso2/carbon-mediation/blob/master/components/mediation-monitor/mediation-data-publisher/org.wso2.carbon.das.messageflow.data.publisher/src/main/java/org/wso2/carbon/das/messageflow/data/publisher/publish/Publisher.java#L203 Thanks, Supun -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
