[Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] ESB Mediator for WSO2 Machine Learner
Does this update the model time to time or use the same modal throughout the ESB up time? It is better if we can update the model while the ESB is up and running. There may be some use cases related to multiple domains right? So I think would be more generic and useful if we can apply the mediator not based on the domain but with a custom argument expression. For example if the target service is SalesService and the operation is getTotalSalesCount we want to use the modal to analyze all incoming requests from all domains. WDYT? Thanks. *Maninda Edirisooriya* Senior Software Engineer *WSO2, Inc.*lean.enterprise.middleware. *Blog* : http://maninda.blogspot.com/ *E-mail* : mani...@wso2.com *Skype* : @manindae *Twitter* : @maninda On Tue, Mar 24, 2015 at 12:00 PM, Manorama Perera manor...@wso2.com wrote: Hi, ESB mediator for WSO2 ML enables making use of the models generated by WSO2 Machine Leaner, to predict values in message mediation in ESB. - Machine Learner feature needs to be installed in ESB prior to configure the ML mediator. - Mediator will extract the data from the incoming messages and send to the ML model as features. - The mediator use the Machine Learner features to predict the output value for every message passing through the mediator. *Proposed Mediator Configuration* ml model storage-location=string/ feature name=string expression=xpath/+ prediction expression=xpath/ /ml *model* storage-location : Storage location of the generated model. *feature* name : Feature name according to the generated model. expression : XPath expression used to determine the feature value. *prediction* expression : XPath expression to determine the location to insert the predicted value. The generated models can be applied to predict only the data coming from the same domain, as the data used to train the model. So the mediator should be configured only to mediate messages which carry data coming from the same domain, as the data used to train the model. Any suggestion are appreciated. Thanks. -- Manorama Perera Software Engineer WSO2, Inc.; http://wso2.com/ Mobile : +94716436216 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] ESB Mediator for WSO2 Machine Learner
Hi Maninda, With the initial release of ML, the model would not update on the fly. If we need to update the model, it has to be done in ML by training a model with the same configurations, but with the new data. In future releases we will be looking at this for possible improvements. Here the term domain refers to the definition of the data which use to train the model. In the case you mentioned, If the model was trained using the data collected from all incoming requests; then domain is all sales. and can be used to handle all incoming requests. But if the model was trained using the data collected from the incoming requests of, say Seller_A, then it cannot be used to predict/handle incoming requests of all sellers, which is a different domain. When we say domain, it also includes the Schema (set of features/variables) of the dataset on which the prediction is going to be done. Handling multiple schema is tricky. If the data from a different domain doesn't contain some of the features (variables in the dataset) used to train the model, then that model cannot be used to predict on that particular data. So we have to work with this limitation, hence all the data has to be on the same domain (i.e. should have the same schema). Regards, Supun On Tue, Mar 24, 2015 at 3:10 PM, Maninda Edirisooriya mani...@wso2.com wrote: Does this update the model time to time or use the same modal throughout the ESB up time? It is better if we can update the model while the ESB is up and running. There may be some use cases related to multiple domains right? So I think would be more generic and useful if we can apply the mediator not based on the domain but with a custom argument expression. For example if the target service is SalesService and the operation is getTotalSalesCount we want to use the modal to analyze all incoming requests from all domains. WDYT? Thanks. *Maninda Edirisooriya* Senior Software Engineer *WSO2, Inc.*lean.enterprise.middleware. *Blog* : http://maninda.blogspot.com/ *E-mail* : mani...@wso2.com *Skype* : @manindae *Twitter* : @maninda On Tue, Mar 24, 2015 at 12:00 PM, Manorama Perera manor...@wso2.com wrote: Hi, ESB mediator for WSO2 ML enables making use of the models generated by WSO2 Machine Leaner, to predict values in message mediation in ESB. - Machine Learner feature needs to be installed in ESB prior to configure the ML mediator. - Mediator will extract the data from the incoming messages and send to the ML model as features. - The mediator use the Machine Learner features to predict the output value for every message passing through the mediator. *Proposed Mediator Configuration* ml model storage-location=string/ feature name=string expression=xpath/+ prediction expression=xpath/ /ml *model* storage-location : Storage location of the generated model. *feature* name : Feature name according to the generated model. expression : XPath expression used to determine the feature value. *prediction* expression : XPath expression to determine the location to insert the predicted value. The generated models can be applied to predict only the data coming from the same domain, as the data used to train the model. So the mediator should be configured only to mediate messages which carry data coming from the same domain, as the data used to train the model. Any suggestion are appreciated. Thanks. -- Manorama Perera Software Engineer WSO2, Inc.; http://wso2.com/ Mobile : +94716436216 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
On Tue, Mar 24, 2015 at 8:14 PM, danush...@wso2.com wrote: I think same behavior is there in APIM too. They have the API list and separate list called Recently added. To get recently added to API list takes few seconds. Can’t we implement similar thing here? I don't see an issue to fix in indexing . There is no way to index the resource just after adding. Indexing will trigger from the frequency which scheduled in registry.xml file. So if you need reduce the indexing task frequency. -Ajith Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 *From:* Dimuthu Leelarathne dimut...@wso2.com *Sent:* Tuesday, March 24, 2015 10:22 PM *To:* Punnadi Gunarathna punn...@wso2.com *Cc:* architecture architecture@wso2.org Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Ajith Vitharana. WSO2 Inc. - http://wso2.org Email : aji...@wso2.com Mobile : +1-812-360-5335 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
I think same behavior is there in APIM too. They have the API list and separate list called Recently added. To get recently added to API list takes few seconds. Can’t we implement similar thing here? Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 From: Dimuthu Leelarathne Sent: Tuesday, March 24, 2015 10:22 PM To: Punnadi Gunarathna Cc: architecture Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3]https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi Ajith What I was pointing out was this could be a fix in our side (AF). Sorry if I mislead you. Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 On Mar 25, 2015 6:11 AM, Ajith Vitharana aji...@wso2.com wrote: On Tue, Mar 24, 2015 at 8:14 PM, danush...@wso2.com wrote: I think same behavior is there in APIM too. They have the API list and separate list called Recently added. To get recently added to API list takes few seconds. Can’t we implement similar thing here? I don't see an issue to fix in indexing . There is no way to index the resource just after adding. Indexing will trigger from the frequency which scheduled in registry.xml file. So if you need reduce the indexing task frequency. -Ajith Thanks Regards Danushka Fernando Software Engineer WSO2 inc. http://wso2.com/ Mobile : +94716332729 *From:* Dimuthu Leelarathne dimut...@wso2.com *Sent:* Tuesday, March 24, 2015 10:22 PM *To:* Punnadi Gunarathna punn...@wso2.com *Cc:* architecture architecture@wso2.org Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Ajith Vitharana. WSO2 Inc. - http://wso2.org Email : aji...@wso2.com Mobile : +1-812-360-5335 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi Dimuthu, So shall I go-ahead with indexing? On Tue, Mar 24, 2015 at 10:22 PM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] WSO2 Machine Learner - Model Generation Process
Hi Guys I have shared latest markup bundle [1] required for proposed ML changes; [1] https://drive.google.com/open?id=0B4eoLHAhQ246fjc3ZUV5VE1qUnVYMEJuN1A2OXo5aWxoT3ZRMFZTWVRrS0Fsd0NsRW9OOEkauthuser=0 Lets have a chat so I should be able to explain the flow and discuss next steps/further clarification and changes. *New pages* datasets.html create-dataset.html create-version.html images\icons ico-edit.png css custom-theme.css custom.css *Updated Pages* projects.html create-project.html Cheers On Mon, Mar 16, 2015 at 8:03 AM, Nirmal Fernando nir...@wso2.com wrote: Hi Srinath, Noted. Yes, I think we need to consider this when we alter the UI. The process diagram is mainly targeting the users of the REST API and will see how it can be improved too. On Mon, Mar 16, 2015 at 7:41 AM, Srinath Perera srin...@wso2.com wrote: Nirmal, one key thing to worry about is experience for someone getting started and guide him though even if he does not understand the value set, feature set etc, and Analysis. --Srinath On Sat, Mar 14, 2015 at 7:30 PM, Nirmal Fernando nir...@wso2.com wrote: Hi All, Please find the model generation process of WSO2 ML as a diagram below [1]. Model Generation is one of the key use-cases of WSO2 ML. Model Generation When you login to the WSO2 ML, you will be redirected to a page where you can see two main entities of WSO2 ML, namely, Datasets and Projects. When you navigate to the Projects, you could see all the Analyses created for that project. Analysis is a workflow created to cover the full machine learning pipeline from data extraction to model generation. When you create a new Analysis, you could see all the available Value-sets for the logged-in user. You pick a Value set and start the process of analysing that Value-set. Then you will be navigated to the data extraction interface, where you are getting a chance, either to pick an existing customized feature-set or to create a new customized feature-set. Once you have picked a feature-set, you could provide the model configuration. Based on the model configuration, now you could calibrate the corresponding hyper parameters. Once you have a value-set, customized feature-set, a model configuration and a calibrated set of hyper-parameters, you could generate a ML model. [1] -- Thanks regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/ -- Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 -- Thanks regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/ -- Dhanuka De Silva m: +94 773 887816 e: dhanu...@wso2.com w: http://www.wso2.com ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [AppM][AppFactory][AS] AppM Discovery of WebApps from As
Ruwan, I have create the Redmine ticket (https://redmine.wso2.com/issues/3822) for internal tracking the public JIRA (https://wso2.org/jira/browse/APPM-643) as well. On Mon, Mar 16, 2015 at 1:46 PM, Ruwan Abeykoon ruw...@wso2.com wrote: Hi All, It seems that the SVG diagram missing the text. Attaching the same in PNG format. Sorry for the inconvenience. Cheers, Ruwan On Mon, Mar 16, 2015 at 10:38 AM, Ruwan Abeykoon ruw...@wso2.com wrote: Hi All, I am in the process of adding a feature to AppManager to discover Web Applications deployed on AS and publish them on AppM. * Applications are discovered using the Admin Service on AS (use of WebappAdmin Service), which is the same service used by the AS Application Listing. * Publisher (Person) will get a new tab to initiate the discovery process in the AppM. The UI will present a WebApp listing and publisher can selectively publish WebApps discovered from AS. (I will initiate a separate thread to discuss about the UI) * AppM will initiate the normal publish workflow when the Publisher opt for an application to be published. The details (Name, Version, Context, etc.) will be taken from the discovered details. The conflicting details will be handled as same as manual Application creation. * Applications created via App Factory will also be discovered through AS once it is deployed to AS. Please find the attached diagram for the architecture (connectivity) diagram. Please voice your concerns and suggestions. Cheers, -- *Ruwan Abeykoon* *Architect,* *WSO2, Inc. http://wso2.com http://wso2.com/ * *lean.enterprise.middleware.* email: ruw...@wso2.com phone:(+94) 39736 -- *Ruwan Abeykoon* *Architect,* *WSO2, Inc. http://wso2.com http://wso2.com/ * *lean.enterprise.middleware.* email: ruw...@wso2.com phone:(+94) 39736 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- /sumedha m: +94 773017743 b : bit.ly/sumedha ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] Proposed ESB connector scenario - Integrating Stripe with Mailchimp, ZohoCRM, Mandrill and ZohoBooks
Hi All I have done with the mandrill connector review. Please find the related doc below. https://docs.google.com/a/wso2.com/document/d/1hOeAHeqDNK_APkfzC1wudVTvTIHo1KsuHxJTKb7PpR8/edit Thank you, Rgds, On Tue, Mar 10, 2015 at 7:50 PM, Vanjikumaran Sivajothy va...@wso2.com wrote: Since this connector is not released. Please do not send it as separate minor version. On Tuesday, March 10, 2015, Rasika Hettige rasi...@gmail.com wrote: Hi All, Please note that the method *unsubscribeFromList* will be changed to *removeSubscribersFromList* in MailChimp connector to maintain the consistency with version 1.0 methods. Thanks Regards Rasika -- View this message in context: http://wso2-oxygen-tank.10903.n7.nabble.com/Proposed-ESB-connector-scenario-Integrating-Stripe-with-Mailchimp-ZohoCRM-Mandrill-and-ZohoBooks-tp112580p114188.html Sent from the WSO2 Architecture mailing list archive at Nabble.com. ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Sent from Gmail Mobile ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- S.Sriashalya, Associate Software Engineer, WSO2. ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi all, If we are to make use of indexing, we need to check the feasibility to index existing data as well because size of data in cloud setup is huge. However this is a one time task that we can find a solution such as copy the indexed data into server after running indexes offline for dump of live data. It seems that we need to come up with a aggregated solution of indexed data and non-indexed data(new additions) then removing duplicates etc. We can not depend on retries because we can not assume the time taken to index etc. thank you. On Wed, Mar 25, 2015 at 9:57 AM, Punnadi Gunarathna punn...@wso2.com wrote: Hi Dimuthu, So shall I go-ahead with indexing? On Tue, Mar 24, 2015 at 10:22 PM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Manjula Rathnayaka Software Engineer WSO2, Inc. Mobile:+94 77 743 1987 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] Proposed ESB connector scenario - Integrating Stripe with Mailchimp, ZohoCRM, Mandrill and ZohoBooks
adding PDF DOC On Wed, Mar 25, 2015 at 10:40 AM, Sriashalya Srivathsan asha...@wso2.com wrote: Hi All I have done with the mandrill connector review. Please find the related doc below. https://docs.google.com/a/wso2.com/document/d/1hOeAHeqDNK_APkfzC1wudVTvTIHo1KsuHxJTKb7PpR8/edit Thank you, Rgds, On Tue, Mar 10, 2015 at 7:50 PM, Vanjikumaran Sivajothy va...@wso2.com wrote: Since this connector is not released. Please do not send it as separate minor version. On Tuesday, March 10, 2015, Rasika Hettige rasi...@gmail.com wrote: Hi All, Please note that the method *unsubscribeFromList* will be changed to *removeSubscribersFromList* in MailChimp connector to maintain the consistency with version 1.0 methods. Thanks Regards Rasika -- View this message in context: http://wso2-oxygen-tank.10903.n7.nabble.com/Proposed-ESB-connector-scenario-Integrating-Stripe-with-Mailchimp-ZohoCRM-Mandrill-and-ZohoBooks-tp112580p114188.html Sent from the WSO2 Architecture mailing list archive at Nabble.com. ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Sent from Gmail Mobile ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- S.Sriashalya, Associate Software Engineer, WSO2. -- S.Sriashalya, Associate Software Engineer, WSO2. MandrillConnectorReview.pdf Description: Adobe PDF document ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi, We should not make architectural decision fearing a 29 hour data migration :). @Chandana and Greg team - If you guys say this is a fixed thing and cannot be fixed, we'll give priority to our Runtime version database and move ahead. :) We'll still keep meta data in registry. Punnadi and AF team - Lets give priority to our Runtime database because of the 10 second delay ( probably in cloud it will be 30 seconds) thanks, dimuthu On Wed, Mar 25, 2015 at 10:40 AM, Manjula Rathnayake manju...@wso2.com wrote: Hi all, If we are to make use of indexing, we need to check the feasibility to index existing data as well because size of data in cloud setup is huge. However this is a one time task that we can find a solution such as copy the indexed data into server after running indexes offline for dump of live data. It seems that we need to come up with a aggregated solution of indexed data and non-indexed data(new additions) then removing duplicates etc. We can not depend on retries because we can not assume the time taken to index etc. thank you. On Wed, Mar 25, 2015 at 9:57 AM, Punnadi Gunarathna punn...@wso2.com wrote: Hi Dimuthu, So shall I go-ahead with indexing? On Tue, Mar 24, 2015 at 10:22 PM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Manjula Rathnayaka Software Engineer WSO2, Inc. Mobile:+94 77 743 1987 -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi, I wasn't suggesting to index the resource as soon as it is created, but rather whether it is possible to fix the 10 second delay that we have to introduce between creating and reading. artifactManager.createResource() //wait 10 seconds --- This is a problem artifactManager.readResource() @Punnadi - Please stop with indexing for now. Because if we are fixing it we need to fix without retries and needs design discussion. Indexing would only fix the initial loading time. Put it on hold until we do a detail design review. thanks, dimuthu On Wed, Mar 25, 2015 at 9:57 AM, Punnadi Gunarathna punn...@wso2.com wrote: Hi Dimuthu, So shall I go-ahead with indexing? On Tue, Mar 24, 2015 at 10:22 PM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi Dimuthu, As explained by Ajith as well, this is the expected behavior. However, you can optimize your indexing related parameters (indexingFrequencyInSeconds, etc.) according to your requirements and available resources. Regards, Chandana On Wed, Mar 25, 2015 at 10:51 AM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi, We should not make architectural decision fearing a 29 hour data migration :). @Chandana and Greg team - If you guys say this is a fixed thing and cannot be fixed, we'll give priority to our Runtime version database and move ahead. :) We'll still keep meta data in registry. Punnadi and AF team - Lets give priority to our Runtime database because of the 10 second delay ( probably in cloud it will be 30 seconds) thanks, dimuthu On Wed, Mar 25, 2015 at 10:40 AM, Manjula Rathnayake manju...@wso2.com wrote: Hi all, If we are to make use of indexing, we need to check the feasibility to index existing data as well because size of data in cloud setup is huge. However this is a one time task that we can find a solution such as copy the indexed data into server after running indexes offline for dump of live data. It seems that we need to come up with a aggregated solution of indexed data and non-indexed data(new additions) then removing duplicates etc. We can not depend on retries because we can not assume the time taken to index etc. thank you. On Wed, Mar 25, 2015 at 9:57 AM, Punnadi Gunarathna punn...@wso2.com wrote: Hi Dimuthu, So shall I go-ahead with indexing? On Tue, Mar 24, 2015 at 10:22 PM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Manjula Rathnayaka Software Engineer WSO2, Inc. Mobile:+94 77 743 1987 -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware -- *Chandana Napagoda* Senior Software Engineer WSO2 Inc. - http://wso2.org *Email : chand...@wso2.com chand...@wso2.com**Mobile : +94718169299* *Blog :http://cnapagoda.blogspot.com http://cnapagoda.blogspot.com* ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [AppM][AppFactory][AS] AppM Discovery of WebApps from As
Ruwan, As discussed, let's add this capability in an extensible manner. So that we will not limit application discovery to WSO2's application server. On Mon, Mar 16, 2015 at 10:38 AM, Ruwan Abeykoon ruw...@wso2.com wrote: Hi All, I am in the process of adding a feature to AppManager to discover Web Applications deployed on AS and publish them on AppM. * Applications are discovered using the Admin Service on AS (use of WebappAdmin Service), which is the same service used by the AS Application Listing. * Publisher (Person) will get a new tab to initiate the discovery process in the AppM. The UI will present a WebApp listing and publisher can selectively publish WebApps discovered from AS. (I will initiate a separate thread to discuss about the UI) * AppM will initiate the normal publish workflow when the Publisher opt for an application to be published. The details (Name, Version, Context, etc.) will be taken from the discovered details. The conflicting details will be handled as same as manual Application creation. * Applications created via App Factory will also be discovered through AS once it is deployed to AS. Please find the attached diagram for the architecture (connectivity) diagram. Please voice your concerns and suggestions. Cheers, -- *Ruwan Abeykoon* *Architect,* *WSO2, Inc. http://wso2.com http://wso2.com/ * *lean.enterprise.middleware.* email: ruw...@wso2.com phone:(+94) 39736 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- /sumedha m: +94 773017743 b : bit.ly/sumedha ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] WSO2 Machine Learner - Model Generation Process
This is great! Thanks Dhanuka! On Wed, Mar 25, 2015 at 10:00 AM, Upul Bandara u...@wso2.com wrote: Thanks a lot Dhanuka On Wed, Mar 25, 2015 at 9:56 AM, Dhanuka De Silva dhanu...@wso2.com wrote: Hi Guys I have shared latest markup bundle [1] required for proposed ML changes; [1] https://drive.google.com/open?id=0B4eoLHAhQ246fjc3ZUV5VE1qUnVYMEJuN1A2OXo5aWxoT3ZRMFZTWVRrS0Fsd0NsRW9OOEkauthuser=0 Lets have a chat so I should be able to explain the flow and discuss next steps/further clarification and changes. *New pages* datasets.html create-dataset.html create-version.html images\icons ico-edit.png css custom-theme.css custom.css *Updated Pages* projects.html create-project.html Cheers On Mon, Mar 16, 2015 at 8:03 AM, Nirmal Fernando nir...@wso2.com wrote: Hi Srinath, Noted. Yes, I think we need to consider this when we alter the UI. The process diagram is mainly targeting the users of the REST API and will see how it can be improved too. On Mon, Mar 16, 2015 at 7:41 AM, Srinath Perera srin...@wso2.com wrote: Nirmal, one key thing to worry about is experience for someone getting started and guide him though even if he does not understand the value set, feature set etc, and Analysis. --Srinath On Sat, Mar 14, 2015 at 7:30 PM, Nirmal Fernando nir...@wso2.com wrote: Hi All, Please find the model generation process of WSO2 ML as a diagram below [1]. Model Generation is one of the key use-cases of WSO2 ML. Model Generation When you login to the WSO2 ML, you will be redirected to a page where you can see two main entities of WSO2 ML, namely, Datasets and Projects. When you navigate to the Projects, you could see all the Analyses created for that project. Analysis is a workflow created to cover the full machine learning pipeline from data extraction to model generation. When you create a new Analysis, you could see all the available Value-sets for the logged-in user. You pick a Value set and start the process of analysing that Value-set. Then you will be navigated to the data extraction interface, where you are getting a chance, either to pick an existing customized feature-set or to create a new customized feature-set. Once you have picked a feature-set, you could provide the model configuration. Based on the model configuration, now you could calibrate the corresponding hyper parameters. Once you have a value-set, customized feature-set, a model configuration and a calibrated set of hyper-parameters, you could generate a ML model. [1] -- Thanks regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/ -- Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902 -- Thanks regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/ -- Dhanuka De Silva m: +94 773 887816 e: dhanu...@wso2.com w: http://www.wso2.com -- Upul Bandara, Associate Technical Lead, WSO2, Inc., Mob: +94 715 468 345. -- Thanks regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi, On Wed, Mar 25, 2015 at 5:21 AM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi, We should not make architectural decision fearing a 29 hour data migration :). @Chandana and Greg team - If you guys say this is a fixed thing and cannot be fixed, we'll give priority to our Runtime version database and move ahead. :) We'll still keep meta data in registry. Punnadi and AF team - Lets give priority to our Runtime database because of the 10 second delay ( probably in cloud it will be 30 seconds) Please note - Solr is for content indexing. We will use DB indexing. We don't need to do content indexing. thanks, dimuthu thanks, dimuthu On Wed, Mar 25, 2015 at 10:40 AM, Manjula Rathnayake manju...@wso2.com wrote: Hi all, If we are to make use of indexing, we need to check the feasibility to index existing data as well because size of data in cloud setup is huge. However this is a one time task that we can find a solution such as copy the indexed data into server after running indexes offline for dump of live data. It seems that we need to come up with a aggregated solution of indexed data and non-indexed data(new additions) then removing duplicates etc. We can not depend on retries because we can not assume the time taken to index etc. thank you. On Wed, Mar 25, 2015 at 9:57 AM, Punnadi Gunarathna punn...@wso2.com wrote: Hi Dimuthu, So shall I go-ahead with indexing? On Tue, Mar 24, 2015 at 10:22 PM, Dimuthu Leelarathne dimut...@wso2.com wrote: Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- Manjula Rathnayaka Software Engineer WSO2, Inc. Mobile:+94 77 743 1987 -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] Analytics Facets APIs in AnalyticsDataService
Gimantha, it is better if you can give some possible use cases for each of
the features we have listed here, so people can get an better understanding
of what to use where.
Cheers,
Anjana.
On Thu, Mar 19, 2015 at 8:23 PM, Gimantha Bandara giman...@wso2.com wrote:
Hi all,
Analytics facets APIs provide indexing capabilities for hierarchical
categorization of table entries in New analytics data service (Please refer
to [Architecture] BAM 3.0 REST APIs for AnalyticsDataService / Indexing /
Search for more information). Using facet APIs, an user can define
multiple categories as indices for a table and later can be used to search
table entries based on categories. These APIs will be generic, so the user
can assign a weight for each category when indexing, combine a mathematical
function to calculate weights,
*Facet Counts*
As an example in log analysis, consider the following
E.g. log-time : 2015/mar/12/ 20:30:23, 2015/jan/16 13:34:76, 2015/jan/11
01:34:76 ( in 3 different log lines)
In the above example the log time can be defined as a hierarchical facet
as year/month/date. Later if the user wants to get the counts of log
entries by year/month, API would return
2015/jan - Count :2
2015/mar - Count 1
If the user wants to get the total count of log entries by year, API
would return
2015 - Count :3
If the user wants to get the count of log entries by year/month/date,
API returns,
2015/jan/11 - Count :1
2015/jan/16 - Count :1
2015/mar/12 - Count : 1
*Drill-Down capabilities*
Dill down capabilities are provided by Facets APIs. User can drill down
through the facet hierarchy of the index and search table entries. User
also can combine a search query so he can filter out the table entries. As
an example, in above example, User queries for the total count of log lines
in 2015/jan/11 ( he gets 1 as the count) and then he wants to view the
other attributes of the log line ( TID, Component name, log level, ..etc).
*REST APIs for Facets*
Users will be able to use facets API through REST APIs. Users can create
facets indices via the usual Analytics indexing REST APIs and insert
hierarchical category information through Analytics REST APIs, Following
are the updated Analytics REST APIs.
1. Drill-down through a facets hierarchy
/analytics/drilldown or /analytics/drilldown-count
{
tableName :
categories : [{
name : hierarchy name e.g. Publish date
categoryPath : [ ], hierarchy as an array e.g.
[2001, March, 02]
}],
language : lucene or regex
query : lucene query or regular expression
scoreFunction : Javascript function to define scoring function
scoreParams : [] Array of docvalue fields used as parameters for
scoring function
}
2. Querying for Ranges (Additional to facets)
/analytics/searchrange or /analytics/rangecount
{
tableName : sample-table-name,
ranges : [{
label:
from :
to:
minInclusive:
maxInclusive:
}],
language :
query :
}
In addition to the existing index types two more are introduced. They are
FACET and SCOREPARAM. FACET is used to define a hierarchical facet
field and SCOREPARAM is used to define scoring parameters for score
function.
*Adding Facet fields and score fields*
*to a table/tables*
Facet fields and score fields need to be defined using indexing APIs.
/analytics/tables/table-name/indices
{
field : STRING,
facetField : FACET,
scoreField : SCOREPARAM
}
Later user can add facet and score fields using POST to,
/analytics/tables/table-name
[
{
values : {
field : value,
facetField : {
weight :
categoryPath : [ ]
},
scoreField : numeric-value
}
}
]
or /analytics/records
[
{
tableName :
values : {
field : value,
facetField : {
weight :
categoryPath : [ ]
},
scoreField : numeric-value
}
}
]
Feedback and suggestions are appreciated.
--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919
--
*Anjana Fernando*
Senior Technical Lead
WSO2 Inc. | http://wso2.com
lean . enterprise . middleware
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] ESB Mediator for WSO2 Machine Learner
+1 This configuration looks very neat and concise. If the ML mediator will be primary used on a single domain, would it make sense to include an optional domain attribute to the ML node? I could see two uses for an optional domain attribute: 1. It could be useful for logging, annotation and troubleshooting. 2. Perhaps it could be used as a default filter? ml domain=string model storage-location=string/ feature name=string expression=xpath/+ prediction expression=xpath/ /ml Perhaps 'target-domain' or 'domain-name' would be better than simply 'domain'. Perhaps this is just duplicating functionality of the 'name' attribute and filter mediator. WDYT? Thanks, Colin Roy-Ehri Software Engineer *WSO2, Inc. : wso2.com http://wso2.com/* *Mobile* : 812-219-6517 On Tue, Mar 24, 2015 at 12:00 PM, Manorama Perera manor...@wso2.com wrote: Hi, ESB mediator for WSO2 ML enables making use of the models generated by WSO2 Machine Leaner, to predict values in message mediation in ESB. - Machine Learner feature needs to be installed in ESB prior to configure the ML mediator. - Mediator will extract the data from the incoming messages and send to the ML model as features. - The mediator use the Machine Learner features to predict the output value for every message passing through the mediator. *Proposed Mediator Configuration* ml model storage-location=string/ feature name=string expression=xpath/+ prediction expression=xpath/ /ml *model* storage-location : Storage location of the generated model. *feature* name : Feature name according to the generated model. expression : XPath expression used to determine the feature value. *prediction* expression : XPath expression to determine the location to insert the predicted value. The generated models can be applied to predict only the data coming from the same domain, as the data used to train the model. So the mediator should be configured only to mediate messages which carry data coming from the same domain, as the data used to train the model. Any suggestion are appreciated. Thanks. -- Manorama Perera Software Engineer WSO2, Inc.; http://wso2.com/ Mobile : +94716436216 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
Re: [Architecture] ESB Mediator for WSO2 Machine Learner
Hi Manorama, I guess that the names are not finalized yet. The name ml does not make much impression on the users. Please give a valid name like MachineLearner and that would make life easier for ESB developers. All our mediators has self explanatory names. Thanks, Chanaka On Tue, Mar 24, 2015 at 4:09 PM, Supun Sethunga sup...@wso2.com wrote: Hi Maninda, With the initial release of ML, the model would not update on the fly. If we need to update the model, it has to be done in ML by training a model with the same configurations, but with the new data. In future releases we will be looking at this for possible improvements. Here the term domain refers to the definition of the data which use to train the model. In the case you mentioned, If the model was trained using the data collected from all incoming requests; then domain is all sales. and can be used to handle all incoming requests. But if the model was trained using the data collected from the incoming requests of, say Seller_A, then it cannot be used to predict/handle incoming requests of all sellers, which is a different domain. When we say domain, it also includes the Schema (set of features/variables) of the dataset on which the prediction is going to be done. Handling multiple schema is tricky. If the data from a different domain doesn't contain some of the features (variables in the dataset) used to train the model, then that model cannot be used to predict on that particular data. So we have to work with this limitation, hence all the data has to be on the same domain (i.e. should have the same schema). Regards, Supun On Tue, Mar 24, 2015 at 3:10 PM, Maninda Edirisooriya mani...@wso2.com wrote: Does this update the model time to time or use the same modal throughout the ESB up time? It is better if we can update the model while the ESB is up and running. There may be some use cases related to multiple domains right? So I think would be more generic and useful if we can apply the mediator not based on the domain but with a custom argument expression. For example if the target service is SalesService and the operation is getTotalSalesCount we want to use the modal to analyze all incoming requests from all domains. WDYT? Thanks. *Maninda Edirisooriya* Senior Software Engineer *WSO2, Inc.*lean.enterprise.middleware. *Blog* : http://maninda.blogspot.com/ *E-mail* : mani...@wso2.com *Skype* : @manindae *Twitter* : @maninda On Tue, Mar 24, 2015 at 12:00 PM, Manorama Perera manor...@wso2.com wrote: Hi, ESB mediator for WSO2 ML enables making use of the models generated by WSO2 Machine Leaner, to predict values in message mediation in ESB. - Machine Learner feature needs to be installed in ESB prior to configure the ML mediator. - Mediator will extract the data from the incoming messages and send to the ML model as features. - The mediator use the Machine Learner features to predict the output value for every message passing through the mediator. *Proposed Mediator Configuration* ml model storage-location=string/ feature name=string expression=xpath/+ prediction expression=xpath/ /ml *model* storage-location : Storage location of the generated model. *feature* name : Feature name according to the generated model. expression : XPath expression used to determine the feature value. *prediction* expression : XPath expression to determine the location to insert the predicted value. The generated models can be applied to predict only the data coming from the same domain, as the data used to train the model. So the mediator should be configured only to mediate messages which carry data coming from the same domain, as the data used to train the model. Any suggestion are appreciated. Thanks. -- Manorama Perera Software Engineer WSO2, Inc.; http://wso2.com/ Mobile : +94716436216 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture -- -- Chanaka Fernando Technical Lead WSO2, Inc.; http://wso2.com lean.enterprise.middleware mobile: +94 773337238 Blog : http://soatutorials.blogspot.com LinkedIn:http://www.linkedin.com/pub/chanaka-fernando/19/a20/5b0 Twitter:https://twitter.com/chanakaudaya Wordpress:http://chanakaudaya.wordpress.com ___ Architecture mailing list Architecture@wso2.org
Re: [Architecture] [App Fac] Performance Improvements - Issues in enabling GReg resource Indexing in App Factory
Hi Chandana, Could we get fixes from Registry side for this? thanks, dimuthu On Tue, Mar 24, 2015 at 7:29 PM, Punnadi Gunarathna punn...@wso2.com wrote: Hi All, While fixing a cloud bug which is AF code tries to retrieve information of irrelevant apps when inside an app [1] , I found that passing a Map object rather than GenericArtifactFilter object to GenericArtifactManager would solve the problem [2]. When I changed the GenericArtifactFilter object to a Map object [3], it threw some exceptions. With the help of Greg team it was found out that the App Factory resources weren't indexed which result those exceptions. So I indexed the resource [5]. But after applying indexing I'm facing more problems as follows: 1. When an application is created, the 'trunk' created via 'appversion.rxt' will not be returned from getAppVersionRXTFromRegistry method in RxtManager.java until for few seconds (such as 10s), So it requires to implement retry attempts until it returns the version 'trunk' to proceed further in app creation process. 2. When an app version is created, newly created verson will not be returned by the same method mentioned in 1, due to the same reason and it will require some code changes to fix the exceptions introduced later in execution path. Moreover this might be a significant issue in Cloud as well. Please advice whether to keep the old implementation or to godhead with indexing and sort the remaining issues. [1] https://wso2.org/jira/browse/APPFAC-3009 [2][Dev] [Greg] What is the difference between overloaded findGenericArtifacts method in GenericArtifactManager - mail [3] https://github.com/punnadih/product-af/commit/dff7920619d25a134daef8f0134744908ae1e402 [4][Dev][App Fac] Registry resources Indexing issue fixed - mail -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com http://wso2 Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 http://lalajisureshika.blogspot.com/ -- Dimuthu Leelarathne Architect Product Lead of App Factory WSO2, Inc. (http://wso2.com) email: dimut...@wso2.com Mobile : 0773661935 Lean . Enterprise . Middleware ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
[Architecture] RFC: RESTFul API for API Manager
Hi All We are planing to implement a RESTFul API to expose the API Manager functionality. This will be a replacement to the currently provided Store and Publisher APIs [1] https://docs.wso2.com/display/AM180/Publisher+APIs [2] https://docs.wso2.com/display/AM180/Store+APIs. Main Motivation. 1. The current APIs are not RESTful and they do not cover all the functionality. 2. To make it easy to integrate and automate API manager functionality with 3rd party systems. 3. To provide better security with Oauth. 4. To provide better versioning and documentation with the API. As a start we have written a draft version of the API definition which you can find here [3] http://hevayo.github.io/restful-apim/. Following is a rough implementation plan. 1. Work on the API Definition, get feed back from users and finalize. 2. Implementation. ( Architecture , Jax-RS ?) 3. Adding Security. ( O-auth, scopes ? ) 4. Testing. 5. Documentation. API definition was written with Swagger 2 once completed we can use it to generate server stubs, client stubs and documentation. Please share your thoughts. Thanks Jo [1] https://docs.wso2.com/display/AM180/Publisher+APIs [2] https://docs.wso2.com/display/AM180/Store+APIs [3] http://hevayo.github.io/restful-apim/ -- *Joseph Fonseka* WSO2 Inc.; http://wso2.com lean.enterprise.middleware mobile: +94 772 512 430 skype: jpfonseka * http://lk.linkedin.com/in/rumeshbandara* ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
[Architecture] ESB Mediator for WSO2 Machine Learner
Hi, ESB mediator for WSO2 ML enables making use of the models generated by WSO2 Machine Leaner, to predict values in message mediation in ESB. - Machine Learner feature needs to be installed in ESB prior to configure the ML mediator. - Mediator will extract the data from the incoming messages and send to the ML model as features. - The mediator use the Machine Learner features to predict the output value for every message passing through the mediator. *Proposed Mediator Configuration* ml model storage-location=string/ feature name=string expression=xpath/+ prediction expression=xpath/ /ml *model* storage-location : Storage location of the generated model. *feature* name : Feature name according to the generated model. expression : XPath expression used to determine the feature value. *prediction* expression : XPath expression to determine the location to insert the predicted value. The generated models can be applied to predict only the data coming from the same domain, as the data used to train the model. So the mediator should be configured only to mediate messages which carry data coming from the same domain, as the data used to train the model. Any suggestion are appreciated. Thanks. -- Manorama Perera Software Engineer WSO2, Inc.; http://wso2.com/ Mobile : +94716436216 ___ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
