date:20161013

Re: [Architecture] Configuration files in C5

2016-10-13 Thread Jayanga Dissanayake

@Sidath: as of now we haven't identified, which are the configurations that
are going to be frequently changing in the C5. But with the past
experience, we know that all the configurations are not going to be changed
in all deployments. So, the idea is to expose those configurations (which
are likely to be frequently changed) to the users via
deployment.properties.

@Niranjan: you are correct. It seems that wording is misleading. I was
referring to some functionalities that are present in some profiles but not
needed on other profiles. thanks for pointing this.

@Imesh/@Azeez: I also believe that merging all the configurations into one
file would complicate the configuration process.
eg: in APIM we have around 30 different xml files in the conf directory
(excluding tomcat and axis2). So, combining all these into one file would
complicate the user experience IMO.

Thanks,
Jayanga.

*Jayanga Dissanayake*
Associate Technical Lead
WSO2 Inc. - http://wso2.com/
lean . enterprise . middleware
email: jaya...@wso2.com
mobile: +94772207259


On Fri, Oct 14, 2016 at 10:50 AM, Afkham Azeez  wrote:

> I think Imesh's suggestion merges all the config files and complicates
> stuff a lot. With the deployment.properties file we are including only the
> bits that most users will be concerned about and will provide a simple way
> to configure such stuff.
>
> On Fri, Oct 14, 2016 at 9:50 AM, Isuru Perera  wrote:
>
>> +1 for using a YAML file instead of a properties file.
>>
>> On Fri, Oct 14, 2016 at 8:45 AM, Imesh Gunaratne  wrote:
>>
>>> I would like to propose to use a single YAML file for each distribution
>>> (product/profile) to make the configuration process easier.
>>>
>>> I understand that we are trying to do something similar using a
>>> properties file (by overriding configurations in separate files), however
>>> IMO a properties file might not suite well for this purpose. A YAML file or
>>> any other type of a file which is more readable and designed for managing
>>> hierarchical data structures would work well. More importantly having a
>>> single configuration file would make the configuration process more simpler
>>> and clean. WDYT?
>>>
>>> Thanks
>>>
>>>
>>> On Thursday, October 13, 2016, Sidath Weerasinghe 
>>> wrote:
>>>
 Hi Jayanga,

 What are the most frequently changing configurations in C5 which are
 going to store in the deployment.properties" file ?

 On Thu, Oct 13, 2016 at 5:07 PM, Jayanga Dissanayake 
 wrote:

> Hi All,
>
> With C5, we introduced "ConfigResolver" which enhances the user
> experience in changing configuration values. With the previous C4x
> approach, users had to know where the configuration files are and to,
> change several configuration files to get the product working in some
> scenarios.
>
> With "ConfigResolver" it allows us to have more frequently changing
> configurations in one location "deployment.properties" file.
>
> A product has set of configurations that are needed to be changed in
> the deployments and there are some other configurations that we don't
> change unless there is a complex situation. Hence, ideally,
> deployment.properties file should contain only the configurations that are
> frequently used and can add more entries if a requirement arise.
>
> But with the requirements coming in with the "profile" support [1]. we
> have to rethink the way config resolver handle the configuration files.
>
> eg:
> 1. We need to enable indexing in API store and publisher, not in other
> profiles.
> 2. Enabling certain handlers in particular profiles.
>
> At present, there is no configuration to enable/disable these
> features. We have to rethink the way we define configurations in features
> in future. We have to have a way to enable/disable certain features so 
> that
> those could be disabled in certain profiles.
>
> Any idea/questions/clarifications are highly appreciated as it will
> help to model the new configurations story in C5.
>
> [1] "Multiple profile support for C5 based products."
>
> Thanks,
> *Jayanga Dissanayake*
> Associate Technical Lead
> WSO2 Inc. - http://wso2.com/
> lean . enterprise . middleware
> email: jaya...@wso2.com
> mobile: +94772207259
> 
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


 --
 Thank You,
 Best Regards,

 Sidath Weerasinghe


 *Intern*

 *WSO2, Inc. *

 *lean . enterprise . middleware *


 *Mobile: +94719802550 <%2B94719802550>*

 *Email: *sid...@wso2.com

Re: [Architecture] Storing configs in database for C5

2016-10-13 Thread Uvindra Dias Jayasinha

Let me summarize this thread so far.

The functionality provided by a DB makes it easy to implement reading,
updating and sharing configs, but there have been some legitimate concerns
that have been raised in this thread regarding that.

As has already been outlined deployment based configs should be part of the
container and so should be file based. We also have limitations with
certain components such as the Gateway which needs to be deployed in the
DMZ and so cannot connect with a DB in that case.

At the end of the day a file is a persistent storage just like a DB, albeit
much less sophisticated and localized. So maybe we can achieve some of the
advantages I outlined in my initial mail using file configs. The underlying
storage really does not matter(just an implementation detail) as long as we
can achieve the benefits that we need.


On 14 October 2016 at 00:11, Lahiru Sandaruwan  wrote:

> Hi All,
>
> If we try to categorize the configurations we have in C4, using the way we
> manage,
>
>1. *Runtime configurations*
>   - Any config you login to an UI and do
>   - E.g. API creation/ subscription in APIM, Mediation flow in
>   Integration Server, Service providers/ secondary user stores in IS, 
> etc.)
>   - We are using file system, direct databases, and registry as per
>   the requirements for this type
>   2. *One time configurations*
>   - Any one time config you configure using the files
>   - E.g. Configs we have in carbon.xml, axis2.xml, user-mgt.xml
>   - We only use file system for this type
>
>
> With C5 tenant approach, tenants will get to configure a considerable set
> of configs out of #2, but not all. IMO we should use a database for that
> set only, keeping the other one time configurations in the file system from
> #2.
>
> Even hot deployable files are an option for suitable use cases. (Not sure
> if we planning to move away from hot deployable files, with the container
> adaption). A GUI should be provided for managing them in the database or
> hot deployable files.
>
> In other words, we could move those configurations to #1 from #2.
> Wdyt?
>
> Thanks.
>
>
>
>
>
> On Thu, Oct 13, 2016 at 11:39 AM, Lakmal Warusawithana 
> wrote:
>
>>
>>
>> On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:
>>
>>>
>>>
>>> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera 
>>> wrote:
>>>
 I believe the plan is that for server configs, there is no (admin) UI
 in C5. It can be changed only via config files. End user scenarios such as
 API publisher might have UIs for configs.  ( We should agree here or have a
 meeting if we want to change that)

>>>
>>> On the API Cloud (and other clouds) we will have tenants changing some
>>> of their configs. So for that we will have to provide a UI since we won't
>>> have any other way to make them change files.
>>>
>>
>> @NuwanD, are these admin portal based config? like defining/changing
>> policies etc?
>>
>>
>>>
 What is an example of case where configs will take effect later?

 IMO storing configs in the dtabase make the deployment complicated.
 Also it reduce the scalability as all server point to one DB. I think it
 also make docker usecase complex.


 --Srinath

 On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:

>
>
> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta  > wrote:
>
>>
>>
>> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>>
>>> There are challenges when moving configs to the DB. We experienced
>>> it once when we moved the analytics configs to the registry. And then we
>>> moved it back again to the FS because it was too painful to maintain.
>>>
>>> 1. The nodes have to keep polling the DB in a fast enough interval.
>>> This is a unnecessary performance overhead. Because in practise, someone
>>> will only change these configs once. But to support that use case, we 
>>> have
>>> to keep polling the DB for life.
>>>
>>> 2. Gateways don't have access to the DB. So say you're enabling
>>> analytics (data publishing). You have to propagate that change to the
>>> Gateway nodes using some mechanism. And with no clustering on C5, this 
>>> is a
>>> challenge.
>>>
>>> If the objective of this is to make the Cloud (tenant) experience
>>> better, I think we should just restart the tenant's containers with the
>>> relevant configs in place.
>>>
>>
>> Still we have a problem with regard to how we are going to allow the
>> tenants to do the configuration changes. Currently we do it through the
>> registry which will not work for C5.
>>
>
> Yes, so my idea is to provide a UI to do the configs. Those configs we
> can store anywhere (maybe in a table) just for the sake of rendering that

Re: [Architecture] Configuration files in C5

2016-10-13 Thread Afkham Azeez

I think Imesh's suggestion merges all the config files and complicates
stuff a lot. With the deployment.properties file we are including only the
bits that most users will be concerned about and will provide a simple way
to configure such stuff.

On Fri, Oct 14, 2016 at 9:50 AM, Isuru Perera  wrote:

> +1 for using a YAML file instead of a properties file.
>
> On Fri, Oct 14, 2016 at 8:45 AM, Imesh Gunaratne  wrote:
>
>> I would like to propose to use a single YAML file for each distribution
>> (product/profile) to make the configuration process easier.
>>
>> I understand that we are trying to do something similar using a
>> properties file (by overriding configurations in separate files), however
>> IMO a properties file might not suite well for this purpose. A YAML file or
>> any other type of a file which is more readable and designed for managing
>> hierarchical data structures would work well. More importantly having a
>> single configuration file would make the configuration process more simpler
>> and clean. WDYT?
>>
>> Thanks
>>
>>
>> On Thursday, October 13, 2016, Sidath Weerasinghe 
>> wrote:
>>
>>> Hi Jayanga,
>>>
>>> What are the most frequently changing configurations in C5 which are
>>> going to store in the deployment.properties" file ?
>>>
>>> On Thu, Oct 13, 2016 at 5:07 PM, Jayanga Dissanayake 
>>> wrote:
>>>
 Hi All,

 With C5, we introduced "ConfigResolver" which enhances the user
 experience in changing configuration values. With the previous C4x
 approach, users had to know where the configuration files are and to,
 change several configuration files to get the product working in some
 scenarios.

 With "ConfigResolver" it allows us to have more frequently changing
 configurations in one location "deployment.properties" file.

 A product has set of configurations that are needed to be changed in
 the deployments and there are some other configurations that we don't
 change unless there is a complex situation. Hence, ideally,
 deployment.properties file should contain only the configurations that are
 frequently used and can add more entries if a requirement arise.

 But with the requirements coming in with the "profile" support [1]. we
 have to rethink the way config resolver handle the configuration files.

 eg:
 1. We need to enable indexing in API store and publisher, not in other
 profiles.
 2. Enabling certain handlers in particular profiles.

 At present, there is no configuration to enable/disable these features.
 We have to rethink the way we define configurations in features in future.
 We have to have a way to enable/disable certain features so that those
 could be disabled in certain profiles.

 Any idea/questions/clarifications are highly appreciated as it will
 help to model the new configurations story in C5.

 [1] "Multiple profile support for C5 based products."

 Thanks,
 *Jayanga Dissanayake*
 Associate Technical Lead
 WSO2 Inc. - http://wso2.com/
 lean . enterprise . middleware
 email: jaya...@wso2.com
 mobile: +94772207259

 ___
 Architecture mailing list
 Architecture@wso2.org
 https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

>>>
>>>
>>> --
>>> Thank You,
>>> Best Regards,
>>>
>>> Sidath Weerasinghe
>>>
>>>
>>> *Intern*
>>>
>>> *WSO2, Inc. *
>>>
>>> *lean . enterprise . middleware *
>>>
>>>
>>> *Mobile: +94719802550 <%2B94719802550>*
>>>
>>> *Email: *sid...@wso2.com
>>>
>>> Blog: https://medium.com/@sidath
>>>
>>> Linkedin: https://lk.linkedin.com/in/sidathweerasinghe
>>>
>>
>>
>> --
>> *Imesh Gunaratne*
>> Software Architect
>> WSO2 Inc: http://wso2.com
>> T: +94 11 214 5345 M: +94 77 374 2057
>> W: https://medium.com/@imesh TW: @imesh
>> lean. enterprise. middleware
>>
>>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Isuru Perera
> Associate Technical Lead | WSO2, Inc. | http://wso2.com/
> Lean . Enterprise . Middleware
>
> about.me/chrishantha
> Contact: +IsuruPereraWSO2 
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>

-- 
*Afkham Azeez*
Director of Architecture; WSO2, Inc.; http://wso2.com
Member; Apache Software Foundation; http://www.apache.org/
* *
*email: **az...@wso2.com* 
* cell: +94 77 3320919blog: **http://blog.afkham.org*

*twitter: **http://twitter.com/afkham_azeez*

*linked-in:

Re: [Architecture] Configuration files in C5

2016-10-13 Thread Isuru Perera

+1 for using a YAML file instead of a properties file.

On Fri, Oct 14, 2016 at 8:45 AM, Imesh Gunaratne  wrote:

> I would like to propose to use a single YAML file for each distribution
> (product/profile) to make the configuration process easier.
>
> I understand that we are trying to do something similar using a properties
> file (by overriding configurations in separate files), however IMO a
> properties file might not suite well for this purpose. A YAML file or any
> other type of a file which is more readable and designed for managing
> hierarchical data structures would work well. More importantly having a
> single configuration file would make the configuration process more simpler
> and clean. WDYT?
>
> Thanks
>
>
> On Thursday, October 13, 2016, Sidath Weerasinghe  wrote:
>
>> Hi Jayanga,
>>
>> What are the most frequently changing configurations in C5 which are
>> going to store in the deployment.properties" file ?
>>
>> On Thu, Oct 13, 2016 at 5:07 PM, Jayanga Dissanayake 
>> wrote:
>>
>>> Hi All,
>>>
>>> With C5, we introduced "ConfigResolver" which enhances the user
>>> experience in changing configuration values. With the previous C4x
>>> approach, users had to know where the configuration files are and to,
>>> change several configuration files to get the product working in some
>>> scenarios.
>>>
>>> With "ConfigResolver" it allows us to have more frequently changing
>>> configurations in one location "deployment.properties" file.
>>>
>>> A product has set of configurations that are needed to be changed in the
>>> deployments and there are some other configurations that we don't change
>>> unless there is a complex situation. Hence, ideally, deployment.properties
>>> file should contain only the configurations that are frequently used and
>>> can add more entries if a requirement arise.
>>>
>>> But with the requirements coming in with the "profile" support [1]. we
>>> have to rethink the way config resolver handle the configuration files.
>>>
>>> eg:
>>> 1. We need to enable indexing in API store and publisher, not in other
>>> profiles.
>>> 2. Enabling certain handlers in particular profiles.
>>>
>>> At present, there is no configuration to enable/disable these features.
>>> We have to rethink the way we define configurations in features in future.
>>> We have to have a way to enable/disable certain features so that those
>>> could be disabled in certain profiles.
>>>
>>> Any idea/questions/clarifications are highly appreciated as it will help
>>> to model the new configurations story in C5.
>>>
>>> [1] "Multiple profile support for C5 based products."
>>>
>>> Thanks,
>>> *Jayanga Dissanayake*
>>> Associate Technical Lead
>>> WSO2 Inc. - http://wso2.com/
>>> lean . enterprise . middleware
>>> email: jaya...@wso2.com
>>> mobile: +94772207259
>>> 
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> Thank You,
>> Best Regards,
>>
>> Sidath Weerasinghe
>>
>>
>> *Intern*
>>
>> *WSO2, Inc. *
>>
>> *lean . enterprise . middleware *
>>
>>
>> *Mobile: +94719802550 <%2B94719802550>*
>>
>> *Email: *sid...@wso2.com
>>
>> Blog: https://medium.com/@sidath
>>
>> Linkedin: https://lk.linkedin.com/in/sidathweerasinghe
>>
>
>
> --
> *Imesh Gunaratne*
> Software Architect
> WSO2 Inc: http://wso2.com
> T: +94 11 214 5345 M: +94 77 374 2057
> W: https://medium.com/@imesh TW: @imesh
> lean. enterprise. middleware
>
>
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha
Contact: +IsuruPereraWSO2 
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] Using HdrHistogram as a reservoir in Carbon Metrics

2016-10-13 Thread Isuru Perera

Hi,

In Carbon Metrics [1], the TImers and Histograms use the default reservoir
provided by the Dropwizard Metrics [2] library.

Currently there is no support to change this reservoir implementation in
Carbon Metrics. Therefore we are working on supporting a set of reservoirs
in Carbon Metrics.

The HdrHistogram [3] is a popular histogram implementation and it has lot
of useful features for us. For more info, see [4]. One major advantage for
us is that HDR Histogram maintains a fixed cost in both space and time.

When using HdrHistogram as reservoir, we can also a solve a problem with
the default Exponentially Decaying Reservoir when publishing metrics events
to WSO2 Data Analytics Server. The problem is explained in [5]. With HDR
Histogram, we can reset the snapshot when reporting to WSO2 DAS.

We are going to integrate HDR Histogram into the next Carbon Metrics
release, which is based on Carbon 5.x

We don't have plans to do any release of Carbon Metrics based on Carbon
4.x. If anyone needs the HDR Histogram as a reservoir, we'll be able to do
changes in 1.x.x [6] branch and do a release.

Thanks!

Best Regards,

[1] https://github.com/wso2/carbon-metrics
[2] http://metrics.dropwizard.io
[3] https://github.com/HdrHistogram/HdrHistogram
[4] http://hdrhistogram.github.io/HdrHistogram/
[5] http://taint.org/2014/01/16/145944a.html
[6] https://github.com/wso2/carbon-metrics/branches

-- 
Isuru Perera
Associate Technical Lead | WSO2, Inc. | http://wso2.com/
Lean . Enterprise . Middleware

about.me/chrishantha
Contact: +IsuruPereraWSO2 
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Configuration files in C5

2016-10-13 Thread Niranjan Karunanandham

Hi Jayanga,

On Thu, Oct 13, 2016 at 5:07 PM, Jayanga Dissanayake 
wrote:

> Hi All,
>
> With C5, we introduced "ConfigResolver" which enhances the user experience
> in changing configuration values. With the previous C4x approach, users had
> to know where the configuration files are and to, change several
> configuration files to get the product working in some scenarios.
>
> With "ConfigResolver" it allows us to have more frequently changing
> configurations in one location "deployment.properties" file.
>
> A product has set of configurations that are needed to be changed in the
> deployments and there are some other configurations that we don't change
> unless there is a complex situation. Hence, ideally, deployment.properties
> file should contain only the configurations that are frequently used and
> can add more entries if a requirement arise.
>
> But with the requirements coming in with the "profile" support [1]. we
> have to rethink the way config resolver handle the configuration files.
>
> eg:
> 1. We need to enable indexing in API store and publisher, not in other
> profiles.
> 2. Enabling certain handlers in particular profiles.
>
> At present, there is no configuration to enable/disable these features. We
> have to rethink the way we define configurations in features in future. We
> have to have a way to enable/disable certain features so that those could
> be disabled in certain profiles.
>
The above two examples that you have mentioned cannot be called features
(please correct me if am wrong). AFAIU those are functionalities which are
specific to profiles, i.e., certain features have multiple options
(configurations) based on which the features functionality changes.
Therefore the configurations can change from feature to feature?


>
> Any idea/questions/clarifications are highly appreciated as it will help
> to model the new configurations story in C5.
>
> [1] "Multiple profile support for C5 based products."
>
> Thanks,
> *Jayanga Dissanayake*
> Associate Technical Lead
> WSO2 Inc. - http://wso2.com/
> lean . enterprise . middleware
> email: jaya...@wso2.com
> mobile: +94772207259
> 
>

Regards,
Nira

-- 


*Niranjan Karunanandham*
Associate Technical Lead - WSO2 Inc.
WSO2 Inc.: http://www.wso2.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Configuration files in C5

2016-10-13 Thread Imesh Gunaratne

I would like to propose to use a single YAML file for each distribution
(product/profile) to make the configuration process easier.

I understand that we are trying to do something similar using a properties
file (by overriding configurations in separate files), however IMO a
properties file might not suite well for this purpose. A YAML file or any
other type of a file which is more readable and designed for managing
hierarchical data structures would work well. More importantly having a
single configuration file would make the configuration process more simpler
and clean. WDYT?

Thanks

On Thursday, October 13, 2016, Sidath Weerasinghe  wrote:

> Hi Jayanga,
>
> What are the most frequently changing configurations in C5 which are going
> to store in the deployment.properties" file ?
>
> On Thu, Oct 13, 2016 at 5:07 PM, Jayanga Dissanayake  > wrote:
>
>> Hi All,
>>
>> With C5, we introduced "ConfigResolver" which enhances the user
>> experience in changing configuration values. With the previous C4x
>> approach, users had to know where the configuration files are and to,
>> change several configuration files to get the product working in some
>> scenarios.
>>
>> With "ConfigResolver" it allows us to have more frequently changing
>> configurations in one location "deployment.properties" file.
>>
>> A product has set of configurations that are needed to be changed in the
>> deployments and there are some other configurations that we don't change
>> unless there is a complex situation. Hence, ideally, deployment.properties
>> file should contain only the configurations that are frequently used and
>> can add more entries if a requirement arise.
>>
>> But with the requirements coming in with the "profile" support [1]. we
>> have to rethink the way config resolver handle the configuration files.
>>
>> eg:
>> 1. We need to enable indexing in API store and publisher, not in other
>> profiles.
>> 2. Enabling certain handlers in particular profiles.
>>
>> At present, there is no configuration to enable/disable these features.
>> We have to rethink the way we define configurations in features in future.
>> We have to have a way to enable/disable certain features so that those
>> could be disabled in certain profiles.
>>
>> Any idea/questions/clarifications are highly appreciated as it will help
>> to model the new configurations story in C5.
>>
>> [1] "Multiple profile support for C5 based products."
>>
>> Thanks,
>> *Jayanga Dissanayake*
>> Associate Technical Lead
>> WSO2 Inc. - http://wso2.com/
>> lean . enterprise . middleware
>> email: jaya...@wso2.com
>> 
>> mobile: +94772207259
>> 
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> 
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Thank You,
> Best Regards,
>
> Sidath Weerasinghe
>
>
> *Intern*
>
> *WSO2, Inc. *
>
> *lean . enterprise . middleware *
>
>
> *Mobile: +94719802550*
>
> *Email: *sid...@wso2.com 
>
> Blog: https://medium.com/@sidath
>
> Linkedin: https://lk.linkedin.com/in/sidathweerasinghe
>


-- 
*Imesh Gunaratne*
Software Architect
WSO2 Inc: http://wso2.com
T: +94 11 214 5345 M: +94 77 374 2057
W: https://medium.com/@imesh TW: @imesh
lean. enterprise. middleware
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] ESB connector auto generation tool

2016-10-13 Thread Malaka Silva

I think we should have both options and user can control the behavior.

Also we should automate init method for known use cases.

On Thu, Oct 13, 2016 at 4:08 PM, Keerthika Mahendralingam <
keerth...@wso2.com> wrote:

> IMO we need to go ahead with the first approach since there may be some
> cases we need to have some additional properties, headers, or host, etc.
> which are not common for all connectors. These are depend on the backend
> service.
>
>
> Thanks,
> Keerthika.
>
> On Thu, Oct 13, 2016 at 3:55 PM, Rajjaz Mohammed  wrote:
>
>> Hi All,
>>
>> We have two options in the output of auto-generation tool.
>>
>>1. source code of the connector
>>2. connector zip
>>
>> If we generate connector zip file directly we can reduce the manual works
>> but if we generate the source code from this tool
>>
>>- add init method manually for authentication
>>- modify some special cases like adding DISABLE_CHUNKING etc.
>>- integration test
>>
>> WDYT?
>>
>> On Wed, Oct 12, 2016 at 3:28 PM, Malaka Silva  wrote:
>>
>>> Yes Authorization is bit tricky and specific to vendors. So we can have
>>> predefined templates (eg: Salesforce custom web services) and make this
>>> plug-able?
>>>
>>> On Wed, Oct 12, 2016 at 2:44 PM, Rajjaz Mohammed 
>>> wrote:
>>>
 Hi All,

 I have done some testings on this area for that I created an apex
 class[1] to create an account in salesforce and tested with simple java
 client using WSDL file and it works fine and can able to extract the common
 information from WSDL (attached).

 Our next step will create the connector based on the information
 extracted from WSDL. But we can only create the synapse template
 configuration which is general. Another challenge is the authentication
 part so we plan to create the init as the separate module  because it will
 not same to all connectors.

 [1]

 global class ConnectorAutomation{
 global class RequestClass{
 webService String accountName;
 }
 global class ResponseClass{
 webService String responseResultID;
 webService String responseResultName;
 webService String responseResultRecordType;
 }
 webService static ResponseClass addAccount(String name){
 Account acct = new Account();
 acct.Name = name;
 insert acct;
 ResponseClass resClass = new ResponseClass();
 resClass.responseResultID = acct.Id;
 resClass.responseResultName = acct.Name;
 resClass.responseResultRecordType = acct.AccountNumber;
 return resClass;
 }
 }



 On Wed, Sep 21, 2016 at 10:41 AM, Malaka Silva  wrote:

> Yes Kasun this is what our end goal is. We are starting this with soap
> based services. Also great if you can share any work done to speed up our
> r
>
> @Ajanthan
>  - Yes we are currently planning to have this a command line tool,
> however this is not yet finalized.
>
> On Wed, Sep 21, 2016 at 2:09 AM, Kasun Indrasiri 
> wrote:
>
>> Hi Malaka,
>>
>> We did a PoC project on generating a connector based on a given
>> Swagger definition. Is this a similar requirement?
>>
>> On Tue, Sep 20, 2016 at 10:51 AM, Ajanthan Balachandran <
>> ajant...@wso2.com> wrote:
>>
>>> What do you mean by a tool? Is it command line tool  or maven plugin
>>> or eclipse plugin?
>>>
>>> On Fri, Sep 9, 2016 at 2:07 AM, Rajjaz Mohammed 
>>> wrote:
>>>

 Hi,
>
> We have currently 150+ connectors in store
> . Using those we can easily build
> integration use cases with WSO2 ESB.
>
> However there are some apis that resides on premises and specific
> to some users. If we need to integrate such services, we either need 
> to
> manually do the integration with ESB or develop a connector and use 
> it.
>
> The idea of this project is to automate the development of
> connectors that makes the integration tasks more productive.
>
> So we are planning to start this with soap based connectors and
> move to rest based support later.
>
> For soap based connector generation we basically need to parse the
> wsdl and generate a connector operation per soap operation.
>
> For that we can use WSDL4J. Using this we can get the required
> operations and request/response messages required. Using this 
> information
> we can build the connector operations.(Sequence Templates)
>
> eg:
> String wsdlPath = "/home/wso2/Desktop/ConnectorTest.wsdl";
> WSDLReader

Re: [Architecture] Storing configs in database for C5

2016-10-13 Thread Lahiru Sandaruwan

Hi All,

If we try to categorize the configurations we have in C4, using the way we
manage,

   1. *Runtime configurations*
  - Any config you login to an UI and do
  - E.g. API creation/ subscription in APIM, Mediation flow in
  Integration Server, Service providers/ secondary user stores in IS, etc.)
  - We are using file system, direct databases, and registry as per the
  requirements for this type
  2. *One time configurations*
  - Any one time config you configure using the files
  - E.g. Configs we have in carbon.xml, axis2.xml, user-mgt.xml
  - We only use file system for this type


With C5 tenant approach, tenants will get to configure a considerable set
of configs out of #2, but not all. IMO we should use a database for that
set only, keeping the other one time configurations in the file system from
#2.

Even hot deployable files are an option for suitable use cases. (Not sure
if we planning to move away from hot deployable files, with the container
adaption). A GUI should be provided for managing them in the database or
hot deployable files.

In other words, we could move those configurations to #1 from #2.
Wdyt?

Thanks.





On Thu, Oct 13, 2016 at 11:39 AM, Lakmal Warusawithana 
wrote:

>
>
> On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:
>
>>
>>
>> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera  wrote:
>>
>>> I believe the plan is that for server configs, there is no (admin) UI in
>>> C5. It can be changed only via config files. End user scenarios such as API
>>> publisher might have UIs for configs.  ( We should agree here or have a
>>> meeting if we want to change that)
>>>
>>
>> On the API Cloud (and other clouds) we will have tenants changing some of
>> their configs. So for that we will have to provide a UI since we won't have
>> any other way to make them change files.
>>
>
> @NuwanD, are these admin portal based config? like defining/changing
> policies etc?
>
>
>>
>>> What is an example of case where configs will take effect later?
>>>
>>> IMO storing configs in the dtabase make the deployment complicated. Also
>>> it reduce the scalability as all server point to one DB. I think it also
>>> make docker usecase complex.
>>>
>>>
>>> --Srinath
>>>
>>> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:
>>>


 On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
 wrote:

>
>
> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>
>> There are challenges when moving configs to the DB. We experienced it
>> once when we moved the analytics configs to the registry. And then we 
>> moved
>> it back again to the FS because it was too painful to maintain.
>>
>> 1. The nodes have to keep polling the DB in a fast enough interval.
>> This is a unnecessary performance overhead. Because in practise, someone
>> will only change these configs once. But to support that use case, we 
>> have
>> to keep polling the DB for life.
>>
>> 2. Gateways don't have access to the DB. So say you're enabling
>> analytics (data publishing). You have to propagate that change to the
>> Gateway nodes using some mechanism. And with no clustering on C5, this 
>> is a
>> challenge.
>>
>> If the objective of this is to make the Cloud (tenant) experience
>> better, I think we should just restart the tenant's containers with the
>> relevant configs in place.
>>
>
> Still we have a problem with regard to how we are going to allow the
> tenants to do the configuration changes. Currently we do it through the
> registry which will not work for C5.
>

 Yes, so my idea is to provide a UI to do the configs. Those configs we
 can store anywhere (maybe in a table) just for the sake of rendering that
 UI. The product code will still read from the config files. When you apply
 those configs through the press of a button, the container should get
 rebuilt and restarted with the necessary modification to the config files.

>
> Thanks,
> Lakmali
>
>>
>> Thanks,
>> NuwanD.
>>
>> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta <
>> lakm...@wso2.com> wrote:
>>
>>>
>>>
>>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha <
>>> uvin...@wso2.com> wrote:
>>>
 Hi Sajith,

 Yes even though the boot up time is not an issue in C5 the other
 advantages I have outlined are still there to be gained. There is a 
 huge
 effort we have to do on dev ops side to maintain those images you are
 talking about because of having everything at file level.

 Some examples from API Manager I can think of are turning
 notifications on/off, enable monetization, enable/disable stats, 
 configure

Re: [Architecture] Configuration files in C5

2016-10-13 Thread Sidath Weerasinghe

Hi Jayanga,

What are the most frequently changing configurations in C5 which are going
to store in the deployment.properties" file ?

On Thu, Oct 13, 2016 at 5:07 PM, Jayanga Dissanayake 
wrote:

> Hi All,
>
> With C5, we introduced "ConfigResolver" which enhances the user experience
> in changing configuration values. With the previous C4x approach, users had
> to know where the configuration files are and to, change several
> configuration files to get the product working in some scenarios.
>
> With "ConfigResolver" it allows us to have more frequently changing
> configurations in one location "deployment.properties" file.
>
> A product has set of configurations that are needed to be changed in the
> deployments and there are some other configurations that we don't change
> unless there is a complex situation. Hence, ideally, deployment.properties
> file should contain only the configurations that are frequently used and
> can add more entries if a requirement arise.
>
> But with the requirements coming in with the "profile" support [1]. we
> have to rethink the way config resolver handle the configuration files.
>
> eg:
> 1. We need to enable indexing in API store and publisher, not in other
> profiles.
> 2. Enabling certain handlers in particular profiles.
>
> At present, there is no configuration to enable/disable these features. We
> have to rethink the way we define configurations in features in future. We
> have to have a way to enable/disable certain features so that those could
> be disabled in certain profiles.
>
> Any idea/questions/clarifications are highly appreciated as it will help
> to model the new configurations story in C5.
>
> [1] "Multiple profile support for C5 based products."
>
> Thanks,
> *Jayanga Dissanayake*
> Associate Technical Lead
> WSO2 Inc. - http://wso2.com/
> lean . enterprise . middleware
> email: jaya...@wso2.com
> mobile: +94772207259
> 
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Thank You,
Best Regards,

Sidath Weerasinghe


*Intern*

*WSO2, Inc. *

*lean . enterprise . middleware *


*Mobile: +94719802550*

*Email: *sid...@wso2.com

Blog: https://medium.com/@sidath

Linkedin: https://lk.linkedin.com/in/sidathweerasinghe
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] [APIM] Life cycle Management

2016-10-13 Thread Rajith Roshan

Hi,

The current implementation keeps the data in a separate database. For now
life cycle  component has its own data source. I think its better to read
the data source name from a configuration so the tables related to life
cycle operations can be populated in an existing database as well. For ex:
if the config specifies the AM _DB then these tables will be created in API
Manager database, without using separate database.

Thanks!
Rajith

On Mon, Oct 10, 2016 at 3:49 PM, Prasanna Dangalla 
wrote:

>
> Hi Rajith,
>
> On Mon, Oct 10, 2016 at 3:30 PM, Rajith Roshan  wrote:
>
>> Hi,
>>>
>> please find the images below
>>
>>> On Mon, Oct 10, 2016 at 3:16 PM, Rajith Roshan  wrote:
>>>
 Hi all,

 Life cycle is an integral part in any product. Each SOA governance
 related artifacts can have their own life cycle. Capabilities provided in
 order to manage life cycles not only allow you to properly organize your
 assets but also provides many extensibilities (For ex through custom
 executors)

 RequirementCurrent API life cycle of API manager completely depends
  on the life cycle implementation provided by the registry. Since we are
 moving away from the registry concept we need a completely new life cycle
 management framework which cater for life cycle management within API
 Manager and should be able to ship with other products which require life
 cycle management.

 Proposed SolutionThe basic idea is to completely decouple the life
 cycle framework from the system to which its provide life cycle
 capabilities. I.e the system which uses life cycle will not change the
 behavior of the life cycle framework and only vice versa will be
 applicable. The framework should exist independent of the asset type to
 which it provides life cycle capability

 The mapping should be maintained by asset type in order to associate
 with life cycle data. To be more specific in database schema level each
 asset type should update their tables (add extra columns to maintain
 mapping with life cycle data) in order to map  life cycle data.

 The external systems which uses life cycle service should connect with
 the service in order to have a unique life cycle id which is generated
 by the service. This id should be stored in the external system in order to
 maintain the mapping (Each asset should have its own life cycle id). On the
 other hand life cycle framework will also store this id in order to provide
 all the life cycle related operations for a particular asset.

 Basically, we will be providing a class (ManagedLifecycle) with all the
 required basic operations. Any asset which requires life cycle management
 can extend this class.
 Further this can be extended to support features like check list items
 as well
 Features supported

-

Custom Inputs - User should  provide with the capability in order
to save custom values per each state, which can be passed to executors .
- Executors - Which are custom classes executed during life cycle
state change operation

 Please provide you valuable inputs.

 Thanks!
 Rajith

 --
 Rajith Roshan
 Software Engineer, WSO2 Inc.
 Mobile: +94-72-642-8350 <%2B94-71-554-8430>

>>>
>>
>>
>> --
>> Rajith Roshan
>> Software Engineer, WSO2 Inc.
>> Mobile: +94-72-642-8350 <%2B94-71-554-8430>
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
> Since this proposed feature is running separately, are we keeping the db
> tables in a separate database? If so, we can separate them from products
> and bind them to the feature. IMHO it will be an advantage.
>
> Regards,
>
> *Prasanna Dangalla*
> Senior Software Engineer, WSO2, Inc.; http://wso2.com/
> lean.enterprise.middleware
>
>
> *cell: +94 718 11 27 51*
> *twitter: @prasa77*
>
>

-- 
Rajith Roshan
Software Engineer, WSO2 Inc.
Mobile: +94-72-642-8350 <%2B94-71-554-8430>
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

[Architecture] Configuration files in C5

2016-10-13 Thread Jayanga Dissanayake

Hi All,

With C5, we introduced "ConfigResolver" which enhances the user experience
in changing configuration values. With the previous C4x approach, users had
to know where the configuration files are and to, change several
configuration files to get the product working in some scenarios.

With "ConfigResolver" it allows us to have more frequently changing
configurations in one location "deployment.properties" file.

A product has set of configurations that are needed to be changed in the
deployments and there are some other configurations that we don't change
unless there is a complex situation. Hence, ideally, deployment.properties
file should contain only the configurations that are frequently used and
can add more entries if a requirement arise.

But with the requirements coming in with the "profile" support [1]. we have
to rethink the way config resolver handle the configuration files.

eg:
1. We need to enable indexing in API store and publisher, not in other
profiles.
2. Enabling certain handlers in particular profiles.

At present, there is no configuration to enable/disable these features. We
have to rethink the way we define configurations in features in future. We
have to have a way to enable/disable certain features so that those could
be disabled in certain profiles.

Any idea/questions/clarifications are highly appreciated as it will help to
model the new configurations story in C5.

[1] "Multiple profile support for C5 based products."

Thanks,
*Jayanga Dissanayake*
Associate Technical Lead
WSO2 Inc. - http://wso2.com/
lean . enterprise . middleware
email: jaya...@wso2.com
mobile: +94772207259

___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] ESB connector auto generation tool

2016-10-13 Thread Keerthika Mahendralingam

IMO we need to go ahead with the first approach since there may be some
cases we need to have some additional properties, headers, or host, etc.
which are not common for all connectors. These are depend on the backend
service.


Thanks,
Keerthika.

On Thu, Oct 13, 2016 at 3:55 PM, Rajjaz Mohammed  wrote:

> Hi All,
>
> We have two options in the output of auto-generation tool.
>
>1. source code of the connector
>2. connector zip
>
> If we generate connector zip file directly we can reduce the manual works
> but if we generate the source code from this tool
>
>- add init method manually for authentication
>- modify some special cases like adding DISABLE_CHUNKING etc.
>- integration test
>
> WDYT?
>
> On Wed, Oct 12, 2016 at 3:28 PM, Malaka Silva  wrote:
>
>> Yes Authorization is bit tricky and specific to vendors. So we can have
>> predefined templates (eg: Salesforce custom web services) and make this
>> plug-able?
>>
>> On Wed, Oct 12, 2016 at 2:44 PM, Rajjaz Mohammed  wrote:
>>
>>> Hi All,
>>>
>>> I have done some testings on this area for that I created an apex
>>> class[1] to create an account in salesforce and tested with simple java
>>> client using WSDL file and it works fine and can able to extract the common
>>> information from WSDL (attached).
>>>
>>> Our next step will create the connector based on the information
>>> extracted from WSDL. But we can only create the synapse template
>>> configuration which is general. Another challenge is the authentication
>>> part so we plan to create the init as the separate module  because it will
>>> not same to all connectors.
>>>
>>> [1]
>>>
>>> global class ConnectorAutomation{
>>> global class RequestClass{
>>> webService String accountName;
>>> }
>>> global class ResponseClass{
>>> webService String responseResultID;
>>> webService String responseResultName;
>>> webService String responseResultRecordType;
>>> }
>>> webService static ResponseClass addAccount(String name){
>>> Account acct = new Account();
>>> acct.Name = name;
>>> insert acct;
>>> ResponseClass resClass = new ResponseClass();
>>> resClass.responseResultID = acct.Id;
>>> resClass.responseResultName = acct.Name;
>>> resClass.responseResultRecordType = acct.AccountNumber;
>>> return resClass;
>>> }
>>> }
>>>
>>>
>>>
>>> On Wed, Sep 21, 2016 at 10:41 AM, Malaka Silva  wrote:
>>>
 Yes Kasun this is what our end goal is. We are starting this with soap
 based services. Also great if you can share any work done to speed up our
 r

 @Ajanthan
  - Yes we are currently planning to have this a command line tool,
 however this is not yet finalized.

 On Wed, Sep 21, 2016 at 2:09 AM, Kasun Indrasiri 
 wrote:

> Hi Malaka,
>
> We did a PoC project on generating a connector based on a given
> Swagger definition. Is this a similar requirement?
>
> On Tue, Sep 20, 2016 at 10:51 AM, Ajanthan Balachandran <
> ajant...@wso2.com> wrote:
>
>> What do you mean by a tool? Is it command line tool  or maven plugin
>> or eclipse plugin?
>>
>> On Fri, Sep 9, 2016 at 2:07 AM, Rajjaz Mohammed 
>> wrote:
>>
>>>
>>> Hi,

 We have currently 150+ connectors in store
 . Using those we can easily build
 integration use cases with WSO2 ESB.

 However there are some apis that resides on premises and specific
 to some users. If we need to integrate such services, we either need to
 manually do the integration with ESB or develop a connector and use it.

 The idea of this project is to automate the development of
 connectors that makes the integration tasks more productive.

 So we are planning to start this with soap based connectors and
 move to rest based support later.

 For soap based connector generation we basically need to parse the
 wsdl and generate a connector operation per soap operation.

 For that we can use WSDL4J. Using this we can get the required
 operations and request/response messages required. Using this 
 information
 we can build the connector operations.(Sequence Templates)

 eg:
 String wsdlPath = "/home/wso2/Desktop/ConnectorTest.wsdl";
 WSDLReader reader = javax.wsdl.factory.WSDLFactory
 .newInstance().newWSDLReader();
 javax.wsdl.Definition defn = reader.readWSDL(wsdlPath);

 Map tmp =
 defn.getAllServices();

 for(javax.xml.namespace.QName  key:tmp.keySet()){
 ServiceImpl serviceImpl = tmp.get(key);
 Map  mPorts =

Re: [Architecture] ESB connector auto generation tool

2016-10-13 Thread Rajjaz Mohammed

Hi All,

We have two options in the output of auto-generation tool.

   1. source code of the connector
   2. connector zip

If we generate connector zip file directly we can reduce the manual works
but if we generate the source code from this tool

   - add init method manually for authentication
   - modify some special cases like adding DISABLE_CHUNKING etc.
   - integration test

WDYT?

On Wed, Oct 12, 2016 at 3:28 PM, Malaka Silva  wrote:

> Yes Authorization is bit tricky and specific to vendors. So we can have
> predefined templates (eg: Salesforce custom web services) and make this
> plug-able?
>
> On Wed, Oct 12, 2016 at 2:44 PM, Rajjaz Mohammed  wrote:
>
>> Hi All,
>>
>> I have done some testings on this area for that I created an apex
>> class[1] to create an account in salesforce and tested with simple java
>> client using WSDL file and it works fine and can able to extract the common
>> information from WSDL (attached).
>>
>> Our next step will create the connector based on the information
>> extracted from WSDL. But we can only create the synapse template
>> configuration which is general. Another challenge is the authentication
>> part so we plan to create the init as the separate module  because it will
>> not same to all connectors.
>>
>> [1]
>>
>> global class ConnectorAutomation{
>> global class RequestClass{
>> webService String accountName;
>> }
>> global class ResponseClass{
>> webService String responseResultID;
>> webService String responseResultName;
>> webService String responseResultRecordType;
>> }
>> webService static ResponseClass addAccount(String name){
>> Account acct = new Account();
>> acct.Name = name;
>> insert acct;
>> ResponseClass resClass = new ResponseClass();
>> resClass.responseResultID = acct.Id;
>> resClass.responseResultName = acct.Name;
>> resClass.responseResultRecordType = acct.AccountNumber;
>> return resClass;
>> }
>> }
>>
>>
>>
>> On Wed, Sep 21, 2016 at 10:41 AM, Malaka Silva  wrote:
>>
>>> Yes Kasun this is what our end goal is. We are starting this with soap
>>> based services. Also great if you can share any work done to speed up our
>>> r
>>>
>>> @Ajanthan
>>>  - Yes we are currently planning to have this a command line tool,
>>> however this is not yet finalized.
>>>
>>> On Wed, Sep 21, 2016 at 2:09 AM, Kasun Indrasiri  wrote:
>>>
 Hi Malaka,

 We did a PoC project on generating a connector based on a given Swagger
 definition. Is this a similar requirement?

 On Tue, Sep 20, 2016 at 10:51 AM, Ajanthan Balachandran <
 ajant...@wso2.com> wrote:

> What do you mean by a tool? Is it command line tool  or maven plugin
> or eclipse plugin?
>
> On Fri, Sep 9, 2016 at 2:07 AM, Rajjaz Mohammed 
> wrote:
>
>>
>> Hi,
>>>
>>> We have currently 150+ connectors in store
>>> . Using those we can easily build
>>> integration use cases with WSO2 ESB.
>>>
>>> However there are some apis that resides on premises and specific to
>>> some users. If we need to integrate such services, we either need to
>>> manually do the integration with ESB or develop a connector and use it.
>>>
>>> The idea of this project is to automate the development of
>>> connectors that makes the integration tasks more productive.
>>>
>>> So we are planning to start this with soap based connectors and move
>>> to rest based support later.
>>>
>>> For soap based connector generation we basically need to parse the
>>> wsdl and generate a connector operation per soap operation.
>>>
>>> For that we can use WSDL4J. Using this we can get the required
>>> operations and request/response messages required. Using this 
>>> information
>>> we can build the connector operations.(Sequence Templates)
>>>
>>> eg:
>>> String wsdlPath = "/home/wso2/Desktop/ConnectorTest.wsdl";
>>> WSDLReader reader = javax.wsdl.factory.WSDLFactory
>>> .newInstance().newWSDLReader();
>>> javax.wsdl.Definition defn = reader.readWSDL(wsdlPath);
>>>
>>> Map tmp =
>>> defn.getAllServices();
>>>
>>> for(javax.xml.namespace.QName  key:tmp.keySet()){
>>> ServiceImpl serviceImpl = tmp.get(key);
>>> Map  mPorts = serviceImpl.getPorts();
>>> for(String k1:mPorts.keySet()){
>>> PortImpl portImpl = mPorts.get(k1);
>>> List bindingOperations =
>>> portImpl.getBinding().getBindingOperations();
>>> for(BindingOperationImpl bindingOperation:bindingOperations){
>>> System.out.println("operation:" + bindingOperation.getName());
>>> BindingInput bindingInput = bindingOperation.getBindingInput();
>>> }
>>> }
>>> }
>>> Map messages = defn.getMessages();

Re: [Architecture] Multiple profile support for C5 based products.

2016-10-13 Thread Afkham Azeez

Please note that during a discussion this week, we decided that instead of
all carbon servers running on the same ports, each of the 5 products will
have their own well known ports. For example, APIM GW port will be 8084
(just an example). So while we would still have portOffset support, we
won't need to specify the offset when running multiple products on the same
machine and they will be able to discover each other on these well known
ports.

On Wed, Oct 12, 2016 at 6:29 PM, Nuwan Dias  wrote:

> Do we really need multi-profile support (at an osgi level) on C5? What if
> we have separate dedicated runtimes per profile? Which means that we have a
> gateway runtime, store runtime, etc. within the same distribution? Each
> will run on its own port, maybe using offsets by default (no need to worry
> about that if each profile is a container).
>
> Also if we have profiles, do we also have a global profile which runs all
> components in one? I doubt we'll be able to achieve the target of 1 to 2
> sec startup time if we do that.
>
> Thanks,
> NuwanD.
>
> On Wed, Oct 12, 2016 at 3:40 PM, Muhammed Shariq  wrote:
>
>> With regards to the discussion on improving some of the limitations in
>> the our current product profiles support [1], we had a discussion to
>> rethink how we can improve the support for running different profiles in C5.
>>
>> Participants - Lakmal, Azeez, Imesh, Kishanthan, Jayanga, Chandana, Rohan
>>
>> *Limitations with the current approach*
>>
>> - We only consider the bundles.info and load the necessary bundles,
>> however there might be certain configs and deployment artifacts that's not
>> required
>> - Some functionalities that aren't required for certain profiles are
>> enabled (e.g. transports)
>> - Distribution size is not optimal for container native architecture
>>
>> *Suggestion for profile support in C5*
>>
>> Considering the above limitation, we decided on building a profile
>> specific distribution from the base distribution. The base distribution
>> will pack all artifacts and also profile specific descriptors
>>
>> If there are any configs specific to a profile, those configs can be
>> changed using the deployment.properties file. The base distribution will
>> have deployment.properties file specific to the profiles, that way we don't
>> have to duplicate any config files.
>>
>> During the runtime, each profile will run as an independent process - for
>> this, we'll be moving away from default 9763/9443 ports, instead each
>> product will have an assigned port range.
>>
>> We will provide a tool that will look into the profile descriptors and
>> build the bare minimum runtime corresponding to the profile. This tool will
>> copy the required bundles, config files etc by looking into the meta info.
>> We can use the same tool to start the runtimes as well.
>>
>> The directory structure of the base distribution will be something like;
>>
>> wso2-am
>> |-- bin
>> |-- osgi
>>  |-- plugins
>> |-- conf
>> |-- profiles
>>  |-- gateway
>>   |-- metadata.yaml
>>   |-- deployment.properties
>>  |-- km
>>   |-- metadata.yaml
>>   |-- deployment.properties
>> |-- *runtime* --> wiill be created by the tool
>>  |-- *wso2-am-gateway*
>>  |-- *wso2-am-km*
>>
>> As shown in the folder structure, the tool will create the profile
>> specific runtime by reading the bundles.info, metadata file etc. In the
>> metadata.yaml file, we can specify what are the config files and other
>> resources needed for that particular profile. So a given runtime will only
>> have the minimum required jars and configs.
>>
>> Additionally, the tool provides the option to create docker-compose yaml
>> that would take the created runtimes and deploy it in containers.
>>
>> We have started implementing the tool, please share your thoughts /
>> suggestions.
>>
>> [1] - [Architecture] How can we improve our profiles story?
>>
>> --
>> Thanks,
>> Shariq
>> Associate Technical Lead
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Nuwan Dias
>
> Software Architect - WSO2, Inc. http://wso2.com
> email : nuw...@wso2.com
> Phone : +94 777 775 729
>



-- 
*Afkham Azeez*
Director of Architecture; WSO2, Inc.; http://wso2.com
Member; Apache Software Foundation; http://www.apache.org/
* *
*email: **az...@wso2.com* 
* cell: +94 77 3320919blog: **http://blog.afkham.org*

*twitter: **http://twitter.com/afkham_azeez*

*linked-in: **http://lk.linkedin.com/in/afkhamazeez
*

*Lean . Enterprise . Middleware*
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Multiple profile support for C5 based products.

2016-10-13 Thread Muhammed Shariq

The options are either build all the different distributions at runtime
itself or provide a tool to create the different runtime from a base
distribution. If we build all the individual runtimes at build time and
package it into a single archive, the download time will be bigger cz
there'll be many duplicates. Apart from the kernel jars, most of the
third-party libraries and our own common plugins plus the config files will
be duplicated and IINM a distribution size will easily be over 750-800MB.
On the other hand, if we go with the tool we can still look to keep the
base distribution size under 300MB. If we have the ability to reduce the
download size, we should go for it IMO. Ideally the user will only have to
run the tool once and we'll create the runtimes, so it's not burden on the
user's part. Plus the tool can also provide the option to launch the
runtimes in containers.

On Thu, Oct 13, 2016 at 10:26 AM, Nuwan Dias  wrote:

>
>
> On Wed, Oct 12, 2016 at 7:36 PM, Kishanthan Thangarajah <
> kishant...@wso2.com> wrote:
>
>>
>>
>> On Wed, Oct 12, 2016 at 7:00 PM, Nuwan Dias  wrote:
>>
>>>
>>>
>>> On Wed, Oct 12, 2016 at 6:55 PM, Kishanthan Thangarajah <
>>> kishant...@wso2.com> wrote:
>>>

 On Wed, Oct 12, 2016 at 6:29 PM, Nuwan Dias  wrote:

> Do we really need multi-profile support (at an osgi level) on C5? What
> if we have separate dedicated runtimes per profile? Which means that we
> have a gateway runtime, store runtime, etc. within the same distribution?
> Each will run on its own port, maybe using offsets by default (no need to
> worry about that if each profile is a container).
>

 This is the similar thing we are trying propose. The "profiles" concept
 is C5 is actually runtime we are talking about. It is not only about osgi
 bundles, but everything a runtime needs to start on its own.

 For example, api-manager store runtime needs set of bundles,
 configuration files and store UI component written using uuf/jaggery. The
 tool will create this runtime by reading the descriptor file and launch the
 instance as a separate JVM.

>>>
>>> But why would we need a tool to do that? What if we build the product
>>> itself in that manner.
>>>
>>
>> It's not only about API-Manager product but the same requirement is
>> needed for IoT product (and IS may also have the requirement when running
>> the user/admin portal separately) that require multiple runtimes. So isn't
>> it better to provide a common way to do this?
>>
>
> Well, by providing a tool aren't we pushing the complication to the
> customer's end? Specially when it comes to automating (puppet, chef)
> customer specific deployment patterns and all, won't it be tricky to play
> around with this tool to achieve it? Even for the case of IoTS and IS and
> maybe even for the new ESB, I was thinking that all should build their own
> distros in that way :).
>
>>
>>
>>> Ex: Store will have all its bundles, config files, etc in a single
>>> directory. And it'll have its own startup script (.sh). Same will be for
>>> the publisher and other runtimes. It will result in a certain amount of
>>> duplicate jars in those directories (carbon kernel), but that isn't much of
>>> a problem isn't it? Do you see any other problems in doing that?
>>>
>>
>> The duplication will be high here IMO based on the passed experience. The
>> pack size will increase due to this.
>>
>
> Well, our focus should be to reduce the size of the runtime right? Not
> really the pack. As far as the size of store, publisher, etc remain small,
> it doesn't really matter if their accumulated size is large. Because the
> objective of making the pack size small is to make them more container
> friendly. So in practise, people will only execute individual runtimes in a
> container and not the whole distribution (accumulated pack). Therefore as
> far as the individual runtime is small, it should be fine I think.
>
>>
>>
>
 Additionally if we have option of docker compose given, then it will
 launch a docker instance with the store runtime.

> Also if we have profiles, do we also have a global profile which runs
> all components in one?
>

 No, with the above model, there will not be a single profile (or global
 profile) which has all the runtime components in it and we have to move
 away from single product running all the components together in the same
 JVM. Each runtime will be started as different processes/JVM (can be
 OSGi-java process and just another java process).

 If a runtime needs a non-OSGi java process (say for example you want to
 start activemq as part of the product), then you can also do that using
 this proposed approach.

> I doubt we'll be able to achieve the target of 1 to 2 sec startup time
> if we do that.
>
> Thanks,
> NuwanD.
>
>

Re: [Architecture] Storing configs in database for C5

2016-10-13 Thread Lakmal Warusawithana

On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:

>
>
> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera  wrote:
>
>> I believe the plan is that for server configs, there is no (admin) UI in
>> C5. It can be changed only via config files. End user scenarios such as API
>> publisher might have UIs for configs.  ( We should agree here or have a
>> meeting if we want to change that)
>>
>
> On the API Cloud (and other clouds) we will have tenants changing some of
> their configs. So for that we will have to provide a UI since we won't have
> any other way to make them change files.
>

@NuwanD, are these admin portal based config? like defining/changing
policies etc?


>
>> What is an example of case where configs will take effect later?
>>
>> IMO storing configs in the dtabase make the deployment complicated. Also
>> it reduce the scalability as all server point to one DB. I think it also
>> make docker usecase complex.
>>
>>
>> --Srinath
>>
>> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:
>>
>>>
>>>
>>> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
>>> wrote:
>>>


 On 12 October 2016 at 14:31, Nuwan Dias  wrote:

> There are challenges when moving configs to the DB. We experienced it
> once when we moved the analytics configs to the registry. And then we 
> moved
> it back again to the FS because it was too painful to maintain.
>
> 1. The nodes have to keep polling the DB in a fast enough interval.
> This is a unnecessary performance overhead. Because in practise, someone
> will only change these configs once. But to support that use case, we have
> to keep polling the DB for life.
>
> 2. Gateways don't have access to the DB. So say you're enabling
> analytics (data publishing). You have to propagate that change to the
> Gateway nodes using some mechanism. And with no clustering on C5, this is 
> a
> challenge.
>
> If the objective of this is to make the Cloud (tenant) experience
> better, I think we should just restart the tenant's containers with the
> relevant configs in place.
>

 Still we have a problem with regard to how we are going to allow the
 tenants to do the configuration changes. Currently we do it through the
 registry which will not work for C5.

>>>
>>> Yes, so my idea is to provide a UI to do the configs. Those configs we
>>> can store anywhere (maybe in a table) just for the sake of rendering that
>>> UI. The product code will still read from the config files. When you apply
>>> those configs through the press of a button, the container should get
>>> rebuilt and restarted with the necessary modification to the config files.
>>>

 Thanks,
 Lakmali

>
> Thanks,
> NuwanD.
>
> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta  > wrote:
>
>>
>>
>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha > > wrote:
>>
>>> Hi Sajith,
>>>
>>> Yes even though the boot up time is not an issue in C5 the other
>>> advantages I have outlined are still there to be gained. There is a huge
>>> effort we have to do on dev ops side to maintain those images you are
>>> talking about because of having everything at file level.
>>>
>>> Some examples from API Manager I can think of are turning
>>> notifications on/off, enable monetization, enable/disable stats, 
>>> configure
>>> work flows, Enable/Disable JWT token header.
>>>
>>
>> +1 to move feature related configurations to the database and make
>> them configurable through the UI.
>>
>> Thanks,
>> Lakmali
>>
>>>
>>>
>>>
>>> On 12 October 2016 at 12:58, Sajith Kariyawasam 
>>> wrote:
>>>
 Hi Uvindra,

 With cloud deployment in mind, the idea is to boot up the nodes in
 quick time, therefore the docker images are pre-configured with all the
 configuration values, which will speed up the node start up. A change 
 of
 configuration means a new docker image will be created with the new
 configs, and re-spawn the cluster.

 Therefore, IMO a node restart for a config change is not relevant,
 also no need of a periodic config checks.

 Btw, can you give me some example configuration you were thinking
 of?

 Thanks,
 Sajith

 On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
 uvin...@wso2.com> wrote:

> Was wondering about $subject
>
> Traditionally we have stored our product configs, be it
> carbon.xml, api-manager.xml, identity.xml, etc. at file level. Some
> configs, such as "port offset" are inherently bound to the

Re: [Architecture] Configuration files in C5

Re: [Architecture] Storing configs in database for C5

Re: [Architecture] Configuration files in C5

Re: [Architecture] Configuration files in C5

[Architecture] Using HdrHistogram as a reservoir in Carbon Metrics

Re: [Architecture] Configuration files in C5

Re: [Architecture] Configuration files in C5

Re: [Architecture] ESB connector auto generation tool

Re: [Architecture] Storing configs in database for C5

Re: [Architecture] Configuration files in C5

Re: [Architecture] [APIM] Life cycle Management

[Architecture] Configuration files in C5

Re: [Architecture] ESB connector auto generation tool

Re: [Architecture] ESB connector auto generation tool

Re: [Architecture] Multiple profile support for C5 based products.

Re: [Architecture] Multiple profile support for C5 based products.

Re: [Architecture] Storing configs in database for C5

17 matches

Site Navigation

Mail list logo

Footer information