Re: [Architecture] Storing configs in database for C5

[Ruwan Abeykoon]

Hi All,
I think we need to consider two type of configs.
1. Configs which are only read at the system startup. Code is not written
to allow re-read the properties and re-configure at runtime.
2. Configs which are changed at runtime, and dynamically affect the system.

I think the type (1) configs better remain at the file system. There will
be no difference in file or database storage for the end user. It is easy
for sys-admin or the build scripts to build the containers with these
static configs. This reduces the complexities and unwanted UI.

On the other hand type(2) configs need to be stored in shared DB, and
should have a proper UI to change. Code should allow them to be dynamically
applied(without restart).

Cheers,
Ruwan





On Fri, Oct 14, 2016 at 1:20 PM, Uvindra Dias Jayasinha 
wrote:

> I think we should not be in the mind set that all configs are changed by
> system admins. That kind of thinking will not help us in our cloud first
> approach. Some configs govern how a particular feature may behave and its
> the end user who will need to set those. These are the exact kind of
> configs we need to make more accessible because it makes our products more
> user friendly.
>
> I've seen distributed systems that stored their configs in DB's and
> functioned without any issue. But I understand that this may not be the
> best fit for our products.
>
> Im fine with configs being at file level as long as the UX does not
> suffer. We should strive to improve usability more than ever before.
>
>
> On 14 October 2016 at 12:34, Sagara Gunathunga  wrote:
>
>>
>>
>> On Fri, Oct 14, 2016 at 10:55 AM, Uvindra Dias Jayasinha <
>> uvin...@wso2.com> wrote:
>>
>>> Let me summarize this thread so far.
>>>
>>> The functionality provided by a DB makes it easy to implement reading,
>>> updating and sharing configs, but there have been some legitimate concerns
>>> that have been raised in this thread regarding that.
>>>
>>
>> First, we need to separate out service configs from end user specific
>> configs.
>>
>> For end user specific configs such as Publisher/Store we should evaluate
>> them case by case through a user story driven approach, in such cases if
>> storing config values in a DB and populating  through a UI is the right
>> design/solution then we should go for that but it should not be a rule.
>>
>> Generally storing server configs in a DB is an anti-pattern for me.
>>
>> - Storing server configs in a DB involves great amount of development
>> work compare to maintaining simple configuration file such as designing and
>> implementing UIs, Writing JDBC/ORM code, maintaining DB schema for each
>> vendor, writing UI tests etc. When you change or introduce new config means
>> you have to modify all of above layers again.
>>
>> - People who suppose to change these server configs are system admins,
>> they usually love config files over UIs.
>>
>> - Compare to number of people who will use these UIs are very limited so
>> it's OK to let them to change config files while we can focus on Store or
>> Publisher where user experience is more critical.
>>
>> - As stated this is not Container friendly
>>
>> - Previously we had very bad experience storing config in Registry DB
>>
>> - This seems like a re-inventing another Registry, if we really want to
>> go in this path, we should use existing key-value store instead of writing
>> from the scratch
>>
>>
>> Sharing configs among several node is a valid use case to use DB, but
>> it's kind of a lazy workaround to bring a DB instead of engineering the
>> sharing configs among nodes problem properly :)
>>
>> Thanks !
>>
>>
>>
>>
>>>
>>> As has already been outlined deployment based configs should be part of
>>> the container and so should be file based. We also have limitations with
>>> certain components such as the Gateway which needs to be deployed in the
>>> DMZ and so cannot connect with a DB in that case.
>>>
>>> At the end of the day a file is a persistent storage just like a DB,
>>> albeit much less sophisticated and localized. So maybe we can achieve some
>>> of the advantages I outlined in my initial mail using file configs. The
>>> underlying storage really does not matter(just an implementation detail) as
>>> long as we can achieve the benefits that we need.
>>>
>>>
>>> On 14 October 2016 at 00:11, Lahiru Sandaruwan  wrote:
>>>
 Hi All,

 If we try to categorize the configurations we have in C4, using the way
 we manage,

1. *Runtime configurations*
   - Any config you login to an UI and do
   - E.g. API creation/ subscription in APIM, Mediation flow in
   Integration Server, Service providers/ secondary user stores in IS, 
 etc.)
   - We are using file system, direct databases, and registry as
   per the requirements for this type
   2. *One time configurations*
   - Any one time config you configure using the files
   

Re: [Architecture] Storing configs in database for C5

[Uvindra Dias Jayasinha]

I think we should not be in the mind set that all configs are changed by
system admins. That kind of thinking will not help us in our cloud first
approach. Some configs govern how a particular feature may behave and its
the end user who will need to set those. These are the exact kind of
configs we need to make more accessible because it makes our products more
user friendly.

I've seen distributed systems that stored their configs in DB's and
functioned without any issue. But I understand that this may not be the
best fit for our products.

Im fine with configs being at file level as long as the UX does not suffer.
We should strive to improve usability more than ever before.

On 14 October 2016 at 12:34, Sagara Gunathunga  wrote:

>
>
> On Fri, Oct 14, 2016 at 10:55 AM, Uvindra Dias Jayasinha  > wrote:
>
>> Let me summarize this thread so far.
>>
>> The functionality provided by a DB makes it easy to implement reading,
>> updating and sharing configs, but there have been some legitimate concerns
>> that have been raised in this thread regarding that.
>>
>
> First, we need to separate out service configs from end user specific
> configs.
>
> For end user specific configs such as Publisher/Store we should evaluate
> them case by case through a user story driven approach, in such cases if
> storing config values in a DB and populating  through a UI is the right
> design/solution then we should go for that but it should not be a rule.
>
> Generally storing server configs in a DB is an anti-pattern for me.
>
> - Storing server configs in a DB involves great amount of development work
> compare to maintaining simple configuration file such as designing and
> implementing UIs, Writing JDBC/ORM code, maintaining DB schema for each
> vendor, writing UI tests etc. When you change or introduce new config means
> you have to modify all of above layers again.
>
> - People who suppose to change these server configs are system admins,
> they usually love config files over UIs.
>
> - Compare to number of people who will use these UIs are very limited so
> it's OK to let them to change config files while we can focus on Store or
> Publisher where user experience is more critical.
>
> - As stated this is not Container friendly
>
> - Previously we had very bad experience storing config in Registry DB
>
> - This seems like a re-inventing another Registry, if we really want to go
> in this path, we should use existing key-value store instead of writing
> from the scratch
>
>
> Sharing configs among several node is a valid use case to use DB, but it's
> kind of a lazy workaround to bring a DB instead of engineering the sharing
> configs among nodes problem properly :)
>
> Thanks !
>
>
>
>
>>
>> As has already been outlined deployment based configs should be part of
>> the container and so should be file based. We also have limitations with
>> certain components such as the Gateway which needs to be deployed in the
>> DMZ and so cannot connect with a DB in that case.
>>
>> At the end of the day a file is a persistent storage just like a DB,
>> albeit much less sophisticated and localized. So maybe we can achieve some
>> of the advantages I outlined in my initial mail using file configs. The
>> underlying storage really does not matter(just an implementation detail) as
>> long as we can achieve the benefits that we need.
>>
>>
>> On 14 October 2016 at 00:11, Lahiru Sandaruwan  wrote:
>>
>>> Hi All,
>>>
>>> If we try to categorize the configurations we have in C4, using the way
>>> we manage,
>>>
>>>1. *Runtime configurations*
>>>   - Any config you login to an UI and do
>>>   - E.g. API creation/ subscription in APIM, Mediation flow in
>>>   Integration Server, Service providers/ secondary user stores in IS, 
>>> etc.)
>>>   - We are using file system, direct databases, and registry as per
>>>   the requirements for this type
>>>   2. *One time configurations*
>>>   - Any one time config you configure using the files
>>>   - E.g. Configs we have in carbon.xml, axis2.xml, user-mgt.xml
>>>   - We only use file system for this type
>>>
>>>
>>> With C5 tenant approach, tenants will get to configure a considerable
>>> set of configs out of #2, but not all. IMO we should use a database for
>>> that set only, keeping the other one time configurations in the file system
>>> from #2.
>>>
>>> Even hot deployable files are an option for suitable use cases. (Not
>>> sure if we planning to move away from hot deployable files, with the
>>> container adaption). A GUI should be provided for managing them in the
>>> database or hot deployable files.
>>>
>>> In other words, we could move those configurations to #1 from #2.
>>> Wdyt?
>>>
>>> Thanks.
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Oct 13, 2016 at 11:39 AM, Lakmal Warusawithana 
>>> wrote:
>>>


 On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:

>

Re: [Architecture] Storing configs in database for C5

[Sagara Gunathunga]

On Fri, Oct 14, 2016 at 10:55 AM, Uvindra Dias Jayasinha 
wrote:

> Let me summarize this thread so far.
>
> The functionality provided by a DB makes it easy to implement reading,
> updating and sharing configs, but there have been some legitimate concerns
> that have been raised in this thread regarding that.
>

First, we need to separate out service configs from end user specific
configs.

For end user specific configs such as Publisher/Store we should evaluate
them case by case through a user story driven approach, in such cases if
storing config values in a DB and populating  through a UI is the right
design/solution then we should go for that but it should not be a rule.

Generally storing server configs in a DB is an anti-pattern for me.

- Storing server configs in a DB involves great amount of development work
compare to maintaining simple configuration file such as designing and
implementing UIs, Writing JDBC/ORM code, maintaining DB schema for each
vendor, writing UI tests etc. When you change or introduce new config means
you have to modify all of above layers again.

- People who suppose to change these server configs are system admins, they
usually love config files over UIs.

- Compare to number of people who will use these UIs are very limited so
it's OK to let them to change config files while we can focus on Store or
Publisher where user experience is more critical.

- As stated this is not Container friendly

- Previously we had very bad experience storing config in Registry DB

- This seems like a re-inventing another Registry, if we really want to go
in this path, we should use existing key-value store instead of writing
from the scratch


Sharing configs among several node is a valid use case to use DB, but it's
kind of a lazy workaround to bring a DB instead of engineering the sharing
configs among nodes problem properly :)

Thanks !




>
> As has already been outlined deployment based configs should be part of
> the container and so should be file based. We also have limitations with
> certain components such as the Gateway which needs to be deployed in the
> DMZ and so cannot connect with a DB in that case.
>
> At the end of the day a file is a persistent storage just like a DB,
> albeit much less sophisticated and localized. So maybe we can achieve some
> of the advantages I outlined in my initial mail using file configs. The
> underlying storage really does not matter(just an implementation detail) as
> long as we can achieve the benefits that we need.
>
>
> On 14 October 2016 at 00:11, Lahiru Sandaruwan  wrote:
>
>> Hi All,
>>
>> If we try to categorize the configurations we have in C4, using the way
>> we manage,
>>
>>1. *Runtime configurations*
>>   - Any config you login to an UI and do
>>   - E.g. API creation/ subscription in APIM, Mediation flow in
>>   Integration Server, Service providers/ secondary user stores in IS, 
>> etc.)
>>   - We are using file system, direct databases, and registry as per
>>   the requirements for this type
>>   2. *One time configurations*
>>   - Any one time config you configure using the files
>>   - E.g. Configs we have in carbon.xml, axis2.xml, user-mgt.xml
>>   - We only use file system for this type
>>
>>
>> With C5 tenant approach, tenants will get to configure a considerable set
>> of configs out of #2, but not all. IMO we should use a database for that
>> set only, keeping the other one time configurations in the file system from
>> #2.
>>
>> Even hot deployable files are an option for suitable use cases. (Not sure
>> if we planning to move away from hot deployable files, with the container
>> adaption). A GUI should be provided for managing them in the database or
>> hot deployable files.
>>
>> In other words, we could move those configurations to #1 from #2.
>> Wdyt?
>>
>> Thanks.
>>
>>
>>
>>
>>
>> On Thu, Oct 13, 2016 at 11:39 AM, Lakmal Warusawithana 
>> wrote:
>>
>>>
>>>
>>> On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:
>>>


 On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera 
 wrote:

> I believe the plan is that for server configs, there is no (admin) UI
> in C5. It can be changed only via config files. End user scenarios such as
> API publisher might have UIs for configs.  ( We should agree here or have 
> a
> meeting if we want to change that)
>

 On the API Cloud (and other clouds) we will have tenants changing some
 of their configs. So for that we will have to provide a UI since we won't
 have any other way to make them change files.

>>>
>>> @NuwanD, are these admin portal based config? like defining/changing
>>> policies etc?
>>>
>>>

> What is an example of case where configs will take effect later?
>
> IMO storing configs in the dtabase make the deployment complicated.
> Also it reduce the scalability as 

Re: [Architecture] Storing configs in database for C5

[Nuwan Dias]

On Thu, Oct 13, 2016 at 11:39 AM, Lakmal Warusawithana 
wrote:

>
>
> On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:
>
>>
>>
>> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera  wrote:
>>
>>> I believe the plan is that for server configs, there is no (admin) UI in
>>> C5. It can be changed only via config files. End user scenarios such as API
>>> publisher might have UIs for configs.  ( We should agree here or have a
>>> meeting if we want to change that)
>>>
>>
>> On the API Cloud (and other clouds) we will have tenants changing some of
>> their configs. So for that we will have to provide a UI since we won't have
>> any other way to make them change files.
>>
>
> @NuwanD, are these admin portal based config? like defining/changing
> policies etc?
>

Yes, that's correct. Enabling workflows, Enabling Monetization, etc. These
are some capabilities that differ from tenant to tenant based on their
preferences.

>
>
>>
>>> What is an example of case where configs will take effect later?
>>>
>>> IMO storing configs in the dtabase make the deployment complicated. Also
>>> it reduce the scalability as all server point to one DB. I think it also
>>> make docker usecase complex.
>>>
>>>
>>> --Srinath
>>>
>>> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:
>>>


 On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
 wrote:

>
>
> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>
>> There are challenges when moving configs to the DB. We experienced it
>> once when we moved the analytics configs to the registry. And then we 
>> moved
>> it back again to the FS because it was too painful to maintain.
>>
>> 1. The nodes have to keep polling the DB in a fast enough interval.
>> This is a unnecessary performance overhead. Because in practise, someone
>> will only change these configs once. But to support that use case, we 
>> have
>> to keep polling the DB for life.
>>
>> 2. Gateways don't have access to the DB. So say you're enabling
>> analytics (data publishing). You have to propagate that change to the
>> Gateway nodes using some mechanism. And with no clustering on C5, this 
>> is a
>> challenge.
>>
>> If the objective of this is to make the Cloud (tenant) experience
>> better, I think we should just restart the tenant's containers with the
>> relevant configs in place.
>>
>
> Still we have a problem with regard to how we are going to allow the
> tenants to do the configuration changes. Currently we do it through the
> registry which will not work for C5.
>

 Yes, so my idea is to provide a UI to do the configs. Those configs we
 can store anywhere (maybe in a table) just for the sake of rendering that
 UI. The product code will still read from the config files. When you apply
 those configs through the press of a button, the container should get
 rebuilt and restarted with the necessary modification to the config files.

>
> Thanks,
> Lakmali
>
>>
>> Thanks,
>> NuwanD.
>>
>> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta <
>> lakm...@wso2.com> wrote:
>>
>>>
>>>
>>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha <
>>> uvin...@wso2.com> wrote:
>>>
 Hi Sajith,

 Yes even though the boot up time is not an issue in C5 the other
 advantages I have outlined are still there to be gained. There is a 
 huge
 effort we have to do on dev ops side to maintain those images you are
 talking about because of having everything at file level.

 Some examples from API Manager I can think of are turning
 notifications on/off, enable monetization, enable/disable stats, 
 configure
 work flows, Enable/Disable JWT token header.

>>>
>>> +1 to move feature related configurations to the database and make
>>> them configurable through the UI.
>>>
>>> Thanks,
>>> Lakmali
>>>



 On 12 October 2016 at 12:58, Sajith Kariyawasam 
 wrote:

> Hi Uvindra,
>
> With cloud deployment in mind, the idea is to boot up the nodes in
> quick time, therefore the docker images are pre-configured with all 
> the
> configuration values, which will speed up the node start up. A change 
> of
> configuration means a new docker image will be created with the new
> configs, and re-spawn the cluster.
>
> Therefore, IMO a node restart for a config change is not relevant,
> also no need of a periodic config checks.
>
> Btw, can you give me some example configuration you were thinking
> of?
>

Re: [Architecture] Storing configs in database for C5

[Uvindra Dias Jayasinha]

Let me summarize this thread so far.

The functionality provided by a DB makes it easy to implement reading,
updating and sharing configs, but there have been some legitimate concerns
that have been raised in this thread regarding that.

As has already been outlined deployment based configs should be part of the
container and so should be file based. We also have limitations with
certain components such as the Gateway which needs to be deployed in the
DMZ and so cannot connect with a DB in that case.

At the end of the day a file is a persistent storage just like a DB, albeit
much less sophisticated and localized. So maybe we can achieve some of the
advantages I outlined in my initial mail using file configs. The underlying
storage really does not matter(just an implementation detail) as long as we
can achieve the benefits that we need.


On 14 October 2016 at 00:11, Lahiru Sandaruwan  wrote:

> Hi All,
>
> If we try to categorize the configurations we have in C4, using the way we
> manage,
>
>1. *Runtime configurations*
>   - Any config you login to an UI and do
>   - E.g. API creation/ subscription in APIM, Mediation flow in
>   Integration Server, Service providers/ secondary user stores in IS, 
> etc.)
>   - We are using file system, direct databases, and registry as per
>   the requirements for this type
>   2. *One time configurations*
>   - Any one time config you configure using the files
>   - E.g. Configs we have in carbon.xml, axis2.xml, user-mgt.xml
>   - We only use file system for this type
>
>
> With C5 tenant approach, tenants will get to configure a considerable set
> of configs out of #2, but not all. IMO we should use a database for that
> set only, keeping the other one time configurations in the file system from
> #2.
>
> Even hot deployable files are an option for suitable use cases. (Not sure
> if we planning to move away from hot deployable files, with the container
> adaption). A GUI should be provided for managing them in the database or
> hot deployable files.
>
> In other words, we could move those configurations to #1 from #2.
> Wdyt?
>
> Thanks.
>
>
>
>
>
> On Thu, Oct 13, 2016 at 11:39 AM, Lakmal Warusawithana 
> wrote:
>
>>
>>
>> On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:
>>
>>>
>>>
>>> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera 
>>> wrote:
>>>
 I believe the plan is that for server configs, there is no (admin) UI
 in C5. It can be changed only via config files. End user scenarios such as
 API publisher might have UIs for configs.  ( We should agree here or have a
 meeting if we want to change that)

>>>
>>> On the API Cloud (and other clouds) we will have tenants changing some
>>> of their configs. So for that we will have to provide a UI since we won't
>>> have any other way to make them change files.
>>>
>>
>> @NuwanD, are these admin portal based config? like defining/changing
>> policies etc?
>>
>>
>>>
 What is an example of case where configs will take effect later?

 IMO storing configs in the dtabase make the deployment complicated.
 Also it reduce the scalability as all server point to one DB. I think it
 also make docker usecase complex.


 --Srinath

 On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:

>
>
> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta  > wrote:
>
>>
>>
>> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>>
>>> There are challenges when moving configs to the DB. We experienced
>>> it once when we moved the analytics configs to the registry. And then we
>>> moved it back again to the FS because it was too painful to maintain.
>>>
>>> 1. The nodes have to keep polling the DB in a fast enough interval.
>>> This is a unnecessary performance overhead. Because in practise, someone
>>> will only change these configs once. But to support that use case, we 
>>> have
>>> to keep polling the DB for life.
>>>
>>> 2. Gateways don't have access to the DB. So say you're enabling
>>> analytics (data publishing). You have to propagate that change to the
>>> Gateway nodes using some mechanism. And with no clustering on C5, this 
>>> is a
>>> challenge.
>>>
>>> If the objective of this is to make the Cloud (tenant) experience
>>> better, I think we should just restart the tenant's containers with the
>>> relevant configs in place.
>>>
>>
>> Still we have a problem with regard to how we are going to allow the
>> tenants to do the configuration changes. Currently we do it through the
>> registry which will not work for C5.
>>
>
> Yes, so my idea is to provide a UI to do the configs. Those configs we
> can store anywhere (maybe in a table) just for the sake of rendering that

Re: [Architecture] Storing configs in database for C5

[Lahiru Sandaruwan]

Hi All,

If we try to categorize the configurations we have in C4, using the way we
manage,

   1. *Runtime configurations*
  - Any config you login to an UI and do
  - E.g. API creation/ subscription in APIM, Mediation flow in
  Integration Server, Service providers/ secondary user stores in IS, etc.)
  - We are using file system, direct databases, and registry as per the
  requirements for this type
  2. *One time configurations*
  - Any one time config you configure using the files
  - E.g. Configs we have in carbon.xml, axis2.xml, user-mgt.xml
  - We only use file system for this type


With C5 tenant approach, tenants will get to configure a considerable set
of configs out of #2, but not all. IMO we should use a database for that
set only, keeping the other one time configurations in the file system from
#2.

Even hot deployable files are an option for suitable use cases. (Not sure
if we planning to move away from hot deployable files, with the container
adaption). A GUI should be provided for managing them in the database or
hot deployable files.

In other words, we could move those configurations to #1 from #2.
Wdyt?

Thanks.





On Thu, Oct 13, 2016 at 11:39 AM, Lakmal Warusawithana 
wrote:

>
>
> On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:
>
>>
>>
>> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera  wrote:
>>
>>> I believe the plan is that for server configs, there is no (admin) UI in
>>> C5. It can be changed only via config files. End user scenarios such as API
>>> publisher might have UIs for configs.  ( We should agree here or have a
>>> meeting if we want to change that)
>>>
>>
>> On the API Cloud (and other clouds) we will have tenants changing some of
>> their configs. So for that we will have to provide a UI since we won't have
>> any other way to make them change files.
>>
>
> @NuwanD, are these admin portal based config? like defining/changing
> policies etc?
>
>
>>
>>> What is an example of case where configs will take effect later?
>>>
>>> IMO storing configs in the dtabase make the deployment complicated. Also
>>> it reduce the scalability as all server point to one DB. I think it also
>>> make docker usecase complex.
>>>
>>>
>>> --Srinath
>>>
>>> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:
>>>


 On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
 wrote:

>
>
> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>
>> There are challenges when moving configs to the DB. We experienced it
>> once when we moved the analytics configs to the registry. And then we 
>> moved
>> it back again to the FS because it was too painful to maintain.
>>
>> 1. The nodes have to keep polling the DB in a fast enough interval.
>> This is a unnecessary performance overhead. Because in practise, someone
>> will only change these configs once. But to support that use case, we 
>> have
>> to keep polling the DB for life.
>>
>> 2. Gateways don't have access to the DB. So say you're enabling
>> analytics (data publishing). You have to propagate that change to the
>> Gateway nodes using some mechanism. And with no clustering on C5, this 
>> is a
>> challenge.
>>
>> If the objective of this is to make the Cloud (tenant) experience
>> better, I think we should just restart the tenant's containers with the
>> relevant configs in place.
>>
>
> Still we have a problem with regard to how we are going to allow the
> tenants to do the configuration changes. Currently we do it through the
> registry which will not work for C5.
>

 Yes, so my idea is to provide a UI to do the configs. Those configs we
 can store anywhere (maybe in a table) just for the sake of rendering that
 UI. The product code will still read from the config files. When you apply
 those configs through the press of a button, the container should get
 rebuilt and restarted with the necessary modification to the config files.

>
> Thanks,
> Lakmali
>
>>
>> Thanks,
>> NuwanD.
>>
>> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta <
>> lakm...@wso2.com> wrote:
>>
>>>
>>>
>>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha <
>>> uvin...@wso2.com> wrote:
>>>
 Hi Sajith,

 Yes even though the boot up time is not an issue in C5 the other
 advantages I have outlined are still there to be gained. There is a 
 huge
 effort we have to do on dev ops side to maintain those images you are
 talking about because of having everything at file level.

 Some examples from API Manager I can think of are turning
 notifications on/off, enable monetization, enable/disable stats, 
 configure
 

Re: [Architecture] Storing configs in database for C5

[Lakmal Warusawithana]

On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:

>
>
> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera  wrote:
>
>> I believe the plan is that for server configs, there is no (admin) UI in
>> C5. It can be changed only via config files. End user scenarios such as API
>> publisher might have UIs for configs.  ( We should agree here or have a
>> meeting if we want to change that)
>>
>
> On the API Cloud (and other clouds) we will have tenants changing some of
> their configs. So for that we will have to provide a UI since we won't have
> any other way to make them change files.
>

@NuwanD, are these admin portal based config? like defining/changing
policies etc?


>
>> What is an example of case where configs will take effect later?
>>
>> IMO storing configs in the dtabase make the deployment complicated. Also
>> it reduce the scalability as all server point to one DB. I think it also
>> make docker usecase complex.
>>
>>
>> --Srinath
>>
>> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:
>>
>>>
>>>
>>> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
>>> wrote:
>>>


 On 12 October 2016 at 14:31, Nuwan Dias  wrote:

> There are challenges when moving configs to the DB. We experienced it
> once when we moved the analytics configs to the registry. And then we 
> moved
> it back again to the FS because it was too painful to maintain.
>
> 1. The nodes have to keep polling the DB in a fast enough interval.
> This is a unnecessary performance overhead. Because in practise, someone
> will only change these configs once. But to support that use case, we have
> to keep polling the DB for life.
>
> 2. Gateways don't have access to the DB. So say you're enabling
> analytics (data publishing). You have to propagate that change to the
> Gateway nodes using some mechanism. And with no clustering on C5, this is 
> a
> challenge.
>
> If the objective of this is to make the Cloud (tenant) experience
> better, I think we should just restart the tenant's containers with the
> relevant configs in place.
>

 Still we have a problem with regard to how we are going to allow the
 tenants to do the configuration changes. Currently we do it through the
 registry which will not work for C5.

>>>
>>> Yes, so my idea is to provide a UI to do the configs. Those configs we
>>> can store anywhere (maybe in a table) just for the sake of rendering that
>>> UI. The product code will still read from the config files. When you apply
>>> those configs through the press of a button, the container should get
>>> rebuilt and restarted with the necessary modification to the config files.
>>>

 Thanks,
 Lakmali

>
> Thanks,
> NuwanD.
>
> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta  > wrote:
>
>>
>>
>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha > > wrote:
>>
>>> Hi Sajith,
>>>
>>> Yes even though the boot up time is not an issue in C5 the other
>>> advantages I have outlined are still there to be gained. There is a huge
>>> effort we have to do on dev ops side to maintain those images you are
>>> talking about because of having everything at file level.
>>>
>>> Some examples from API Manager I can think of are turning
>>> notifications on/off, enable monetization, enable/disable stats, 
>>> configure
>>> work flows, Enable/Disable JWT token header.
>>>
>>
>> +1 to move feature related configurations to the database and make
>> them configurable through the UI.
>>
>> Thanks,
>> Lakmali
>>
>>>
>>>
>>>
>>> On 12 October 2016 at 12:58, Sajith Kariyawasam 
>>> wrote:
>>>
 Hi Uvindra,

 With cloud deployment in mind, the idea is to boot up the nodes in
 quick time, therefore the docker images are pre-configured with all the
 configuration values, which will speed up the node start up. A change 
 of
 configuration means a new docker image will be created with the new
 configs, and re-spawn the cluster.

 Therefore, IMO a node restart for a config change is not relevant,
 also no need of a periodic config checks.

 Btw, can you give me some example configuration you were thinking
 of?

 Thanks,
 Sajith

 On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
 uvin...@wso2.com> wrote:

> Was wondering about $subject
>
> Traditionally we have stored our product configs, be it
> carbon.xml, api-manager.xml, identity.xml, etc. at file level. Some
> configs, such as "port offset" are inherently bound to the 

Re: [Architecture] Storing configs in database for C5

[Kishanthan Thangarajah]

On Wed, Oct 12, 2016 at 6:36 PM, Nuwan Dias  wrote:

>
>
> On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera  wrote:
>
>> I believe the plan is that for server configs, there is no (admin) UI in
>> C5. It can be changed only via config files. End user scenarios such as API
>> publisher might have UIs for configs.  ( We should agree here or have a
>> meeting if we want to change that)
>>
>
> On the API Cloud (and other clouds) we will have tenants changing some of
> their configs. So for that we will have to provide a UI since we won't have
> any other way to make them change files.
>

Shouldn't those configuration changes be env or sys variables and passed to
the container when launching them? You take the config values from the
user, and set them as env/sys variables for the container and start it.
This is how app cloud currently launches apps which require config changes
passed from user.



>> What is an example of case where configs will take effect later?
>>
>> IMO storing configs in the dtabase make the deployment complicated. Also
>> it reduce the scalability as all server point to one DB. I think it also
>> make docker usecase complex.
>>
>>
>> --Srinath
>>
>> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:
>>
>>>
>>>
>>> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
>>> wrote:
>>>


 On 12 October 2016 at 14:31, Nuwan Dias  wrote:

> There are challenges when moving configs to the DB. We experienced it
> once when we moved the analytics configs to the registry. And then we 
> moved
> it back again to the FS because it was too painful to maintain.
>
> 1. The nodes have to keep polling the DB in a fast enough interval.
> This is a unnecessary performance overhead. Because in practise, someone
> will only change these configs once. But to support that use case, we have
> to keep polling the DB for life.
>
> 2. Gateways don't have access to the DB. So say you're enabling
> analytics (data publishing). You have to propagate that change to the
> Gateway nodes using some mechanism. And with no clustering on C5, this is 
> a
> challenge.
>
> If the objective of this is to make the Cloud (tenant) experience
> better, I think we should just restart the tenant's containers with the
> relevant configs in place.
>

 Still we have a problem with regard to how we are going to allow the
 tenants to do the configuration changes. Currently we do it through the
 registry which will not work for C5.

>>>
>>> Yes, so my idea is to provide a UI to do the configs. Those configs we
>>> can store anywhere (maybe in a table) just for the sake of rendering that
>>> UI. The product code will still read from the config files. When you apply
>>> those configs through the press of a button, the container should get
>>> rebuilt and restarted with the necessary modification to the config files.
>>>

 Thanks,
 Lakmali

>
> Thanks,
> NuwanD.
>
> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta  > wrote:
>
>>
>>
>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha > > wrote:
>>
>>> Hi Sajith,
>>>
>>> Yes even though the boot up time is not an issue in C5 the other
>>> advantages I have outlined are still there to be gained. There is a huge
>>> effort we have to do on dev ops side to maintain those images you are
>>> talking about because of having everything at file level.
>>>
>>> Some examples from API Manager I can think of are turning
>>> notifications on/off, enable monetization, enable/disable stats, 
>>> configure
>>> work flows, Enable/Disable JWT token header.
>>>
>>
>> +1 to move feature related configurations to the database and make
>> them configurable through the UI.
>>
>> Thanks,
>> Lakmali
>>
>>>
>>>
>>>
>>> On 12 October 2016 at 12:58, Sajith Kariyawasam 
>>> wrote:
>>>
 Hi Uvindra,

 With cloud deployment in mind, the idea is to boot up the nodes in
 quick time, therefore the docker images are pre-configured with all the
 configuration values, which will speed up the node start up. A change 
 of
 configuration means a new docker image will be created with the new
 configs, and re-spawn the cluster.

 Therefore, IMO a node restart for a config change is not relevant,
 also no need of a periodic config checks.

 Btw, can you give me some example configuration you were thinking
 of?

 Thanks,
 Sajith

 On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
 uvin...@wso2.com> wrote:

> Was wondering about 

Re: [Architecture] Storing configs in database for C5

[Nuwan Dias]

On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera  wrote:

> I believe the plan is that for server configs, there is no (admin) UI in
> C5. It can be changed only via config files. End user scenarios such as API
> publisher might have UIs for configs.  ( We should agree here or have a
> meeting if we want to change that)
>

On the API Cloud (and other clouds) we will have tenants changing some of
their configs. So for that we will have to provide a UI since we won't have
any other way to make them change files.

>
> What is an example of case where configs will take effect later?
>
> IMO storing configs in the dtabase make the deployment complicated. Also
> it reduce the scalability as all server point to one DB. I think it also
> make docker usecase complex.
>
>
> --Srinath
>
> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:
>
>>
>>
>> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
>> wrote:
>>
>>>
>>>
>>> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>>>
 There are challenges when moving configs to the DB. We experienced it
 once when we moved the analytics configs to the registry. And then we moved
 it back again to the FS because it was too painful to maintain.

 1. The nodes have to keep polling the DB in a fast enough interval.
 This is a unnecessary performance overhead. Because in practise, someone
 will only change these configs once. But to support that use case, we have
 to keep polling the DB for life.

 2. Gateways don't have access to the DB. So say you're enabling
 analytics (data publishing). You have to propagate that change to the
 Gateway nodes using some mechanism. And with no clustering on C5, this is a
 challenge.

 If the objective of this is to make the Cloud (tenant) experience
 better, I think we should just restart the tenant's containers with the
 relevant configs in place.

>>>
>>> Still we have a problem with regard to how we are going to allow the
>>> tenants to do the configuration changes. Currently we do it through the
>>> registry which will not work for C5.
>>>
>>
>> Yes, so my idea is to provide a UI to do the configs. Those configs we
>> can store anywhere (maybe in a table) just for the sake of rendering that
>> UI. The product code will still read from the config files. When you apply
>> those configs through the press of a button, the container should get
>> rebuilt and restarted with the necessary modification to the config files.
>>
>>>
>>> Thanks,
>>> Lakmali
>>>

 Thanks,
 NuwanD.

 On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta 
 wrote:

>
>
> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha 
> wrote:
>
>> Hi Sajith,
>>
>> Yes even though the boot up time is not an issue in C5 the other
>> advantages I have outlined are still there to be gained. There is a huge
>> effort we have to do on dev ops side to maintain those images you are
>> talking about because of having everything at file level.
>>
>> Some examples from API Manager I can think of are turning
>> notifications on/off, enable monetization, enable/disable stats, 
>> configure
>> work flows, Enable/Disable JWT token header.
>>
>
> +1 to move feature related configurations to the database and make
> them configurable through the UI.
>
> Thanks,
> Lakmali
>
>>
>>
>>
>> On 12 October 2016 at 12:58, Sajith Kariyawasam 
>> wrote:
>>
>>> Hi Uvindra,
>>>
>>> With cloud deployment in mind, the idea is to boot up the nodes in
>>> quick time, therefore the docker images are pre-configured with all the
>>> configuration values, which will speed up the node start up. A change of
>>> configuration means a new docker image will be created with the new
>>> configs, and re-spawn the cluster.
>>>
>>> Therefore, IMO a node restart for a config change is not relevant,
>>> also no need of a periodic config checks.
>>>
>>> Btw, can you give me some example configuration you were thinking
>>> of?
>>>
>>> Thanks,
>>> Sajith
>>>
>>> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
>>> uvin...@wso2.com> wrote:
>>>
 Was wondering about $subject

 Traditionally we have stored our product configs, be it carbon.xml,
 api-manager.xml, identity.xml, etc. at file level. Some configs, such 
 as
 "port offset" are inherently bound to the server startup so it makes 
 sense
 for them to be at file level, since they come into affect during the
 startup. But certain runtime configs actually get engaged only when a 
 given
 feature is used. But having those configs at file level require a 
 restart

Re: [Architecture] Storing configs in database for C5

[Lakmal Warusawithana]

Yes, storing configs in the database make the deployment complicated. In
container world, config, dependancies, artifacts etc all bundle into docker
image to make service start fast. Also it give as immutable container
capabilities which make production deployment more robust.

On Wed, Oct 12, 2016 at 5:04 PM, Srinath Perera  wrote:

> I believe the plan is that for server configs, there is no (admin) UI in
> C5. It can be changed only via config files. End user scenarios such as API
> publisher might have UIs for configs.  ( We should agree here or have a
> meeting if we want to change that)
>
> What is an example of case where configs will take effect later?
>
> IMO storing configs in the dtabase make the deployment complicated. Also
> it reduce the scalability as all server point to one DB. I think it also
> make docker usecase complex.
>
>
> --Srinath
>
> On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:
>
>>
>>
>> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
>> wrote:
>>
>>>
>>>
>>> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>>>
 There are challenges when moving configs to the DB. We experienced it
 once when we moved the analytics configs to the registry. And then we moved
 it back again to the FS because it was too painful to maintain.

 1. The nodes have to keep polling the DB in a fast enough interval.
 This is a unnecessary performance overhead. Because in practise, someone
 will only change these configs once. But to support that use case, we have
 to keep polling the DB for life.

 2. Gateways don't have access to the DB. So say you're enabling
 analytics (data publishing). You have to propagate that change to the
 Gateway nodes using some mechanism. And with no clustering on C5, this is a
 challenge.

 If the objective of this is to make the Cloud (tenant) experience
 better, I think we should just restart the tenant's containers with the
 relevant configs in place.

>>>
>>> Still we have a problem with regard to how we are going to allow the
>>> tenants to do the configuration changes. Currently we do it through the
>>> registry which will not work for C5.
>>>
>>
>> Yes, so my idea is to provide a UI to do the configs. Those configs we
>> can store anywhere (maybe in a table) just for the sake of rendering that
>> UI. The product code will still read from the config files. When you apply
>> those configs through the press of a button, the container should get
>> rebuilt and restarted with the necessary modification to the config files.
>>
>>>
>>> Thanks,
>>> Lakmali
>>>

 Thanks,
 NuwanD.

 On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta 
 wrote:

>
>
> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha 
> wrote:
>
>> Hi Sajith,
>>
>> Yes even though the boot up time is not an issue in C5 the other
>> advantages I have outlined are still there to be gained. There is a huge
>> effort we have to do on dev ops side to maintain those images you are
>> talking about because of having everything at file level.
>>
>> Some examples from API Manager I can think of are turning
>> notifications on/off, enable monetization, enable/disable stats, 
>> configure
>> work flows, Enable/Disable JWT token header.
>>
>
> +1 to move feature related configurations to the database and make
> them configurable through the UI.
>
> Thanks,
> Lakmali
>
>>
>>
>>
>> On 12 October 2016 at 12:58, Sajith Kariyawasam 
>> wrote:
>>
>>> Hi Uvindra,
>>>
>>> With cloud deployment in mind, the idea is to boot up the nodes in
>>> quick time, therefore the docker images are pre-configured with all the
>>> configuration values, which will speed up the node start up. A change of
>>> configuration means a new docker image will be created with the new
>>> configs, and re-spawn the cluster.
>>>
>>> Therefore, IMO a node restart for a config change is not relevant,
>>> also no need of a periodic config checks.
>>>
>>> Btw, can you give me some example configuration you were thinking
>>> of?
>>>
>>> Thanks,
>>> Sajith
>>>
>>> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
>>> uvin...@wso2.com> wrote:
>>>
 Was wondering about $subject

 Traditionally we have stored our product configs, be it carbon.xml,
 api-manager.xml, identity.xml, etc. at file level. Some configs, such 
 as
 "port offset" are inherently bound to the server startup so it makes 
 sense
 for them to be at file level, since they come into affect during the
 startup. But certain runtime configs actually get engaged only when a 
 given
 

Re: [Architecture] Storing configs in database for C5

[Srinath Perera]

I believe the plan is that for server configs, there is no (admin) UI in
C5. It can be changed only via config files. End user scenarios such as API
publisher might have UIs for configs.  ( We should agree here or have a
meeting if we want to change that)

What is an example of case where configs will take effect later?

IMO storing configs in the dtabase make the deployment complicated. Also it
reduce the scalability as all server point to one DB. I think it also make
docker usecase complex.


--Srinath

On Wed, Oct 12, 2016 at 3:17 AM, Nuwan Dias  wrote:

>
>
> On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
> wrote:
>
>>
>>
>> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>>
>>> There are challenges when moving configs to the DB. We experienced it
>>> once when we moved the analytics configs to the registry. And then we moved
>>> it back again to the FS because it was too painful to maintain.
>>>
>>> 1. The nodes have to keep polling the DB in a fast enough interval. This
>>> is a unnecessary performance overhead. Because in practise, someone will
>>> only change these configs once. But to support that use case, we have to
>>> keep polling the DB for life.
>>>
>>> 2. Gateways don't have access to the DB. So say you're enabling
>>> analytics (data publishing). You have to propagate that change to the
>>> Gateway nodes using some mechanism. And with no clustering on C5, this is a
>>> challenge.
>>>
>>> If the objective of this is to make the Cloud (tenant) experience
>>> better, I think we should just restart the tenant's containers with the
>>> relevant configs in place.
>>>
>>
>> Still we have a problem with regard to how we are going to allow the
>> tenants to do the configuration changes. Currently we do it through the
>> registry which will not work for C5.
>>
>
> Yes, so my idea is to provide a UI to do the configs. Those configs we can
> store anywhere (maybe in a table) just for the sake of rendering that UI.
> The product code will still read from the config files. When you apply
> those configs through the press of a button, the container should get
> rebuilt and restarted with the necessary modification to the config files.
>
>>
>> Thanks,
>> Lakmali
>>
>>>
>>> Thanks,
>>> NuwanD.
>>>
>>> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta 
>>> wrote:
>>>


 On 12 October 2016 at 13:47, Uvindra Dias Jayasinha 
 wrote:

> Hi Sajith,
>
> Yes even though the boot up time is not an issue in C5 the other
> advantages I have outlined are still there to be gained. There is a huge
> effort we have to do on dev ops side to maintain those images you are
> talking about because of having everything at file level.
>
> Some examples from API Manager I can think of are turning
> notifications on/off, enable monetization, enable/disable stats, configure
> work flows, Enable/Disable JWT token header.
>

 +1 to move feature related configurations to the database and make them
 configurable through the UI.

 Thanks,
 Lakmali

>
>
>
> On 12 October 2016 at 12:58, Sajith Kariyawasam 
> wrote:
>
>> Hi Uvindra,
>>
>> With cloud deployment in mind, the idea is to boot up the nodes in
>> quick time, therefore the docker images are pre-configured with all the
>> configuration values, which will speed up the node start up. A change of
>> configuration means a new docker image will be created with the new
>> configs, and re-spawn the cluster.
>>
>> Therefore, IMO a node restart for a config change is not relevant,
>> also no need of a periodic config checks.
>>
>> Btw, can you give me some example configuration you were thinking of?
>>
>> Thanks,
>> Sajith
>>
>> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
>> uvin...@wso2.com> wrote:
>>
>>> Was wondering about $subject
>>>
>>> Traditionally we have stored our product configs, be it carbon.xml,
>>> api-manager.xml, identity.xml, etc. at file level. Some configs, such as
>>> "port offset" are inherently bound to the server startup so it makes 
>>> sense
>>> for them to be at file level, since they come into affect during the
>>> startup. But certain runtime configs actually get engaged only when a 
>>> given
>>> feature is used. But having those configs at file level require a 
>>> restart
>>> for the changes to take affect. In C4 API Manager avoided doing restarts
>>> for certain config changes, like adding mediation extensions, by storing
>>> them in the registry.
>>>
>>> For C5 a reusable implementation can exist at each node which
>>> periodically reads the table(say once a minute) and updates the config
>>> values in memory. Products communicate with this config library to get 

Re: [Architecture] Storing configs in database for C5

[Nuwan Dias]

On Wed, Oct 12, 2016 at 3:28 PM, Lakmali Baminiwatta 
wrote:

>
>
> On 12 October 2016 at 14:31, Nuwan Dias  wrote:
>
>> There are challenges when moving configs to the DB. We experienced it
>> once when we moved the analytics configs to the registry. And then we moved
>> it back again to the FS because it was too painful to maintain.
>>
>> 1. The nodes have to keep polling the DB in a fast enough interval. This
>> is a unnecessary performance overhead. Because in practise, someone will
>> only change these configs once. But to support that use case, we have to
>> keep polling the DB for life.
>>
>> 2. Gateways don't have access to the DB. So say you're enabling analytics
>> (data publishing). You have to propagate that change to the Gateway nodes
>> using some mechanism. And with no clustering on C5, this is a challenge.
>>
>> If the objective of this is to make the Cloud (tenant) experience better,
>> I think we should just restart the tenant's containers with the relevant
>> configs in place.
>>
>
> Still we have a problem with regard to how we are going to allow the
> tenants to do the configuration changes. Currently we do it through the
> registry which will not work for C5.
>

Yes, so my idea is to provide a UI to do the configs. Those configs we can
store anywhere (maybe in a table) just for the sake of rendering that UI.
The product code will still read from the config files. When you apply
those configs through the press of a button, the container should get
rebuilt and restarted with the necessary modification to the config files.

>
> Thanks,
> Lakmali
>
>>
>> Thanks,
>> NuwanD.
>>
>> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta 
>> wrote:
>>
>>>
>>>
>>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha 
>>> wrote:
>>>
 Hi Sajith,

 Yes even though the boot up time is not an issue in C5 the other
 advantages I have outlined are still there to be gained. There is a huge
 effort we have to do on dev ops side to maintain those images you are
 talking about because of having everything at file level.

 Some examples from API Manager I can think of are turning notifications
 on/off, enable monetization, enable/disable stats, configure work flows,
 Enable/Disable JWT token header.

>>>
>>> +1 to move feature related configurations to the database and make them
>>> configurable through the UI.
>>>
>>> Thanks,
>>> Lakmali
>>>



 On 12 October 2016 at 12:58, Sajith Kariyawasam 
 wrote:

> Hi Uvindra,
>
> With cloud deployment in mind, the idea is to boot up the nodes in
> quick time, therefore the docker images are pre-configured with all the
> configuration values, which will speed up the node start up. A change of
> configuration means a new docker image will be created with the new
> configs, and re-spawn the cluster.
>
> Therefore, IMO a node restart for a config change is not relevant,
> also no need of a periodic config checks.
>
> Btw, can you give me some example configuration you were thinking of?
>
> Thanks,
> Sajith
>
> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
> uvin...@wso2.com> wrote:
>
>> Was wondering about $subject
>>
>> Traditionally we have stored our product configs, be it carbon.xml,
>> api-manager.xml, identity.xml, etc. at file level. Some configs, such as
>> "port offset" are inherently bound to the server startup so it makes 
>> sense
>> for them to be at file level, since they come into affect during the
>> startup. But certain runtime configs actually get engaged only when a 
>> given
>> feature is used. But having those configs at file level require a restart
>> for the changes to take affect. In C4 API Manager avoided doing restarts
>> for certain config changes, like adding mediation extensions, by storing
>> them in the registry.
>>
>> For C5 a reusable implementation can exist at each node which
>> periodically reads the table(say once a minute) and updates the config
>> values in memory. Products communicate with this config library to get 
>> the
>> values of a given config. So eventually they will read the updated value 
>> in
>> a short time. If we were to store at least certain configs at DB level
>> there are several advantages.
>>
>> 1. Eliminate need for a restart for changes to take affect. I realize
>> in C5 a restart is relatively cheap so this might not be a big deal, but
>> you still need someone to initiate the restart after the config change.
>>
>> 2. Since the config DB table has a known structure a UI can be easily
>> developed to do CRUD operations for config changes and used by all
>> products. This is a lot more user friendly than asking users to change
>> files.
>>

Re: [Architecture] Storing configs in database for C5

[Lakmali Baminiwatta]

On 12 October 2016 at 14:31, Nuwan Dias  wrote:

> There are challenges when moving configs to the DB. We experienced it once
> when we moved the analytics configs to the registry. And then we moved it
> back again to the FS because it was too painful to maintain.
>
> 1. The nodes have to keep polling the DB in a fast enough interval. This
> is a unnecessary performance overhead. Because in practise, someone will
> only change these configs once. But to support that use case, we have to
> keep polling the DB for life.
>
> 2. Gateways don't have access to the DB. So say you're enabling analytics
> (data publishing). You have to propagate that change to the Gateway nodes
> using some mechanism. And with no clustering on C5, this is a challenge.
>
> If the objective of this is to make the Cloud (tenant) experience better,
> I think we should just restart the tenant's containers with the relevant
> configs in place.
>

Still we have a problem with regard to how we are going to allow the
tenants to do the configuration changes. Currently we do it through the
registry which will not work for C5.

Thanks,
Lakmali

>
> Thanks,
> NuwanD.
>
> On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta 
> wrote:
>
>>
>>
>> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha 
>> wrote:
>>
>>> Hi Sajith,
>>>
>>> Yes even though the boot up time is not an issue in C5 the other
>>> advantages I have outlined are still there to be gained. There is a huge
>>> effort we have to do on dev ops side to maintain those images you are
>>> talking about because of having everything at file level.
>>>
>>> Some examples from API Manager I can think of are turning notifications
>>> on/off, enable monetization, enable/disable stats, configure work flows,
>>> Enable/Disable JWT token header.
>>>
>>
>> +1 to move feature related configurations to the database and make them
>> configurable through the UI.
>>
>> Thanks,
>> Lakmali
>>
>>>
>>>
>>>
>>> On 12 October 2016 at 12:58, Sajith Kariyawasam  wrote:
>>>
 Hi Uvindra,

 With cloud deployment in mind, the idea is to boot up the nodes in
 quick time, therefore the docker images are pre-configured with all the
 configuration values, which will speed up the node start up. A change of
 configuration means a new docker image will be created with the new
 configs, and re-spawn the cluster.

 Therefore, IMO a node restart for a config change is not relevant, also
 no need of a periodic config checks.

 Btw, can you give me some example configuration you were thinking of?

 Thanks,
 Sajith

 On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
 uvin...@wso2.com> wrote:

> Was wondering about $subject
>
> Traditionally we have stored our product configs, be it carbon.xml,
> api-manager.xml, identity.xml, etc. at file level. Some configs, such as
> "port offset" are inherently bound to the server startup so it makes sense
> for them to be at file level, since they come into affect during the
> startup. But certain runtime configs actually get engaged only when a 
> given
> feature is used. But having those configs at file level require a restart
> for the changes to take affect. In C4 API Manager avoided doing restarts
> for certain config changes, like adding mediation extensions, by storing
> them in the registry.
>
> For C5 a reusable implementation can exist at each node which
> periodically reads the table(say once a minute) and updates the config
> values in memory. Products communicate with this config library to get the
> values of a given config. So eventually they will read the updated value 
> in
> a short time. If we were to store at least certain configs at DB level
> there are several advantages.
>
> 1. Eliminate need for a restart for changes to take affect. I realize
> in C5 a restart is relatively cheap so this might not be a big deal, but
> you still need someone to initiate the restart after the config change.
>
> 2. Since the config DB table has a known structure a UI can be easily
> developed to do CRUD operations for config changes and used by all
> products. This is a lot more user friendly than asking users to change
> files.
>
> 3. We can provide a REST API to allow config changes to be done on the
> DB table alternatively.
>
> 4. Simplify dev ops by eliminating complicated puppet config templates
> that need to constantly maintained with new releases.
>
> 5. Since configs are in a central DB its easy to manage them since all
> nodes will read from the same table.
>
> 6. Configs can be backed up by simply backing up the table
>
>
> Doing this makes sense for certain use cases of API Manger, I'm sure
> there maybe similar benefits for other products as 

Re: [Architecture] Storing configs in database for C5

[Inosh Goonewardena]

Storing configurations in DB has some other advantages like security (where
users cannot alter settings or read sensitive data).

However, IMO it will add an unnecessary overhead because we have to rely on
DB connection and reading overhead is also there.

But, I agree on the point about the enable/disable configurations
(monetization, stats, etc,) where it require a server restart after config
changes atm. In this case, can't we watch the config file for changes (ex:
using WatchService) and reload if there are changes to config properties?

On Wed, Oct 12, 2016 at 1:47 PM, Uvindra Dias Jayasinha 
wrote:

> Hi Sajith,
>
> Yes even though the boot up time is not an issue in C5 the other
> advantages I have outlined are still there to be gained. There is a huge
> effort we have to do on dev ops side to maintain those images you are
> talking about because of having everything at file level.
>
> Some examples from API Manager I can think of are turning notifications
> on/off, enable monetization, enable/disable stats, configure work flows,
> Enable/Disable JWT token header.
>
>
>
> On 12 October 2016 at 12:58, Sajith Kariyawasam  wrote:
>
>> Hi Uvindra,
>>
>> With cloud deployment in mind, the idea is to boot up the nodes in quick
>> time, therefore the docker images are pre-configured with all the
>> configuration values, which will speed up the node start up. A change of
>> configuration means a new docker image will be created with the new
>> configs, and re-spawn the cluster.
>>
>> Therefore, IMO a node restart for a config change is not relevant, also
>> no need of a periodic config checks.
>>
>> Btw, can you give me some example configuration you were thinking of?
>>
>> Thanks,
>> Sajith
>>
>> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
>> uvin...@wso2.com> wrote:
>>
>>> Was wondering about $subject
>>>
>>> Traditionally we have stored our product configs, be it carbon.xml,
>>> api-manager.xml, identity.xml, etc. at file level. Some configs, such as
>>> "port offset" are inherently bound to the server startup so it makes sense
>>> for them to be at file level, since they come into affect during the
>>> startup. But certain runtime configs actually get engaged only when a given
>>> feature is used. But having those configs at file level require a restart
>>> for the changes to take affect. In C4 API Manager avoided doing restarts
>>> for certain config changes, like adding mediation extensions, by storing
>>> them in the registry.
>>>
>>> For C5 a reusable implementation can exist at each node which
>>> periodically reads the table(say once a minute) and updates the config
>>> values in memory. Products communicate with this config library to get the
>>> values of a given config. So eventually they will read the updated value in
>>> a short time. If we were to store at least certain configs at DB level
>>> there are several advantages.
>>>
>>> 1. Eliminate need for a restart for changes to take affect. I realize in
>>> C5 a restart is relatively cheap so this might not be a big deal, but you
>>> still need someone to initiate the restart after the config change.
>>>
>>> 2. Since the config DB table has a known structure a UI can be easily
>>> developed to do CRUD operations for config changes and used by all
>>> products. This is a lot more user friendly than asking users to change
>>> files.
>>>
>>> 3. We can provide a REST API to allow config changes to be done on the
>>> DB table alternatively.
>>>
>>> 4. Simplify dev ops by eliminating complicated puppet config templates
>>> that need to constantly maintained with new releases.
>>>
>>> 5. Since configs are in a central DB its easy to manage them since all
>>> nodes will read from the same table.
>>>
>>> 6. Configs can be backed up by simply backing up the table
>>>
>>>
>>> Doing this makes sense for certain use cases of API Manger, I'm sure
>>> there maybe similar benefits for other products as well. It may not make
>>> sense for all configs but at least for some that govern feature
>>> functionality its great to have. WDYT?
>>>
>>> --
>>> Regards,
>>> Uvindra
>>>
>>> Mobile: 33962
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> Sajith Kariyawasam
>> *Associate Tech Lead*
>> *WSO2 Inc.; http://wso2.com *
>> *Committer and PMC member, Apache Stratos *
>> *AMIE (SL)*
>> *Mobile: 0772269575*
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Regards,
> Uvindra
>
> Mobile: 33962
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Thanks & Regards,

Inosh Goonewardena

Re: [Architecture] Storing configs in database for C5

[Nuwan Dias]

There are challenges when moving configs to the DB. We experienced it once
when we moved the analytics configs to the registry. And then we moved it
back again to the FS because it was too painful to maintain.

1. The nodes have to keep polling the DB in a fast enough interval. This is
a unnecessary performance overhead. Because in practise, someone will only
change these configs once. But to support that use case, we have to keep
polling the DB for life.

2. Gateways don't have access to the DB. So say you're enabling analytics
(data publishing). You have to propagate that change to the Gateway nodes
using some mechanism. And with no clustering on C5, this is a challenge.

If the objective of this is to make the Cloud (tenant) experience better, I
think we should just restart the tenant's containers with the relevant
configs in place.

Thanks,
NuwanD.

On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta 
wrote:

>
>
> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha 
> wrote:
>
>> Hi Sajith,
>>
>> Yes even though the boot up time is not an issue in C5 the other
>> advantages I have outlined are still there to be gained. There is a huge
>> effort we have to do on dev ops side to maintain those images you are
>> talking about because of having everything at file level.
>>
>> Some examples from API Manager I can think of are turning notifications
>> on/off, enable monetization, enable/disable stats, configure work flows,
>> Enable/Disable JWT token header.
>>
>
> +1 to move feature related configurations to the database and make them
> configurable through the UI.
>
> Thanks,
> Lakmali
>
>>
>>
>>
>> On 12 October 2016 at 12:58, Sajith Kariyawasam  wrote:
>>
>>> Hi Uvindra,
>>>
>>> With cloud deployment in mind, the idea is to boot up the nodes in quick
>>> time, therefore the docker images are pre-configured with all the
>>> configuration values, which will speed up the node start up. A change of
>>> configuration means a new docker image will be created with the new
>>> configs, and re-spawn the cluster.
>>>
>>> Therefore, IMO a node restart for a config change is not relevant, also
>>> no need of a periodic config checks.
>>>
>>> Btw, can you give me some example configuration you were thinking of?
>>>
>>> Thanks,
>>> Sajith
>>>
>>> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
>>> uvin...@wso2.com> wrote:
>>>
 Was wondering about $subject

 Traditionally we have stored our product configs, be it carbon.xml,
 api-manager.xml, identity.xml, etc. at file level. Some configs, such as
 "port offset" are inherently bound to the server startup so it makes sense
 for them to be at file level, since they come into affect during the
 startup. But certain runtime configs actually get engaged only when a given
 feature is used. But having those configs at file level require a restart
 for the changes to take affect. In C4 API Manager avoided doing restarts
 for certain config changes, like adding mediation extensions, by storing
 them in the registry.

 For C5 a reusable implementation can exist at each node which
 periodically reads the table(say once a minute) and updates the config
 values in memory. Products communicate with this config library to get the
 values of a given config. So eventually they will read the updated value in
 a short time. If we were to store at least certain configs at DB level
 there are several advantages.

 1. Eliminate need for a restart for changes to take affect. I realize
 in C5 a restart is relatively cheap so this might not be a big deal, but
 you still need someone to initiate the restart after the config change.

 2. Since the config DB table has a known structure a UI can be easily
 developed to do CRUD operations for config changes and used by all
 products. This is a lot more user friendly than asking users to change
 files.

 3. We can provide a REST API to allow config changes to be done on the
 DB table alternatively.

 4. Simplify dev ops by eliminating complicated puppet config templates
 that need to constantly maintained with new releases.

 5. Since configs are in a central DB its easy to manage them since all
 nodes will read from the same table.

 6. Configs can be backed up by simply backing up the table


 Doing this makes sense for certain use cases of API Manger, I'm sure
 there maybe similar benefits for other products as well. It may not make
 sense for all configs but at least for some that govern feature
 functionality its great to have. WDYT?

 --
 Regards,
 Uvindra

 Mobile: 33962

 ___
 Architecture mailing list
 Architecture@wso2.org
 https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


>>>
>>>
>>> --
>>> 

Re: [Architecture] Storing configs in database for C5

[Lahiru Cooray]

This will also make the life easier when we need to make such
configurations tenant wise configurable.

On Wed, Oct 12, 2016 at 2:12 PM, Lakmali Baminiwatta 
wrote:

>
>
> On 12 October 2016 at 13:47, Uvindra Dias Jayasinha 
> wrote:
>
>> Hi Sajith,
>>
>> Yes even though the boot up time is not an issue in C5 the other
>> advantages I have outlined are still there to be gained. There is a huge
>> effort we have to do on dev ops side to maintain those images you are
>> talking about because of having everything at file level.
>>
>> Some examples from API Manager I can think of are turning notifications
>> on/off, enable monetization, enable/disable stats, configure work flows,
>> Enable/Disable JWT token header.
>>
>
> +1 to move feature related configurations to the database and make them
> configurable through the UI.
>
> Thanks,
> Lakmali
>
>>
>>
>>
>> On 12 October 2016 at 12:58, Sajith Kariyawasam  wrote:
>>
>>> Hi Uvindra,
>>>
>>> With cloud deployment in mind, the idea is to boot up the nodes in quick
>>> time, therefore the docker images are pre-configured with all the
>>> configuration values, which will speed up the node start up. A change of
>>> configuration means a new docker image will be created with the new
>>> configs, and re-spawn the cluster.
>>>
>>> Therefore, IMO a node restart for a config change is not relevant, also
>>> no need of a periodic config checks.
>>>
>>> Btw, can you give me some example configuration you were thinking of?
>>>
>>> Thanks,
>>> Sajith
>>>
>>> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
>>> uvin...@wso2.com> wrote:
>>>
 Was wondering about $subject

 Traditionally we have stored our product configs, be it carbon.xml,
 api-manager.xml, identity.xml, etc. at file level. Some configs, such as
 "port offset" are inherently bound to the server startup so it makes sense
 for them to be at file level, since they come into affect during the
 startup. But certain runtime configs actually get engaged only when a given
 feature is used. But having those configs at file level require a restart
 for the changes to take affect. In C4 API Manager avoided doing restarts
 for certain config changes, like adding mediation extensions, by storing
 them in the registry.

 For C5 a reusable implementation can exist at each node which
 periodically reads the table(say once a minute) and updates the config
 values in memory. Products communicate with this config library to get the
 values of a given config. So eventually they will read the updated value in
 a short time. If we were to store at least certain configs at DB level
 there are several advantages.

 1. Eliminate need for a restart for changes to take affect. I realize
 in C5 a restart is relatively cheap so this might not be a big deal, but
 you still need someone to initiate the restart after the config change.

 2. Since the config DB table has a known structure a UI can be easily
 developed to do CRUD operations for config changes and used by all
 products. This is a lot more user friendly than asking users to change
 files.

 3. We can provide a REST API to allow config changes to be done on the
 DB table alternatively.

 4. Simplify dev ops by eliminating complicated puppet config templates
 that need to constantly maintained with new releases.

 5. Since configs are in a central DB its easy to manage them since all
 nodes will read from the same table.

 6. Configs can be backed up by simply backing up the table


 Doing this makes sense for certain use cases of API Manger, I'm sure
 there maybe similar benefits for other products as well. It may not make
 sense for all configs but at least for some that govern feature
 functionality its great to have. WDYT?

 --
 Regards,
 Uvindra

 Mobile: 33962

 ___
 Architecture mailing list
 Architecture@wso2.org
 https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture


>>>
>>>
>>> --
>>> Sajith Kariyawasam
>>> *Associate Tech Lead*
>>> *WSO2 Inc.; http://wso2.com *
>>> *Committer and PMC member, Apache Stratos *
>>> *AMIE (SL)*
>>> *Mobile: 0772269575*
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> Regards,
>> Uvindra
>>
>> Mobile: 33962
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Lakmali Baminiwatta
> Associate Technical Lead
> WSO2, Inc.: http://wso2.com
> lean.enterprise.middleware
> mobile:  +94 71 2335936
> blog : 

Re: [Architecture] Storing configs in database for C5

[Lakmali Baminiwatta]

On 12 October 2016 at 13:47, Uvindra Dias Jayasinha 
wrote:

> Hi Sajith,
>
> Yes even though the boot up time is not an issue in C5 the other
> advantages I have outlined are still there to be gained. There is a huge
> effort we have to do on dev ops side to maintain those images you are
> talking about because of having everything at file level.
>
> Some examples from API Manager I can think of are turning notifications
> on/off, enable monetization, enable/disable stats, configure work flows,
> Enable/Disable JWT token header.
>

+1 to move feature related configurations to the database and make them
configurable through the UI.

Thanks,
Lakmali

>
>
>
> On 12 October 2016 at 12:58, Sajith Kariyawasam  wrote:
>
>> Hi Uvindra,
>>
>> With cloud deployment in mind, the idea is to boot up the nodes in quick
>> time, therefore the docker images are pre-configured with all the
>> configuration values, which will speed up the node start up. A change of
>> configuration means a new docker image will be created with the new
>> configs, and re-spawn the cluster.
>>
>> Therefore, IMO a node restart for a config change is not relevant, also
>> no need of a periodic config checks.
>>
>> Btw, can you give me some example configuration you were thinking of?
>>
>> Thanks,
>> Sajith
>>
>> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha <
>> uvin...@wso2.com> wrote:
>>
>>> Was wondering about $subject
>>>
>>> Traditionally we have stored our product configs, be it carbon.xml,
>>> api-manager.xml, identity.xml, etc. at file level. Some configs, such as
>>> "port offset" are inherently bound to the server startup so it makes sense
>>> for them to be at file level, since they come into affect during the
>>> startup. But certain runtime configs actually get engaged only when a given
>>> feature is used. But having those configs at file level require a restart
>>> for the changes to take affect. In C4 API Manager avoided doing restarts
>>> for certain config changes, like adding mediation extensions, by storing
>>> them in the registry.
>>>
>>> For C5 a reusable implementation can exist at each node which
>>> periodically reads the table(say once a minute) and updates the config
>>> values in memory. Products communicate with this config library to get the
>>> values of a given config. So eventually they will read the updated value in
>>> a short time. If we were to store at least certain configs at DB level
>>> there are several advantages.
>>>
>>> 1. Eliminate need for a restart for changes to take affect. I realize in
>>> C5 a restart is relatively cheap so this might not be a big deal, but you
>>> still need someone to initiate the restart after the config change.
>>>
>>> 2. Since the config DB table has a known structure a UI can be easily
>>> developed to do CRUD operations for config changes and used by all
>>> products. This is a lot more user friendly than asking users to change
>>> files.
>>>
>>> 3. We can provide a REST API to allow config changes to be done on the
>>> DB table alternatively.
>>>
>>> 4. Simplify dev ops by eliminating complicated puppet config templates
>>> that need to constantly maintained with new releases.
>>>
>>> 5. Since configs are in a central DB its easy to manage them since all
>>> nodes will read from the same table.
>>>
>>> 6. Configs can be backed up by simply backing up the table
>>>
>>>
>>> Doing this makes sense for certain use cases of API Manger, I'm sure
>>> there maybe similar benefits for other products as well. It may not make
>>> sense for all configs but at least for some that govern feature
>>> functionality its great to have. WDYT?
>>>
>>> --
>>> Regards,
>>> Uvindra
>>>
>>> Mobile: 33962
>>>
>>> ___
>>> Architecture mailing list
>>> Architecture@wso2.org
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> Sajith Kariyawasam
>> *Associate Tech Lead*
>> *WSO2 Inc.; http://wso2.com *
>> *Committer and PMC member, Apache Stratos *
>> *AMIE (SL)*
>> *Mobile: 0772269575*
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Regards,
> Uvindra
>
> Mobile: 33962
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Lakmali Baminiwatta
Associate Technical Lead
WSO2, Inc.: http://wso2.com
lean.enterprise.middleware
mobile:  +94 71 2335936
blog : lakmali.com
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Storing configs in database for C5

[Uvindra Dias Jayasinha]

Hi Sajith,

Yes even though the boot up time is not an issue in C5 the other advantages
I have outlined are still there to be gained. There is a huge effort we
have to do on dev ops side to maintain those images you are talking about
because of having everything at file level.

Some examples from API Manager I can think of are turning notifications
on/off, enable monetization, enable/disable stats, configure work flows,
Enable/Disable JWT token header.



On 12 October 2016 at 12:58, Sajith Kariyawasam  wrote:

> Hi Uvindra,
>
> With cloud deployment in mind, the idea is to boot up the nodes in quick
> time, therefore the docker images are pre-configured with all the
> configuration values, which will speed up the node start up. A change of
> configuration means a new docker image will be created with the new
> configs, and re-spawn the cluster.
>
> Therefore, IMO a node restart for a config change is not relevant, also no
> need of a periodic config checks.
>
> Btw, can you give me some example configuration you were thinking of?
>
> Thanks,
> Sajith
>
> On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha  > wrote:
>
>> Was wondering about $subject
>>
>> Traditionally we have stored our product configs, be it carbon.xml,
>> api-manager.xml, identity.xml, etc. at file level. Some configs, such as
>> "port offset" are inherently bound to the server startup so it makes sense
>> for them to be at file level, since they come into affect during the
>> startup. But certain runtime configs actually get engaged only when a given
>> feature is used. But having those configs at file level require a restart
>> for the changes to take affect. In C4 API Manager avoided doing restarts
>> for certain config changes, like adding mediation extensions, by storing
>> them in the registry.
>>
>> For C5 a reusable implementation can exist at each node which
>> periodically reads the table(say once a minute) and updates the config
>> values in memory. Products communicate with this config library to get the
>> values of a given config. So eventually they will read the updated value in
>> a short time. If we were to store at least certain configs at DB level
>> there are several advantages.
>>
>> 1. Eliminate need for a restart for changes to take affect. I realize in
>> C5 a restart is relatively cheap so this might not be a big deal, but you
>> still need someone to initiate the restart after the config change.
>>
>> 2. Since the config DB table has a known structure a UI can be easily
>> developed to do CRUD operations for config changes and used by all
>> products. This is a lot more user friendly than asking users to change
>> files.
>>
>> 3. We can provide a REST API to allow config changes to be done on the DB
>> table alternatively.
>>
>> 4. Simplify dev ops by eliminating complicated puppet config templates
>> that need to constantly maintained with new releases.
>>
>> 5. Since configs are in a central DB its easy to manage them since all
>> nodes will read from the same table.
>>
>> 6. Configs can be backed up by simply backing up the table
>>
>>
>> Doing this makes sense for certain use cases of API Manger, I'm sure
>> there maybe similar benefits for other products as well. It may not make
>> sense for all configs but at least for some that govern feature
>> functionality its great to have. WDYT?
>>
>> --
>> Regards,
>> Uvindra
>>
>> Mobile: 33962
>>
>> ___
>> Architecture mailing list
>> Architecture@wso2.org
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> Sajith Kariyawasam
> *Associate Tech Lead*
> *WSO2 Inc.; http://wso2.com *
> *Committer and PMC member, Apache Stratos *
> *AMIE (SL)*
> *Mobile: 0772269575*
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Regards,
Uvindra

Mobile: 33962
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Re: [Architecture] Storing configs in database for C5

[Sajith Kariyawasam]

Hi Uvindra,

With cloud deployment in mind, the idea is to boot up the nodes in quick
time, therefore the docker images are pre-configured with all the
configuration values, which will speed up the node start up. A change of
configuration means a new docker image will be created with the new
configs, and re-spawn the cluster.

Therefore, IMO a node restart for a config change is not relevant, also no
need of a periodic config checks.

Btw, can you give me some example configuration you were thinking of?

Thanks,
Sajith

On Wed, Oct 12, 2016 at 11:53 AM, Uvindra Dias Jayasinha 
wrote:

> Was wondering about $subject
>
> Traditionally we have stored our product configs, be it carbon.xml,
> api-manager.xml, identity.xml, etc. at file level. Some configs, such as
> "port offset" are inherently bound to the server startup so it makes sense
> for them to be at file level, since they come into affect during the
> startup. But certain runtime configs actually get engaged only when a given
> feature is used. But having those configs at file level require a restart
> for the changes to take affect. In C4 API Manager avoided doing restarts
> for certain config changes, like adding mediation extensions, by storing
> them in the registry.
>
> For C5 a reusable implementation can exist at each node which periodically
> reads the table(say once a minute) and updates the config values in memory.
> Products communicate with this config library to get the values of a given
> config. So eventually they will read the updated value in a short time. If
> we were to store at least certain configs at DB level there are several
> advantages.
>
> 1. Eliminate need for a restart for changes to take affect. I realize in
> C5 a restart is relatively cheap so this might not be a big deal, but you
> still need someone to initiate the restart after the config change.
>
> 2. Since the config DB table has a known structure a UI can be easily
> developed to do CRUD operations for config changes and used by all
> products. This is a lot more user friendly than asking users to change
> files.
>
> 3. We can provide a REST API to allow config changes to be done on the DB
> table alternatively.
>
> 4. Simplify dev ops by eliminating complicated puppet config templates
> that need to constantly maintained with new releases.
>
> 5. Since configs are in a central DB its easy to manage them since all
> nodes will read from the same table.
>
> 6. Configs can be backed up by simply backing up the table
>
>
> Doing this makes sense for certain use cases of API Manger, I'm sure there
> maybe similar benefits for other products as well. It may not make sense
> for all configs but at least for some that govern feature functionality its
> great to have. WDYT?
>
> --
> Regards,
> Uvindra
>
> Mobile: 33962
>
> ___
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Sajith Kariyawasam
*Associate Tech Lead*
*WSO2 Inc.; http://wso2.com *
*Committer and PMC member, Apache Stratos *
*AMIE (SL)*
*Mobile: 0772269575*
___
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture