Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Samitha Chathuranga]

Hi Tharindu,

On Mon, Feb 3, 2020 at 4:07 PM Tharindu Dharmarathna 
wrote:

> Hi Samitha,
>
> Please find my comments inline below.
>
> On Mon, Feb 3, 2020 at 3:29 PM Samitha Chathuranga 
> wrote:
>
>> Hi,
>>
>> Additionally to this, I could see that there are still the below
>> operations executed in the profile optimization script.
>>
>> * - Config changes in below config files*
>>
>>- api-manager.xml - enable_policy_deploy, enable_data_publishing,
>>enable_blacklist_condition,
>>enable_decision_connection (JMSConnectionDetails)
>>- axis2.xml - transport.ws.sender.enable, transport.wss.sender.enable
>>- registry.xml - indexing.enable
>>
>> These profile specific configurations are already handled by the
>> server_role implementation in infer.json, so no need to do anything in
>> profile optimization.
>>
>> *- Config file replacements*
>>
>>- axis2_TM.xml
>>- axis2_KM.xml
>>- registry_TM.xml
>>
>> This step will need to change the relevant .j2 file according to the
> profile. Therefore we have to keep this step in optimizer as itits.
>

For config file replacement there are the functions replaceAxis2File(),
replaceTenantAxis2File() and replaceRegistryXMLFile() in profileSetup.sh
[1]. And for the replacement of relavent .jt template files, there are
separate functions i.e. replaceAxis2TemplateFile(),
replaceTenantAxis2TemplateFile(), replaceRegistryXMLTemplateFile(). So the
two operations are independent in the script. So I don't see that the above
statement is correct. So we can safely remove the "Config file
replacements" AFAIU.
Please clarify, if you meant something else.

[1]
https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/startup-scripts/profileSetup.sh

Regards,
Samitha

>
>>-
>>
>> These operations are not required since, the config file replacement
>> won't be effective with the new toml based config model. And the related
>> template files are already been replaced via the optimization script.
>>
>> So I will proceed to remove these redundant operations, within this
>> effort itself.
>>
>> Regards,
>> Samitha
>>
>>
>> On Fri, Jan 31, 2020 at 11:31 AM Samitha Chathuranga 
>> wrote:
>>
>>> And regarding name of the option;
>>>
 *What is the ideal name for the new option ()?*
> Suggestions: --override, --overrideConfig
>

 How about --skipConfigOptimization ?

>>> +1 for this
>>> And if we use this, the default behavior should be overriding the
>>> configs.
>>>
>>>
>>> On Fri, Jan 31, 2020 at 9:49 AM Samitha Chathuranga 
>>> wrote:
>>>
 Hi Pubudu,

 IMHO, I prefer to skip the config override when running the profile
> optimization. What if we make this as the default approach? If someone
> wants to override the configs, then with profile optimization, they can 
> use
> a new flag called --override or something similar as Samitha mentioned.
>
 If we make the default behavior to not-override,  then in non
 Docker/K8s scenarios the profile optimization will be done incompletely. We
 have to think about most generic scenario of profile optimization. Is
 Docker/K8s the most generic scenario of customer use cases? I assume not.

 And in the perspective that "the users expected to run the optimization
 first and then do the configuration", overriding the configs should be the
 default option.

 And thinking about preserving the same old behavior too, as Bhathiya
 suggested, overriding configs should be ideal.

 Regards,
 Samitha

 On Thu, Jan 30, 2020 at 10:28 AM Pubudu Gunatilaka 
 wrote:

> Hi,
>
> If we check one of the existing profile toml [1], the values are very
> common and it cannot be used in real production cases.  Sample values
> contain commonly used hostnames and admin credentials. If we are to use
> this feature properly with configuration override, users have to change
> these default templates. And then they can use profile optimize along with
> configs override.
>
> IMHO, I prefer to skip the config override when running the profile
> optimization. What if we make this as the default approach? If someone
> wants to override the configs, then with profile optimization, they can 
> use
> a new flag called --override or something similar as Samitha mentioned.
>
> In K8s, docker use cases we have a single docker image and we are
> having profile based deployment.toml. We can bring the hostnames to the
> deployment.toml as Nuwan suggested and this would simplify the
> deployment.toml. When running the docker image, we can run the profile
> optimize and it won't affect the deployment.tomls that come in config 
> maps.
> At the moment, in docker images, we had to deal with this configuration
> overridden issue.
>
> [1] -
> https://github.com/wso2/product-apim/blob/master/modules/distribution

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Tharindu Dharmarathna]

Hi Samitha,

Please find my comments inline below.

On Mon, Feb 3, 2020 at 3:29 PM Samitha Chathuranga  wrote:

> Hi,
>
> Additionally to this, I could see that there are still the below
> operations executed in the profile optimization script.
>
> * - Config changes in below config files*
>
>- api-manager.xml - enable_policy_deploy, enable_data_publishing,
>enable_blacklist_condition,
>enable_decision_connection (JMSConnectionDetails)
>- axis2.xml - transport.ws.sender.enable, transport.wss.sender.enable
>- registry.xml - indexing.enable
>
> These profile specific configurations are already handled by the
> server_role implementation in infer.json, so no need to do anything in
> profile optimization.
>
> *- Config file replacements*
>
>- axis2_TM.xml
>- axis2_KM.xml
>- registry_TM.xml
>
> This step will need to change the relevant .j2 file according to the
profile. Therefore we have to keep this step in optimizer as itits.

>
>-
>
> These operations are not required since, the config file replacement won't
> be effective with the new toml based config model. And the related template
> files are already been replaced via the optimization script.
>
> So I will proceed to remove these redundant operations, within this effort
> itself.
>
> Regards,
> Samitha
>
>
> On Fri, Jan 31, 2020 at 11:31 AM Samitha Chathuranga 
> wrote:
>
>> And regarding name of the option;
>>
>>> *What is the ideal name for the new option ()?*
 Suggestions: --override, --overrideConfig

>>>
>>> How about --skipConfigOptimization ?
>>>
>> +1 for this
>> And if we use this, the default behavior should be overriding the
>> configs.
>>
>>
>> On Fri, Jan 31, 2020 at 9:49 AM Samitha Chathuranga 
>> wrote:
>>
>>> Hi Pubudu,
>>>
>>> IMHO, I prefer to skip the config override when running the profile
 optimization. What if we make this as the default approach? If someone
 wants to override the configs, then with profile optimization, they can use
 a new flag called --override or something similar as Samitha mentioned.

>>> If we make the default behavior to not-override,  then in non
>>> Docker/K8s scenarios the profile optimization will be done incompletely. We
>>> have to think about most generic scenario of profile optimization. Is
>>> Docker/K8s the most generic scenario of customer use cases? I assume not.
>>>
>>> And in the perspective that "the users expected to run the optimization
>>> first and then do the configuration", overriding the configs should be the
>>> default option.
>>>
>>> And thinking about preserving the same old behavior too, as Bhathiya
>>> suggested, overriding configs should be ideal.
>>>
>>> Regards,
>>> Samitha
>>>
>>> On Thu, Jan 30, 2020 at 10:28 AM Pubudu Gunatilaka 
>>> wrote:
>>>
 Hi,

 If we check one of the existing profile toml [1], the values are very
 common and it cannot be used in real production cases.  Sample values
 contain commonly used hostnames and admin credentials. If we are to use
 this feature properly with configuration override, users have to change
 these default templates. And then they can use profile optimize along with
 configs override.

 IMHO, I prefer to skip the config override when running the profile
 optimization. What if we make this as the default approach? If someone
 wants to override the configs, then with profile optimization, they can use
 a new flag called --override or something similar as Samitha mentioned.

 In K8s, docker use cases we have a single docker image and we are
 having profile based deployment.toml. We can bring the hostnames to the
 deployment.toml as Nuwan suggested and this would simplify the
 deployment.toml. When running the docker image, we can run the profile
 optimize and it won't affect the deployment.tomls that come in config maps.
 At the moment, in docker images, we had to deal with this configuration
 overridden issue.

 [1] -
 https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/resources/conf/deployment-templates/api-publisher.toml

 Thank you!

 On Wed, Jan 29, 2020 at 2:10 PM Harsha Kumara  wrote:

> I'm thinking on real world scenario where customer will keep one base
> product archive and add the configs. isn't it easy for them to copy those
> files just to the profile optimizer tool location?
>
> On Wed, Jan 29, 2020 at 3:15 PM Bhathiya Jayasekara 
> wrote:
>
>> Hi Samitha,
>>
>> On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
>> wrote:
>>
>>> Hi all,
>>>
>>> Our profile optimization [1] script for 3.0.0 replaces [2] already
>>> available deployment.toml with a per-profile specific updated/optimized
>>> toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
>>> process affects already configured setups, to lose the configurations.
>

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Samitha Chathuranga]

Hi,

Additionally to this, I could see that there are still the below operations
executed in the profile optimization script.

* - Config changes in below config files*

   - api-manager.xml - enable_policy_deploy, enable_data_publishing,
   enable_blacklist_condition,
   enable_decision_connection (JMSConnectionDetails)
   - axis2.xml - transport.ws.sender.enable, transport.wss.sender.enable
   - registry.xml - indexing.enable

These profile specific configurations are already handled by the
server_role implementation in infer.json, so no need to do anything in
profile optimization.

*- Config file replacements*

   - axis2_TM.xml
   - axis2_KM.xml
   - registry_TM.xml

These operations are not required since, the config file replacement won't
be effective with the new toml based config model. And the related template
files are already been replaced via the optimization script.

So I will proceed to remove these redundant operations, within this effort
itself.

Regards,
Samitha


On Fri, Jan 31, 2020 at 11:31 AM Samitha Chathuranga 
wrote:

> And regarding name of the option;
>
>> *What is the ideal name for the new option ()?*
>>> Suggestions: --override, --overrideConfig
>>>
>>
>> How about --skipConfigOptimization ?
>>
> +1 for this
> And if we use this, the default behavior should be overriding the configs.
>
>
> On Fri, Jan 31, 2020 at 9:49 AM Samitha Chathuranga 
> wrote:
>
>> Hi Pubudu,
>>
>> IMHO, I prefer to skip the config override when running the profile
>>> optimization. What if we make this as the default approach? If someone
>>> wants to override the configs, then with profile optimization, they can use
>>> a new flag called --override or something similar as Samitha mentioned.
>>>
>> If we make the default behavior to not-override,  then in non Docker/K8s
>> scenarios the profile optimization will be done incompletely. We have to
>> think about most generic scenario of profile optimization. Is Docker/K8s
>> the most generic scenario of customer use cases? I assume not.
>>
>> And in the perspective that "the users expected to run the optimization
>> first and then do the configuration", overriding the configs should be the
>> default option.
>>
>> And thinking about preserving the same old behavior too, as Bhathiya
>> suggested, overriding configs should be ideal.
>>
>> Regards,
>> Samitha
>>
>> On Thu, Jan 30, 2020 at 10:28 AM Pubudu Gunatilaka 
>> wrote:
>>
>>> Hi,
>>>
>>> If we check one of the existing profile toml [1], the values are very
>>> common and it cannot be used in real production cases.  Sample values
>>> contain commonly used hostnames and admin credentials. If we are to use
>>> this feature properly with configuration override, users have to change
>>> these default templates. And then they can use profile optimize along with
>>> configs override.
>>>
>>> IMHO, I prefer to skip the config override when running the profile
>>> optimization. What if we make this as the default approach? If someone
>>> wants to override the configs, then with profile optimization, they can use
>>> a new flag called --override or something similar as Samitha mentioned.
>>>
>>> In K8s, docker use cases we have a single docker image and we are having
>>> profile based deployment.toml. We can bring the hostnames to the
>>> deployment.toml as Nuwan suggested and this would simplify the
>>> deployment.toml. When running the docker image, we can run the profile
>>> optimize and it won't affect the deployment.tomls that come in config maps.
>>> At the moment, in docker images, we had to deal with this configuration
>>> overridden issue.
>>>
>>> [1] -
>>> https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/resources/conf/deployment-templates/api-publisher.toml
>>>
>>> Thank you!
>>>
>>> On Wed, Jan 29, 2020 at 2:10 PM Harsha Kumara  wrote:
>>>
 I'm thinking on real world scenario where customer will keep one base
 product archive and add the configs. isn't it easy for them to copy those
 files just to the profile optimizer tool location?

 On Wed, Jan 29, 2020 at 3:15 PM Bhathiya Jayasekara 
 wrote:

> Hi Samitha,
>
> On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
> wrote:
>
>> Hi all,
>>
>> Our profile optimization [1] script for 3.0.0 replaces [2] already
>> available deployment.toml with a per-profile specific updated/optimized
>> toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
>> process affects already configured setups, to lose the configurations.
>>
>> A user running the profile optimization might follow two approaches.
>>
>> 1. Run profile optimization on a vanilla pack.
>>  - So no any configurations done when the profile optimization is
>> executed
>> 2. Run profile optimization on a configured pack.
>>  - User/deployment specific configurations are already done in
>> the deployment.toml file and user run the pro

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Samitha Chathuranga]

And regarding name of the option;

> *What is the ideal name for the new option ()?*
>> Suggestions: --override, --overrideConfig
>>
>
> How about --skipConfigOptimization ?
>
+1 for this
And if we use this, the default behavior should be overriding the configs.


On Fri, Jan 31, 2020 at 9:49 AM Samitha Chathuranga 
wrote:

> Hi Pubudu,
>
> IMHO, I prefer to skip the config override when running the profile
>> optimization. What if we make this as the default approach? If someone
>> wants to override the configs, then with profile optimization, they can use
>> a new flag called --override or something similar as Samitha mentioned.
>>
> If we make the default behavior to not-override,  then in non Docker/K8s
> scenarios the profile optimization will be done incompletely. We have to
> think about most generic scenario of profile optimization. Is Docker/K8s
> the most generic scenario of customer use cases? I assume not.
>
> And in the perspective that "the users expected to run the optimization
> first and then do the configuration", overriding the configs should be the
> default option.
>
> And thinking about preserving the same old behavior too, as Bhathiya
> suggested, overriding configs should be ideal.
>
> Regards,
> Samitha
>
> On Thu, Jan 30, 2020 at 10:28 AM Pubudu Gunatilaka 
> wrote:
>
>> Hi,
>>
>> If we check one of the existing profile toml [1], the values are very
>> common and it cannot be used in real production cases.  Sample values
>> contain commonly used hostnames and admin credentials. If we are to use
>> this feature properly with configuration override, users have to change
>> these default templates. And then they can use profile optimize along with
>> configs override.
>>
>> IMHO, I prefer to skip the config override when running the profile
>> optimization. What if we make this as the default approach? If someone
>> wants to override the configs, then with profile optimization, they can use
>> a new flag called --override or something similar as Samitha mentioned.
>>
>> In K8s, docker use cases we have a single docker image and we are having
>> profile based deployment.toml. We can bring the hostnames to the
>> deployment.toml as Nuwan suggested and this would simplify the
>> deployment.toml. When running the docker image, we can run the profile
>> optimize and it won't affect the deployment.tomls that come in config maps.
>> At the moment, in docker images, we had to deal with this configuration
>> overridden issue.
>>
>> [1] -
>> https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/resources/conf/deployment-templates/api-publisher.toml
>>
>> Thank you!
>>
>> On Wed, Jan 29, 2020 at 2:10 PM Harsha Kumara  wrote:
>>
>>> I'm thinking on real world scenario where customer will keep one base
>>> product archive and add the configs. isn't it easy for them to copy those
>>> files just to the profile optimizer tool location?
>>>
>>> On Wed, Jan 29, 2020 at 3:15 PM Bhathiya Jayasekara 
>>> wrote:
>>>
 Hi Samitha,

 On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
 wrote:

> Hi all,
>
> Our profile optimization [1] script for 3.0.0 replaces [2] already
> available deployment.toml with a per-profile specific updated/optimized
> toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
> process affects already configured setups, to lose the configurations.
>
> A user running the profile optimization might follow two approaches.
>
> 1. Run profile optimization on a vanilla pack.
>  - So no any configurations done when the profile optimization is
> executed
> 2. Run profile optimization on a configured pack.
>  - User/deployment specific configurations are already done in the
> deployment.toml file and user run the profile optimization after that.
>

 The user is expected to run the optimization first and then do the
 configuration. The real problem here comes with Kubernetes, where we pass
 configs as config maps. These shouldn't be replaced by the optimizer,
 because we expect the optimized configs to come via config maps.


>
> There is no any issue with the Approach-1. But the Approach-2 results
> in the above mentioned issue. A user might have made all his 
> configurations
> in the deployment.toml file. But once the profile optimization is 
> executed,
> he will lose all those config changes.
>
> So the suggested solution is to pass an argument/option when we run
> the optimizing script, which decides whether to replace the 
> deployment.toml
> or not.
>
> sh /bin/wso2server.sh --optimize *-- 
> *-Dprofile=api-publisher
>
> In this case, we have to assume(and mandate) that user has manually
> applied the profile optimization related configurations too, into the
> deployment.toml file while doing the other configurations. So if the "new
>>>

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Samitha Chathuranga]

Hi Pubudu,

IMHO, I prefer to skip the config override when running the profile
> optimization. What if we make this as the default approach? If someone
> wants to override the configs, then with profile optimization, they can use
> a new flag called --override or something similar as Samitha mentioned.
>
If we make the default behavior to not-override,  then in non Docker/K8s
scenarios the profile optimization will be done incompletely. We have to
think about most generic scenario of profile optimization. Is Docker/K8s
the most generic scenario of customer use cases? I assume not.

And in the perspective that "the users expected to run the optimization
first and then do the configuration", overriding the configs should be the
default option.

And thinking about preserving the same old behavior too, as Bhathiya
suggested, overriding configs should be ideal.

Regards,
Samitha

On Thu, Jan 30, 2020 at 10:28 AM Pubudu Gunatilaka  wrote:

> Hi,
>
> If we check one of the existing profile toml [1], the values are very
> common and it cannot be used in real production cases.  Sample values
> contain commonly used hostnames and admin credentials. If we are to use
> this feature properly with configuration override, users have to change
> these default templates. And then they can use profile optimize along with
> configs override.
>
> IMHO, I prefer to skip the config override when running the profile
> optimization. What if we make this as the default approach? If someone
> wants to override the configs, then with profile optimization, they can use
> a new flag called --override or something similar as Samitha mentioned.
>
> In K8s, docker use cases we have a single docker image and we are having
> profile based deployment.toml. We can bring the hostnames to the
> deployment.toml as Nuwan suggested and this would simplify the
> deployment.toml. When running the docker image, we can run the profile
> optimize and it won't affect the deployment.tomls that come in config maps.
> At the moment, in docker images, we had to deal with this configuration
> overridden issue.
>
> [1] -
> https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/resources/conf/deployment-templates/api-publisher.toml
>
> Thank you!
>
> On Wed, Jan 29, 2020 at 2:10 PM Harsha Kumara  wrote:
>
>> I'm thinking on real world scenario where customer will keep one base
>> product archive and add the configs. isn't it easy for them to copy those
>> files just to the profile optimizer tool location?
>>
>> On Wed, Jan 29, 2020 at 3:15 PM Bhathiya Jayasekara 
>> wrote:
>>
>>> Hi Samitha,
>>>
>>> On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
>>> wrote:
>>>
 Hi all,

 Our profile optimization [1] script for 3.0.0 replaces [2] already
 available deployment.toml with a per-profile specific updated/optimized
 toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
 process affects already configured setups, to lose the configurations.

 A user running the profile optimization might follow two approaches.

 1. Run profile optimization on a vanilla pack.
  - So no any configurations done when the profile optimization is
 executed
 2. Run profile optimization on a configured pack.
  - User/deployment specific configurations are already done in the
 deployment.toml file and user run the profile optimization after that.

>>>
>>> The user is expected to run the optimization first and then do the
>>> configuration. The real problem here comes with Kubernetes, where we pass
>>> configs as config maps. These shouldn't be replaced by the optimizer,
>>> because we expect the optimized configs to come via config maps.
>>>
>>>

 There is no any issue with the Approach-1. But the Approach-2 results
 in the above mentioned issue. A user might have made all his configurations
 in the deployment.toml file. But once the profile optimization is executed,
 he will lose all those config changes.

 So the suggested solution is to pass an argument/option when we run the
 optimizing script, which decides whether to replace the deployment.toml or
 not.

 sh /bin/wso2server.sh --optimize *-- 
 *-Dprofile=api-publisher

 In this case, we have to assume(and mandate) that user has manually
 applied the profile optimization related configurations too, into the
 deployment.toml file while doing the other configurations. So if the "new
 option" decides not to override the deployment.toml file, the profile
 optimization steps other than deployment.toml changes will be applied.

 So we have two concerns on how we use this new option.

 *What is the ideal name for the new option ()?*
 Suggestions: --override, --overrideConfig

>>>
>>> How about --skipConfigOptimization ?
>>>
>>>

 *What is the default behavior of the new option?*
 - What is the default beha

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Pubudu Gunatilaka]

Hi,

If we check one of the existing profile toml [1], the values are very
common and it cannot be used in real production cases.  Sample values
contain commonly used hostnames and admin credentials. If we are to use
this feature properly with configuration override, users have to change
these default templates. And then they can use profile optimize along with
configs override.

IMHO, I prefer to skip the config override when running the profile
optimization. What if we make this as the default approach? If someone
wants to override the configs, then with profile optimization, they can use
a new flag called --override or something similar as Samitha mentioned.

In K8s, docker use cases we have a single docker image and we are having
profile based deployment.toml. We can bring the hostnames to the
deployment.toml as Nuwan suggested and this would simplify the
deployment.toml. When running the docker image, we can run the profile
optimize and it won't affect the deployment.tomls that come in config maps.
At the moment, in docker images, we had to deal with this configuration
overridden issue.

[1] -
https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/resources/conf/deployment-templates/api-publisher.toml

Thank you!

On Wed, Jan 29, 2020 at 2:10 PM Harsha Kumara  wrote:

> I'm thinking on real world scenario where customer will keep one base
> product archive and add the configs. isn't it easy for them to copy those
> files just to the profile optimizer tool location?
>
> On Wed, Jan 29, 2020 at 3:15 PM Bhathiya Jayasekara 
> wrote:
>
>> Hi Samitha,
>>
>> On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
>> wrote:
>>
>>> Hi all,
>>>
>>> Our profile optimization [1] script for 3.0.0 replaces [2] already
>>> available deployment.toml with a per-profile specific updated/optimized
>>> toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
>>> process affects already configured setups, to lose the configurations.
>>>
>>> A user running the profile optimization might follow two approaches.
>>>
>>> 1. Run profile optimization on a vanilla pack.
>>>  - So no any configurations done when the profile optimization is
>>> executed
>>> 2. Run profile optimization on a configured pack.
>>>  - User/deployment specific configurations are already done in the
>>> deployment.toml file and user run the profile optimization after that.
>>>
>>
>> The user is expected to run the optimization first and then do the
>> configuration. The real problem here comes with Kubernetes, where we pass
>> configs as config maps. These shouldn't be replaced by the optimizer,
>> because we expect the optimized configs to come via config maps.
>>
>>
>>>
>>> There is no any issue with the Approach-1. But the Approach-2 results in
>>> the above mentioned issue. A user might have made all his configurations in
>>> the deployment.toml file. But once the profile optimization is executed, he
>>> will lose all those config changes.
>>>
>>> So the suggested solution is to pass an argument/option when we run the
>>> optimizing script, which decides whether to replace the deployment.toml or
>>> not.
>>>
>>> sh /bin/wso2server.sh --optimize *-- 
>>> *-Dprofile=api-publisher
>>>
>>> In this case, we have to assume(and mandate) that user has manually
>>> applied the profile optimization related configurations too, into the
>>> deployment.toml file while doing the other configurations. So if the "new
>>> option" decides not to override the deployment.toml file, the profile
>>> optimization steps other than deployment.toml changes will be applied.
>>>
>>> So we have two concerns on how we use this new option.
>>>
>>> *What is the ideal name for the new option ()?*
>>> Suggestions: --override, --overrideConfig
>>>
>>
>> How about --skipConfigOptimization ?
>>
>>
>>>
>>> *What is the default behavior of the new option?*
>>> - What is the default behavior if the new option is not passed when
>>> running the optimization.?
>>>
>>
>> It should be the current behavior.
>>
>> Thanks,
>> Bhathiya
>>
>>
>>> - Should it override the user-made configs with the profile-specific
>>> deployment toml or not?
>>> - Concerns:
>>>   - If override the toml, users manual configs will be completely
>>> removed (may be unknowlingly too).
>>>   - If do not override toml, the profile optimization will be done
>>> incompletely. Anyway we can echo a message conveying that the deployment
>>> toml was not overridiiden.
>>>
>>> Appreciate your input on this.
>>>
>>> [1]
>>> https://apim.docs.wso2.com/en/next/SetupAndInstall/DeployingWSO2APIManager/DistributedDeployment/product-profiles/
>>> [2]
>>> https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/startup-scripts/profileSetup.sh#L302
>>>
>>> Thanks,
>>> Samitha
>>>
>>> --
>>> *Samitha Chathuranga*
>>> *Associate Technical Lead*, *WSO2 Inc.*
>>> lean.enterprise.middleware
>>> Mobile: +94715123761
>>>
>>> [image: http://wso2.com/

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Harsha Kumara]

I'm thinking on real world scenario where customer will keep one base
product archive and add the configs. isn't it easy for them to copy those
files just to the profile optimizer tool location?

On Wed, Jan 29, 2020 at 3:15 PM Bhathiya Jayasekara 
wrote:

> Hi Samitha,
>
> On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
> wrote:
>
>> Hi all,
>>
>> Our profile optimization [1] script for 3.0.0 replaces [2] already
>> available deployment.toml with a per-profile specific updated/optimized
>> toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
>> process affects already configured setups, to lose the configurations.
>>
>> A user running the profile optimization might follow two approaches.
>>
>> 1. Run profile optimization on a vanilla pack.
>>  - So no any configurations done when the profile optimization is
>> executed
>> 2. Run profile optimization on a configured pack.
>>  - User/deployment specific configurations are already done in the
>> deployment.toml file and user run the profile optimization after that.
>>
>
> The user is expected to run the optimization first and then do the
> configuration. The real problem here comes with Kubernetes, where we pass
> configs as config maps. These shouldn't be replaced by the optimizer,
> because we expect the optimized configs to come via config maps.
>
>
>>
>> There is no any issue with the Approach-1. But the Approach-2 results in
>> the above mentioned issue. A user might have made all his configurations in
>> the deployment.toml file. But once the profile optimization is executed, he
>> will lose all those config changes.
>>
>> So the suggested solution is to pass an argument/option when we run the
>> optimizing script, which decides whether to replace the deployment.toml or
>> not.
>>
>> sh /bin/wso2server.sh --optimize *-- 
>> *-Dprofile=api-publisher
>>
>> In this case, we have to assume(and mandate) that user has manually
>> applied the profile optimization related configurations too, into the
>> deployment.toml file while doing the other configurations. So if the "new
>> option" decides not to override the deployment.toml file, the profile
>> optimization steps other than deployment.toml changes will be applied.
>>
>> So we have two concerns on how we use this new option.
>>
>> *What is the ideal name for the new option ()?*
>> Suggestions: --override, --overrideConfig
>>
>
> How about --skipConfigOptimization ?
>
>
>>
>> *What is the default behavior of the new option?*
>> - What is the default behavior if the new option is not passed when
>> running the optimization.?
>>
>
> It should be the current behavior.
>
> Thanks,
> Bhathiya
>
>
>> - Should it override the user-made configs with the profile-specific
>> deployment toml or not?
>> - Concerns:
>>   - If override the toml, users manual configs will be completely
>> removed (may be unknowlingly too).
>>   - If do not override toml, the profile optimization will be done
>> incompletely. Anyway we can echo a message conveying that the deployment
>> toml was not overridiiden.
>>
>> Appreciate your input on this.
>>
>> [1]
>> https://apim.docs.wso2.com/en/next/SetupAndInstall/DeployingWSO2APIManager/DistributedDeployment/product-profiles/
>> [2]
>> https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/startup-scripts/profileSetup.sh#L302
>>
>> Thanks,
>> Samitha
>>
>> --
>> *Samitha Chathuranga*
>> *Associate Technical Lead*, *WSO2 Inc.*
>> lean.enterprise.middleware
>> Mobile: +94715123761
>>
>> [image: http://wso2.com/signature] 
>>
>
>
> --
> *Bhathiya Jayasekara* | Technical Lead | WSO2 Inc.
> (m) +94 71 547 8185  | (e) bhathiya-@t-wso2-d0t-com
>
>
>

-- 

*Harsha Kumara*

Technical Lead, WSO2 Inc.
Mobile: +94775505618
Email: hars...@wso2.coim
Blog: harshcreationz.blogspot.com

GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Bhathiya Jayasekara]

Hi Samitha,

On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
wrote:

> Hi all,
>
> Our profile optimization [1] script for 3.0.0 replaces [2] already
> available deployment.toml with a per-profile specific updated/optimized
> toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
> process affects already configured setups, to lose the configurations.
>
> A user running the profile optimization might follow two approaches.
>
> 1. Run profile optimization on a vanilla pack.
>  - So no any configurations done when the profile optimization is
> executed
> 2. Run profile optimization on a configured pack.
>  - User/deployment specific configurations are already done in the
> deployment.toml file and user run the profile optimization after that.
>

The user is expected to run the optimization first and then do the
configuration. The real problem here comes with Kubernetes, where we pass
configs as config maps. These shouldn't be replaced by the optimizer,
because we expect the optimized configs to come via config maps.


>
> There is no any issue with the Approach-1. But the Approach-2 results in
> the above mentioned issue. A user might have made all his configurations in
> the deployment.toml file. But once the profile optimization is executed, he
> will lose all those config changes.
>
> So the suggested solution is to pass an argument/option when we run the
> optimizing script, which decides whether to replace the deployment.toml or
> not.
>
> sh /bin/wso2server.sh --optimize *-- 
> *-Dprofile=api-publisher
>
> In this case, we have to assume(and mandate) that user has manually
> applied the profile optimization related configurations too, into the
> deployment.toml file while doing the other configurations. So if the "new
> option" decides not to override the deployment.toml file, the profile
> optimization steps other than deployment.toml changes will be applied.
>
> So we have two concerns on how we use this new option.
>
> *What is the ideal name for the new option ()?*
> Suggestions: --override, --overrideConfig
>

How about --skipConfigOptimization ?


>
> *What is the default behavior of the new option?*
> - What is the default behavior if the new option is not passed when
> running the optimization.?
>

It should be the current behavior.

Thanks,
Bhathiya


> - Should it override the user-made configs with the profile-specific
> deployment toml or not?
> - Concerns:
>   - If override the toml, users manual configs will be completely
> removed (may be unknowlingly too).
>   - If do not override toml, the profile optimization will be done
> incompletely. Anyway we can echo a message conveying that the deployment
> toml was not overridiiden.
>
> Appreciate your input on this.
>
> [1]
> https://apim.docs.wso2.com/en/next/SetupAndInstall/DeployingWSO2APIManager/DistributedDeployment/product-profiles/
> [2]
> https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/startup-scripts/profileSetup.sh#L302
>
> Thanks,
> Samitha
>
> --
> *Samitha Chathuranga*
> *Associate Technical Lead*, *WSO2 Inc.*
> lean.enterprise.middleware
> Mobile: +94715123761
>
> [image: http://wso2.com/signature] 
>


-- 
*Bhathiya Jayasekara* | Technical Lead | WSO2 Inc.
(m) +94 71 547 8185  | (e) bhathiya-@t-wso2-d0t-com
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Samitha Chathuranga]

Hi Harsha,

On Wed, Jan 29, 2020 at 2:39 PM Harsha Kumara  wrote:

> Aren't there any place where we can place the profile specific
> deployment.toml which tool eventually replace it rather than giving a
> separate option?
>

*/wso2am-3.0.0/repository/resources/conf/deployment-templates/*  is the
location where the profile specific deployment.toml files are located.
The issue is that some users don't plan about profile optimization
initially, so do the configurations on the normal deployment.toml file.
In this case, users will have to merge our profile-specific deployment toml
file with their updated general deployment.toml file and place it into the
above mentioned path manually, before running the profile optimization.
Anyway the merge operation would be difficult and so it is then better to
ask user to forget about his configurations and redo them after doing the
profile optimization. Then this addition la new-option is not required.

But the objective of this effort is to allow users who has already done the
profile optimization steps in deployment.toml file, to run the optimization
script to execute the other optimization steps.
@Tharindu Dharmarathna  , @Pubudu Gunatilaka
 , @Bhathiya Jayasekara  I think this
is the original expectation according to the initial suggestion on this
matter.

Regards,
Samitha


> On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
> wrote:
>
>> Hi all,
>>
>> Our profile optimization [1] script for 3.0.0 replaces [2] already
>> available deployment.toml with a per-profile specific updated/optimized
>> toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
>> process affects already configured setups, to lose the configurations.
>>
>> A user running the profile optimization might follow two approaches.
>>
>> 1. Run profile optimization on a vanilla pack.
>>  - So no any configurations done when the profile optimization is
>> executed
>> 2. Run profile optimization on a configured pack.
>>  - User/deployment specific configurations are already done in the
>> deployment.toml file and user run the profile optimization after that.
>>
>> There is no any issue with the Approach-1. But the Approach-2 results in
>> the above mentioned issue. A user might have made all his configurations in
>> the deployment.toml file. But once the profile optimization is executed, he
>> will lose all those config changes.
>>
>> So the suggested solution is to pass an argument/option when we run the
>> optimizing script, which decides whether to replace the deployment.toml or
>> not.
>>
>> sh /bin/wso2server.sh --optimize *-- 
>> *-Dprofile=api-publisher
>>
>> In this case, we have to assume(and mandate) that user has manually
>> applied the profile optimization related configurations too, into the
>> deployment.toml file while doing the other configurations. So if the "new
>> option" decides not to override the deployment.toml file, the profile
>> optimization steps other than deployment.toml changes will be applied.
>>
>> So we have two concerns on how we use this new option.
>>
>> *What is the ideal name for the new option ()?*
>> Suggestions: --override, --overrideConfig
>>
>> *What is the default behavior of the new option?*
>> - What is the default behavior if the new option is not passed when
>> running the optimization.?
>> - Should it override the user-made configs with the profile-specific
>> deployment toml or not?
>> - Concerns:
>>   - If override the toml, users manual configs will be completely
>> removed (may be unknowlingly too).
>>   - If do not override toml, the profile optimization will be done
>> incompletely. Anyway we can echo a message conveying that the deployment
>> toml was not overridiiden.
>>
>> Appreciate your input on this.
>>
>> [1]
>> https://apim.docs.wso2.com/en/next/SetupAndInstall/DeployingWSO2APIManager/DistributedDeployment/product-profiles/
>> [2]
>> https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/startup-scripts/profileSetup.sh#L302
>>
>> Thanks,
>> Samitha
>>
>> --
>> *Samitha Chathuranga*
>> *Associate Technical Lead*, *WSO2 Inc.*
>> lean.enterprise.middleware
>> Mobile: +94715123761
>>
>> [image: http://wso2.com/signature] 
>>
>
>
> --
>
> *Harsha Kumara*
>
> Technical Lead, WSO2 Inc.
> Mobile: +94775505618
> Email: hars...@wso2.coim
> Blog: harshcreationz.blogspot.com
>
> GET INTEGRATION AGILE
> Integration Agility for Digitally Driven Business
>


-- 
*Samitha Chathuranga*
*Associate Technical Lead*, *WSO2 Inc.*
lean.enterprise.middleware
Mobile: +94715123761

[image: http://wso2.com/signature] 
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] APIM Profile optimization script - deployment.toml replacement

[Harsha Kumara]

Aren't there any place where we can place the profile specific
deployment.toml which tool eventually replace it rather than giving a
separate option?

On Wed, Jan 29, 2020 at 2:29 PM Samitha Chathuranga 
wrote:

> Hi all,
>
> Our profile optimization [1] script for 3.0.0 replaces [2] already
> available deployment.toml with a per-profile specific updated/optimized
> toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
> process affects already configured setups, to lose the configurations.
>
> A user running the profile optimization might follow two approaches.
>
> 1. Run profile optimization on a vanilla pack.
>  - So no any configurations done when the profile optimization is
> executed
> 2. Run profile optimization on a configured pack.
>  - User/deployment specific configurations are already done in the
> deployment.toml file and user run the profile optimization after that.
>
> There is no any issue with the Approach-1. But the Approach-2 results in
> the above mentioned issue. A user might have made all his configurations in
> the deployment.toml file. But once the profile optimization is executed, he
> will lose all those config changes.
>
> So the suggested solution is to pass an argument/option when we run the
> optimizing script, which decides whether to replace the deployment.toml or
> not.
>
> sh /bin/wso2server.sh --optimize *-- 
> *-Dprofile=api-publisher
>
> In this case, we have to assume(and mandate) that user has manually
> applied the profile optimization related configurations too, into the
> deployment.toml file while doing the other configurations. So if the "new
> option" decides not to override the deployment.toml file, the profile
> optimization steps other than deployment.toml changes will be applied.
>
> So we have two concerns on how we use this new option.
>
> *What is the ideal name for the new option ()?*
> Suggestions: --override, --overrideConfig
>
> *What is the default behavior of the new option?*
> - What is the default behavior if the new option is not passed when
> running the optimization.?
> - Should it override the user-made configs with the profile-specific
> deployment toml or not?
> - Concerns:
>   - If override the toml, users manual configs will be completely
> removed (may be unknowlingly too).
>   - If do not override toml, the profile optimization will be done
> incompletely. Anyway we can echo a message conveying that the deployment
> toml was not overridiiden.
>
> Appreciate your input on this.
>
> [1]
> https://apim.docs.wso2.com/en/next/SetupAndInstall/DeployingWSO2APIManager/DistributedDeployment/product-profiles/
> [2]
> https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/startup-scripts/profileSetup.sh#L302
>
> Thanks,
> Samitha
>
> --
> *Samitha Chathuranga*
> *Associate Technical Lead*, *WSO2 Inc.*
> lean.enterprise.middleware
> Mobile: +94715123761
>
> [image: http://wso2.com/signature] 
>


-- 

*Harsha Kumara*

Technical Lead, WSO2 Inc.
Mobile: +94775505618
Email: hars...@wso2.coim
Blog: harshcreationz.blogspot.com

GET INTEGRATION AGILE
Integration Agility for Digitally Driven Business
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] APIM Profile optimization script - deployment.toml replacement

[Samitha Chathuranga]

Hi all,

Our profile optimization [1] script for 3.0.0 replaces [2] already
available deployment.toml with a per-profile specific updated/optimized
toml file. (i.e. api-devportal.toml, gateway-worker.toml, etc.). This
process affects already configured setups, to lose the configurations.

A user running the profile optimization might follow two approaches.

1. Run profile optimization on a vanilla pack.
 - So no any configurations done when the profile optimization is
executed
2. Run profile optimization on a configured pack.
 - User/deployment specific configurations are already done in the
deployment.toml file and user run the profile optimization after that.

There is no any issue with the Approach-1. But the Approach-2 results in
the above mentioned issue. A user might have made all his configurations in
the deployment.toml file. But once the profile optimization is executed, he
will lose all those config changes.

So the suggested solution is to pass an argument/option when we run the
optimizing script, which decides whether to replace the deployment.toml or
not.

sh /bin/wso2server.sh --optimize *--
*-Dprofile=api-publisher

In this case, we have to assume(and mandate) that user has manually applied
the profile optimization related configurations too, into the
deployment.toml file while doing the other configurations. So if the "new
option" decides not to override the deployment.toml file, the profile
optimization steps other than deployment.toml changes will be applied.

So we have two concerns on how we use this new option.

*What is the ideal name for the new option ()?*
Suggestions: --override, --overrideConfig

*What is the default behavior of the new option?*
- What is the default behavior if the new option is not passed when running
the optimization.?
- Should it override the user-made configs with the profile-specific
deployment toml or not?
- Concerns:
  - If override the toml, users manual configs will be completely
removed (may be unknowlingly too).
  - If do not override toml, the profile optimization will be done
incompletely. Anyway we can echo a message conveying that the deployment
toml was not overridiiden.

Appreciate your input on this.

[1]
https://apim.docs.wso2.com/en/next/SetupAndInstall/DeployingWSO2APIManager/DistributedDeployment/product-profiles/
[2]
https://github.com/wso2/product-apim/blob/master/modules/distribution/product/src/main/startup-scripts/profileSetup.sh#L302

Thanks,
Samitha

-- 
*Samitha Chathuranga*
*Associate Technical Lead*, *WSO2 Inc.*
lean.enterprise.middleware
Mobile: +94715123761

[image: http://wso2.com/signature] 
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev