Kind reminder on the meeting at 2.00PM at 7th Floor Banksy room.
Regards, Chamila de Alwis Committer and PMC Member - Apache Stratos Senior Software Engineer | WSO2 Blog: https://medium.com/@chamilad On Tue, Aug 29, 2017 at 9:58 AM, Chamila De Alwis <[email protected]> wrote: > Please note I've pushed the meeting again to Thursday based on > availability. Sorry for the constant notifications. > > > Regards, > Chamila de Alwis > Committer and PMC Member - Apache Stratos > Senior Software Engineer | WSO2 > Blog: https://medium.com/@chamilad > > > > On Mon, Aug 28, 2017 at 10:39 AM, Chamila De Alwis <[email protected]> > wrote: > >> I've rescheduled the meeting to 30th August (Wednesday) 1pm-2pm, as >> Lakmal is unavailable today. >> >> >> Regards, >> Chamila de Alwis >> Committer and PMC Member - Apache Stratos >> Senior Software Engineer | WSO2 >> Blog: https://medium.com/@chamilad >> >> >> >> On Wed, Aug 23, 2017 at 5:31 PM, Chamila De Alwis <[email protected]> >> wrote: >> >>> I agree. We could omit Secure Vaulted or possible fields like "password" >>> values from being logged. >>> >>> I've scheduled a meeting on 28th Monday 2.00PM. Let's consolidate these >>> into a plan then. >>> >>> >>> Regards, >>> Chamila de Alwis >>> Committer and PMC Member - Apache Stratos >>> Senior Software Engineer | WSO2 >>> Blog: https://medium.com/@chamilad >>> >>> >>> >>> On Wed, Aug 23, 2017 at 5:26 PM, Nuwan Dias <[email protected]> wrote: >>> >>>> Logging might be tricky. Some of these configs we're talking about are >>>> related to credentials. We shouldn't log credentials in any place. So we >>>> probably will have to find a way to omit logging certain variables if we go >>>> down the path of logging. >>>> >>>> On Wed, Aug 23, 2017 at 5:21 PM, Chamila De Alwis <[email protected]> >>>> wrote: >>>> >>>>> I had an offline chat with IsuruH and Jayanga, during which following >>>>> points came into light. >>>>> >>>>> 1. To address the troubleshooting story, a separate log can be written >>>>> at the startup where the ConfigProvider executes that would contain the >>>>> configurations overridden for the particular product. The security of this >>>>> log should be handled in the same way for other logs. For debugging, this >>>>> log file can be requested from a deployment. >>>>> 2. There is a possibility that product start-up time might degrade if >>>>> this logging is introduced. >>>>> 3. Not all configurations need to be set via environment variables. >>>>> There can be an attribute in the config annotation which denotes the >>>>> environment variable that can be used to override the particular >>>>> configuration. This can also help in filling the documentation gap that is >>>>> introduced, i.e. the list of environment variables that can be used in a >>>>> particular product can be autogenerated when configuration doc generation >>>>> story is completed. >>>>> 4. Complex types do not need to be supported in environment variable >>>>> story. This would vary depending on the likelihood that configuration >>>>> would >>>>> be overridden through an environment variable. >>>>> >>>>> Let's schedule a meeting to discuss the concerns, including the above, >>>>> and come to a conclusion. >>>>> >>>>> >>>>> Regards, >>>>> Chamila de Alwis >>>>> Committer and PMC Member - Apache Stratos >>>>> Senior Software Engineer | WSO2 >>>>> Blog: https://medium.com/@chamilad >>>>> >>>>> >>>>> >>>>> On Wed, Aug 23, 2017 at 3:01 PM, Chamila De Alwis <[email protected]> >>>>> wrote: >>>>> >>>>>> IMO logging values would be a security risk. It would be more >>>>>> manageable if information on where certain values are retrieved from can >>>>>> be >>>>>> displayed if that log level is enabled. >>>>>> >>>>>> >>>>>> Regards, >>>>>> Chamila de Alwis >>>>>> Committer and PMC Member - Apache Stratos >>>>>> Senior Software Engineer | WSO2 >>>>>> Blog: https://medium.com/@chamilad >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Aug 23, 2017 at 3:00 PM, Isuru Haththotuwa <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 23, 2017 at 2:46 PM, Lakmal Warusawithana < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 23, 2017 at 2:39 PM, Isuru Haththotuwa <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> While I fully agree that its necessary to be able to inject >>>>>>>>> configs via environment variables to be container friendly, I think >>>>>>>>> Jayanga >>>>>>>>> do have a point. When we are troubleshooting an issue, the first >>>>>>>>> thing to >>>>>>>>> check is the configurations. When up configuration values are looked >>>>>>>>> up via >>>>>>>>> environment variables, how do we know what are the configuration >>>>>>>>> parameters >>>>>>>>> that are used in the system? One possible option is to write to a log >>>>>>>>> as >>>>>>>>> Chamila has mentioned, or else write the currently used config params >>>>>>>>> to a >>>>>>>>> log file at startup. WDYT? However this does not take into account >>>>>>>>> the >>>>>>>>> possibility of changing an environment value even after the server has >>>>>>>>> started. >>>>>>>>> >>>>>>>> >>>>>>>> @Isuru, it is coming with correct namespace right? and first >>>>>>>> priority is given to environment variables. So there should not be an >>>>>>>> issue >>>>>>>> with identifying the our config from environment variables IMO. >>>>>>>> >>>>>>> Yes. What I want to highlight is the troubleshooting perspective. We >>>>>>> can have a particular set of config values coming from environment >>>>>>> variables, another set of config values from deployment.yaml and >>>>>>> defaults >>>>>>> from the code. If we have log file file with all the currently used >>>>>>> configurations values (irrespective of whether they are coming from >>>>>>> environment variables, deployment.yaml or defaults in code), which can >>>>>>> be >>>>>>> written at server startup, it would be useful for the troubleshooting >>>>>>> purpose IMHO. >>>>>>> >>>>>>>> >>>>>>>>> On Wed, Aug 23, 2017 at 1:35 PM, Chamila De Alwis < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi Jayanga, >>>>>>>>>> >>>>>>>>>> From Support's point of view, requesting all the data that can >>>>>>>>>> have an effect is not an overhead IMO. Especially since we already >>>>>>>>>> do that >>>>>>>>>> now, we request for possible environmental factors, at least after >>>>>>>>>> all >>>>>>>>>> other efforts are exhausted. When this change is introduced, we >>>>>>>>>> would not >>>>>>>>>> additionally ask for anything. >>>>>>>>>> >>>>>>>>>> Furthermore, environment variables would mainly be used in >>>>>>>>>> scenarios where Container Cluster Management systems are used, in >>>>>>>>>> which >>>>>>>>>> case there is a clear cut definition of which environment variables >>>>>>>>>> with >>>>>>>>>> what values are being applied to Containers. This is how most >>>>>>>>>> systems are >>>>>>>>>> debugged in CMSs like Kubernetes even now, where commands like >>>>>>>>>> `kubectl >>>>>>>>>> describe` would give you all the runtime information. >>>>>>>>>> >>>>>>>>>> Non-Containerized environments like VMs would mostly use the file >>>>>>>>>> based approach where it is easy to be coupled with Config Automation >>>>>>>>>> tools >>>>>>>>>> like Puppet. >>>>>>>>>> >>>>>>>>>> The concern of troubleshooting can be tackled by enabling a >>>>>>>>>> verbose log that would display the environment variable name from >>>>>>>>>> which a >>>>>>>>>> certain value was retrieved, at the time configs are loaded. This >>>>>>>>>> would >>>>>>>>>> obviously have to be verbose than INFO, however it would greatly >>>>>>>>>> help any >>>>>>>>>> debugging process. This is how tools like Puppet handle environment >>>>>>>>>> variable injected configs. >>>>>>>>>> >>>>>>>>>> The argument for file-less environment variables in the lookup >>>>>>>>>> hierarchy is how quickly it becomes cumbersome to handle >>>>>>>>>> configuration >>>>>>>>>> files across different Container images. Having files would soon >>>>>>>>>> present us >>>>>>>>>> with a situation similar to what we handle now in C4 with >>>>>>>>>> configuration >>>>>>>>>> repositories for different "patterns" of deployment in Containers, >>>>>>>>>> that >>>>>>>>>> requires a separate effort to be maintained for different releases. >>>>>>>>>> Having >>>>>>>>>> low-config convention-based environment variables support is a >>>>>>>>>> valuable >>>>>>>>>> addition to the C5 configuration lookup model since we've already >>>>>>>>>> reduced >>>>>>>>>> the number of configuration files. deployment.yaml will anyway be >>>>>>>>>> there, >>>>>>>>>> and I'm willing to bet that most clients would use that to make >>>>>>>>>> configuration changes. However, there are a set of clients who would >>>>>>>>>> require the convention-based environment variables for a >>>>>>>>>> smoother integration to Containerization than others. >>>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Chamila de Alwis >>>>>>>>>> Committer and PMC Member - Apache Stratos >>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>> Blog: https://medium.com/@chamilad >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Aug 23, 2017 at 1:03 PM, Jayanga Dissanayake < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> @Nuwan, configs should be able to inject via >>>>>>>>>>> environment properties. What I am saying is all these configs >>>>>>>>>>> should be >>>>>>>>>>> specified the in the deployment.yaml as {env:placeholder name}. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Jayanga. >>>>>>>>>>> >>>>>>>>>>> *Jayanga Dissanayake* >>>>>>>>>>> Associate Technical Lead >>>>>>>>>>> WSO2 Inc. - http://wso2.com/ >>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>> email: [email protected] >>>>>>>>>>> mobile: +94772207259 <+94%2077%20220%207259> >>>>>>>>>>> <http://wso2.com/signature> >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 23, 2017 at 12:51 PM, Nuwan Dias <[email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> @Jayanga, how do you make the server run elegantly on >>>>>>>>>>>> containers without being able to inject properties via env >>>>>>>>>>>> variables? >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Aug 23, 2017 at 12:43 PM, Jayanga Dissanayake < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> -1, for injecting configs without specifying it in the >>>>>>>>>>>>> deployment.yaml >>>>>>>>>>>>> >>>>>>>>>>>>> Reason: When analyzing most of the issues we request config >>>>>>>>>>>>> files (with C5 deployment.yaml) from the users to identify what >>>>>>>>>>>>> are the >>>>>>>>>>>>> configuration changes. If we have at-least the place holders in >>>>>>>>>>>>> the config >>>>>>>>>>>>> files, there is a guarantee that we know what are the injected >>>>>>>>>>>>> configuration and we could request the actual values of the >>>>>>>>>>>>> configs we need. >>>>>>>>>>>>> >>>>>>>>>>>>> But with the suggested approach, we have to request the >>>>>>>>>>>>> 1)config files, 2)all the environment variables and get the union >>>>>>>>>>>>> to >>>>>>>>>>>>> identify the changed configurations. IMO an additional effort. >>>>>>>>>>>>> >>>>>>>>>>>>> Hence considering the overhead on the support front I am -1 >>>>>>>>>>>>> for the suggested approach. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Jayanga. >>>>>>>>>>>>> >>>>>>>>>>>>> *Jayanga Dissanayake* >>>>>>>>>>>>> Associate Technical Lead >>>>>>>>>>>>> WSO2 Inc. - http://wso2.com/ >>>>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>>>> email: [email protected] >>>>>>>>>>>>> mobile: +94772207259 <+94%2077%20220%207259> >>>>>>>>>>>>> <http://wso2.com/signature> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 23, 2017 at 9:16 AM, Danesh Kuruppu < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Created git issue[1] to track this improvement. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. https://github.com/wso2/carbon-config/issues/48 >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Aug 22, 2017 at 9:50 AM, Danesh Kuruppu < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Chamila, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 for the suggested approach. So we can have both ways to >>>>>>>>>>>>>>> set environment variables either from placeholder ${env:<key>} >>>>>>>>>>>>>>> where we can >>>>>>>>>>>>>>> specify any key for the variable or from the new approach. For >>>>>>>>>>>>>>> the new >>>>>>>>>>>>>>> approach, we need adhere to a certain format as suggested. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Danesh >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 5:59 PM, Chamila De Alwis < >>>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In C5, the configuration lookup is done in the environment >>>>>>>>>>>>>>>> variables, system properties, deployment.yaml file, and the >>>>>>>>>>>>>>>> value provided >>>>>>>>>>>>>>>> in the configuration bean class, in that order. However, it >>>>>>>>>>>>>>>> looks like >>>>>>>>>>>>>>>> environment variable lookup happens only if the particular >>>>>>>>>>>>>>>> configuration is >>>>>>>>>>>>>>>> noted to do so in the deployment.yaml, specifically if the >>>>>>>>>>>>>>>> value of the >>>>>>>>>>>>>>>> config is prefixed by a placeholder env:. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> gateWayEndpoint: ${env:gateWayEndpoint} >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> While this pattern may be easy to implement, it is >>>>>>>>>>>>>>>> cumbersome to be followed in a Containerization approach and >>>>>>>>>>>>>>>> IMO breaks the >>>>>>>>>>>>>>>> 12Factor approach to configuration. This is because of the >>>>>>>>>>>>>>>> need to edit a >>>>>>>>>>>>>>>> physical file in order for the environment variables to be >>>>>>>>>>>>>>>> queried. This >>>>>>>>>>>>>>>> makes it harder to maintain a single Container Image for >>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>> deployments, and manipulate the Container runtimes via >>>>>>>>>>>>>>>> environment >>>>>>>>>>>>>>>> variables. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> AFAIU, this pattern emerges from the requirements in >>>>>>>>>>>>>>>> SecureVault; having a ${sec:} prefix can mark the >>>>>>>>>>>>>>>> configuration value to be >>>>>>>>>>>>>>>> resolved through Secure Vault resolver. However IMO this, >>>>>>>>>>>>>>>> value resolution, >>>>>>>>>>>>>>>> is not the same as value lookup. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> IMO, the following should be the lookup order, without >>>>>>>>>>>>>>>> having to mark a particular config in the deployment.yaml file. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. Environment variables: >>>>>>>>>>>>>>>> PREFIX_wso2.NAMESPACE_CONFIGKEY="value", >>>>>>>>>>>>>>>> PREFIX_wso2.NAMESPACE_CONFIGPARENT_CONFIG2="value2" >>>>>>>>>>>>>>>> 2. deployment.yaml: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wso2.NAMESPACE: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CONFIG_KEY: value >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CONFIG_PARENT: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> CONFIG2: valu2 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. Default value in the code >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This approach would be most Container friendly one as there >>>>>>>>>>>>>>>> is no need to change files depending on deployment patterns, >>>>>>>>>>>>>>>> environments, >>>>>>>>>>>>>>>> etc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> Chamila de Alwis >>>>>>>>>>>>>>>> Committer and PMC Member - Apache Stratos >>>>>>>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>>>>>>> Blog: https://medium.com/@chamilad >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> *Danesh Kuruppu* >>>>>>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>>> Mobile: +94 (77) 1690552 <+94%2077%20169%200552> >>>>>>>>>>>>>>> Web: WSO2 Inc <https://wso2.com/signature> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Danesh Kuruppu* >>>>>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>>> Mobile: +94 (77) 1690552 <+94%2077%20169%200552> >>>>>>>>>>>>>> Web: WSO2 Inc <https://wso2.com/signature> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Nuwan Dias >>>>>>>>>>>> >>>>>>>>>>>> Software Architect - WSO2, Inc. http://wso2.com >>>>>>>>>>>> email : [email protected] >>>>>>>>>>>> Phone : +94 777 775 729 <+94%2077%20777%205729> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thanks and Regards, >>>>>>>>> >>>>>>>>> Isuru H. >>>>>>>>> +94 716 358 048 <+94%2071%20635%208048>* <http://wso2.com/>* >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Lakmal Warusawithana >>>>>>>> Director - Cloud Architecture; WSO2 Inc. >>>>>>>> Mobile : +94714289692 <071%20428%209692> >>>>>>>> Blogs : https://medium.com/@lakwarus/ >>>>>>>> http://lakmalsview.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks and Regards, >>>>>>> >>>>>>> Isuru H. >>>>>>> +94 716 358 048 <+94%2071%20635%208048>* <http://wso2.com/>* >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> Nuwan Dias >>>> >>>> Software Architect - WSO2, Inc. http://wso2.com >>>> email : [email protected] >>>> Phone : +94 777 775 729 <+94%2077%20777%205729> >>>> >>> >>> >> >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
