I've rescheduled the meeting to 30th August (Wednesday) 1pm-2pm, as Lakmal is unavailable today.
Regards, Chamila de Alwis Committer and PMC Member - Apache Stratos Senior Software Engineer | WSO2 Blog: https://medium.com/@chamilad On Wed, Aug 23, 2017 at 5:31 PM, Chamila De Alwis <[email protected]> wrote: > I agree. We could omit Secure Vaulted or possible fields like "password" > values from being logged. > > I've scheduled a meeting on 28th Monday 2.00PM. Let's consolidate these > into a plan then. > > > Regards, > Chamila de Alwis > Committer and PMC Member - Apache Stratos > Senior Software Engineer | WSO2 > Blog: https://medium.com/@chamilad > > > > On Wed, Aug 23, 2017 at 5:26 PM, Nuwan Dias <[email protected]> wrote: > >> Logging might be tricky. Some of these configs we're talking about are >> related to credentials. We shouldn't log credentials in any place. So we >> probably will have to find a way to omit logging certain variables if we go >> down the path of logging. >> >> On Wed, Aug 23, 2017 at 5:21 PM, Chamila De Alwis <[email protected]> >> wrote: >> >>> I had an offline chat with IsuruH and Jayanga, during which following >>> points came into light. >>> >>> 1. To address the troubleshooting story, a separate log can be written >>> at the startup where the ConfigProvider executes that would contain the >>> configurations overridden for the particular product. The security of this >>> log should be handled in the same way for other logs. For debugging, this >>> log file can be requested from a deployment. >>> 2. There is a possibility that product start-up time might degrade if >>> this logging is introduced. >>> 3. Not all configurations need to be set via environment variables. >>> There can be an attribute in the config annotation which denotes the >>> environment variable that can be used to override the particular >>> configuration. This can also help in filling the documentation gap that is >>> introduced, i.e. the list of environment variables that can be used in a >>> particular product can be autogenerated when configuration doc generation >>> story is completed. >>> 4. Complex types do not need to be supported in environment variable >>> story. This would vary depending on the likelihood that configuration would >>> be overridden through an environment variable. >>> >>> Let's schedule a meeting to discuss the concerns, including the above, >>> and come to a conclusion. >>> >>> >>> Regards, >>> Chamila de Alwis >>> Committer and PMC Member - Apache Stratos >>> Senior Software Engineer | WSO2 >>> Blog: https://medium.com/@chamilad >>> >>> >>> >>> On Wed, Aug 23, 2017 at 3:01 PM, Chamila De Alwis <[email protected]> >>> wrote: >>> >>>> IMO logging values would be a security risk. It would be more >>>> manageable if information on where certain values are retrieved from can be >>>> displayed if that log level is enabled. >>>> >>>> >>>> Regards, >>>> Chamila de Alwis >>>> Committer and PMC Member - Apache Stratos >>>> Senior Software Engineer | WSO2 >>>> Blog: https://medium.com/@chamilad >>>> >>>> >>>> >>>> On Wed, Aug 23, 2017 at 3:00 PM, Isuru Haththotuwa <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Aug 23, 2017 at 2:46 PM, Lakmal Warusawithana <[email protected] >>>>> > wrote: >>>>> >>>>>> >>>>>> >>>>>> On Wed, Aug 23, 2017 at 2:39 PM, Isuru Haththotuwa <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> While I fully agree that its necessary to be able to inject configs >>>>>>> via environment variables to be container friendly, I think Jayanga do >>>>>>> have >>>>>>> a point. When we are troubleshooting an issue, the first thing to check >>>>>>> is >>>>>>> the configurations. When up configuration values are looked up via >>>>>>> environment variables, how do we know what are the configuration >>>>>>> parameters >>>>>>> that are used in the system? One possible option is to write to a log as >>>>>>> Chamila has mentioned, or else write the currently used config params >>>>>>> to a >>>>>>> log file at startup. WDYT? However this does not take into account the >>>>>>> possibility of changing an environment value even after the server has >>>>>>> started. >>>>>>> >>>>>> >>>>>> @Isuru, it is coming with correct namespace right? and first priority >>>>>> is given to environment variables. So there should not be an issue with >>>>>> identifying the our config from environment variables IMO. >>>>>> >>>>> Yes. What I want to highlight is the troubleshooting perspective. We >>>>> can have a particular set of config values coming from environment >>>>> variables, another set of config values from deployment.yaml and defaults >>>>> from the code. If we have log file file with all the currently used >>>>> configurations values (irrespective of whether they are coming from >>>>> environment variables, deployment.yaml or defaults in code), which can be >>>>> written at server startup, it would be useful for the troubleshooting >>>>> purpose IMHO. >>>>> >>>>>> >>>>>>> On Wed, Aug 23, 2017 at 1:35 PM, Chamila De Alwis <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi Jayanga, >>>>>>>> >>>>>>>> From Support's point of view, requesting all the data that can have >>>>>>>> an effect is not an overhead IMO. Especially since we already do that >>>>>>>> now, >>>>>>>> we request for possible environmental factors, at least after all other >>>>>>>> efforts are exhausted. When this change is introduced, we would not >>>>>>>> additionally ask for anything. >>>>>>>> >>>>>>>> Furthermore, environment variables would mainly be used in >>>>>>>> scenarios where Container Cluster Management systems are used, in which >>>>>>>> case there is a clear cut definition of which environment variables >>>>>>>> with >>>>>>>> what values are being applied to Containers. This is how most systems >>>>>>>> are >>>>>>>> debugged in CMSs like Kubernetes even now, where commands like `kubectl >>>>>>>> describe` would give you all the runtime information. >>>>>>>> >>>>>>>> Non-Containerized environments like VMs would mostly use the file >>>>>>>> based approach where it is easy to be coupled with Config Automation >>>>>>>> tools >>>>>>>> like Puppet. >>>>>>>> >>>>>>>> The concern of troubleshooting can be tackled by enabling a verbose >>>>>>>> log that would display the environment variable name from which a >>>>>>>> certain >>>>>>>> value was retrieved, at the time configs are loaded. This would >>>>>>>> obviously >>>>>>>> have to be verbose than INFO, however it would greatly help any >>>>>>>> debugging >>>>>>>> process. This is how tools like Puppet handle environment variable >>>>>>>> injected >>>>>>>> configs. >>>>>>>> >>>>>>>> The argument for file-less environment variables in the lookup >>>>>>>> hierarchy is how quickly it becomes cumbersome to handle configuration >>>>>>>> files across different Container images. Having files would soon >>>>>>>> present us >>>>>>>> with a situation similar to what we handle now in C4 with configuration >>>>>>>> repositories for different "patterns" of deployment in Containers, that >>>>>>>> requires a separate effort to be maintained for different releases. >>>>>>>> Having >>>>>>>> low-config convention-based environment variables support is a valuable >>>>>>>> addition to the C5 configuration lookup model since we've already >>>>>>>> reduced >>>>>>>> the number of configuration files. deployment.yaml will anyway be >>>>>>>> there, >>>>>>>> and I'm willing to bet that most clients would use that to make >>>>>>>> configuration changes. However, there are a set of clients who would >>>>>>>> require the convention-based environment variables for a >>>>>>>> smoother integration to Containerization than others. >>>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Chamila de Alwis >>>>>>>> Committer and PMC Member - Apache Stratos >>>>>>>> Senior Software Engineer | WSO2 >>>>>>>> Blog: https://medium.com/@chamilad >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Aug 23, 2017 at 1:03 PM, Jayanga Dissanayake < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> @Nuwan, configs should be able to inject via >>>>>>>>> environment properties. What I am saying is all these configs should >>>>>>>>> be >>>>>>>>> specified the in the deployment.yaml as {env:placeholder name}. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Jayanga. >>>>>>>>> >>>>>>>>> *Jayanga Dissanayake* >>>>>>>>> Associate Technical Lead >>>>>>>>> WSO2 Inc. - http://wso2.com/ >>>>>>>>> lean . enterprise . middleware >>>>>>>>> email: [email protected] >>>>>>>>> mobile: +94772207259 <+94%2077%20220%207259> >>>>>>>>> <http://wso2.com/signature> >>>>>>>>> >>>>>>>>> On Wed, Aug 23, 2017 at 12:51 PM, Nuwan Dias <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> @Jayanga, how do you make the server run elegantly on containers >>>>>>>>>> without being able to inject properties via env variables? >>>>>>>>>> >>>>>>>>>> On Wed, Aug 23, 2017 at 12:43 PM, Jayanga Dissanayake < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi All, >>>>>>>>>>> >>>>>>>>>>> -1, for injecting configs without specifying it in the >>>>>>>>>>> deployment.yaml >>>>>>>>>>> >>>>>>>>>>> Reason: When analyzing most of the issues we request config >>>>>>>>>>> files (with C5 deployment.yaml) from the users to identify what are >>>>>>>>>>> the >>>>>>>>>>> configuration changes. If we have at-least the place holders in the >>>>>>>>>>> config >>>>>>>>>>> files, there is a guarantee that we know what are the injected >>>>>>>>>>> configuration and we could request the actual values of the configs >>>>>>>>>>> we need. >>>>>>>>>>> >>>>>>>>>>> But with the suggested approach, we have to request the 1)config >>>>>>>>>>> files, 2)all the environment variables and get the union to >>>>>>>>>>> identify the >>>>>>>>>>> changed configurations. IMO an additional effort. >>>>>>>>>>> >>>>>>>>>>> Hence considering the overhead on the support front I am -1 for >>>>>>>>>>> the suggested approach. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Jayanga. >>>>>>>>>>> >>>>>>>>>>> *Jayanga Dissanayake* >>>>>>>>>>> Associate Technical Lead >>>>>>>>>>> WSO2 Inc. - http://wso2.com/ >>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>> email: [email protected] >>>>>>>>>>> mobile: +94772207259 <+94%2077%20220%207259> >>>>>>>>>>> <http://wso2.com/signature> >>>>>>>>>>> >>>>>>>>>>> On Wed, Aug 23, 2017 at 9:16 AM, Danesh Kuruppu <[email protected] >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> Created git issue[1] to track this improvement. >>>>>>>>>>>> >>>>>>>>>>>> 1. https://github.com/wso2/carbon-config/issues/48 >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Aug 22, 2017 at 9:50 AM, Danesh Kuruppu < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Chamila, >>>>>>>>>>>>> >>>>>>>>>>>>> +1 for the suggested approach. So we can have both ways to set >>>>>>>>>>>>> environment variables either from placeholder ${env:<key>} where >>>>>>>>>>>>> we can >>>>>>>>>>>>> specify any key for the variable or from the new approach. For >>>>>>>>>>>>> the new >>>>>>>>>>>>> approach, we need adhere to a certain format as suggested. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks >>>>>>>>>>>>> Danesh >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Aug 21, 2017 at 5:59 PM, Chamila De Alwis < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> In C5, the configuration lookup is done in the environment >>>>>>>>>>>>>> variables, system properties, deployment.yaml file, and the >>>>>>>>>>>>>> value provided >>>>>>>>>>>>>> in the configuration bean class, in that order. However, it >>>>>>>>>>>>>> looks like >>>>>>>>>>>>>> environment variable lookup happens only if the particular >>>>>>>>>>>>>> configuration is >>>>>>>>>>>>>> noted to do so in the deployment.yaml, specifically if the value >>>>>>>>>>>>>> of the >>>>>>>>>>>>>> config is prefixed by a placeholder env:. >>>>>>>>>>>>>> >>>>>>>>>>>>>> gateWayEndpoint: ${env:gateWayEndpoint} >>>>>>>>>>>>>> >>>>>>>>>>>>>> While this pattern may be easy to implement, it is cumbersome >>>>>>>>>>>>>> to be followed in a Containerization approach and IMO breaks the >>>>>>>>>>>>>> 12Factor >>>>>>>>>>>>>> approach to configuration. This is because of the need to edit a >>>>>>>>>>>>>> physical >>>>>>>>>>>>>> file in order for the environment variables to be queried. This >>>>>>>>>>>>>> makes it >>>>>>>>>>>>>> harder to maintain a single Container Image for different >>>>>>>>>>>>>> deployments, and >>>>>>>>>>>>>> manipulate the Container runtimes via environment variables. >>>>>>>>>>>>>> >>>>>>>>>>>>>> AFAIU, this pattern emerges from the requirements in >>>>>>>>>>>>>> SecureVault; having a ${sec:} prefix can mark the configuration >>>>>>>>>>>>>> value to be >>>>>>>>>>>>>> resolved through Secure Vault resolver. However IMO this, value >>>>>>>>>>>>>> resolution, >>>>>>>>>>>>>> is not the same as value lookup. >>>>>>>>>>>>>> >>>>>>>>>>>>>> IMO, the following should be the lookup order, without having >>>>>>>>>>>>>> to mark a particular config in the deployment.yaml file. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Environment variables: >>>>>>>>>>>>>> PREFIX_wso2.NAMESPACE_CONFIGKEY="value", >>>>>>>>>>>>>> PREFIX_wso2.NAMESPACE_CONFIGPARENT_CONFIG2="value2" >>>>>>>>>>>>>> 2. deployment.yaml: >>>>>>>>>>>>>> >>>>>>>>>>>>>> wso2.NAMESPACE: >>>>>>>>>>>>>> >>>>>>>>>>>>>> CONFIG_KEY: value >>>>>>>>>>>>>> >>>>>>>>>>>>>> CONFIG_PARENT: >>>>>>>>>>>>>> >>>>>>>>>>>>>> CONFIG2: valu2 >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3. Default value in the code >>>>>>>>>>>>>> >>>>>>>>>>>>>> This approach would be most Container friendly one as there >>>>>>>>>>>>>> is no need to change files depending on deployment patterns, >>>>>>>>>>>>>> environments, >>>>>>>>>>>>>> etc. >>>>>>>>>>>>>> >>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Chamila de Alwis >>>>>>>>>>>>>> Committer and PMC Member - Apache Stratos >>>>>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>>>>> Blog: https://medium.com/@chamilad >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> *Danesh Kuruppu* >>>>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>>>> >>>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>>> Mobile: +94 (77) 1690552 <+94%2077%20169%200552> >>>>>>>>>>>>> Web: WSO2 Inc <https://wso2.com/signature> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> *Danesh Kuruppu* >>>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>>> >>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>> Mobile: +94 (77) 1690552 <+94%2077%20169%200552> >>>>>>>>>>>> Web: WSO2 Inc <https://wso2.com/signature> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Architecture mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Nuwan Dias >>>>>>>>>> >>>>>>>>>> Software Architect - WSO2, Inc. http://wso2.com >>>>>>>>>> email : [email protected] >>>>>>>>>> Phone : +94 777 775 729 <+94%2077%20777%205729> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thanks and Regards, >>>>>>> >>>>>>> Isuru H. >>>>>>> +94 716 358 048 <+94%2071%20635%208048>* <http://wso2.com/>* >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Lakmal Warusawithana >>>>>> Director - Cloud Architecture; WSO2 Inc. >>>>>> Mobile : +94714289692 <071%20428%209692> >>>>>> Blogs : https://medium.com/@lakwarus/ >>>>>> http://lakmalsview.blogspot.com/ >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thanks and Regards, >>>>> >>>>> Isuru H. >>>>> +94 716 358 048 <+94%2071%20635%208048>* <http://wso2.com/>* >>>>> >>>>> >>>>> >>>> >>> >> >> >> -- >> Nuwan Dias >> >> Software Architect - WSO2, Inc. http://wso2.com >> email : [email protected] >> Phone : +94 777 775 729 <+94%2077%20777%205729> >> > >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
