I agree. We could omit Secure Vaulted or possible fields like "password" values from being logged.
I've scheduled a meeting on 28th Monday 2.00PM. Let's consolidate these into a plan then. Regards, Chamila de Alwis Committer and PMC Member - Apache Stratos Senior Software Engineer | WSO2 Blog: https://medium.com/@chamilad On Wed, Aug 23, 2017 at 5:26 PM, Nuwan Dias <[email protected]> wrote: > Logging might be tricky. Some of these configs we're talking about are > related to credentials. We shouldn't log credentials in any place. So we > probably will have to find a way to omit logging certain variables if we go > down the path of logging. > > On Wed, Aug 23, 2017 at 5:21 PM, Chamila De Alwis <[email protected]> > wrote: > >> I had an offline chat with IsuruH and Jayanga, during which following >> points came into light. >> >> 1. To address the troubleshooting story, a separate log can be written at >> the startup where the ConfigProvider executes that would contain the >> configurations overridden for the particular product. The security of this >> log should be handled in the same way for other logs. For debugging, this >> log file can be requested from a deployment. >> 2. There is a possibility that product start-up time might degrade if >> this logging is introduced. >> 3. Not all configurations need to be set via environment variables. There >> can be an attribute in the config annotation which denotes the environment >> variable that can be used to override the particular configuration. This >> can also help in filling the documentation gap that is introduced, i.e. the >> list of environment variables that can be used in a particular product can >> be autogenerated when configuration doc generation story is completed. >> 4. Complex types do not need to be supported in environment variable >> story. This would vary depending on the likelihood that configuration would >> be overridden through an environment variable. >> >> Let's schedule a meeting to discuss the concerns, including the above, >> and come to a conclusion. >> >> >> Regards, >> Chamila de Alwis >> Committer and PMC Member - Apache Stratos >> Senior Software Engineer | WSO2 >> Blog: https://medium.com/@chamilad >> >> >> >> On Wed, Aug 23, 2017 at 3:01 PM, Chamila De Alwis <[email protected]> >> wrote: >> >>> IMO logging values would be a security risk. It would be more manageable >>> if information on where certain values are retrieved from can be displayed >>> if that log level is enabled. >>> >>> >>> Regards, >>> Chamila de Alwis >>> Committer and PMC Member - Apache Stratos >>> Senior Software Engineer | WSO2 >>> Blog: https://medium.com/@chamilad >>> >>> >>> >>> On Wed, Aug 23, 2017 at 3:00 PM, Isuru Haththotuwa <[email protected]> >>> wrote: >>> >>>> >>>> >>>> On Wed, Aug 23, 2017 at 2:46 PM, Lakmal Warusawithana <[email protected]> >>>> wrote: >>>> >>>>> >>>>> >>>>> On Wed, Aug 23, 2017 at 2:39 PM, Isuru Haththotuwa <[email protected]> >>>>> wrote: >>>>> >>>>>> While I fully agree that its necessary to be able to inject configs >>>>>> via environment variables to be container friendly, I think Jayanga do >>>>>> have >>>>>> a point. When we are troubleshooting an issue, the first thing to check >>>>>> is >>>>>> the configurations. When up configuration values are looked up via >>>>>> environment variables, how do we know what are the configuration >>>>>> parameters >>>>>> that are used in the system? One possible option is to write to a log as >>>>>> Chamila has mentioned, or else write the currently used config params to >>>>>> a >>>>>> log file at startup. WDYT? However this does not take into account the >>>>>> possibility of changing an environment value even after the server has >>>>>> started. >>>>>> >>>>> >>>>> @Isuru, it is coming with correct namespace right? and first priority >>>>> is given to environment variables. So there should not be an issue with >>>>> identifying the our config from environment variables IMO. >>>>> >>>> Yes. What I want to highlight is the troubleshooting perspective. We >>>> can have a particular set of config values coming from environment >>>> variables, another set of config values from deployment.yaml and defaults >>>> from the code. If we have log file file with all the currently used >>>> configurations values (irrespective of whether they are coming from >>>> environment variables, deployment.yaml or defaults in code), which can be >>>> written at server startup, it would be useful for the troubleshooting >>>> purpose IMHO. >>>> >>>>> >>>>>> On Wed, Aug 23, 2017 at 1:35 PM, Chamila De Alwis <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Jayanga, >>>>>>> >>>>>>> From Support's point of view, requesting all the data that can have >>>>>>> an effect is not an overhead IMO. Especially since we already do that >>>>>>> now, >>>>>>> we request for possible environmental factors, at least after all other >>>>>>> efforts are exhausted. When this change is introduced, we would not >>>>>>> additionally ask for anything. >>>>>>> >>>>>>> Furthermore, environment variables would mainly be used in scenarios >>>>>>> where Container Cluster Management systems are used, in which case >>>>>>> there is >>>>>>> a clear cut definition of which environment variables with what values >>>>>>> are >>>>>>> being applied to Containers. This is how most systems are debugged in >>>>>>> CMSs >>>>>>> like Kubernetes even now, where commands like `kubectl describe` would >>>>>>> give >>>>>>> you all the runtime information. >>>>>>> >>>>>>> Non-Containerized environments like VMs would mostly use the file >>>>>>> based approach where it is easy to be coupled with Config Automation >>>>>>> tools >>>>>>> like Puppet. >>>>>>> >>>>>>> The concern of troubleshooting can be tackled by enabling a verbose >>>>>>> log that would display the environment variable name from which a >>>>>>> certain >>>>>>> value was retrieved, at the time configs are loaded. This would >>>>>>> obviously >>>>>>> have to be verbose than INFO, however it would greatly help any >>>>>>> debugging >>>>>>> process. This is how tools like Puppet handle environment variable >>>>>>> injected >>>>>>> configs. >>>>>>> >>>>>>> The argument for file-less environment variables in the lookup >>>>>>> hierarchy is how quickly it becomes cumbersome to handle configuration >>>>>>> files across different Container images. Having files would soon >>>>>>> present us >>>>>>> with a situation similar to what we handle now in C4 with configuration >>>>>>> repositories for different "patterns" of deployment in Containers, that >>>>>>> requires a separate effort to be maintained for different releases. >>>>>>> Having >>>>>>> low-config convention-based environment variables support is a valuable >>>>>>> addition to the C5 configuration lookup model since we've already >>>>>>> reduced >>>>>>> the number of configuration files. deployment.yaml will anyway be there, >>>>>>> and I'm willing to bet that most clients would use that to make >>>>>>> configuration changes. However, there are a set of clients who would >>>>>>> require the convention-based environment variables for a >>>>>>> smoother integration to Containerization than others. >>>>>>> >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Chamila de Alwis >>>>>>> Committer and PMC Member - Apache Stratos >>>>>>> Senior Software Engineer | WSO2 >>>>>>> Blog: https://medium.com/@chamilad >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Aug 23, 2017 at 1:03 PM, Jayanga Dissanayake < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> @Nuwan, configs should be able to inject via >>>>>>>> environment properties. What I am saying is all these configs should be >>>>>>>> specified the in the deployment.yaml as {env:placeholder name}. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Jayanga. >>>>>>>> >>>>>>>> *Jayanga Dissanayake* >>>>>>>> Associate Technical Lead >>>>>>>> WSO2 Inc. - http://wso2.com/ >>>>>>>> lean . enterprise . middleware >>>>>>>> email: [email protected] >>>>>>>> mobile: +94772207259 <+94%2077%20220%207259> >>>>>>>> <http://wso2.com/signature> >>>>>>>> >>>>>>>> On Wed, Aug 23, 2017 at 12:51 PM, Nuwan Dias <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> @Jayanga, how do you make the server run elegantly on containers >>>>>>>>> without being able to inject properties via env variables? >>>>>>>>> >>>>>>>>> On Wed, Aug 23, 2017 at 12:43 PM, Jayanga Dissanayake < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> -1, for injecting configs without specifying it in the >>>>>>>>>> deployment.yaml >>>>>>>>>> >>>>>>>>>> Reason: When analyzing most of the issues we request config files >>>>>>>>>> (with C5 deployment.yaml) from the users to identify what are the >>>>>>>>>> configuration changes. If we have at-least the place holders in the >>>>>>>>>> config >>>>>>>>>> files, there is a guarantee that we know what are the injected >>>>>>>>>> configuration and we could request the actual values of the configs >>>>>>>>>> we need. >>>>>>>>>> >>>>>>>>>> But with the suggested approach, we have to request the 1)config >>>>>>>>>> files, 2)all the environment variables and get the union to identify >>>>>>>>>> the >>>>>>>>>> changed configurations. IMO an additional effort. >>>>>>>>>> >>>>>>>>>> Hence considering the overhead on the support front I am -1 for >>>>>>>>>> the suggested approach. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Jayanga. >>>>>>>>>> >>>>>>>>>> *Jayanga Dissanayake* >>>>>>>>>> Associate Technical Lead >>>>>>>>>> WSO2 Inc. - http://wso2.com/ >>>>>>>>>> lean . enterprise . middleware >>>>>>>>>> email: [email protected] >>>>>>>>>> mobile: +94772207259 <+94%2077%20220%207259> >>>>>>>>>> <http://wso2.com/signature> >>>>>>>>>> >>>>>>>>>> On Wed, Aug 23, 2017 at 9:16 AM, Danesh Kuruppu <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Created git issue[1] to track this improvement. >>>>>>>>>>> >>>>>>>>>>> 1. https://github.com/wso2/carbon-config/issues/48 >>>>>>>>>>> >>>>>>>>>>> On Tue, Aug 22, 2017 at 9:50 AM, Danesh Kuruppu <[email protected] >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Chamila, >>>>>>>>>>>> >>>>>>>>>>>> +1 for the suggested approach. So we can have both ways to set >>>>>>>>>>>> environment variables either from placeholder ${env:<key>} where >>>>>>>>>>>> we can >>>>>>>>>>>> specify any key for the variable or from the new approach. For the >>>>>>>>>>>> new >>>>>>>>>>>> approach, we need adhere to a certain format as suggested. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Danesh >>>>>>>>>>>> >>>>>>>>>>>> On Mon, Aug 21, 2017 at 5:59 PM, Chamila De Alwis < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> In C5, the configuration lookup is done in the environment >>>>>>>>>>>>> variables, system properties, deployment.yaml file, and the value >>>>>>>>>>>>> provided >>>>>>>>>>>>> in the configuration bean class, in that order. However, it looks >>>>>>>>>>>>> like >>>>>>>>>>>>> environment variable lookup happens only if the particular >>>>>>>>>>>>> configuration is >>>>>>>>>>>>> noted to do so in the deployment.yaml, specifically if the value >>>>>>>>>>>>> of the >>>>>>>>>>>>> config is prefixed by a placeholder env:. >>>>>>>>>>>>> >>>>>>>>>>>>> gateWayEndpoint: ${env:gateWayEndpoint} >>>>>>>>>>>>> >>>>>>>>>>>>> While this pattern may be easy to implement, it is cumbersome >>>>>>>>>>>>> to be followed in a Containerization approach and IMO breaks the >>>>>>>>>>>>> 12Factor >>>>>>>>>>>>> approach to configuration. This is because of the need to edit a >>>>>>>>>>>>> physical >>>>>>>>>>>>> file in order for the environment variables to be queried. This >>>>>>>>>>>>> makes it >>>>>>>>>>>>> harder to maintain a single Container Image for different >>>>>>>>>>>>> deployments, and >>>>>>>>>>>>> manipulate the Container runtimes via environment variables. >>>>>>>>>>>>> >>>>>>>>>>>>> AFAIU, this pattern emerges from the requirements in >>>>>>>>>>>>> SecureVault; having a ${sec:} prefix can mark the configuration >>>>>>>>>>>>> value to be >>>>>>>>>>>>> resolved through Secure Vault resolver. However IMO this, value >>>>>>>>>>>>> resolution, >>>>>>>>>>>>> is not the same as value lookup. >>>>>>>>>>>>> >>>>>>>>>>>>> IMO, the following should be the lookup order, without having >>>>>>>>>>>>> to mark a particular config in the deployment.yaml file. >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Environment variables: PREFIX_wso2.NAMESPACE_CONFIGKEY="value", >>>>>>>>>>>>> PREFIX_wso2.NAMESPACE_CONFIGPARENT_CONFIG2="value2" >>>>>>>>>>>>> 2. deployment.yaml: >>>>>>>>>>>>> >>>>>>>>>>>>> wso2.NAMESPACE: >>>>>>>>>>>>> >>>>>>>>>>>>> CONFIG_KEY: value >>>>>>>>>>>>> >>>>>>>>>>>>> CONFIG_PARENT: >>>>>>>>>>>>> >>>>>>>>>>>>> CONFIG2: valu2 >>>>>>>>>>>>> >>>>>>>>>>>>> 3. Default value in the code >>>>>>>>>>>>> >>>>>>>>>>>>> This approach would be most Container friendly one as there is >>>>>>>>>>>>> no need to change files depending on deployment patterns, >>>>>>>>>>>>> environments, etc. >>>>>>>>>>>>> >>>>>>>>>>>>> WDYT? >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Chamila de Alwis >>>>>>>>>>>>> Committer and PMC Member - Apache Stratos >>>>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>>>> Blog: https://medium.com/@chamilad >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> *Danesh Kuruppu* >>>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>>> >>>>>>>>>>>> Email: [email protected] >>>>>>>>>>>> Mobile: +94 (77) 1690552 <+94%2077%20169%200552> >>>>>>>>>>>> Web: WSO2 Inc <https://wso2.com/signature> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> *Danesh Kuruppu* >>>>>>>>>>> Senior Software Engineer | WSO2 >>>>>>>>>>> >>>>>>>>>>> Email: [email protected] >>>>>>>>>>> Mobile: +94 (77) 1690552 <+94%2077%20169%200552> >>>>>>>>>>> Web: WSO2 Inc <https://wso2.com/signature> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> [email protected] >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Nuwan Dias >>>>>>>>> >>>>>>>>> Software Architect - WSO2, Inc. http://wso2.com >>>>>>>>> email : [email protected] >>>>>>>>> Phone : +94 777 775 729 <+94%2077%20777%205729> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Architecture mailing list >>>>>>>>> [email protected] >>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> [email protected] >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Thanks and Regards, >>>>>> >>>>>> Isuru H. >>>>>> +94 716 358 048 <+94%2071%20635%208048>* <http://wso2.com/>* >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Lakmal Warusawithana >>>>> Director - Cloud Architecture; WSO2 Inc. >>>>> Mobile : +94714289692 <071%20428%209692> >>>>> Blogs : https://medium.com/@lakwarus/ >>>>> http://lakmalsview.blogspot.com/ >>>>> >>>>> >>>> >>>> >>>> -- >>>> Thanks and Regards, >>>> >>>> Isuru H. >>>> +94 716 358 048 <+94%2071%20635%208048>* <http://wso2.com/>* >>>> >>>> >>>> >>> >> > > > -- > Nuwan Dias > > Software Architect - WSO2, Inc. http://wso2.com > email : [email protected] > Phone : +94 777 775 729 <+94%2077%20777%205729> >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
