Ignore the question Isura, I think Ruwan's reply contains the answer. Regards, Johann.
On Thu, Jul 4, 2019 at 8:48 AM Johann Nallathamby <joh...@wso2.com> wrote: > Hi Isura, > > On Fri, Jun 7, 2019 at 9:16 AM Isura Karunaratne <is...@wso2.com> wrote: > >> >> >> On Wed, Jun 5, 2019 at 9:34 AM Ruwan Abeykoon <ruw...@wso2.com> wrote: >> >>> Hi All, >>> The "User Store" configuration can be considered as a deployment >>> artifact if we look at the following aspects. >>> >>> a) "Secondary User Stores" are added, removed and updated per tenant >>> basis. (Same as SP, and IdP configs) >>> b) "User Store", "IdP" and "SP", XACML policies, etc behaves as a >>> collection of business rules, defining the authentication and authorization >>> flows per the tenant. >>> c) A change in a particular "User Store" usually affect the >>> "Authentication" decisions done via SP config. Hence they have tight >>> coupling. >>> d) All "User Store", "SP", "IdP", etc, need to be taken as one unit, >>> when we consider environment to environment promotion of these configs. >>> (Dev->QA->staging->Prod) >>> >>> Hence IMHO, treating "User Store" as the file-based artifact was a right >>> decision, when our products have been designed for deployment on bear-metal >>> or VM. However moving forward to container, and cloud nativeness, posses >>> the challenge on sharing these artifacts. >>> >>> Also, considering the CI/CD pipelines, the governance aspect of changing >>> the configurations, etc, these type of configs need to be considered as >>> artifacts. We might need to version control these artifacts in future and >>> may need to push and pull them from VCS systems. >>> >>> What we need to do is to have a delegation pattern implemented for all >>> current file based (and registry based artifacts), where we can switch the >>> repository from file based one to different system. DB based repository >>> would be the first such(simple) implementation. We may need to implement >>> Git based repository when we properly support cloud use cases for example. >>> We can abstract the storage system, and retain all the parsing and >>> generation logic unchanged for artifacts. it would be a minimal change and >>> most versatile way to extend IMO. >>> >>> We would need to implement property or "environment variable" binding >>> logic, to get proper support for environment to environment promotion of >>> artifacts. yet, it can be done with a separate effort than this IMO. >>> >>> Hence +1 to treat >>> >>> - Persist data as a blob (marshalled to text form) >>> - In a separate table structure. >>> >>> Cheers, >>> Ruwan A >>> >>> >>> On Tue, Jun 4, 2019 at 3:50 PM Johann Nallathamby <joh...@wso2.com> >>> wrote: >>> >>>> +1 to get rid of the artifacts for user stores. I think this was a >>>> wrong decision we made early on. >>>> >>>> On Tue, Jun 4, 2019 at 1:19 PM Hasanthi Purnima Dissanayake < >>>> hasan...@wso2.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> *Problem * >>>>> Currently, some artifacts like userstores , tenants' data, etc are >>>>> stored in the file system (not in the database). So when using a clustered >>>>> setup those artifacts should be shared among all the nodes by using one of >>>>> the following file sharing mechanisms. >>>>> >>>>> - Dep Sync >>>>> - rSync >>>>> - Shared File System >>>>> >>>>> >>>>> *Solution * >>>>> In order to avoid a shared file system and to reduce the deployment >>>>> and maintenance overhead, those artifacts ca be persisted in the database >>>>> itself. >>>>> >>>>> *Approach* >>>>> After discussing with @Ruwan Abeykoon <ruw...@wso2.com> and @Isura >>>>> Karunaratne <is...@wso2.com> we have two options to persist above >>>>> discussed artifact details. >>>>> >>>>> - In the configuration store which is already implemented as >>>>> discussed in [1][2]. >>>>> - In a separate table structure. >>>>> >>>>> If we are to go with option 01, then we need to consider the artifacts >>>>> as configurations and persist in the existing schema. >>>>> >>>> The advantage of using this is we can re-use the existing >>>>> implementation including the database schema and existing rest APIs and >>>>> functionalities (pagination, searching, etc) . >>>>> >>>> >>>> >>> >> Hi all, >> >> >>> The drawback is the conceptual difference between an artifact and >>>>> configuration. >>>>> >>>> >>>> I think this is not a problem. In fact I believe we made a wrong >>>> decision of considering user stores as artifacts. I can't remember exactly >>>> as to why we decided like that. User stores are not development artifacts; >>>> they are one time configurations. They don't have metadata, versioning, >>>> lifecycles or any other properties associated with other artifacts in WSO2. >>>> >>>> >>>>> Further if we are to use the configuration store there is no way to >>>>> include specific input validations for the user store configuration >>>>> feature. >>>>> >>>> >>>> I am not sure I completely understand this. But please check the SAML2 >>>> metadata for identity provider feature where I believe we've done some >>>> validations on the properties. I think they were done through interceptors >>>> in identity provider service. >>>> >>> Yes. If we have an interception layer, we can do the validation in that >> level. Currently, we don't have an intercepting layer for configuration >> management, we can treat this as an improvement to the config management >> feature. >> >> >>>>> If we are to go with the option 02, then the flow will be as follows. >>>>> >>>>> *Existing Flow* >>>>> >>>>> [image: Untitled Diagram (9).png] >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *Suggested Flow* >>>>> >>>>> [image: Untitled Diagram (10).png] >>>>> >>>>> >>>>> >>>>> - With the suggested approach, as the storage mechanisms, file >>>>> system and database can be used and any other storage mechanism is >>>>> pluggable. >>>>> >>>>> I don't agree with classifying user stores as artifacts. Therefore I >>>> guess for me this option doesn't qualify at all :). >>>> >>>> However, just to understand this option better, have we come across any >>>> customer who has asked us to support multiple types of storage mechanisms >>>> for artifacts and/or configurations? Where is this requirement coming from? >>>> I've only seen such requirements for user stores. For configurations I feel >>>> this is just over engineering. May be it is a valid requirement for >>>> artifacts? Even if we agree that there are valid reasons to support this >>>> then it has to be supported for all configurations and/or artifacts. >>>> >>> > *I think you've missed the important question here :). Where is this > requirement coming from? To have multiple storage locations for > configurations?* > > Regards, > Johann. > >> >>>>> - There should be a way to identify the repository where the data >>>>> is loaded from. The repository can be the file system, database or any >>>>> other storage mechanism. >>>>> >>>>> It sounds like this can get too complicated. >>>> >>> I think this is just a flag which we can set while loading the user >> stores. This will help to show where the user store is configured. Ex. file >> system, database or GIT location. >> >>> >>>>> - In both the read write operations the enduser should have the >>>>> control to decide the storage mechanism. >>>>> >>>>> Hmm.. this sounds more like a requirement to optimize database read >>>> write performance. Doesn't sound right for artifacts. >>>> >>> This way we can support the backward compatibility. It is true that we >> can support backward compatibility via a configuration, but this is more >> flexible. >> >>> >>>>> - If the user needs to migrate a userstore from one storage >>>>> mechanism (file system) to another then they can do it via UI. >>>>> >>>>> Again too many options for the user can make the product fragile. >>>> >>> Users may require to migrate the existing file-based configuration to >> JDBC tables. In such cases, this will help to migrate configuration rather >> than doing them manually. >> >>> >>>> >>>>> When persisting the data in the database there are two options we can >>>>> use : >>>>> >>>>> - Persist data as a blob >>>>> >>>>> If we persist as blob then we lose the granular control over each >>>> property for validation, transformation, etc. >>>> >>>>> >>>>> - Persists data as key value pair >>>>> >>>>> +1 for this. >>>> >>>> >>>>> If we are to go with the option one then we can persist the file as a >>>>> blob and reuse most of the existing parsing logics. >>>>> >>>> >>>> Given the understanding I think I prefer option 1 with properties. >>>> >>>> Thanks & Regards, >>>> Johann. >>>> >>> >> Cheers, >> Isura. >> >>> >>>> >>>>> >>>>> Highly appreciate your suggestions and feedbacks on the above approach. >>>>> >>>>> [1] [Architecture][IAM][JDBC based Configuration Store] Database Schema >>>>> [2] [Architecture] [IS] JDBC based Configuration Store for WSO2 IS >>>>> >>>>> Thanks, >>>>> Hasanthi >>>>> >>>>> -- >>>>> >>>>> Hasanthi Dissanayake | Senior Software Engineer | WSO2 Inc. >>>>> (m) +94718407133 | (w) +94112145345 | Email: hasan...@wso2.com >>>>> >>>>> >>>> >>>> -- >>>> *Johann Dilantha Nallathamby* | Associate Director/Solutions Architect >>>> | WSO2 Inc. >>>> (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com >>>> [image: Signature.jpg] >>>> >>> >>> >>> -- >>> Ruwan Abeykoon | Director/Architect | WSO2 Inc. >>> (w) +947435800 | Email: ruw...@wso2.com >>> >>> >> >> -- >> >> *Isura Dilhara Karunaratne* >> Technical Lead | WSO2 <http://wso2.com/> >> *lean.enterprise.middleware* >> Email: is...@wso2.com >> Mob : +94 772 254 810 >> Blog : https://medium.com/@isurakarunaratne >> >> >> >> > > -- > *Johann Dilantha Nallathamby* | Associate Director/Solutions Architect | > WSO2 Inc. > (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com > [image: Signature.jpg] > -- *Johann Dilantha Nallathamby* | Associate Director/Solutions Architect | WSO2 Inc. (m) +94 (77) 7776950 | (w) +94 (11) 2145345 | (e) joh...@wso2.com [image: Signature.jpg]
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
